arXiv Artificial Intelligence

When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation

9 days ago 01.09.2026 07:00 👁 1 views
When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation

Quick summary

arXiv:2509.26600v3 Announce Type: replace-cross Abstract: As LLMs rapidly saturate existing benchmarks, automated benchmark creation using LLMs (LLM as a benchmark) where a model generates test inputs (LLM as a testset) and evaluates outputs (LLM as an evaluator) has gained traction as a cheap alternative to human curation. We show that this paradigm has a fundamental problem: LLM-generated benchmarks systematically favor the model that created them. Using machine translation as our primary testbed, we find that self bias arises from two compounding sources, LLM as a testset and LLM as an eval

Key takeaways

  • arXiv:2509.26600v3 Announce Type: replace-cross Abstract: As LLMs rapidly saturate existing benchmarks, automated benchmark creation using LLMs (LLM as a benchmark) where a model generates test inputs (LLM as a testset) and evaluates outputs (LLM as an evaluator) has gained traction as a cheap alternative to human curation.
  • We show that this paradigm has a fundamental problem: LLM-generated benchmarks systematically favor the model that created them.
  • Using machine translation as our primary testbed, we find that self bias arises from two compounding sources, LLM as a testset and LLM as an eval

Why it matters

“When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation” highlights the need for repeatable measurement rather than a single impressive demonstration. Independent validation across datasets and clearly stated limitations determine whether a result can guide product decisions.

Kaynak sitede devamını oku: arXiv Artificial Intelligence ↗