Commit-first LLM judging inherits the judge's own errors
Quick summary
arXiv:2609.00088v1 Announce Type: cross Abstract: LLM judges, models that score another system's output, can be gamed by the systems they score. Recent work identifies one defence that works: the judge solves the task itself first and commits to that answer, then accepts a candidate only if the two match. We call this commit-first judging, and ask whether shipped software implements it, and what it costs. We audit the default judge configurations of eight widely used evaluation frameworks. Of the 24 configurations in scope, none implement it. Nine implement a variant the literature measures as
Key takeaways
- arXiv:2609.00088v1 Announce Type: cross Abstract: LLM judges, models that score another system's output, can be gamed by the systems they score.
- Recent work identifies one defence that works: the judge solves the task itself first and commits to that answer, then accepts a candidate only if the two match.
- We call this commit-first judging, and ask whether shipped software implements it, and what it costs.
Why it matters
The value of this work lies as much in how it was tested as in the claim itself. Sample design, baselines, uncertainty and replication help separate a laboratory result from real-world impact.
Member comments