Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization
Quick summary
arXiv:2606.31002v2 Announce Type: replace Abstract: Lean verifies that a generated declaration is well typed, but not that it expresses the statement a user intended. We study two questions for autoformalization without canonical Lean targets: whether LLM judges can provide a usable proxy for human semantic review, and how much compilation overstates faithfulness across systems. Our criterion combines Lean compilation with strict semantic consensus between GPT-5.2 and Gemini-2.5-Pro. On an independently audited random sample, it agrees with human majority on 89.7\% of cases (Wilson 95\% CI: 82
Key takeaways
- arXiv:2606.31002v2 Announce Type: replace Abstract: Lean verifies that a generated declaration is well typed, but not that it expresses the statement a user intended.
- We study two questions for autoformalization without canonical Lean targets: whether LLM judges can provide a usable proxy for human semantic review, and how much compilation overstates faithfulness across systems.
- Our criterion combines Lean compilation with strict semantic consensus between GPT-5.2 and Gemini-2.5-Pro.
Why it matters
The value of this work lies as much in how it was tested as in the claim itself. Sample design, baselines, uncertainty and replication help separate a laboratory result from real-world impact.
Member comments