autoformalization-problem-statements-into-lean-4-improves-sharply-formalizer CLM-0294observationsingle paperReviewed by AI
Autoformalization of problem statements into Lean 4 improves sharply when the formalizer can call the Lean compiler for syntax and a multi-model judge panel for meaning and revise on the feedback: a 32B model's consistency Pass@1 on CombiBench rose from about 24% without tools to about 65% with them, and an 8B distilled version reached about 91% on FormalMath-Lite, with the judge ensemble holding false-positive consistency below 6%.
Capability: Formalizing and machine-checking mathematics
Observed on
ATF-32B and ATF-8B-Distilled, Lean 4, FormalMath-Lite, CombiBench and similar, 2025; consistency judged by an LLM panel; cold-start, expert iteration and DPO training..
Sources
- Validated by: Syntactic validity and consistency Pass@1/Pass@16 across benchmarks against no-tool and prior formalizer baselines; judge-panel false-positive rate measured against human labels.. Consistency is judged by models, which the paper partly validates; the gain from tool feedback is large and ablated, which is the useful part.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On text-to-SQL benchmarks, aggregate accuracy hides a steep difficulty gradient: pre-trained-language-model-based parsers resolved roughly 80% of Spider cases overall but only about 50% of its extra-hard cases, and as natural-language understanding improved the residual errors shifted toward database-side problems — large schemas, ambiguous columns, dirty cell values — rather than language parsing.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- On natural-language logical reasoning benchmarks, auto-formalizing each reasoning step into SMT-LIB and regenerating from the first step the solver cannot verify raises both the share of solver-verified steps and final-answer accuracy over plain chain-of-thought, and greedy regeneration at the failure point beats beam search or MCTS on step verifiability.Carrying a chain of inference through to its conclusion · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
- When an 8B model verifies natural-language claims against knowledge-graph triples, scoring the evidence twice under opposing prompts (argue it supports, argue it refutes) and adjudicating between the two arguments avoids the bias toward a "Supported" verdict that a single-perspective assessment shows, and is the largest single contributor to accuracy in that pipeline.Checking claims against evidence · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Statements whose semantic mismatch the judge panel systematically misses, so consistency scores overstate faithfulness.