lean-prover-trained-large-scale-agentic-reinforcement-learning-against-lean CLM-0309observationsingle paperReviewed by AI
A Lean prover trained with large-scale agentic reinforcement learning against the Lean checker, paired with a test-time workflow that drafts in natural language before formalizing, solved 88% of PutnamBench (undergraduate), 80% of Fate-H (graduate) and 33% of Fate-X (PhD-level), and 11 of 12 Putnam 2025 problems, at a smaller compute budget than prior systems, so formal proving at undergraduate level is largely solved while PhD-level remains open.
Capability: Formalizing and machine-checking mathematics
Observed on
Seed-Prover 1.5 (ByteDance), Lean 4, PutnamBench, Fate-H, Fate-X, Putnam 2025, late 2025; results are the developers' own..
Sources
- Validated by: Pass rates on PutnamBench, Fate-H, Fate-X and Putnam 2025 against prior provers at stated compute budgets; RL training accuracy curve from about 50% to near 90%; developer-reported.. Developer-reported results without independent evaluation; the level gradient (88 → 80 → 33) is consistent with RLMEval's research-level gap.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On 1,000 Lean 4 targets from computational and applied mathematics textbooks (optimization, numerical linear algebra, numerical analysis), whose statements depend on locally introduced definitions, current provers solve about 62-72% of a 200-problem sample, so applied-math formalization is tractable but not saturated and requires recovering textbook context to state problems faithfully.Formalizing and machine-checking mathematics · reviewed by AI
- On human-annotated step-level verification of frontier-model proofs to recent IMO and Putnam problems, most of 29 generative critics and process reward models do little better than chance at finding the first wrong step, a leading math PRM (Qwen2.5-Math-PRM-72B) reaches only about 37% balanced F1, and beyond a few standouts open-source verifiers lag closed models; verification is harder than generation for the same frontier models.Spotting an error in a given artifact · reviewed by AI
- When training small instruction-tuned models with reinforcement learning plus factual supervision, routing each atomic fact's verification score only to the tokens that produced it, and down-weighting verifier judgements that do not change when their key evidence is removed, improves factuality benchmark scores over trajectory-level or reasoning- step-level factual rewards.Checking claims against evidence · reviewed by AI
- Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
- On 613 research-level theorems from six real Lean Blueprint formalization projects, the best current prover reaches only about a 10% pass rate, so progress on curated benchmarks such as miniF2F does not carry over to theorems embedded in live formalization projects with their own definitions and lemmas.Formalizing and machine-checking mathematics · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Independent replication failing to reproduce the PutnamBench rate, or contamination of Putnam problems in training data.