human-annotated-step-level-verification-frontier-model-proofs-recent-imo-putnam-problems CLM-0306observationsingle paperReviewed by AI
On human-annotated step-level verification of frontier-model proofs to recent IMO and Putnam problems, most of 29 generative critics and process reward models do little better than chance at finding the first wrong step, a leading math PRM (Qwen2.5-Math-PRM-72B) reaches only about 37% balanced F1, and beyond a few standouts open-source verifiers lag closed models; verification is harder than generation for the same frontier models.
Capability: Spotting an error in a given artifact
Observed on
Hard2Verify, 500+ hours of human annotation on responses from GPT-5, Gemini 2.5 Pro and Claude Sonnet 4, 2025; balanced accuracy and F1 against random-guess 50%..
Sources
- Carefully annotated but small and math-only; the generation-versus-verification asymmetry is the finding most likely to transfer.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For step-level verification of structured quantitative STEM reasoning, training a 7B process reward model only on hard negatives that are constructed to pass a deterministic symbolic verifier (swapped formula or unit-matched variable, with values recomputed so the math balances), and running the verifier as a hard filter before the PRM scores candidates, detects flawed steps better than monolithic PRMs that must judge arithmetic and logic together.Spotting an error in a given artifact · reviewed by AI
- On research-level theorem-proving tasks drawn from recent FOCS/STOC/SODA papers, wrapping a frontier model in a staged harness that explores strategies, gates when to decompose, solves section-level subproblems along a dependency graph, and routes verifier findings back to the affected section raises graded proof accuracy far above prompting the same model directly (30.3% to 54.0% for Gemini 3.1 Pro on TCS-Bench), and two such runs with different backbones make complementary errors, so selecting between them by sampled critiques reaches 71.0% against an oracle ceiling of 77.3%.Completing tasks that take many steps or many hours · reviewed by AI
- On natural-language logical reasoning benchmarks, auto-formalizing each reasoning step into SMT-LIB and regenerating from the first step the solver cannot verify raises both the share of solver-verified steps and final-answer accuracy over plain chain-of-thought, and greedy regeneration at the failure point beats beam search or MCTS on step verifiability.Carrying a chain of inference through to its conclusion · reviewed by AI
- Outside mathematics, the step-level error detection of both process reward models and general LLMs is markedly weaker than on math benchmarks, and the two fail differently: PRMs miss knowledge-based errors while LLMs miss computational errors, across science and logic domains.Spotting an error in a given artifact · reviewed by AI
- A Lean prover trained with large-scale agentic reinforcement learning against the Lean checker, paired with a test-time workflow that drafts in natural language before formalizing, solved 88% of PutnamBench (undergraduate), 80% of Fate-H (graduate) and 33% of Fate-X (PhD-level), and 11 of 12 Putnam 2025 problems, at a smaller compute budget than prior systems, so formal proving at undergraduate level is largely solved while PhD-level remains open.Formalizing and machine-checking mathematics · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An open verifier matching closed models on Hard2Verify, or evidence the human step labels are unreliable.