natural-language-logical-reasoning-benchmarks-auto-formalizing-each-reasoning-step-into CLM-0457On natural-language logical reasoning benchmarks, auto-formalizing each reasoning step into SMT-LIB and regenerating from the first step the solver cannot verify raises both the share of solver-verified steps and final-answer accuracy over plain chain-of-thought, and greedy regeneration at the failure point beats beam search or MCTS on step verifiability.
Capability: Carrying a chain of inference through to its conclusion
Observed on
Eight NL logic/entailment benchmarks (FOLIO, e-SNLI, LogiQA, ProntoQA, ProofWriter, QASC, SARA, SemEval) across seven models, 7B to frontier; tasks must be close enough to formal l.
Sources
- supportsLogicTrack: Auditing Reasoning Trajectories of Large Language Models with Formal Logic SolversValidated by: Compared against vanilla CoT on 168 benchmark-metric cells across 8 datasets and 7 models, improving in 133; unverified step ratio and verification-weighted accuracy reported per cell; also compared against LogicReward (OA 0.4338 to 0.5765) and against beam search and MCTS variants on SARA.. Measured on eight benchmarks and seven models against a vanilla CoT baseline only; no matched-compute baseline, so part of the gain may come from extra generation attempts rather than from solver signal. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For step-level verification of structured quantitative STEM reasoning, training a 7B process reward model only on hard negatives that are constructed to pass a deterministic symbolic verifier (swapped formula or unit-matched variable, with values recomputed so the math balances), and running the verifier as a hard filter before the PRM scores candidates, detects flawed steps better than monolithic PRMs that must judge arithmetic and logic together.Spotting an error in a given artifact · reviewed by AI
- Verifying a chain of thought by decomposing it into nodes and asking binary consistency questions per node localizes errors better than judging the whole chain at once (F1 up 10-25% over CoT-based verifiers, error-localization accuracy far above an 8-vote end-to-end judge) while using 6-58 times fewer tokens, because whole-chain judging dilutes attention and multi-sampling pays for long generations.Carrying a chain of inference through to its conclusion · reviewed by AI
- Having a model score its confidence step by step detects failures in multi-step reasoning better than one holistic confidence score for the whole answer, with up to 15% higher AUROC on two multi-step benchmarks, so where a chain fails is more visible per step than in the final answer.Carrying a chain of inference through to its conclusion · reviewed by AI
- On human-annotated step-level verification of frontier-model proofs to recent IMO and Putnam problems, most of 29 generative critics and process reward models do little better than chance at finding the first wrong step, a leading math PRM (Qwen2.5-Math-PRM-72B) reaches only about 37% balanced F1, and beyond a few standouts open-source verifiers lag closed models; verification is harder than generation for the same frontier models.Spotting an error in a given artifact · reviewed by AI
- For factual paired-candidate QA where the model already holds the needed knowledge, hallucination can be detected by removing the query keyword that most drives the chosen answer and seeing whether the prediction systematically recovers to the alternative — erroneous answers show that recovery while correct answers merely become ambiguous — but the signal weakens when the knowledge is absent and largely fails on multi-step arithmetic reasoning, where errors are distributed across steps rather than bound to one keyword.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A controlled comparison showing step-wise solver verification with immediate backtracking gives no better verified-step ratio or answer accuracy than post-hoc whole-chain verification at matched sampling budget, or that the gain disappears once autoformalization failures are accounted for.