verifying-chain-thought-decomposing-into-nodes-asking-binary-consistency CLM-0327mechanismsingle paperReviewed by AI
Verifying a chain of thought by decomposing it into nodes and asking binary consistency questions per node localizes errors better than judging the whole chain at once (F1 up 10-25% over CoT-based verifiers, error-localization accuracy far above an 8-vote end-to-end judge) while using 6-58 times fewer tokens, because whole-chain judging dilutes attention and multi-sampling pays for long generations.
Capability: Carrying a chain of inference through to its conclusion
Observed on
Public multi-step reasoning datasets, training-free verification with open models, 2025; F1 and localization accuracy versus CoT verifiers and voting..
Sources
- Validated by: F1 and error-localization accuracy against end-to-end CoT verifiers with and without voting, with token counts, on public datasets.. Method paper; the cost ratio is large and the attention-dilution explanation is plausible but not directly tested.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a long chain of thought is verified by classifying a graph of its steps, the predictive signal comes almost entirely from the semantic content of the steps, not the graph shape: replacing step embeddings with purely structural metadata (child/parent counts, chain index, position) drops accuracy to near chance on math, science and multiple-choice reasoning, while remaining above chance only on code.Carrying a chain of inference through to its conclusion · reviewed by AI
- For 0.6B-8B Qwen3 models on math, logic and multiple-choice reasoning, quantifying reasoning uncertainty from a directed acyclic graph built by sampling several chains and merging semantically equivalent steps with an NLI model separates correct from incorrect generations better than sequence-level entropy, likelihood and self-prompting uncertainty measures.Carrying a chain of inference through to its conclusion · reviewed by AI
- On natural-language logical reasoning benchmarks, auto-formalizing each reasoning step into SMT-LIB and regenerating from the first step the solver cannot verify raises both the share of solver-verified steps and final-answer accuracy over plain chain-of-thought, and greedy regeneration at the failure point beats beam search or MCTS on step verifiability.Carrying a chain of inference through to its conclusion · reviewed by AI
- When an error is injected into a supplied chain of thought, models mostly carry it to the answer: a planted arithmetic error costs small models 50-60% accuracy and 18% on average, though the largest models lose only 3-7%; unit-conversion errors cost about 9% at every scale; skipped steps and sycophantic steps cost small models about 10%; extra correct steps cost almost nothing (0-6%).Carrying a chain of inference through to its conclusion · reviewed by AI
- General-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Chains where the error is global (wrong strategy) and every node is locally consistent.