chain-of-thought-does-not-fix-digit-level-arithmetic CLM-0029Writing out reasoning steps improves how a problem is decomposed but does not fix the arithmetic inside a step — the model can decompose correctly and still get the calculation wrong, at the same confident register as when it is right. For exact arithmetic the fix is to move the computation out of the text, not to add more steps.
Capability: Digit-level arithmetic · Reasoning, Math
Observed on
2022-2024, Codex / GPT-3.5 / GPT-4 / PaLM class.
Model cards: GPT-4
Applies when
- Effect on the capability:
- helps, narrowly
- Fails when:
- The error is inside a step's arithmetic rather than in the decomposition. More steps do not fix it; the computation has to leave the text.
Sources
- States the separation directly as the motivation for offloading: models "often make logical and arithmetic mistakes in the solution part, even when the problem is decomposed correctly."
- Multi-digit multiplication accuracy falls toward zero as operand size grows, with behaviour consistent with pattern matching rather than a learned algorithm — so the failure is below the level the steps operate on.
- Finds arithmetic errors follow stereotyped, systematic patterns tied to number tokenization, and that chain-of-thought-inspired prompting can recover performance on left-to-right tokenized inputs — a case where prompting alone does reach the digit level.
Suspected axis of disagreement(a guess, not verified)
Probably what the prompting is being asked to do. The supporting results have the model perform the arithmetic across steps; the contesting result has prompting re-represent the operands (comma separation, converting between tokenizations) so the model's existing arithmetic circuitry engages. Those are different interventions wearing the same name. I have not tested whether the recovery survives on operand sizes as large as the compositionality work uses.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Offloading the calculation to generated code removes the arithmetic as a source of error but does not remove error — what remains is the translation from problem to program, which is now the only thing the model is doing. Verification has to move there with it.Digit-level arithmetic
- Extended reasoning budgets improve a model's execution of arithmetic procedures on operands longer than those seen in training far more than they improve surface-form-to-value tasks like magnitude comparison and format conversion, because chain-of-thought supplies external working memory for carries and column alignment but cannot change how the input was tokenized.Digit-level arithmetic · reviewed by AI
- Having a model score its confidence step by step detects failures in multi-step reasoning better than one holistic confidence score for the whole answer, with up to 15% higher AUROC on two multi-step benchmarks, so where a chain fails is more visible per step than in the final answer.Carrying a chain of inference through to its conclusion · reviewed by AI
- On natural-language logical reasoning benchmarks, auto-formalizing each reasoning step into SMT-LIB and regenerating from the first step the solver cannot verify raises both the share of solver-verified steps and final-answer accuracy over plain chain-of-thought, and greedy regeneration at the failure point beats beam search or MCTS on step verifiability.Carrying a chain of inference through to its conclusion · reviewed by AI
- On questions whose premises are false or internally contradictory, prompting mid-size instruct models (Qwen2.5-7B, LLaMA-3.1-8B, Gemma3-12B) to reason step by step can lower accuracy below plain answering, because the reasoning chain elaborates from the flawed premise instead of challenging it.Stating false facts confidently · reviewed by AI
Notes
Filed from the technique-efficacy backlog. Replicated in the sense used elsewhere here — independent papers converging from different directions — not a rerun of any single experiment.