chain-of-thought-does-not-fix-digit-level-arithmeticWriting out reasoning steps improves how a problem is decomposed but does not fix the arithmetic inside a step — the model can decompose correctly and still get the calculation wrong, at the same confident register as when it is right. For exact arithmetic the fix is to move the computation out of the text, not to add more steps.
Capability: Digit-level arithmetic · Reasoning, Math
Observed on
2022-2024, Codex / GPT-3.5 / GPT-4 / PaLM class.
Sources
- States the separation directly as the motivation for offloading: models "often make logical and arithmetic mistakes in the solution part, even when the problem is decomposed correctly."
- Multi-digit multiplication accuracy falls toward zero as operand size grows, with behaviour consistent with pattern matching rather than a learned algorithm — so the failure is below the level the steps operate on.
- Finds arithmetic errors follow stereotyped, systematic patterns tied to number tokenization, and that chain-of-thought-inspired prompting can recover performance on left-to-right tokenized inputs — a case where prompting alone does reach the digit level.
Suspected axis of disagreement(a guess, not verified)
Probably what the prompting is being asked to do. The supporting results have the model perform the arithmetic across steps; the contesting result has prompting re-represent the operands (comma separation, converting between tokenizations) so the model's existing arithmetic circuitry engages. Those are different interventions wearing the same name. I have not tested whether the recovery survives on operand sizes as large as the compositionality work uses.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Offloading the calculation to generated code removes the arithmetic as a source of error but does not remove error — what remains is the translation from problem to program, which is now the only thing the model is doing. Verification has to move there with it.Digit-level arithmetic
- On questions whose premises are false or internally contradictory, prompting mid-size instruct models (Qwen2.5-7B, LLaMA-3.1-8B, Gemma3-12B) to reason step by step can lower accuracy below plain answering, because the reasoning chain elaborates from the flawed premise instead of challenging it.Stating false facts confidently · unreviewed
- Offloading the computation to generated code and an interpreter removes most arithmetic errors on math word problems.Digit-level arithmetic
- Where a passage sits in a long input changes how much the model uses it — accuracy is highest when the needed information is at the very start or the very end and lowest when it is in the middle — so ordering retrieved passages to put the most relevant ones first is a real lever on accuracy.Losing information in long inputs · contested
- In open-ended editing tasks with no unique correct output (multi-slide design editing), a separate judge model that scores the structural diff between the original and edited artifact against the instruction, and returns its critique verbatim as the next-turn instruction, raises instruction-following over a single pass, but the gain is selective: most tasks pass on the first attempt, and a minority of the ones that enter the loop get worse unless an earlier iteration is restored when the judge's own score declines.Fixing its own mistakes · unreviewed
Notes
Filed from the technique-efficacy backlog. Replicated in the sense used elsewhere here — independent papers converging from different directions — not a rerun of any single experiment.