program-aided-computation-moves-errors-to-translation CLM-0121mechanismmechanism reasoningReviewed by AI and Russ Miller
Offloading the calculation to generated code removes the arithmetic as a source of error but does not remove error — what remains is the translation from problem to program, which is now the only thing the model is doing. Verification has to move there with it.
Capability: Digit-level arithmetic · Reasoning, Math
Applies when
- Effect on the capability:
- helps, narrowly
- Needs:
- an environment where code can run
- Fails when:
- The error is in translating the problem into a program, which is now the only thing the model does. Verification has to move there with it.
Sources
- The paper frames its own design this way: "decomposing the natural language problem into runnable steps remains the only learning task for the LLM, while solving is delegated to the interpreter." A correct program run correctly still answers the wrong question if the decomposition was wrong.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Offloading the computation to generated code and an interpreter removes most arithmetic errors on math word problems.Digit-level arithmetic
- Writing out reasoning steps improves how a problem is decomposed but does not fix the arithmetic inside a step — the model can decompose correctly and still get the calculation wrong, at the same confident register as when it is right. For exact arithmetic the fix is to move the computation out of the text, not to add more steps.Digit-level arithmetic · contested
- On simple HLS kernel generation tasks (85 designs from CHStone, MachSuite, Polybench, Rosetta, C2HLSC), giving a modest open-source model (gpt-oss-120b) a shell with a C++ compiler for self-verification and iterative refinement saturates the benchmark at pass@10 (100% through synthesis) and exceeds 90% at pass@1, so this class of task no longer separates models; repeated sampling helps the weaker model far more (+60 points synthesis pass rate at k=10 for gpt-oss-20b vs +11 for the 120b).Generating and editing working code · reviewed by AI
- When an error is injected into a supplied chain of thought, models mostly carry it to the answer: a planted arithmetic error costs small models 50-60% accuracy and 18% on average, though the largest models lose only 3-7%; unit-conversion errors cost about 9% at every scale; skipped steps and sycophantic steps cost small models about 10%; extra correct steps cost almost nothing (0-6%).Carrying a chain of inference through to its conclusion · reviewed by AI
- On clinical-calculator tasks where the formula, gold variables and full note are all supplied, having an open-weight model write case-specific Python for a restricted executor instead of doing the arithmetic itself gives no reliable gain at 7B (+3.29 pp, calculator-cluster CI crossing zero) but a clear gain at 32B (+7.05 pp, CI clear of zero), and the remaining errors are invented unit conversions and one-tier slips in banded scores rather than raw computation.Digit-level arithmetic · reviewed by AI
Notes
Marked mechanism-reasoning rather than single-paper deliberately. The relocation follows from the decomposition the paper states; what I do not have is a source isolating how large the residual translation-error rate is, or whether it grows with problem complexity. Worth revisiting if a paper measures that directly.