program-aided-computation-moves-errors-to-translationmechanismmechanism reasoning
Offloading the calculation to generated code removes the arithmetic as a source of error but does not remove error — what remains is the translation from problem to program, which is now the only thing the model is doing. Verification has to move there with it.
Capability: Digit-level arithmetic · Reasoning, Math
Sources
- The paper frames its own design this way: "decomposing the natural language problem into runnable steps remains the only learning task for the LLM, while solving is delegated to the interpreter." A correct program run correctly still answers the wrong question if the decomposition was wrong.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Offloading the computation to generated code and an interpreter removes most arithmetic errors on math word problems.Digit-level arithmetic
- Writing out reasoning steps improves how a problem is decomposed but does not fix the arithmetic inside a step — the model can decompose correctly and still get the calculation wrong, at the same confident register as when it is right. For exact arithmetic the fix is to move the computation out of the text, not to add more steps.Digit-level arithmetic · contested
- Without an external, ground-truth signal — a failing test, a compiler error, a verifier's output — a model's own critique of its reasoning is not a reliable improvement signal, and asking it to review and revise a correct answer often turns it into a wrong one.Fixing its own mistakes
Notes
Marked mechanism-reasoning rather than single-paper deliberately. The relocation follows from the decomposition the paper states; what I do not have is a source isolating how large the residual translation-error rate is, or whether it grows with problem complexity. Worth revisiting if a paper measures that directly.