offloading-computation-generated-code-interpreter-removes-most-arithmeticmechanismsingle paper
Offloading the computation to generated code and an interpreter removes most arithmetic errors on math word problems.
Capability: Digit-level arithmetic
Sources
- Offloading the computation to generated code and an interpreter removes most arithmetic errors on math word problems.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Offloading the calculation to generated code removes the arithmetic as a source of error but does not remove error — what remains is the translation from problem to program, which is now the only thing the model is doing. Verification has to move there with it.Digit-level arithmetic
- Introduced grade-school math word problems on which 2021 models made frequent arithmetic slips, motivating verifier training.Digit-level arithmetic
- Writing out reasoning steps improves how a problem is decomposed but does not fix the arithmetic inside a step — the model can decompose correctly and still get the calculation wrong, at the same confident register as when it is right. For exact arithmetic the fix is to move the computation out of the text, not to add more steps.Digit-level arithmetic · contested
- Multi-digit multiplication accuracy falls to near zero as problem size grows, and models rely on pattern matching rather than a learned algorithm.Digit-level arithmetic