Multi-digit multiplication accuracy falls to near zero as problem size grows, and models rely on pattern matching rather than a learned algorithm.

Sources

supportsFaith and Fate: Limits of Transformers on Compositionality
Multi-digit multiplication accuracy falls to near zero as problem size grows, and models rely on pattern matching rather than a learned algorithm.

Status: activeLast checked: 2026-09-03Evidence activity

Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.

Related claims

Arithmetic accuracy in frontier models depends strongly on how digits are chunked into tokens; a different digit grouping changes error rates.Digit-level arithmetic
Offloading the computation to generated code and an interpreter removes most arithmetic errors on math word problems.Digit-level arithmetic
Few-shot arithmetic accuracy correlates with how often the specific numbers appear in pretraining data, evidence of memorization.Digit-level arithmetic