Few-shot arithmetic accuracy correlates with how often the specific numbers appear in pretraining data, evidence of memorization.

Sources

supportsImpact of Pretraining Term Frequencies on Few-Shot Reasoning
Few-shot arithmetic accuracy correlates with how often the specific numbers appear in pretraining data, evidence of memorization.

Status: activeLast checked: 2026-09-03Evidence activity

Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.

Related claims

Accuracy on a fact scales with how many pretraining documents mention it, so long-tail facts are systematically unreliable.Stating false facts confidently
Multi-digit multiplication accuracy falls to near zero as problem size grows, and models rely on pattern matching rather than a learned algorithm.Digit-level arithmetic