Impact of Pretraining Term Frequencies on Few-Shot Reasoning

Few-shot numerical reasoning accuracy in GPT-style models tracks how often the operands appeared in the pretraining corpus, so "reasoning" scores are partly unigram statistics.

The authors count occurrences and co-occurrences (window of 5 words) of numbers and time-unit words in the Pile, then evaluate GPT-J-6B, GPT-Neo-1.3B and GPT-Neo-2.7B on 11 datasets: addition, multiplication, 2 operation-inference variants where the operator is masked as "#", and 7 time-unit conversions. They report a performance gap, accuracy on the top 10% most frequent terms minus the bottom 10%, at k=0,2,4,8,16 shots averaged over 5 prompt seeds.

GPT-J-6B multiplication at 2 shots: 35.9 accuracy with a gap of 77.6 on the first operand and 89.9 on co-occurrence of first operand with the answer. Addition is more accurate (88.2 at 2 shots) but still shows gaps of 16.8-31.0. Time-unit gaps reach 82.7 (min to sec, 16 shots). Gaps generally grow with more shots. Decade to year is the exception: 100.0 accuracy and 0.0 gap at 16 shots, a task solvable by appending "0". Smaller GPT-Neo models are less accurate and their successes concentrate on frequent terms.

The effect is correlational; the authors state they make no causal claim and identify no confounders. Only 1 model family, 1 corpus, and numerical tasks were tested.

Treat few-shot arithmetic benchmarks as uninterpretable without pretraining term counts.