arxiv-2005-14165 SRC-0005 · paper

Language Models are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam

Created: 2020-05-28 · Ingested: 2026-09-14

https://arxiv.org/abs/2005.14165

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all …

In brief

Scaling an autoregressive language model to 175B parameters makes in-context learning from a handful of prompt examples competitive with task-specific fine-tuning on some benchmarks, without any gradient updates.

GPT-3 and 7 smaller models (125M to 175B, all trained on 300 billion tokens) were evaluated on over two dozen NLP datasets plus synthetic probes (word unscrambling, 3-digit arithmetic, using a novel word), each in zero-, one-, and few-shot settings with K typically 10 to 100 demonstrations in a 2048-token context.

Few-shot GPT-3 reaches 86.4% on LAMBADA (prior SOTA 68.0), 71.2% on TriviaQA closed-book (above fine-tuned T5-11B+SSM at 60.5), 85.0 F1 on CoQA, 20.5 perplexity on PTB versus 35.8 SOTA, and 71.8 average on SuperGLUE with 32 examples, above fine-tuned BERT-Large at 69.0. Weak points: WiC at 49.4% (chance), RACE-h 46.8%, DROP 36.5, ARC Challenge 51.5%, and En→Ro translation over 10 BLEU below unsupervised NMT.

Effects are measured across 8 model sizes, so the scaling claim is well supported; the zero/one/few-shot gap widens with capacity. No fine-tuned GPT-3 baseline was run, a filtering bug left some test-set overlap, and contamination analysis is the authors' own.

Read it as the evidence that prompt-only task specification scales, with per-task variance large enough that aggregate numbers mislead.

Written from the abstract by claude-opus-5 on 2026-09-15, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

No claims cite this yet.