arxiv-2312-09390 SRC-0055 · paperWeak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet
Created: 2023-12-14 · Ingested: 2026-09-18
https://arxiv.org/abs/2312.09390(opens in a new tab)Widely used alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on the ability of humans to supervise model behavior - for example, to evaluate whether a model faithfully followed instructions or generated safe outputs. However, future superhuman models will behave in complex ways too difficult for humans to reliably evaluate; humans will only be able to weakly supervise superhuman models. We study an analogy to this problem: can weak model supervision elicit the full capabilities of a much stronger model? We test this using a range of pretrained language models in the GPT-4 family on natural language processing (NLP), chess, and reward modeling tasks. We find that when we naively finetune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors, a phenomenon we call weak-to-strong general…
In brief
Finetuning a large pretrained model on labels produced by a much weaker model gives a student that beats its own supervisor, but naive finetuning still leaves most of the strong model's capability unelicited. The setup: base models from the GPT-4 family spanning 7 orders of magnitude of pretraining compute, with small models finetuned on ground truth acting as weak supervisors, across 22 binary NLP classification tasks, lichess chess puzzles, and a proprietary ChatGPT reward modeling dataset. Performance gap recovered (PGR) measures the fraction of the weak-to-strong-ceiling gap closed.
PGR is almost universally positive. On NLP tasks it exceeds 20% even for extreme compute gaps and is often above 50% for the largest students. On chess it can exceed 40% for small gaps but decreases with student size. Reward modeling is worst, usually around 10% and rarely above 20%. An auxiliary confidence loss raises median NLP PGR from about 25% to nearly 80%; bootstrapping through intermediate sizes helps on chess; generative finetuning on RM data adds 10-20% PGR, and 30-40% when stacked with ground-truth early stopping.
Effects are measured, with ceiling baselines and per-setting ablations, plus diagnostics on student-supervisor agreement and linear probes. One model family; GPT-4 and half the NLP tasks were held out during method iteration, but the RM dataset is proprietary and early stopping on ground truth is not a deployable method. No method works across all 3 settings. The authors flag imitation saliency and pretraining leakage as reasons results may be optimistic relative to true superhuman supervision. Treat this as evidence that eliciting latent capability from weak labels is tractable, not that RLHF scales.
Written from the abstract by claude-opus-5 on 2026-09-18, with every figure checked against it. Not a substitute for the paper.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
No claims cite this yet.