arxiv-2211-03540 SRC-0015 · paperMeasuring Progress on Scalable Oversight for Large Language Models
Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell
Created: 2022-11-04 · Ingested: 2026-09-18
https://arxiv.org/abs/2211.03540(opens in a new tab)Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broadly exceed our abilities. This paper discusses one of the major ways we think about this problem, with a focus on ways it can be studied empirically. We first present an experimental design centered on tasks for which human specialists succeed but unaided humans and current general AI systems fail. We then present a proof-of-concept experiment meant to demonstrate a key feature of this experimental design and show its viability with two question-answering tasks: MMLU and time-limited QuALITY. On these tasks, we find that human participants who interact with an unreliable larg…
In brief
Non-expert humans chatting with an unreliable RLHF dialog model beat both the model alone and their own unaided performance on questions neither can reliably answer. This is the first empirical test of the sandwiching setup for scalable oversight.
Anthropic ran 2 question-answering tasks with a 52B helpful-only RLHF assistant: MMLU exam questions, and QuALITY reading comprehension under a 5-minute limit on ~5,000-word stories. 10 Surge AI participants (2 outliers dropped) each answered 100 validation questions assisted and 100 unassisted, with a chat pane and a conversation reset button.
Individual accuracy: 57.2 unassisted vs 65.6 for best-of-20 chain-of-thought vs 75.4 human+model on MMLU; 48.6 vs 66.9 vs 76.8 on QuALITY. Weighted majority vote reached 78.0 and 86.0, still under expert estimates of 90.0 and 93.5. Calibration got worse with assistance (CE 12 and 18 on MMLU).
The comparison is measured and within-participant, with automatic model baselines including 5-shot and best-of-20 CoT. But 2 of 10 participants were removed for outperforming the model, creating a selection effect toward positive results, answers were revealed after each question, and 1 model and 1 size were tested.
Nothing here covers fine-tuning-based oversight, free-form tasks, or adversarial models; multiple-choice QA is close to training data.
Treat this as evidence the paradigm is runnable, not that chat-based oversight is sufficient.
Written from the abstract by claude-opus-5 on 2026-09-18, with every figure checked against it. Not a substitute for the paper.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
No claims cite this yet.