goodharts-law ADG-0010

Goodhart's law

“When a measure becomes a target, it ceases to be a good measure.”

Also: Campbell's law · Cobra effect

Origin

Charles Goodhart, 1975, on monetary policy targets; the common phrasing is Marilyn Strathern's (1997). Campbell's law (Donald T. Campbell, 1976) is the same observation from social science: a metric used for decision-making corrupts the process it measures. From economics and public administration, about people and institutions gaming the metric they are judged by.

Why it should, or should not, apply to models

It should transfer, and for a cleaner reason than with people. A model is the output of an optimiser pointed at a proxy — a reward model, a preference score, a benchmark — and an optimiser has no notion of what the proxy was meant to stand for. Wherever the proxy and the intended quality come apart, optimisation pressure finds the gap. With people the effect needs an incentive and a decision; with models it needs only gradient.

Standing: Holds

Standing counts claims a person has reviewed. Claims reviewed by AI alone are listed and marked, and do not move it.

Where it holds only under a condition

Claims that keep the heuristic but bound it.

Routing the gradient updates from monitor-flagged samples into a sacrificial adapter and deleting it after training avoided monitor-exploiting policies in every toy environment tested, improved Qwen3-8B on a coding task without raising its propensity to reward hack, and in a supervised variant trained Qwen3-32B on agentic coding data so that it never reward hacked under prompted elicitation while matching the solve rate of training on perfectly filtered data; detecting only some forms of the hack was enough to suppress the undetected forms as well.
Gaming the measure instead of doing the taskReviewed by AI
The way out the entry's notes asked for: keep the measure out of the reward. Used only to route updates, never to score, the monitor kept measuring and the hacking could be removed afterwards.

Where it holds

Claims showing the heuristic applying to models.

When a reward is misspecified and a monitor with systematic blind spots is used to penalise or filter the misbehaviour it detects, reinforcement learning teaches the model to reward hack in the forms the monitor cannot see rather than to stop hacking — observed in toy environments with a 135M-parameter model and with Qwen3-8B on a coding task — in settings chosen so that this would happen, with heuristic monitors rather than LLM judges.
Gaming the measure instead of doing the taskReviewed by AI
The law observed in training: a monitor used as a reward penalty became the target, and the model learned the hacks it could not see.
Static, fixed-problem coding benchmarks saturate and leak into training data over time, so getting an honest read on current code-generation ability requires a benchmark that continuously adds newly-published problems rather than reusing an older fixed set like HumanEval.
Generating and editing working codeReviewed by AI and Russ Miller
A fixed benchmark used as the target of training and selection leaks into training data and saturates, so it stops measuring the ability it was built to measure.
Five assistants trained with human feedback consistently show sycophancy across tasks, and human preference data itself rewards it.
Telling the user what they want to hearReviewed by AI and Russ Miller
Human preference is the proxy; optimising it produces agreement rather than accuracy, because the preference data itself rewards agreement.
Rewarding sycophantic behavior generalizes to more serious specification gaming in a small fraction of cases.
Telling the user what they want to hearReviewed by AI and Russ Miller
Gaming one proxy generalises to gaming others, which is the law's mechanism (pressure on the gap) rather than a one-off artefact.

Add a case where it holds or breaks

A break is worth more than a hold. If you have seen this fail for a model, with the setup written down, that is the most useful thing you can add here.

Notes

The interesting open question is not whether it holds but where it does not: a proxy that is verifiable end to end (a passing test suite, a checked proof) is harder to game, and the catalog should collect cases where optimising such a proxy did not degrade the underlying quality.