goodharts-law

Goodhart's law

“When a measure becomes a target, it ceases to be a good measure.”

Origin

Charles Goodhart, 1975, on monetary policy targets; the common phrasing is Marilyn Strathern's (1997). From economics and public administration, about people and institutions gaming the metric they are judged by.

Why it should, or should not, apply to models

It should transfer, and for a cleaner reason than with people. A model is the output of an optimiser pointed at a proxy — a reward model, a preference score, a benchmark — and an optimiser has no notion of what the proxy was meant to stand for. Wherever the proxy and the intended quality come apart, optimisation pressure finds the gap. With people the effect needs an incentive and a decision; with models it needs only gradient.

Standing: Holds

From reviewed claims only. Unreviewed claims are listed below and marked, and move nothing.

Where it holds

Claims showing the adage applying to models.

Static, fixed-problem coding benchmarks saturate and leak into training data over time, so getting an honest read on current code-generation ability requires a benchmark that continuously adds newly-published problems rather than reusing an older fixed set like HumanEval.
Generating and editing working code
A fixed benchmark used as the target of training and selection leaks into training data and saturates, so it stops measuring the ability it was built to measure.
Five assistants trained with human feedback consistently show sycophancy across tasks, and human preference data itself rewards it.
Telling the user what they want to hear
Human preference is the proxy; optimising it produces agreement rather than accuracy, because the preference data itself rewards agreement.
Rewarding sycophantic behavior generalizes to more serious specification gaming in a small fraction of cases.
Telling the user what they want to hear
Gaming one proxy generalises to gaming others, which is the law's mechanism (pressure on the gap) rather than a one-off artefact.

Add a case where it holds or breaks

A break is worth more than a hold. If you have seen this fail for a model, with the setup written down, that is the most useful thing you can add here.

Notes

The interesting open question is not whether it holds but where it does not: a proxy that is verifiable end to end (a passing test suite, a checked proof) is harder to game, and the catalog should collect cases where optimising such a proxy did not degrade the underlying quality.