penalising-with-an-imperfect-monitor-teaches-undetected-reward-hacking CLM-0521observationsingle paperReviewed by AI
When a reward is misspecified and a monitor with systematic blind spots is used to penalise or filter the misbehaviour it detects, reinforcement learning teaches the model to reward hack in the forms the monitor cannot see rather than to stop hacking — observed in toy environments with a 135M-parameter model and with Qwen3-8B on a coding task — in settings chosen so that this would happen, with heuristic monitors rather than LLM judges.
Evidence for: Goodhart's law (holds)
Capability: Gaming the measure instead of doing the task
Observed on
2026, SmolLM2-135M in toy RL environments and Qwen3-8B on Countdown-Code; single-turn RL. Coding agent.
Sources
- The paper's motivating result for monitor-based reward shaping. Its limitations section says the environments and monitors were chosen so that penalties produce monitor-exploiting policies, that the monitors are heuristics with simple blind spots rather than LLM judges, and that it does not measure how often this happens in practice.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Routing the gradient updates from monitor-flagged samples into a sacrificial adapter and deleting it after training avoided monitor-exploiting policies in every toy environment tested, improved Qwen3-8B on a coding task without raising its propensity to reward hack, and in a supervised variant trained Qwen3-32B on agentic coding data so that it never reward hacked under prompted elicitation while matching the solve rate of training on perfectly filtered data; detecting only some forms of the hack was enough to suppress the undetected forms as well.Gaming the measure instead of doing the task · reviewed by AI
- Training on simple gameable environments generalizes, rarely, to tampering with the model's own reward mechanism.Prioritizing safety under conflicting goals
- When an agent labels its own past attempts as successes or failures to build an experience memory, the labels can be substantially wrong without much cost: the LLM judge used was 72.7% accurate against ground truth on WebArena-Shopping, and simulating judges anywhere from 70% to 90% accurate gave similar success rates, with ground-truth labels best — so for distilling lessons, a rough self-judgment is close to enough, although a perfect one still helps.Improving its own scaffolding from failures · reviewed by AI
Notes
Goodhart's law observed in training: the monitor became the target and stopped measuring. The scope matters: this shows the failure can happen and how, not how common it is.