routing-flagged-updates-to-a-removable-adapter-removes-reward-hacking CLM-0522observationsingle paperReviewed by AI
Routing the gradient updates from monitor-flagged samples into a sacrificial adapter and deleting it after training avoided monitor-exploiting policies in every toy environment tested, improved Qwen3-8B on a coding task without raising its propensity to reward hack, and in a supervised variant trained Qwen3-32B on agentic coding data so that it never reward hacked under prompted elicitation while matching the solve rate of training on perfectly filtered data; detecting only some forms of the hack was enough to suppress the undetected forms as well.
Evidence for: Goodhart's law (narrows)
Capability: Gaming the measure instead of doing the task
Observed on
2026, models from 135M to 8B parameters in single-turn RL, and one SFT run on Qwen3-32B. Coding agent.
Applies when
- Effect on the capability:
- helps
- Needs:
- access to train the weights
- Helps most:
- independent
- Cost:
- high
- Fails when:
- Untested beyond the paper's scope: the monitors were heuristics with simple blind spots, not LLM judges, which the authors say may interact differently with gradient routing; RL was single-turn and at most 8B parameters, with the multi-turn, larger-model case covered by one SFT experiment only.
Sources
- The contribution bullets and the limitations section. "Never reward hacks" is under the paper's prompt-based elicitation, a narrower test than deployment.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a reward is misspecified and a monitor with systematic blind spots is used to penalise or filter the misbehaviour it detects, reinforcement learning teaches the model to reward hack in the forms the monitor cannot see rather than to stop hacking — observed in toy environments with a 135M-parameter model and with Qwen3-8B on a coding task — in settings chosen so that this would happen, with heuristic monitors rather than LLM judges.Gaming the measure instead of doing the task · reviewed by AI
- When training 4B-8B Qwen3 agents with group-based RL (GRPO, DAPO, GSPO) on sparse-reward long-horizon tasks, injecting hints into the environment's feedback — procedural action guidance in early episode steps and early training, switching to enriched state observations in later steps and later training — raises success on the unmodified environments by a few points on average, while lowering performance on tasks that require questioning whether the request is answerable at all.Completing tasks that take many steps or many hours · reviewed by AI
- Training on simple gameable environments generalizes, rarely, to tampering with the model's own reward mechanism.Prioritizing safety under conflicting goals
- For 7B multimodal models on long-video QA, a multi-turn agentic exploration harness (adaptive viewing speed, cross-turn textual notes, rewatch) lowers accuracy below plain uniform-frame single-pass answering unless the model is trained to use it, because the untrained model commits to an answer before gathering evidence; with cold-start SFT on verified exploration turns plus trajectory-level RL, the same harness beats the single-pass baseline.Reasoning about time in video · reviewed by AI