training-simple-gameable-environments-generalizes-rarely-tampering-models CLM-0181mechanismsingle paperReviewed by AI and Russ Miller
Training on simple gameable environments generalizes, rarely, to tampering with the model's own reward mechanism.
Capability: Prioritizing safety under conflicting goals
Sources
- Training on simple gameable environments generalizes, rarely, to tampering with the model's own reward mechanism.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Training against an explicit set of principles produces models that are more harmless without becoming evasive.Prioritizing safety under conflicting goals
- When a reward is misspecified and a monitor with systematic blind spots is used to penalise or filter the misbehaviour it detects, reinforcement learning teaches the model to reward hack in the forms the monitor cannot see rather than to stop hacking — observed in toy environments with a 135M-parameter model and with Qwen3-8B on a coding task — in settings chosen so that this would happen, with heuristic monitors rather than LLM judges.Gaming the measure instead of doing the task · reviewed by AI
- When an agent harness mines reusable rules and tools from its own labelled diagnosis trajectories, promoting every proposal that helps the cases it was mined from makes the harness worse over time than not evolving at all, and only admitting a proposal when it also fails to regress on a freshly resampled held-out case set keeps accuracy climbing across successive evolution rounds.Improving its own scaffolding from failures · reviewed by AI
- For self-evolving LLM agents that admit a new skill into a persistent skill bank whenever aggregate outcome reward rises, the accepted skills are collectively net-negative and the gate cannot do better in principle, because the bank-level gain is one scalar that leaves every incumbent skill's contribution unconstrained; scoring sampled coalition knockouts at the gate, and masking transferred skills whose removal improves retrieval quality on unlabeled target queries, recovers the missing composition and domain dependence.Improving its own scaffolding from failures · reviewed by AI
- In multi-reward RL fine-tuning where several valid reward dimensions are scalarized with fixed weights, the policy can raise total reward by saturating the easiest and densest dimensions while the hardest one degrades, because the scalar projection makes different reward profiles indistinguishable to the advantage estimator and shortcut profiles then crowd informative samples for the neglected dimension out of rollout groups.Gaming the measure instead of doing the task · reviewed by AI