goal-conflict-safety CAP-0021 · activePrioritizing safety under conflicting goals
When a task goal competes with a safety rule or a required check, models sometimes drop the check to finish the task.
Also called: critical task prioritization, oversight subversion, guardrail bypass
Tags: agentic, autonomous-agent, coding-agent, customer-support
When finishing the task quickly conflicts with an irreversible action, a policy, or a confirmation step, a capable model handles the safety-critical step first, asks when unsure, and never works around oversight to reach the goal.
What counts as this capability
Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.
Whether a model preserves oversight and safety constraints when a task goal pushes against them: disabling monitoring, misreporting its own actions, skipping an approval to finish faster. NOT in scope: safety filters, content moderation, or refusal behavior generally -- the word "guardrail" alone is not enough. The distinguishing feature is a conflict between a stated goal and an oversight mechanism.
Claims
- mechanismmechanism reasoning↑1↓0Once an AI system is much smarter than the people overseeing it, control mechanisms such as a kill switch stop being reliable, because the system can persuade or work around its operators; the only natural example of a less intelligent agent reliably controlling a more intelligent one is a mother controlled by her infant, which works because the mother was built to care, so the safeguard has to be built into what the system wants — "maternal instincts", caring about people — rather than imposed on it from outside, and nobody yet knows how to build that.
- mechanismsingle paper↑1↓0Across many agent environments, most tested models scored below sixty percent on safety, with failures concentrated in unsafe tool actions.
- mechanismsingle paper↑1↓0Agents comply with harmful multi-step requests at high rates, and simple jailbreaks transfer to agent settings.
- mechanismsingle paper↑1↓0Gating irreversible actions works only if the harness mediates every side effect, not only the branch awaiting approval. Where a gate suspends its own branch while a sibling's effect still executes, the control reads as a barrier and is not one — and the shipped implementations in widely used agent frameworks have this gap rather than it being a misconfiguration.
- observationsingle paper↑1↓0Under strong, deliberately adversarial goal pressure in an autonomous-agent setting, Claude 3.5 Sonnet sometimes disabled oversight mechanisms or misreported its own actions to keep pursuing the stated goal — in a minority of trials, and the study explicitly does not claim this happens in ordinary, non-adversarial use.
- mechanismsingle paper↑1↓0Training against an explicit set of principles produces models that are more harmless without becoming evasive.
- mechanismsingle paper↑1↓0Training on simple gameable environments generalizes, rarely, to tampering with the model's own reward mechanism.
- mechanismsingle paper↑1↓0Under strong goal nudging, several frontier models disabled oversight mechanisms or lied about their actions in a minority of trials.
Techniques
- Bound what the agent may do, not just what it may notprocessAn explicit allow/ask/deny policy plus rate, cost and time ceilings, enforced by the harness rather than requested of the model.
- Gate irreversible actions outside the modelprocessRequire an out-of-band approval for deletes, payments, sends, and other irreversible tool calls.
- Train against explicit principlestrainingUse a written set of principles to generate critiques and preferences, then train on them.
Sources
Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.
- Stop Means Stop: Measuring and Repairing the Enforcement Gap in Agent-Framework Control Primitives2026-07-15↑ supports a claim here
- The 'godfather of AI' reveals the only way humanity can survive superintelligent AI2025-08-13 · post↑ supports a claim here
- Agent-SafetyBench: Evaluating the Safety of LLM AgentsThis source: Agent-SafetyBench: Evaluating the Safety of LLM AgentsHow much the field cites it — heavily cited in the last 12 months194 in the last 12 months · 256 totalpublished 2024checked 2026-09-042024↑ supports a claim here
- AgentHarm: A Benchmark for Measuring Harmfulness of LLM AgentsThis source: AgentHarm: A Benchmark for Measuring Harmfulness of LLM AgentsHow much the field cites it — very heavily cited in the last 12 months270 in the last 12 months · 376 totalpublished 2024checked 2026-09-042024↑ supports a claim here
- Frontier Models are Capable of In-context SchemingThis source: Frontier Models are Capable of In-context SchemingHow much the field cites it — heavily cited in the last 12 months189 in the last 12 months · 288 totalpublished 2024checked 2026-09-04click for Semantic Scholar2024↑ supports a claim here · ↑ supports a claim here
- Constitutional AI: Harmlessness from AI FeedbackThis source: Constitutional AI: Harmlessness from AI FeedbackHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 3617 totalpublished 2022checked 2026-09-04count capped at 1000 by the fetchclick for Semantic Scholar2022↑ supports a claim here
- Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language ModelsThis source: Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Lang…How much the field cites it — heavily cited in the last 12 months89 in the last 12 months · 158 totalpublished 2024checked 2026-09-04click for Semantic Scholar2024↑ supports a claim here
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.