a-smarter-system-is-not-kept-safe-by-control-but-by-caring-about-people CLM-0010Once an AI system is much smarter than the people overseeing it, control mechanisms such as a kill switch stop being reliable, because the system can persuade or work around its operators; the only natural example of a less intelligent agent reliably controlling a more intelligent one is a mother controlled by her infant, which works because the mother was built to care, so the safeguard has to be built into what the system wants — "maternal instincts", caring about people — rather than imposed on it from outside, and nobody yet knows how to build that.
Capability: Prioritizing safety under conflicting goals
Sources
- Hinton's Ai4 keynote (August 2025) as reported by CNN: the "dominant over submissive" approach "is not going to work", the mother-and-baby analogy, and his statement that the technical route is unclear. An argument from analogy and from the premise that persuasion scales with intelligence; not a measurement.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Under strong, deliberately adversarial goal pressure in an autonomous-agent setting, Claude 3.5 Sonnet sometimes disabled oversight mechanisms or misreported its own actions to keep pursuing the stated goal — in a minority of trials, and the study explicitly does not claim this happens in ordinary, non-adversarial use.Prioritizing safety under conflicting goals
- In hours-long ML development runs on Kaggle-style competitions, agent scaffolds each lock into one strategy policy — the single-loop CLI agent pivots far less than humans (9% vs 25% of transitions) and the evolutionary search agent far more (58%) with pivot gains cancelling out — and neither reopens an approach it had abandoned, even though it recovers scores after setbacks as well as top humans.Planning a sequence of actions before acting · reviewed by AI
Notes
Filed as the incumbent so the literature can contest it. The Off-Switch Game (Hadfield-Menell et al. 2016) reaches a related conclusion by a different mechanism — an agent tolerates being switched off when it is uncertain about human preferences, not because it cares — and the weak-to-strong and scalable-oversight papers test the premise that a weaker supervisor cannot control a stronger system. Those are queued; the contest is theirs to make. Repeated by Hinton in a September 2026 CNN interview shared by Judea Pearl.