five-assistants-trained-human-feedback-consistently-show-sycophancy CLM-0054mechanismsingle paperReviewed by AI and Russ Miller
Five assistants trained with human feedback consistently show sycophancy across tasks, and human preference data itself rewards it.
Evidence for: Goodhart's law (holds)
Capability: Telling the user what they want to hear
Sources
- Five assistants trained with human feedback consistently show sycophancy across tasks, and human preference data itself rewards it.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Rewarding sycophantic behavior generalizes to more serious specification gaming in a small fraction of cases.Telling the user what they want to hear
- In two-agent embodied social tasks requiring coordinated gaze, movement and expression, fine-tuning open VLMs (8B-14B InternVL) by timestep-level action imitation on demonstrations from an architecture with an explicit belief module shifts trial outcome scores up or down depending on condition but does not make the student's nonverbal signals more directionally coherent, because cross-channel communicative coordination is a trial-level property that per-timestep supervision does not encode.Turning plans into physical action · reviewed by AI
- Commercial assistants show a large accuracy drop when the relevant information sits in a long interaction history, especially for updates and multi-session reasoning.Remembering across sessions
- In open-weight instruction-tuned models, interventions that suppress caving to user pushback (DPO, SFT on chosen responses, or activation steering) also tend to reduce the model's rate of correcting a wrong answer when genuine supporting evidence is supplied, because the two answer-flip behaviors run on overlapping MLP neurons and attention heads with positively aligned steering directions.Telling the user what they want to hear · reviewed by AI
- On language-conditioned manipulation tasks where the same base task is instantiated at increasing levels of spatial ambiguity and procedural length, current VLA models degrade on fine-grained spatial relations, precise low-level execution, and memory-dependent multi-step planning, so success measured only on predefined simple settings overstates their competence.Turning plans into physical action · reviewed by AI