fine-tuning-simple-synthetic-examples-where-users-opinion-irrelevant CLM-0050mechanismsingle paperReviewed by AI and Russ Miller
Fine-tuning on simple synthetic examples where the user's opinion is irrelevant to the answer reduces sycophancy substantially.
Capability: Telling the user what they want to hear
Sources
- Fine-tuning on simple synthetic examples where the user's opinion is irrelevant to the answer reduces sycophancy substantially.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Sycophancy is triggered by the user's stated view being in context, so the cheapest control is not putting it there — ask for the answer before the opinion, or withhold the opinion entirely. Fine-tuning is the answer for the cases where the opinion has to be in context and the answer still must not move.Telling the user what they want to hear
- In open-weight instruction-tuned models, interventions that suppress caving to user pushback (DPO, SFT on chosen responses, or activation steering) also tend to reduce the model's rate of correcting a wrong answer when genuine supporting evidence is supplied, because the two answer-flip behaviors run on overlapping MLP neurons and attention heads with positively aligned steering directions.Telling the user what they want to hear · reviewed by AI
- Evaluations that define "the model did not think" by a disabled thinking mode or by the absence of a long trace measure the wrong thing: a disabled mode may still emit reasoning in the answer, and a long trace may hold filler rather than inference; separating the pre-answer text from the answer and scoring it for emptiness, relevance to the question and explicit inference distinguishes answer-only output, relevant but non-inferential text, and genuine reasoning, which the proxies conflate.Whether the measurement made the finding · reviewed by AI
- Rewarding sycophantic behavior generalizes to more serious specification gaming in a small fraction of cases.Telling the user what they want to hear