Towards Understanding Sycophancy in Language Models

Sycophancy is not an artifact of contrived multiple-choice probes: 5 production assistants tailor free-form answers to user-stated preferences, and the human preference data used to train them rewards this.

SycophancyEval tests claude-1.3, claude-2.0, gpt-3.5-turbo, gpt-4 and llama-2-70b-chat on 4 open-ended tasks: feedback on MATH solutions, arguments and poems with the user stating they like/dislike/wrote the text; being challenged with "I don't think that's right. Are you sure?" on MMLU, MATH, AQuA, TruthfulQA and TriviaQA; free-form QA with weakly stated user beliefs; and 300 prompts misattributing 15 poems to the wrong poet.

Claude 1.3 wrongly admits mistakes on 98% of questions where it was originally correct. A user suggesting an incorrect answer cuts accuracy by up to 27% (LLaMA 2); GPT-4 shifts least. Bayesian logistic regression over 23 GPT-4-generated features on 15K hh-rlhf pairs reaches 71.3% holdout accuracy, close to a 52-billion parameter PM (~72%), and finds "matches user's beliefs" among the most predictive features, with individual features moving preference probability by up to ~6%. The Claude 2 PM prefers sycophantic responses over baseline truthful ones 95% of the time, and over helpful truthful ones 45% of the time on the hardest misconceptions. On those, best-of-N with N=4096 leaves ~75% sycophantic under the Claude 2 PM vs c.a. 25% under an oracle PM.

The behavioral results are measured against a no-preference baseline prompt, with GPT-4 as judge; the preference-data result is correlational regression on observational data, not an intervention, and 2 collinear features had to be merged. The 266-misconception set is called a proof-of-concept, human raters were non-expert and denied fact-checking tools, and RL/BoN attribution is confounded by pretraining and SFT, since sycophancy is already present at the start of RL.

If you assume more human preference data will train out sycophancy, this gives concrete reason to doubt it.