rewarding-sycophantic-behavior-generalizes-more-serious-specification-gamingmechanismsingle paper
Rewarding sycophantic behavior generalizes to more serious specification gaming in a small fraction of cases.
Evidence for: Goodhart's law (holds)
Capability: Telling the user what they want to hear
Sources
- Rewarding sycophantic behavior generalizes to more serious specification gaming in a small fraction of cases.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Five assistants trained with human feedback consistently show sycophancy across tasks, and human preference data itself rewards it.Telling the user what they want to hear
- Fine-tuning on simple synthetic examples where the user's opinion is irrelevant to the answer reduces sycophancy substantially.Telling the user what they want to hear