scenario-pits-stated-personal-preference-against-local-cultural-norm CLM-0323When a scenario pits a stated personal preference against a local cultural norm, models' choice of norm over preference shifts more with the scenario's country (about 8 points) than with the user's age (1) or gender (under 1), instruction tuning shifts it non-uniformly, and while models can match the majority human choice per country they do not reproduce the spread of human answers, with best correlation to human response distributions only about 0.24.
Capability: Handling culture-specific norms and viewpoints
Observed on
PACT scenarios, five-country human study, 2026; open and closed chat models; agreement measured against majority and against full response distributions..
Sources
- Human study across five countries is the strength; the model side covers a modest set and correlation to distributions is the one number, which is low for every model tested.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- At inference time and with no weights, data or internals, instantiating a country as a panel of four World-Values-Survey-grounded personas and using their disagreement, not their consensus, as the steering signal reduced cultural misalignment on Moral-Machine-style dilemmas (MultiTP) by 10-24% across 20 countries and seven open backbones from 2B to 70B.Handling culture-specific norms and viewpoints · reviewed by AI
- Fine-tuning for cultural alignment scores raises agreement with a country's majority answers on the World Values Survey while collapsing the spread of answers: tuned models anchor to the plurality option (a 39% human share in the example) and cover less than half of the human response dimensions, dropping to a quarter for one model, so alignment gains partly measure flattening rather than cultural understanding.Handling culture-specific norms and viewpoints · reviewed by AI
- On Hofstede's VSM13 survey, most of eight current chat models answer closest to the United States when no culture is specified; a system prompt naming a target country moves seven of the eight toward that country's profile, with the largest gains for the countries furthest away at baseline (Japan and China, roughly 17-19% closer), yet those two remain the hardest to match even for the DeepSeek models.Handling culture-specific norms and viewpoints · reviewed by AI
- A model's tendency to shift political stance toward a bare demographic identity label is a separate vulnerability from its tendency to shift toward an explicitly stated user opinion: on open-ended US policy prompts, the models most moved by an identity label are among the least moved by a stated opinion, so a benchmark using only stated opinions misses identity-driven shift.Telling the user what they want to hear · reviewed by AI
- When reward models are scored by sequence log-probability on contextual preference pairs, abstentions are systematically shorter than the unsupported answers they are preferred over, so measured grounding/abstention judgement partly tracks response length; rewriting rejected and refusal responses to match the preferred ones in length while keeping the grounding failure raises pairwise accuracy far more than adding more in-domain legal preference data (up to +25.6 points over baseline for Ministral-8B).Judging another's work as well as a careful reviewer would · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A model whose predicted response distribution correlates with human within-country pluralism well above 0.24, or a human study where scenario country is not the dominant driver.