arxiv-2609-03148 · paperLarge Language Models in Resolving Contextual Knowledge Conflicts
Created: 2026-09-02 · Ingested: 2026-09-07
https://arxiv.org/abs/2609.03148(opens in a new tab)Studies conflicts that arise within the context itself rather than between context and parametric knowledge, across six conflict types (factual, inferential, temporal, granularity, perspective, ambiguity) on a dataset called ContextConflict. Finds a first-evidence positional bias that is visible in residual-stream activations, survives random reordering of the evidence, and can be reduced by steering activations toward an evidence-neutral centroid.
In brief
Models detect conflicts between context passages internally, yet still weight whichever evidence came first, and this positional preference is the main obstacle to resolving the conflict.
ContextConflict is a 5,781-sample dataset spanning 6 conflict types built from 10 base datasets: misinformation, inferential and temporal (scored by accuracy) plus granularity, perspective and ambiguity (scored by a Shapley-based Balance metric, lower is more even evidence use). 7 models were tested, from llama-3.1-8b-instruct to gpt-5.
Inferential conflicts stay below 50% for every model (24.8% on llama-3.1-8b-instruct, 44.3% on claude-4.5-sonnet); temporal peaks at 68.0% and misinformation at 61.5%. Linear probes separate conflicting from consistent inputs at AUC > 0.85 in mid-to-late layers, so awareness is present without resolution. Shapley attribution gives Evidence 1 69.0% of contributions in Ambiguity, 38.4% in Granularity, 35.0% in Perspective, and Balance does not improve with scale (GPT-OSS-120B 31.9 vs 20B 39.8). A training-free steering direction toward the mean of single-evidence activations gives +19.2 points on Temporal for Llama-3.1-8B-Instruct.
The causal claim is supported: an intervention check, a CAS steering baseline that the method beats in 5 of 6 reasoning comparisons, and replication of the mechanistic trends on GPT-OSS-20B. The Balance metric agrees with humans only at κ = 0.50–0.52, matching inter-human κ = 0.52, so it is coarse. Steering was tested at 8B and 20B only, needs white-box access, and the data is partly synthetic rather than real retrieval.
If you build RAG over disagreeing sources, treat first-position dominance as a representation-level defect, not a prompting problem.
Written from the abstract by claude-opus-5 on 2026-09-09, with every figure checked against it. Not a substitute for the paper.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
- supportsGiven several conflicting but equally legitimate passages, a model's answer is driven disproportionately by whichever passage comes first, and the skew sits in the representation of the combined prompt rather than in decoding — which is why reordering the passages does not remove it and why steering the activations does.
- contestsWhere a passage sits in a long input changes how much the model uses it — accuracy is highest when the needed information is at the very start or the very end and lowest when it is in the middle — so ordering retrieved passages to put the most relevant ones first is a real lever on accuracy.