putting-relevant-passages-first-mitigates-position-effectsWhere a passage sits in a long input changes how much the model uses it — accuracy is highest when the needed information is at the very start or the very end and lowest when it is in the middle — so ordering retrieved passages to put the most relevant ones first is a real lever on accuracy.
Capability: Losing information in long inputs · Context and memory, Retrieval-augmented QA
Observed on
2023, GPT-3.5 / Claude / open long-context models.
Sources
- Measures the U-shaped curve directly: multi-document QA and key-value retrieval accuracy drops sharply when the relevant document sits in the middle of the context, and is highest at the beginning or the end.
- Contests the remedy rather than the effect. When several passages genuinely conflict, whichever is placed first dominates the output, and randomly permuting the evidence does not move that dominance — the skew is already present in the residual-stream representation of the combined prompt, so reordering cannot be the fix.
- A third position, from the builder. Anthropic reports that earlier models "could sometimes need repeated instructions or be more likely to listen to instructions at the end of their context window than at the start", and that for the Claude 5 generation they deleted those repeats with no measurable loss. If the positional effect has weakened in newer models, the technique is a remedy for a property that is being trained away rather than a durable one. Unverifiable from outside: the evaluations are not published.
Suspected axis of disagreement(a guess, not verified)
Two axes now, because there are three positions. The first is what the other passages are. The supporting result places one relevant passage among distractors, where moving it out of the middle recovers accuracy that position was costing. The 2026 conflict result places several mutually incompatible passages of equal legitimacy, where position is not degrading access to a known answer but deciding which answer wins — and reordering only moves which passage occupies the winning slot. If that is right, both hold, and ordering is a remedy for distraction rather than for conflict. The second is model generation, and it cuts across the first. Anthropic says the end-of-context bias it used to prompt around is largely gone in the Claude 5 generation. So the effect may be real, real still under conflict, and being trained away for ordinary retrieval — three claims that can all hold at once about different models at different times. This is exactly what the mechanism-versus-observation split is for, and it is an argument that this claim is closer to observation than it is currently filed. I have not tested any of the three setups against each other.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Given several conflicting but equally legitimate passages, a model's answer is driven disproportionately by whichever passage comes first, and the skew sits in the representation of the combined prompt rather than in decoding — which is why reordering the passages does not remove it and why steering the activations does.Checking claims against evidence
- In retrieval-augmented answers, splitting an answer into claims, checking each against the retrieved source, and then acting on the flagged claims reduces the share of answers judged to contain unsupported content — but the strategies trade grounding against preservation: deleting unsupported claims reduces unsupported content most while retaining the least original text, and rewriting retains the most while reducing least.Stating false facts confidently · unreviewed
- In synthetic hallucination benchmarks where faithful items are human-written and hallucinated items are LLM-rewrites, a detector's binary score partly measures human-vs-machine style rather than factuality: feeding faithful LLM rewrites of the same truthful articles makes some fine-tuned encoders flag nearly all of them as hallucinated.Whether the measurement made the finding · unreviewed
- Training the model to decide when to retrieve and to critique whether passages support its output improves factuality and citation accuracy.Checking claims against evidence
- General-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Stating false facts confidently · unreviewed
Notes
Filed deliberately as an incumbent position rather than as a held belief. The catalog previously recorded only claims it endorsed, which left contesting evidence with nothing to attach to — the reason the backtest grades known reversals as missed even when the reversing paper is on file. A claim someone reasonably holds, with the evidence against it attached, is more useful than its absence.