arxiv-2307-03172 · paperLost in the Middle: How Language Models Use Long Contexts
Nelson F. Liu, Kevin Lin, John Hewitt, et al.
Created: 2023 · Ingested: 2026-09-02
This source: Lost in the Middle: How Language Models Use Long ContextsHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 4887 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch1000+ citations in the last 12 months · 4887 total · checked 2026-09-04
https://arxiv.org/abs/2307.03172(opens in a new tab)In brief
Where a fact sits inside a long prompt changes whether a model can use it, and the middle is the worst place. Accuracy traces a U-shape: high at the start (primacy) and end (recency), low in between, even for models sold on extended context.
The setup is 2 controlled tasks. Multi-document QA uses 2655 NaturalQuestions-Open queries with 1 answer-bearing Wikipedia passage plus Contriever-retrieved distractors, at 10, 20, and 30 documents, with the gold document moved to each slot. A synthetic key-value retrieval task uses JSON objects of 75, 140, and 300 UUID pairs (500 examples each). Models: MPT-30B-Instruct, LongChat-13B (16K), GPT-3.5-Turbo and its 16K version, Claude-1.3 and its 100K version.
GPT-3.5-Turbo drops by more than 20% depending on position; at 20 and 30 documents, worst-case accuracy falls below its closed-book score of 56.1%. Extended-context variants match their base counterparts almost exactly. On key-value retrieval, Claude-1.3 is near perfect, while GPT-3.5-Turbo (16K) bottoms out at 45.6% before query-aware contextualization, which restores near-perfect accuracy but barely changes multi-document QA.
The design isolates position cleanly: only document order changes, output stays fixed, and controls cover distractor ordering, random distractors, and unambiguous questions. Ablations separate architecture (Flan-UL2 is robust within its 2048-token training window, 1.9% best-worst gap, U-shaped beyond it), instruction tuning (MPT-30B base also U-shaped; tuning narrows the gap from near 10% to about 4%), and scale (Llama-2 7B is only recency-biased; 13B and 70B are U-shaped). GPT-4 was tested only on a subset due to cost.
Open-domain QA saturates well before retriever recall: going from 20 to 50 documents adds ~1.5% for GPT-3.5-Turbo and ~1% for Claude-1.3.
If you build retrieval-augmented systems, rerank so relevant passages land near the edges and stop paying for documents past the saturation point.
Written from the abstract by claude-opus-5 on 2026-09-05, with every figure checked against it. Not a substitute for the paper.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
- supportsOn multi-document question answering with GPT-3.5 Turbo's 16k-context variant, accuracy dropped by more than twenty points when the document containing the answer was moved from the start or end of the context to the middle, with nothing else about the task changed.
- supportsPerformance on multi-document QA is highest when the answer is at the start or end of the context and drops sharply in the middle.
- supportsWhere a passage sits in a long input changes how much the model uses it — accuracy is highest when the needed information is at the very start or the very end and lowest when it is in the middle — so ordering retrieved passages to put the most relevant ones first is a real lever on accuracy.