long-context-accuracy-drops-in-the-middle-gpt-3-5-turbo CLM-0081observationsingle paperReviewed by AI and Russ Miller
On multi-document question answering with GPT-3.5 Turbo's 16k-context variant, accuracy dropped by more than twenty points when the document containing the answer was moved from the start or end of the context to the middle, with nothing else about the task changed.
Capability: Losing information in long inputs
Observed on
GPT. 2023, GPT-3.5 Turbo (16k context). Long document.
Sources
- Multi-document QA accuracy is highest with the answer at the start or end of the context and drops sharply in the middle.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Performance on multi-document QA is highest when the answer is at the start or end of the context and drops sharply in the middle.Losing information in long inputs
- In long-context tasks over 8K-128K-token documents, the middle-of-context penalty is granularity-dependent: word-level QA and paragraph-filling accuracy dip when the evidence sits mid-document, but sentence-cloze accuracy peaks in the middle, because mid-paragraph sentences carry concrete entity and date overlap that models match on instead of reasoning about discourse role.Losing information in long inputs · reviewed by AI
- Grounding generation in retrieved documents improves factual accuracy on knowledge-intensive tasks.Stating false facts confidently
- On long-context tasks over 8K-128K-token documents in the six official UN languages, models answer word-level span questions well (about 73% average accuracy across 12 models) but degrade sharply as the required unit coarsens to sentence selection (about 21%), paragraph filling and document-level summarization and translation, so long-context competence measured by needle-style word retrieval overstates comprehension of larger discourse units.Losing information in long inputs · reviewed by AI
- On 430 tasks distilled from real professional-forum threads across seven domains, the best model (GPT-5) scores about 54%, under the 60% passing line the authors set, with computer science and applied technology lowest at roughly 40-45%, so long-tail practitioner knowledge remains a gap even where standard benchmarks are saturated.Competence in a specialist domain · reviewed by AI