long-context-degradation · activeLosing information in long inputs
Accuracy drops as inputs grow and is worst for information placed in the middle of the context.
Also called: lost in the middle, needle in a haystack, effective context length
Tags: coding-agent, context, long-document, rag-qa
A capable model uses every part of a long input equally well, up to its advertised context length, and its reasoning does not degrade as irrelevant material is added.
Claims
- mechanismsingle paperHolding the task fixed and only lengthening the input degrades reasoning long before the context limit is reached.
- observationsingle paperOn multi-document question answering with GPT-3.5 Turbo's 16k-context variant, accuracy dropped by more than twenty points when the document containing the answer was moved from the start or end of the context to the middle, with nothing else about the task changed.
- mechanismsingle paperMost models claiming long contexts fail well before their advertised length on synthetic retrieval, tracing and aggregation tasks.
- mechanismsingle paperPerformance on multi-document QA is highest when the answer is at the start or end of the context and drops sharply in the middle.
- mechanismsingle papercontestedWhere a passage sits in a long input changes how much the model uses it — accuracy is highest when the needed information is at the very start or the very end and lowest when it is in the middle — so ordering retrieved passages to put the most relevant ones first is a real lever on accuracy.
- mechanismsingle paperOn rubric-graded long-context tasks, most of the gain from a context- compilation harness comes from extracting the context's rules, exact terms and output spec into an explicit checklist placed in the prompt, not from the executable verifiers built on top of it — and the gain appears only on rule-dense tasks and on models with enough capacity, disappearing or reversing on open-ended tasks and on a small-activation model.
- mechanismsingle paperFor safety guardrail classifiers, long-context failure is caused by proportional dilution of the unsafe evidence rather than by absolute input length: as the flagged span's share of the input shrinks, attention mass on it falls, the unsafe-minus-safe logit margin compresses, and unsafe recall collapses, while filling the same length with copies of the unsafe text leaves recall largely intact.
Techniques
- Put important material first or lastpromptingOrder retrieved documents so the most relevant sit at the start and end of the prompt.
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.