Losing information in long inputs

Claims

• mechanismsingle paper↑1↓0
  For a 7B instruction-tuned transformer under a fixed KV-cache budget, training a small policy to pick which cached tokens to keep by rewarding low KL divergence from the full-cache next-token distribution retains more long-context accuracy than attention-heuristic eviction (H2O, SnapKV), and the margin widens as the budget tightens and as context grows.
• mechanismsingle paper↑1↓0
  For an 8B Llama backbone under a fixed 256-slot cache budget, replacing hard KV eviction with a learned soft write that blends a new token's key/value into a routed fixed-size memory slot retains far more long-context retrieval accuracy than sliding-window or sink caches, and unlike full-attention or quantized caches it does not run out of memory at 128K tokens.
• mechanismsingle paper↑1↓0
  Holding the task fixed and only lengthening the input degrades reasoning long before the context limit is reached.
• observationsingle paper↑1↓0
  On multi-document question answering with GPT-3.5 Turbo's 16k-context variant, accuracy dropped by more than twenty points when the document containing the answer was moved from the start or end of the context to the middle, with nothing else about the task changed.
• observationsingle paper↑1↓0
  In long-context tasks over 8K-128K-token documents, the middle-of-context penalty is granularity-dependent: word-level QA and paragraph-filling accuracy dip when the evidence sits mid-document, but sentence-cloze accuracy peaks in the middle, because mid-paragraph sentences carry concrete entity and date overlap that models match on instead of reasoning about discourse role.
• observationsingle paper↑1↓0
  On long-context tasks over 8K-128K-token documents in the six official UN languages, models answer word-level span questions well (about 73% average accuracy across 12 models) but degrade sharply as the required unit coarsens to sentence selection (about 21%), paragraph filling and document-level summarization and translation, so long-context competence measured by needle-style word retrieval overstates comprehension of larger discourse units.
• mechanismsingle paper↑1↓0
  Most models claiming long contexts fail well before their advertised length on synthetic retrieval, tracing and aggregation tasks.
• observationsingle paper↑1↓0
  For multi-hop QA over 28k-896k token haystacks with frozen Qwen2.5 backbones (7B-72B), summarizing the document in a parallel hierarchical tree — overlapping windows scanned ~5x, raw segments scored 'critical' passed straight to the final answer, and one summarization channel per decomposed sub-question — scores higher than sequential memory-agent and BM25 retrieval baselines without training, and beats an RL-trained memory agent on the out-of-distribution dataset while losing to it in-distribution.
• mechanismsingle paper↑1↓0
  Performance on multi-document QA is highest when the answer is at the start or end of the context and drops sharply in the middle.
• mechanismsingle paper↑1↓0
  For precise anomaly localization in long multivariate time-series (16k-131k points), keeping the signal outside the model context as a variable the model queries through code over multiple turns recovers far more localization accuracy than single-pass ingestion, where the recording is serialized, subsampled or compressed into context once.
• mechanismsingle papercontested↑1↓2
  Where a passage sits in a long input changes how much the model uses it — accuracy is highest when the needed information is at the very start or the very end and lowest when it is in the middle — so ordering retrieved passages to put the most relevant ones first is a real lever on accuracy.
• mechanismsingle paper↑1↓0
  On rubric-graded long-context tasks, most of the gain from a context- compilation harness comes from extracting the context's rules, exact terms and output spec into an explicit checklist placed in the prompt, not from the executable verifiers built on top of it — and the gain appears only on rule-dense tasks and on models with enough capacity, disappearing or reversing on open-ended tasks and on a small-activation model.
• mechanismsingle paper↑1↓0
  For safety guardrail classifiers, long-context failure is caused by proportional dilution of the unsafe evidence rather than by absolute input length: as the flagged span's share of the input shrinks, attention mass on it falls, the unsafe-minus-safe logit margin compresses, and unsafe recall collapses, while filling the same length with copies of the unsafe text leaves recall largely intact.
• mechanismsingle paper↑1↓0
  For speech LLMs on long audio, text-centric KV cache eviction methods (StreamingLLM, SnapKV, PyramidKV, ChunkKV) collapse because raw audio tokens have diffuse attention; anchoring compression on ASR transcripts — pooling each transcribed audio span down to its text-token length and adding the text embedding, while keeping untranscribed audio intact — retains over 90% of full-cache accuracy at a 5-10% cache budget where those baselines retain roughly 55-67% at a looser 25% budget.

Techniques

• Blend evicted KV into routed memory slots instead of droppinarchitecture
  Organizes the KV cache as a fixed-size tree of memory buckets, routes each incoming token to a slot via Gumbel-Softmax, and sigmoid-gates a soft merge of the new key/value with the stored one instead of discarding any token, with a learned
• Compress audio KV cache around ASR transcript anchorsarchitecture
  Slice the acoustic feature sequence by ASR timestamps, average-pool each transcribed span to its text length and add the aligned text embedding, keep untranscribed spans raw, then evict remaining tokens by attention scores accumulated with
• Learn KV eviction by matching the full-cache output distribuarchitecture
  A ~200K-parameter MLP scores cached tokens from attention mass, value norm, attention variance, entropy and position, and is trained with REINFORCE against a per-step negative KL between full-cache and pruned-cache next-token distributions.
• Put important material first or lastprompting
  Order retrieved documents so the most relevant sit at the start and end of the prompt.
• Query the time series through code, not in contexttooling
  Store the full recording as a variable in a sandboxed Python REPL and let the model iteratively emit code (and optionally render plots or spawn sub-agents) to inspect segments before committing a structured answer.
• Summarize long context as a parallel hierarchical treeprompting
  Treats a query-prompted frozen LLM as a convolution kernel that summarizes overlapping text windows in parallel, recursing over summaries layer by layer, with high-relevance raw segments skipped forward to the final answer and a separate ch

Sources

Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.

• ARM: Attention with Routed-Memory for Learnable Sparse Control2026-09-21
  ↑ supports a claim here
• VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding2026-09-14
  about this capability · no claim drafted yet
• ConvMem: Convolutional Memory for Long-Context Reasoning2026-09-09
  ↑ supports a claim here
• Large Language Models in Resolving Contextual Knowledge Conflicts2026-09-02
  ↓ contests a claim here
• Compile, Don't Memorize: A Context Compilation Architecture (CCA) for In-Context Learning2026-09-01
  ↑ supports a claim here
• LongGuard: Mechanistic Analysis and Training-Free Mitigation of Long-Context Failure in Safety Guardrails2026-08-27
  ↑ supports a claim here
• MGAL: A Multilingual Granularity-Aware Long-Context Benchmark2026-08-21
  ↑ supports a claim here · ↑ supports a claim here
• DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference2026-08-09
  ↑ supports a claim here
• VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference2026-08-09
  ↑ supports a claim here
• TimeRLM: Recursive Language Models Enable Precise Anomaly Localization in Long-Context Time-Series2026-08-04
  ↑ supports a claim here
• Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language ModelsThis source: Same Task, More Tokens: the Impact of Input Length on the Reasoning Pe…How much the field cites it — heavily cited in the last 12 months104 in the last 12 months · 238 totalpublished 2024checked 2026-09-04click for Semantic Scholar2024
  ↑ supports a claim here
• Lost in the Middle: How Language Models Use Long ContextsThis source: Lost in the Middle: How Language Models Use Long ContextsHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 4887 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch2023
  ↑ supports a claim here · ↑ supports a claim here · ↑ supports a claim here
• RULER: What's the Real Context Size of Your Long-Context Language Models?This source: RULER: What's the Real Context Size of Your Long-Context Language Mode…How much the field cites it — very heavily cited in the last 12 months759 in the last 12 months · 1243 totalpublished 2024checked 2026-09-042024
  ↑ supports a claim here
• The new rules of context engineering for Claude 5 generation models2026 · vendor-doc
  ↓ contests a claim here