long-context-tasks-over-8k-128k-token-documents-six-official-un-languages CLM-0601On long-context tasks over 8K-128K-token documents in the six official UN languages, models answer word-level span questions well (about 73% average accuracy across 12 models) but degrade sharply as the required unit coarsens to sentence selection (about 21%), paragraph filling and document-level summarization and translation, so long-context competence measured by needle-style word retrieval overstates comprehension of larger discourse units.
Capability: Losing information in long inputs
Observed on
UN report documents 8K-128K tokens, six languages, zero-shot, 12 long-context LLMs; coarse-grained tasks scored with ROUGE-L/BLEU, which are themselves weak proxies..
Sources
- Measured across 12 open and proprietary models on 420 query-response pairs per task per language; the gap is large but confounded by metric change between granularities (accuracy vs ROUGE-L vs BLEU), which the design does not isolate.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In long-context tasks over 8K-128K-token documents, the middle-of-context penalty is granularity-dependent: word-level QA and paragraph-filling accuracy dip when the evidence sits mid-document, but sentence-cloze accuracy peaks in the middle, because mid-paragraph sentences carry concrete entity and date overlap that models match on instead of reasoning about discourse role.Losing information in long inputs · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- For spoken language models answering questions over long dialogue histories, converting the history to transcripts beats keeping it as audio when the answer is a lexical detail (exact slot values across dense turns: mean 71.3% vs 42.6%), but loses when the answer depends on how something was said, where audio history wins for six of seven models and by a larger margin once lexical emotion cues are removed.Understanding speech and audio · reviewed by AI
- In ESG-report RAG question answering with retrieval held fixed at a single top-1 chunk, open-weight 2B-30B generators vary widely in how well they stay anchored to the retrieved passage (faithfulness 0.607-0.822) yet all land in a narrow low band of factual correctness (0.387-0.449), and the most faithful model is the least factually correct, so groundedness to context and correctness of extracted ESG facts do not track each other.Competence in a specialist domain · reviewed by AI
- On text-to-SQL benchmarks, aggregate accuracy hides a steep difficulty gradient: pre-trained-language-model-based parsers resolved roughly 80% of Spider cases overall but only about 50% of its extra-hard cases, and as natural-language understanding improved the residual errors shifted toward database-side problems — large schemas, ambiguous columns, dirty cell values — rather than language parsing.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A model family scoring near its word-level accuracy on the sentence-cloze and paragraph-filling tasks of the same benchmark, or showing that the coarse-grained gap disappears under reference-free judging rather than ROUGE-L/BLEU.