rubric-graded-long-context-tasks-most-gain-context-compilation-harness-comes-extracting CLM-0140On rubric-graded long-context tasks, most of the gain from a context- compilation harness comes from extracting the context's rules, exact terms and output spec into an explicit checklist placed in the prompt, not from the executable verifiers built on top of it — and the gain appears only on rule-dense tasks and on models with enough capacity, disappearing or reversing on open-ended tasks and on a small-activation model.
Capability: Losing information in long inputs
Observed on
Long novel contexts (median 20K characters) graded against 5-20 all-or-nothing rubric criteria; four large open models; gains concentrated in rule/procedure-heavy categories and ab.
Sources
- Ablation on 1,899 CL-bench tasks, one model (Kimi K2.5), temperature 0, single GPT-5.1 judge: full pipeline 21.40% vs vanilla 15.40%; removing checklist injection -3.07pp, removing correction loop -1.28pp, verifier execution alone ~0.53pp. Component isolation not repeated on the other three models. LongBench-v2 probe shows aggregate accuracy below vanilla, supporting the scope limit. Absolute pass rates are low throughout, so deltas are on a small base.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For a 7B code model fine-tuned on competitive-programming tasks, training it to emit a fixed-schema requirement specification (goals, input/output constraints, examples, edge cases, dependencies) before the code raises Pass@1 more than training it to emit free-form reasoning or a stepwise implementation plan of similar length, with the largest margin on an out-of-distribution benchmark.Generating and editing working code · reviewed by AI
- Handing an agent context-editing tools (delete, summarize, note, fold) by prompt alone makes mid-size models worse than leaving their context untouched — Qwen3-8B, Qwen3-14B and Gemma4-E4B all scored lower on long-context QA with the tools than without them; only after fine-tuning on trajectories that use the tools does the toolset become a gain.Keeping its own context clean · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- On long-context tasks over 8K-128K-token documents in the six official UN languages, models answer word-level span questions well (about 73% average accuracy across 12 models) but degrade sharply as the required unit coarsens to sentence selection (about 21%), paragraph filling and document-level summarization and translation, so long-context competence measured by needle-style word retrieval overstates comprehension of larger discourse units.Losing information in long inputs · reviewed by AI
- On research-level theorem-proving tasks drawn from recent FOCS/STOC/SODA papers, wrapping a frontier model in a staged harness that explores strategies, gates when to decompose, solves section-level subproblems along a dependency graph, and routes verifier findings back to the affected section raises graded proof accuracy far above prompting the same model directly (30.3% to 54.0% for Gemini 3.1 Pro on TCS-Bench), and two such runs with different backbones make complementary errors, so selecting between them by sampled critiques reaches 71.0% against an oracle ceiling of 77.3%.Completing tasks that take many steps or many hours · reviewed by AI
Notes
Ingested unreviewed on 2026-09-07 and deliberately inert until a human endorses it: it does not move a technique standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Drafted confidence: low. Falsifier as drafted: An ablation on other models or benchmarks where removing the verifier/correction loop costs more pass rate than removing the checklist injection, or where the checklist gives similar gains on open-ended tasks. Proposed technique, not yet catalogued: Compile context into a typed rule IR and inject it as a checklist.