rubric-graded-long-context-tasks-most-gain-context-compilation-harness-comes-extractingOn rubric-graded long-context tasks, most of the gain from a context- compilation harness comes from extracting the context's rules, exact terms and output spec into an explicit checklist placed in the prompt, not from the executable verifiers built on top of it — and the gain appears only on rule-dense tasks and on models with enough capacity, disappearing or reversing on open-ended tasks and on a small-activation model.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Losing information in long inputs
Observed on
Long novel contexts (median 20K characters) graded against 5-20 all-or-nothing rubric criteria; four large open models; gains concentrated in rule/procedure-heavy categories and ab.
Sources
- Ablation on 1,899 CL-bench tasks, one model (Kimi K2.5), temperature 0, single GPT-5.1 judge: full pipeline 21.40% vs vanilla 15.40%; removing checklist injection -3.07pp, removing correction loop -1.28pp, verifier execution alone ~0.53pp. Component isolation not repeated on the other three models. LongBench-v2 probe shows aggregate accuracy below vanilla, supporting the scope limit. Absolute pass rates are low throughout, so deltas are on a small base.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Handing an agent context-editing tools (delete, summarize, note, fold) by prompt alone makes mid-size models worse than leaving their context untouched — Qwen3-8B, Qwen3-14B and Gemma4-E4B all scored lower on long-context QA with the tools than without them; only after fine-tuning on trajectories that use the tools does the toolset become a gain.Keeping its own context clean · unreviewed
- When an agent must work over a very large tool catalogue, wrapping each tool in a natural-language interface that resolves the API schema internally — so raw schemas are retrieved on demand rather than enumerated in the model's prompt — contributes more to multi-step tool- call success than the surrounding planner, router, or verifier stages.Using the tools it is given · unreviewed
- For locally deployed open-weight models driving a stateful, dependency-ordered MCP tool server, cutting tool descriptions from full specifications (purpose, parameter semantics, constraints, failure conditions) down to one sentence each raises the fraction of calls the server rejects for every model tested, while its effect on task coverage is less consistent.Using the tools it is given · unreviewed
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · unreviewed
- When training small instruction-tuned models with reinforcement learning plus factual supervision, routing each atomic fact's verification score only to the tokens that produced it, and down-weighting verifier judgements that do not change when their key evidence is removed, improves factuality benchmark scores over trajectory-level or reasoning- step-level factual rewards.Checking claims against evidence · unreviewed
Notes
Ingested unreviewed on 2026-09-07 and deliberately inert until a human endorses it: it does not move a technique standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Drafted confidence: low. Falsifier as drafted: An ablation on other models or benchmarks where removing the verifier/correction loop costs more pass rate than removing the checklist injection, or where the checklist gives similar gains on open-ended tasks. Proposed technique, not yet catalogued: Compile context into a typed rule IR and inject it as a checklist.