arxiv-2609-00759 · paper

Compile, Don't Memorize: A Context Compilation Architecture (CCA) for In-Context Learning

Created: 2026-09-01 · Ingested: 2026-09-07

Large language models (LLMs) increasingly handle in-context learning (ICL) tasks where a long, novel context defines the rules, knowledge, and output schema for a series of questions. On benchmarks that grade against every detail of the context, even strong open-weights models pass only 12-16% of tasks: a single overlooked rule fails the whole response. We argue this brittleness is structural: the dominant "read- and-reason" paradigm asks the model to extract, plan, generate, and self-verify in one forward pass. We therefore ask whether explicit context compilation can fix it, how it compares t

In brief

Extracting a long context once into a typed JSON structure and injecting it as a checklist does most of the work in rubric-graded in-context learning; the code verifiers and correction loop built on top add less.

CCA compiles each context into fixed slots (rules by polarity, exact_terms, output_spec, tools, data_profile), synthesizes up to 3 Python verifiers, drafts an answer, and re-runs a corrector when 2 or more violations fire. Evaluated on CL-bench (1,899 tasks, 4 domains, 5-20 rubric criteria per task, all-or-nothing pass), with 4 open models and GPT-5.1 as judge, against Vanilla, ReadAgent-P, and Ctx2Skill.

Vanilla pass rates sit at 12-16%. CCA leads on every model: Kimi K2.5 15.4 to 21.4%, GLM-5 16.1 to 21.2%, DeepSeek-V3.2 15.0 to 17.7% (McNemar z = 6.00, 4.89, 3.01), but only +0.5pp on Qwen3-Next-80B (p = 0.70). Token cost is about 3x Vanilla and ~5.7x cheaper than Ctx2Skill.

The attribution comes from a leave-one-out ablation on the full task set, but on 1 model only. Removing IR-as-checklist injection costs -3.07pp; the correction loop -1.28pp; verifier execution alone ~0.53pp. Temperature is fixed at 0.0 throughout; a temp 1.0 check gives 18.75%.

The gain is concentrated in rule-dense sub-categories and disappears on open-ended tasks: on LongBench-v2, CCA scores 53.88% against Vanilla's 57.85%, winning only on Multi-Doc QA (+5.69pp) and Long-Dialog (+10.25pp). Scoring is single-judge.

Treat structured context extraction as a targeted tool for rule-heavy contexts, not a general long-context wrapper.

Written from the abstract by claude-opus-5 on 2026-09-09, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsOn rubric-graded long-context tasks, most of the gain from a context- compilation harness comes from extracting the context's rules, exact terms and output spec into an explicit checklist placed in the prompt, not from the executable verifiers built on top of it — and the gain appears only on rule-dense tasks and on models with enough capacity, disappearing or reversing on open-ended tasks and on a small-activation model.