models-must-apply-abstract-entity-independent-procedural-rules-drawn-shared CLM-0092When models must apply abstract, entity-independent procedural rules drawn from a shared pool, accuracy is limited by locating the right rules rather than by applying them: giving only the gold rules yields much higher exact-match than embedding those same rules in a pool of 100 or 1,000, and accuracy falls as the pool grows.
Capability: Following an unfamiliar procedure
Observed on
Synthetic non-commonsense rules in first-order-logic and natural-language form, single-rule, parallel multi-rule and up to 4-hop QA; pools of 100 to 10,000 rules; tested on Qwen3-3.
Sources
- Gold-rule-only prompting reached 74.15-89.79% exact match for three strong models, while going from 100 to 1,000 rules lowered accuracy across models and both rule representations; retrieval tables show baseline Recall@1 collapsing at 10K rules. All rules are synthetic and deliberately non-commonsense, so the result isolates localization but may overstate mismatch relative to natural policy text. Multi-hop remains weak even with the paper's own method, so localization is not the only bottleneck.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When extracting structured rules from multimodal enterprise guideline documents, wrapping the extraction in deterministic schema validation, an LLM semantic judge, and a duplicate/contradiction consistency filter over a shared rule store lowers the share of extracted rules with no matching source sentence from about 15.7% to 3.2% relative to a one-pass pipeline using the same parser and models.Stating false facts confidently · reviewed by AI
- On multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.Whether the measurement made the finding · reviewed by AI
- On research-level theorem-proving tasks drawn from recent FOCS/STOC/SODA papers, wrapping a frontier model in a staged harness that explores strategies, gates when to decompose, solves section-level subproblems along a dependency graph, and routes verifier findings back to the affected section raises graded proof accuracy far above prompting the same model directly (30.3% to 54.0% for Gemini 3.1 Pro on TCS-Bench), and two such runs with different backbones make complementary errors, so selecting between them by sampled critiques reaches 71.0% against an oracle ceiling of 77.3%.Completing tasks that take many steps or many hours · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- When an LLM judge is given a prior evaluation score as context metadata on a revision, its rubric scores shift toward that uninformative prior — seven of eight tested models (API and open-weight, 0-5 scale, 20 fixed texts) show a negative total shift, and on a categorical compliance task the metadata blocks 48% of error corrections and flips about 10% of previously correct judgments to the anchored wrong label.Judging another's work as well as a careful reviewer would · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Show models scoring no better with gold rules alone than with the same rules hidden in a 1,000-rule pool, or show accuracy flat as the irrelevant-rule pool grows. Proposed technique, not catalogued: Step-triggered rule re-attention in the KV cache.