generated-checklists-make-instruction-requirements-individually-checkable CLM-0062mechanismsingle paperReviewed by AI and Russ Miller
Decomposing an instruction into explicit yes/no items and judging against them beats asking for a holistic score, and the same checklist turned on the model's own draft improves the draft. The gain comes from making each requirement separately checkable, not from the model trying harder — so it is largest on instructions carrying several constraints at once.
Capability: Following an unfamiliar procedure · Behavior, General
Observed on
2024, frontier chat models.
Applies when
- Effect on the capability:
- helps
- Fails when:
- The instruction carries a single constraint. The gain comes from making several requirements separately checkable, so it is smallest when there is only one.
Sources
- Checklist-based judging raised exact agreement with human preference from 46.4% to 52.2% over direct scoring, and self-refinement against generated checklists gained +7.8% absolute on LiveBench reasoning, +6.3% via Best-of-N on WildBench. Human annotators given the same checklists also agreed with each other more.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On rubric-graded long-context tasks, most of the gain from a context- compilation harness comes from extracting the context's rules, exact terms and output spec into an explicit checklist placed in the prompt, not from the executable verifiers built on top of it — and the gain appears only on rule-dense tasks and on models with enough capacity, disappearing or reversing on open-ended tasks and on a small-activation model.Losing information in long inputs · reviewed by AI
- For multi-turn agentic coding sessions, constraints written into a repository policy file (e.g. CLAUDE.md) hold up better over successive turns than the same kinds of constraints stated in user turns, and constraints that replace an earlier constraint are followed less often than newly added ones.Following an unfamiliar procedure · reviewed by AI
- In a prompted text-to-SQL agent, a blanket output-shaping instruction in the system prompt (here "LIMIT 10") is applied by the model to every generated query regardless of the question's intent, truncating answers that legitimately need many rows, so removing it raises execution accuracy most on questions whose correct result set is large.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- When an agent labels its own past attempts as successes or failures to build an experience memory, the labels can be substantially wrong without much cost: the LLM judge used was 72.7% accurate against ground truth on WebArena-Shopping, and simulating judges anywhere from 70% to 90% accurate gave similar success rates, with ground-truth labels best — so for distilling lessons, a rough self-judgment is close to enough, although a perfect one still helps.Improving its own scaffolding from failures · reviewed by AI
- On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Following an unfamiliar procedure · reviewed by AI
Notes
Closest published isolation of the technique, but not an exact match to how it is used in practice. TICK generates the checklist from the instruction and applies it as an evaluation protocol; the practitioner version has the agent write its own list up front and tick it off before finishing, which is nearer to the STICK self-refinement setting than to the judging result. Scope condition worth keeping in mind before quoting the numbers at a coding agent.