generated-checklists-make-instruction-requirements-individually-checkablemechanismsingle paper
Decomposing an instruction into explicit yes/no items and judging against them beats asking for a holistic score, and the same checklist turned on the model's own draft improves the draft. The gain comes from making each requirement separately checkable, not from the model trying harder — so it is largest on instructions carrying several constraints at once.
Capability: Following an unfamiliar procedure · Behavior, General
Observed on
2024, frontier chat models.
Sources
- Checklist-based judging raised exact agreement with human preference from 46.4% to 52.2% over direct scoring, and self-refinement against generated checklists gained +7.8% absolute on LiveBench reasoning, +6.3% via Best-of-N on WildBench. Human annotators given the same checklists also agreed with each other more.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- On rubric-graded long-context tasks, most of the gain from a context- compilation harness comes from extracting the context's rules, exact terms and output spec into an explicit checklist placed in the prompt, not from the executable verifiers built on top of it — and the gain appears only on rule-dense tasks and on models with enough capacity, disappearing or reversing on open-ended tasks and on a small-activation model.Losing information in long inputs · unreviewed
- On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Following an unfamiliar procedure · unreviewed
- In open-ended editing tasks with no unique correct output (multi-slide design editing), a separate judge model that scores the structural diff between the original and edited artifact against the instruction, and returns its critique verbatim as the next-turn instruction, raises instruction-following over a single pass, but the gain is selective: most tasks pass on the first attempt, and a minority of the ones that enter the loop get worse unless an earlier iteration is restored when the judge's own score declines.Fixing its own mistakes · unreviewed
- When fine-tuning a base model on instruction data, removing from the training targets any factual claim the base model cannot consistently recall raises the supported-claim rate of later long-form generations, but the gain comes largely from a more conservative response policy — more refusals and fewer supported claims per answer — not from generating more correct facts at equal coverage.Stating false facts confidently · unreviewed
- When building preference data to teach 7-8B instruct models to follow multi-constraint instructions, perturbing each atomic constraint in several directions so the perturbed instruction's valid-response set contains, partly overlaps, or is disjoint from the original's — and pairing responses drawn from each sub-region — raises constraint adherence more than rejection sampling or teacher-correction pairs, and the gain holds on perturbed instruction variants the baselines barely improve on.Following an unfamiliar procedure · unreviewed
Notes
Closest published isolation of the technique, but not an exact match to how it is used in practice. TICK generates the checklist from the instruction and applies it as an evaluation protocol; the practitioner version has the agent write its own list up front and tick it off before finishing, which is nearer to the STICK self-refinement setting than to the judging result. Scope condition worth keeping in mind before quoting the numbers at a coding agent.