TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation

Making an LLM judge write an instruction-specific YES/NO checklist and answer each item explicitly beats holistic scoring, and feeding those answers back to the generator produces self-correction on tasks where unstructured self-critique degrades performance. TICK generates the checklist and answers it; STICK uses the same model as its own judge.

Evaluation was checked against 612 internal instructions with 3 human-written checklists each, plus InFoBench and WildBench, with GPT-4o, Command-R+ and Llama3.1-70B. GPT-4o checklists matched the chosen human ground truth as closely as alternate human checklists (BLEU 0.759 vs 0.733) and answered questions at 0.826 accuracy against a 3-annotator majority.

On pairwise agreement with human preferences, exact-match rose from 0.464 (direct scoring) to 0.522 (TICK), with Check-then-Score at 0.487 and raw preference prompting at 0.293. Self-refinement gave +6.5% on InFoBench and +7.1% on WildBench, and on LiveBench reasoning +7.8% for Command-R+, while vanilla Self-Refine dropped overall scores 8.3 points for both models. Best-of-8 self-selection: 0.843 to 0.894 DRFR, 64.9 to 71.2 WB-Score, beating the ArmoRM reward model. Human annotators given checklists moved Krippendorff's alpha from 0.194 to 0.256.

The Self-Refine and direct-scoring baselines are matched on task but not on inference tokens, so part of the gain may be extra compute; checklist generation costs more. Gains plateau or regress after 1 iteration on LiveBench and 4 on the others, and agreement stays low in absolute terms. If you run LLM judges, decomposing before scoring is worth the extra calls.