open-ended-editing-tasks-no-unique-correct-output-multi-slide-design CLM-0110In open-ended editing tasks with no unique correct output (multi-slide design editing), a separate judge model that scores the structural diff between the original and edited artifact against the instruction, and returns its critique verbatim as the next-turn instruction, raises instruction-following over a single pass, but the gain is selective: most tasks pass on the first attempt, and a minority of the ones that enter the loop get worse unless an earlier iteration is restored when the judge's own score declines.
Capability: Fixing its own mistakes
Observed on
94 multi-slide slide-editing tasks over a hierarchical scene graph; claude-sonnet-4-6 and gpt-5.5 backbones; gpt-5.5 as in-loop judge; at most 3 iterations; judge sees the origin d.
Applies when
- Effect on the capability:
- helps, narrowly
- Needs:
- a second model with its own context
- Cost:
- moderate
- Fails when:
- The task already passed on the first attempt, or the judge's own score declines across iterations and no earlier iteration is restored; a minority of looped tasks get worse.
Sources
- Measured on a 94-task benchmark adapted from PPTArena: IF 4.23 vs 3.81 for a same-backbone HTML agent (paired Wilcoxon p=.010); self- correction adds +0.41 IF over single-pass ACE; 35/53 tasks halt at iteration 1, and of the 18 entering the loop 13 improve, 2 unchanged, 3 regress by at most 1 point, with strict-peak rollback removing all regressions. Circularity is bounded but not eliminated: out-of-loop judges retain about two-thirds of the gain (+0.61, +0.56 IF), and 26 blind raters prefer self-corrected output 81% of decided cases. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When an agent system evolves its own harness from failure traces, batching failures across tasks before editing buys efficiency but not better failure attribution: pooled across three agent benchmarks, the share of harness edits that merely accommodate the current model rose from 60.0% under per-failure serial editing to 75.6% under cross-task aggregation, and only fell to 45.5% when a separate analyst/critic/engineer refinement stage attributed each recurring pattern before the edit was written.Improving its own scaffolding from failures · reviewed by AI
- In UI-to-code generation, where a rendered screenshot gives genuine external feedback, iterative self-critique still drifts below the initial draft over ten rounds, because a code edit changes the rendering non-locally and a fix for one region breaks regions that were already faithful; restricting each round to one typed, scoped repair target and carrying past targets forward as an avoid-list reduces that drift.Generating and editing working code · reviewed by AI
- Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
- For a frozen LLM run in a generate-critique-revise loop with no external feedback, the accuracy gain is concentrated where single-shot reasoning is already unreliable: on clean multi-step reasoning (BBH, ~74% single-shot) depth makes no difference beyond the confidence interval, while on harder math (MATH, 26.2% single-shot) the same loop gains +14.2 points, with GSM8K in between (+4.2).Fixing its own mistakes · reviewed by AI
- For 8B instruction-tuned backbones, training a LoRA corrector to diagnose a typed, executable error operator (error type, step location, corrupted and repaired proposition) and apply its inverse as a localized edit raises accuracy over free-form self-revision baselines (Self-Refine, RISE, SCoRe) on math, code and commonsense tasks, and dropping either the operator-prediction loss or the adjacent-state reconstruction loss loses most of the gain.Fixing its own mistakes · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Running the same loop with the judge scoring an edit diff shows no instruction-following gain over single-pass, or the gain disappears under judges outside the loop and under blind human comparison, or regressions are as common as improvements among tasks entering the loop. Proposed technique, not catalogued: reference-free instruction-following judge critique as repair signal.