open-ended-editing-tasks-no-unique-correct-output-multi-slide-designIn open-ended editing tasks with no unique correct output (multi-slide design editing), a separate judge model that scores the structural diff between the original and edited artifact against the instruction, and returns its critique verbatim as the next-turn instruction, raises instruction-following over a single pass, but the gain is selective: most tasks pass on the first attempt, and a minority of the ones that enter the loop get worse unless an earlier iteration is restored when the judge's own score declines.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Fixing its own mistakes
Observed on
94 multi-slide slide-editing tasks over a hierarchical scene graph; claude-sonnet-4-6 and gpt-5.5 backbones; gpt-5.5 as in-loop judge; at most 3 iterations; judge sees the origin d.
Sources
- Measured on a 94-task benchmark adapted from PPTArena: IF 4.23 vs 3.81 for a same-backbone HTML agent (paired Wilcoxon p=.010); self- correction adds +0.41 IF over single-pass ACE; 35/53 tasks halt at iteration 1, and of the 18 entering the loop 13 improve, 2 unchanged, 3 regress by at most 1 point, with strict-peak rollback removing all regressions. Circularity is bounded but not eliminated: out-of-loop judges retain about two-thirds of the gain (+0.61, +0.56 IF), and 26 blind raters prefer self-corrected output 81% of decided cases. [truncated]
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- In UI-to-code generation, where a rendered screenshot gives genuine external feedback, iterative self-critique still drifts below the initial draft over ten rounds, because a code edit changes the rendering non-locally and a fix for one region breaks regions that were already faithful; restricting each round to one typed, scoped repair target and carrying past targets forward as an avoid-list reduces that drift.Generating and editing working code · unreviewed
- Used zero-shot as step-level action critics in long-horizon tool-calling tasks, frontier models are over-pessimistic — flagging a large share of correct actions and pushing the actor into revision loops that lower success below no critic at all — whereas small models fine-tuned on action-level verification rationales flag far fewer valid actions and raise repeated-run reliability.Using the tools it is given · unreviewed
- When a video-understanding instruction contains several conditional branches and the model must pick the branch matching what the video shows, both proprietary and open multimodal models pick correctly far more often when the correct branch is listed first, and accuracy falls as the correct branch moves later in the list.Following an unfamiliar procedure · unreviewed
- In open-weight instruction-tuned models, interventions that suppress caving to user pushback (DPO, SFT on chosen responses, or activation steering) also tend to reduce the model's rate of correcting a wrong answer when genuine supporting evidence is supplied, because the two answer-flip behaviors run on overlapping MLP neurons and attention heads with positively aligned steering directions.Telling the user what they want to hear · unreviewed
- In multi-step visual QA, a critic step run by the same model improves final accuracy only when it is shown the candidate evidence alone and denied the generator's textual rationale; conditioned on that rationale, the critic ratifies the generator's initial hypothesis and the loop performs worse than no critic at all.Fixing its own mistakes · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Running the same loop with the judge scoring an edit diff shows no instruction-following gain over single-pass, or the gain disappears under judges outside the loop and under blind human comparison, or regressions are as common as improvements among tasks entering the loop. Proposed technique, not catalogued: reference-free instruction-following judge critique as repair signal.