ui-to-code-generation-where-rendered-screenshot-gives-genuine-external-feedback CLM-0183In UI-to-code generation, where a rendered screenshot gives genuine external feedback, iterative self-critique still drifts below the initial draft over ten rounds, because a code edit changes the rendering non-locally and a fix for one region breaks regions that were already faithful; restricting each round to one typed, scoped repair target and carrying past targets forward as an avoid-list reduces that drift.
Capability: Generating and editing working code
Observed on
HTML/CSS webpage reproduction from a screenshot, three frontier and three open Qwen VLMs, ten refinement rounds, VLM-judge visual-fidelity scoring; effects on open models are mixed.
Sources
- Measured across 6 models x 3 benchmarks (18 settings): scoped rubrics beat naive self-evolution in 15/18 final-round and 14/18 best-round settings, average +1.20 overall judge points; naive refinement fell below the initial draft on Design2Code for both GPT models. A 60-sample human preference study on one benchmark for two models backs the main comparison. Scoring is by VLM judge, which the authors note is unstable on subtle differences; the coupling mechanism is argued and illustrated rather than directly measured.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For 8B instruction-tuned backbones, training a LoRA corrector to diagnose a typed, executable error operator (error type, step location, corrupted and repaired proposition) and apply its inverse as a localized edit raises accuracy over free-form self-revision baselines (Self-Refine, RISE, SCoRe) on math, code and commonsense tasks, and dropping either the operator-prediction loss or the adjacent-state reconstruction loss loses most of the gain.Fixing its own mistakes · reviewed by AI
- In open-ended editing tasks with no unique correct output (multi-slide design editing), a separate judge model that scores the structural diff between the original and edited artifact against the instruction, and returns its critique verbatim as the next-turn instruction, raises instruction-following over a single pass, but the gain is selective: most tasks pass on the first attempt, and a minority of the ones that enter the loop get worse unless an earlier iteration is restored when the judge's own score declines.Fixing its own mistakes · reviewed by AI
- For a frozen LLM run in a generate-critique-revise loop with no external feedback, the accuracy gain is concentrated where single-shot reasoning is already unreliable: on clean multi-step reasoning (BBH, ~74% single-shot) depth makes no difference beyond the confidence interval, while on harder math (MATH, 26.2% single-shot) the same loop gains +14.2 points, with GSM8K in between (+4.2).Fixing its own mistakes · reviewed by AI
- For LLM program-repair workflows built on self-correction, generated-knowledge or multi-agent scaffolding, the gain over plain prompting is unstable across benchmarks and can turn negative once extrinsic defect information (error messages, triggering tests, bug comments) is withheld, while simply putting the error message plus the failing test into the prompt yields more correct repairs than the scaffolds do.Fixing its own mistakes · reviewed by AI
- In LLM-based repository-level program repair, telling the generator to be minimal or constraining its output format does not shrink patches, because verbosity comes from the capability-raising machinery itself (iterative refinement, broad context); a separate post-generation refiner that rewrites an already-plausible patch is what reduces size without losing fixes.Cutting to the chase · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafter linked technique "external-feedback-repair", which does not list this capability in addresses -- recorded, not asserted. Drafted confidence: medium. Falsifier as drafted: Show that naive self-refinement with rendered feedback improves monotonically over rounds on these benchmarks, or that restricting each round to a single scoped repair target gives no advantage over free-form critique at equal round budget. Proposed technique, not catalogued: Rubric-scoped iterative repair with history.