critique-s-worth-measured-whether-fixed-downstream-repair-agent CLM-0276When a critique's worth is measured by whether a fixed downstream repair agent can improve the interface from it, eight frontier models differ meaningfully in how actionable their UX critiques are, lead on different product-surface categories, and vary in reliability fixture by fixture; the best repair lift was about +0.22 on the benchmark's scale, so the task is unsaturated and not one-dimensional.
Capability: Judging another's work as well as a careful reviewer would
Observed on
UXBench, runnable web fixtures over ten product-surface families with coverage-gated browser exploration, 2026; eight frontier models; repair-lift protocol plus a blind human valid.
Sources
- Validated by: Repair-lift scores over seven rubric dimensions for eight models on ten fixture families, cross-checked by blind human validation; single domain (web UX).. The measurement idea is the durable part; the model rankings are 2026 snapshots. One domain, and the repair agent is a model, so lift partly measures the pair.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM multi-agent systems failing on long-horizon web tasks, re-running the whole task from scratch — plain, with self-reflection, or with a critic agent — repairs few failures (at most 6.90% within three attempts, with feedback variants below plain rerun), while reconstructing the recorded execution prefix and intervening once at a symptom-flagged trace node repairs 20.15% with a single attempt.Getting more from several models than from one · reviewed by AI
- For VLA and world-action manipulation policies that score near-ceiling on standard LIBERO, recovery success after a real execution failure falls sharply as recovery requires deeper state reasoning \u2014 retrying or adapting the immediate action works far more often than restoring a displaced task object or clearing an environmental obstruction \u2014 and rankings on the standard benchmark do not predict recovery rankings.Turning plans into physical action · reviewed by AI
- For LLM program-repair workflows built on self-correction, generated-knowledge or multi-agent scaffolding, the gain over plain prompting is unstable across benchmarks and can turn negative once extrinsic defect information (error messages, triggering tests, bug comments) is withheld, while simply putting the error message plus the failing test into the prompt yields more correct repairs than the scaffolds do.Fixing its own mistakes · reviewed by AI
- Used zero-shot as step-level action critics in long-horizon tool-calling tasks, frontier models are over-pessimistic — flagging a large share of correct actions and pushing the actor into revision loops that lower success below no critic at all — whereas small models fine-tuned on action-level verification rationales flag far fewer valid actions and raise repeated-run reliability.Using the tools it is given · reviewed by AI
- For long-horizon repository-level coding agents, a small (4B) critic trained to diagnose each just-completed step with a severity label, error type and causal rationale, and to redirect the next action when severity is critical, repairs more initially-failing tasks with a single branch (14.1% fail-to-pass) than five full resamples of the base agent (13.7%), at about one-fifth the base-agent output tokens.Judging another's work as well as a careful reviewer would · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Repair lift tracking the repair agent's own ability more than the critique, for instance the same critique giving very different lift under two repair agents.