arxiv-2303-17651 · paperSelf-Refine: Iterative Refinement with Self-Feedback
Aman Madaan, Niket Tandon, Prakhar Gupta, et al.
Created: 2023 · Ingested: 2026-09-02
This source: Self-Refine: Iterative Refinement with Self-FeedbackHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 4530 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch1000+ citations in the last 12 months · 4530 total · checked 2026-09-04
https://arxiv.org/abs/2303.17651(opens in a new tab)In brief
A single LLM prompted to critique and rewrite its own output improves on open-ended generation tasks but barely moves on math, where it cannot tell that an answer is wrong. Self-Refine uses 3 few-shot prompts (generate, feedback, refine) with the same model, no training, up to 4 iterations.
Evaluation covers 7 tasks (dialogue response, code optimization, code readability, GSM8K math, sentiment reversal, acronym generation, constrained generation with 20-30 keyword constraints) on GPT-3.5, ChatGPT, GPT-4, plus Codex on code tasks, scored by task metrics, blind human A/B, and GPT-4-as-judge.
Gains average ~20% absolute, range 5-40%. GPT-4 code optimization rises 27.3 to 36.0; GPT-4 dialogue preference 25.4 to 74.6. Math is flat: 64.1 to 64.1 for GPT-3.5, 92.9 to 93.1 for GPT-4, with ChatGPT feedback saying 'everything looks good' on 94% of instances.
Ablations isolate the feedback step: generic feedback drops sentiment reversal 43.2 to 31.2, no feedback to 0. A sampling baseline compares Self-Refine against k=4 ChatGPT samples in 1-vs-k human preference, and Self-Refine wins. Manual analysis of 70 samples attributes 33% of failures to mislocated errors, 61% to bad suggested fixes, 6% to the refiner.
Only closed OpenAI models work; Vicuna-13B could not produce formatted feedback or follow refine prompts. English only, greedy decoding at temperature 0.7, no matched token-budget accounting.
Treat self-refinement as a fix for underspecified generation, not for verifiable reasoning.
Written from the abstract by claude-opus-5 on 2026-09-05, with every figure checked against it. Not a substitute for the paper.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
- contestsReflect-and-retry raises task success when the feedback in the loop is a genuine external signal — a failing test, a compiler error, an environment outcome — but not when the "feedback" is the model's own unaided critique. The grounding, not the reflection step, is what does the work.
- supportsPrompting a model to generate its own feedback on a draft output and revise accordingly improves quality on open-ended generation tasks (dialogue, code, review-writing) over a single-shot attempt.