asked-optimize-code-already-performance-ceiling-frontier-llms-gpt CLM-0198When asked to optimize code that is already at its performance ceiling, frontier LLMs (GPT, Claude, Gemini families) never say it is already optimal under a plain "optimize this code for speed" prompt — they always emit an edit with a confident, unmeasured speedup claim; adding an explicit instruction to abstain unless >90% confident of an improvement produces abstention on a minority of such cases while leaving the edit rate on genuinely sub-optimal code unchanged.
Capability: Stating false facts confidently
Observed on
180 trials, nine models, five paired optimal/sub-optimal LeetCode-style snippets from EffiBench, direct API calls without agent refinement loops, n=5 per model per condition; no ex.
Sources
- Measured: 100% edit rate on optimal code under control across all nine models; 44.4% abstention and 0% false abstention under the penalty prompt. Small pilot (n=5 per model), problems are well-known LeetCode items so memorization may inflate the sub-optimal edit rate, and optimality of the baseline snippets is assumed rather than execution-verified.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In LLM-driven iterative prompt optimization, restricting gradient generation to examples the current prompt gets wrong and accepting a candidate edit only when its gain on hard cases exceeds a penalized regression on a preservation set of initially-correct examples yields higher final accuracy than unfiltered textual-gradient optimization, which degrades on some reasoning tasks below the zero-shot prompt.Improving its own scaffolding from failures · reviewed by AI
- In open-ended editing tasks with no unique correct output (multi-slide design editing), a separate judge model that scores the structural diff between the original and edited artifact against the instruction, and returns its critique verbatim as the next-turn instruction, raises instruction-following over a single pass, but the gain is selective: most tasks pass on the first attempt, and a minority of the ones that enter the loop get worse unless an earlier iteration is restored when the judge's own score declines.Fixing its own mistakes · reviewed by AI
- For a 7B code model fine-tuned on competitive-programming tasks, training it to emit a fixed-schema requirement specification (goals, input/output constraints, examples, edge cases, dependencies) before the code raises Pass@1 more than training it to emit free-form reasoning or a stepwise implementation plan of similar length, with the largest margin on an out-of-distribution benchmark.Generating and editing working code · reviewed by AI
- Large pretrained models are well calibrated on multiple-choice and true/false questions when answer options are shown as lettered choices, and can predict the probability that their own sampled answer is correct (P(True)) or that they know the answer at all (P(IK)); calibration improves with model size and few-shot prompting, self-evaluation improves when the model sees several of its own samples first, and P(IK) generalizes only partly to new tasks, where its calibration degrades.Saying it does not know · reviewed by AI
- For 8B instruction-tuned backbones, training a LoRA corrector to diagnose a typed, executable error operator (error type, step location, corrupted and repaired proposition) and apply its inverse as a localized edit raises accuracy over free-form self-revision baselines (Self-Refine, RISE, SCoRe) on math, code and commonsense tasks, and dropping either the operator-prediction loss or the adjacent-state reconstruction loss loses most of the gain.Fixing its own mistakes · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: On a larger set of verified-optimal snippets, models under a plain optimize prompt decline to edit a substantial share of the time, or the confidence-threshold prompt fails to raise abstention above zero, or it also suppresses edits on genuinely sub-optimal code. Proposed technique, not catalogued: confidence-threshold abstention prompt for code optimization.