search-based-automated-program-repair-simulink-stateflow-models-replacing-hand-crafted CLM-0143In search-based automated program repair of Simulink/Stateflow models, replacing hand-crafted domain-specific mutation operators with an LLM that proposes label rewrites from static component context lowers repair coverage under a fixed wall-clock budget, because unconstrained generative edits widen the candidate space while each candidate costs an expensive simulation and the generator gets no behavioural feedback about why a patch failed.
Capability: Generating and editing working code
Observed on
19 real faulty Stateflow models from four CPS case studies, 1-hour budget per run, 5 runs, GPT-5.5 / GPT-5.4-mini / GPT-4.1-mini as drop-in mutators inside FlowRepair; structural o.
Model cards: GPT-4
Sources
- Measured end-to-end: FlowRepair produced plausible patches for 18 of 19 models and valid for 16; the three LLM variants produced plausible for 4-6 and valid for 4. Same dataset, infrastructure and wall-clock budget as the original, so the comparison is controlled at the budget level but does not separate mutation quality from LLM inference latency (different numbers of candidates evaluated). Benchmark was public, so contamination is not excluded; one success (fridge_2a) matched a few-shot example in the prompt.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM program-repair workflows built on self-correction, generated-knowledge or multi-agent scaffolding, the gain over plain prompting is unstable across benchmarks and can turn negative once extrinsic defect information (error messages, triggering tests, bug comments) is withheld, while simply putting the error message plus the failing test into the prompt yields more correct repairs than the scaffolds do.Fixing its own mistakes · reviewed by AI
- In LLM-based program repair on Defects4J bugs with frontier models (GPT-5, DeepSeek-R1, Claude Sonnet 4.5) prompted zero-shot, passing the developer-written test suite does not mean the patch is grounded in the bug: manual inspection of 812 sampled repairs found ungrounded repairs in 72.7% of cases including test-passing patches, most often from incorrect causal localization (45.9%) rather than a wrong edit style, and the same models frequently misidentify the bug-triggering tests and generate additional tests that miss the triggering condition.Stating false facts confidently · reviewed by AI
- For LLM multi-agent systems failing on long-horizon web tasks, re-running the whole task from scratch — plain, with self-reflection, or with a critic agent — repairs few failures (at most 6.90% within three attempts, with feedback variants below plain rerun), while reconstructing the recorded execution prefix and intervening once at a symptom-flagged trace node repairs 20.15% with a single attempt.Getting more from several models than from one · reviewed by AI
- In zero-shot LLM program repair on Defects4J bugs, passing the developer-written test suite does not indicate that the patch is grounded in a correct understanding of the bug: manual inspection of sampled repairs found ungrounded patches in about three quarters of cases including ones that passed all available tests, with incorrect causal localization the largest category.Stating false facts confidently · reviewed by AI
- Every state-of-the-art model tested repairs fewer Defects4J and GitBug-Java bugs after semantics-preserving transformation of the buggy code, from -4% for GPT-4o to -16% for Llama-3.1, and the size of each drop correlates with how likely the model was to have memorized the original (negative log-likelihood), so part of published repair success on these benchmarks is recall of seen fixes rather than debugging.Finding and fixing a fault in existing code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafter linked technique "external-feedback-repair", which does not list this capability in addresses -- recorded, not asserted. Drafted confidence: medium. Falsifier as drafted: A run of the same benchmark and budget where LLM-generated mutations match or exceed FlowRepair's 18 plausible / 16 valid model coverage, or where adding simulation feedback to the prompt does not close the gap. Drafted stance toward external-feedback-repair-works-only-with-real-grounding: supports -- The LLM mutator generated candidates from static labels with no simulation feedback about failure duration or severity, and repaired far fewer models than the feedback-guided search it replaced. Proposed technique, not catalogued: LLM as mutation operator in search-based repair.