simple-hls-kernel-generation-tasks-85-designs-chstone-machsuite CLM-0485On simple HLS kernel generation tasks (85 designs from CHStone, MachSuite, Polybench, Rosetta, C2HLSC), giving a modest open-source model (gpt-oss-120b) a shell with a C++ compiler for self-verification and iterative refinement saturates the benchmark at pass@10 (100% through synthesis) and exceeds 90% at pass@1, so this class of task no longer separates models; repeated sampling helps the weaker model far more (+60 points synthesis pass rate at k=10 for gpt-oss-20b vs +11 for the 120b).
Capability: Generating and editing working code
Observed on
Single-kernel HLS designs generated from a natural-language spec plus a provided testbench, two gpt-oss models, mini-swe-agent harness with bash and compiler access, N=10 samples; .
Sources
- Measured pass@k at parse, compile, testbench and synthesis stages on 85 designs with 10 samples each, for two models only; no zero-shot control run is reported in this text, so the compiler-access attribution is argued rather than ablated, and no commercial models are tested.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM-generated HLS kernels targeted at real FPGA boards, the two stages of failure respond to different interventions: an evolutionary search over simulation-validated kernel candidates mainly raises synthesizable and executable rates, while a C/C++-level functional verify-and-repair loop mainly raises functional correctness and barely moves synthesizability, so both are needed for end-to-end success.Generating and editing working code · reviewed by AI
- For LLM-generated HLS kernels targeting real FPGA boards, simulation and synthesis success overstates deployability, and the two stages are fixed by different interventions: an evolutionary search over simulated candidates raises synthesizable and executable rates while barely raising functional correctness, whereas C/C++-level verify-and-repair raises functional correctness while leaving synthesizability flat.Generating and editing working code · reviewed by AI
- For 8B instruction-tuned backbones, training a LoRA corrector to diagnose a typed, executable error operator (error type, step location, corrupted and repaired proposition) and apply its inverse as a localized edit raises accuracy over free-form self-revision baselines (Self-Refine, RISE, SCoRe) on math, code and commonsense tasks, and dropping either the operator-prediction loss or the adjacent-state reconstruction loss loses most of the gain.Fixing its own mistakes · reviewed by AI
- In UI-to-code generation, where a rendered screenshot gives genuine external feedback, iterative self-critique still drifts below the initial draft over ten rounds, because a code edit changes the rendering non-locally and a fix for one region breaks regions that were already faithful; restricting each round to one typed, scoped repair target and carrying past targets forward as an avoid-list reduces that drift.Generating and editing working code · reviewed by AI
- In search-based automated program repair of Simulink/Stateflow models, replacing hand-crafted domain-specific mutation operators with an LLM that proposes label rewrites from static component context lowers repair coverage under a fixed wall-clock budget, because unconstrained generative edits widen the candidate space while each candidate costs an expensive simulation and the generator gets no behavioural feedback about why a patch failed.Generating and editing working code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafter linked technique "external-feedback-repair", which does not list this capability in addresses -- recorded, not asserted. Drafted confidence: medium. Falsifier as drafted: A rerun of the same 85 HLS-Eval generation tasks with the same harness and models showing pass@10 well below 100% through synthesis, or showing the weaker model gaining no more from k=10 sampling than the larger one. Drafted stance toward external-feedback-repair-works-only-with-real-grounding: supports -- The authors attribute the saturation to the agent invoking a real C++ compiler for syntactic and functional self-verification at inference time, an external grounding signal absent in the zero-shot baseline.