llm-generated-hls-kernels-targeting-real-fpga-boards-simulation-synthesis CLM-0423For LLM-generated HLS kernels targeting real FPGA boards, simulation and synthesis success overstates deployability, and the two stages are fixed by different interventions: an evolutionary search over simulated candidates raises synthesizable and executable rates while barely raising functional correctness, whereas C/C++-level verify-and-repair raises functional correctness while leaving synthesizability flat.
Capability: Generating and editing working code
Observed on
78 HLS-Eval tasks from natural-language specifications, Vitis HLS 2022.2 and a Xilinx U200 board at 200 MHz; ablation run on one backbone (Qwen3-Coder-30B-A3B)..
Sources
- Validated by: Against zero-shot pass@10 on 78 HLS-Eval tasks across five models: average +16.9% synthesizable, +26.7% executable, +30.6% functionally correct, best 92.3% correct with Gemini 3.1 Pro; ablation isolates the two modules (eoh 80.1/66.7/48.7 vs reflect 57.7/50.0/46.2 vs base 59.0/47.4/37.2).. Measured on one benchmark (78 of 94 instances, interfaces adjusted), one board, one toolchain; ablation on a single mid-size open model, with pass@10 used to match the search budget. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM-generated HLS kernels targeted at real FPGA boards, the two stages of failure respond to different interventions: an evolutionary search over simulation-validated kernel candidates mainly raises synthesizable and executable rates, while a C/C++-level functional verify-and-repair loop mainly raises functional correctness and barely moves synthesizability, so both are needed for end-to-end success.Generating and editing working code · reviewed by AI
- On simple HLS kernel generation tasks (85 designs from CHStone, MachSuite, Polybench, Rosetta, C2HLSC), giving a modest open-source model (gpt-oss-120b) a shell with a C++ compiler for self-verification and iterative refinement saturates the benchmark at pass@10 (100% through synthesis) and exceeds 90% at pass@1, so this class of task no longer separates models; repeated sampling helps the weaker model far more (+60 points synthesis pass rate at k=10 for gpt-oss-20b vs +11 for the 120b).Generating and editing working code · reviewed by AI
- For LLM agents that must compose and then execute domain tool workflows, training separate adapters for the two roles splits the gains cleanly: supervised fine-tuning of the planner on reference tool sequences mainly improves workflow-order fidelity, while node-level RL on argument-match rewards mainly improves tool-argument grounding and final-answer quality, and combining both beats either on a 9B backbone.Planning a sequence of actions before acting · reviewed by AI
- In search-based automated program repair of Simulink/Stateflow models, replacing hand-crafted domain-specific mutation operators with an LLM that proposes label rewrites from static component context lowers repair coverage under a fixed wall-clock budget, because unconstrained generative edits widen the candidate space while each candidate costs an expensive simulation and the generator gets no behavioural feedback about why a patch failed.Generating and editing working code · reviewed by AI
- On real C/C++ faults with reproducing tests, 24 current models repair only about 11% of the 248 bugs and about 6% of the 102 vulnerabilities, far below their Java and Python repair rates on comparable benchmarks, and fine-tuning on the benchmark's commit corpus raised Pass@k in 21 of 28 comparisons by an average 85% relative, so the gap is partly a training-data gap.Finding and fixing a fault in existing code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: An ablation on other backbones or benchmarks where the functional-repair module raises synthesizable/executable rates as much as evolutionary search does, or where simulation-passing designs almost always build and run correctly on hardware.