arxiv-2608-22938 SRC-0109 · paper

Execution-Anchored Hallucination Calibration Reranking for Verilog Code Generation

Created: 2026-08-24 · Ingested: 2026-09-15

Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, yet their performance degrades significantly on low-resource Hardware Description Languages such as Verilog. While multi-candidate sampling improves the likelihood of generating correct solutions, au-tomatically selecting the optimal candidate remains an open challenge. Through a systematic empirical study across nine models and two benchmarks, we identify two critical limitations:(1) existing execution-based reranking methods, which rely on testbench pass/fail outcomes, exhibit poor domain transferabili

In brief

Keeping an LLM judge blind to execution results beats letting it see them, when reranking sampled Verilog candidates. Exposing test outcomes anchors the reasoner on them and costs accuracy.

EAHC reranks 10 samples per problem from 9 generators (GPT-5, DeepSeek-V3, GLM-4.6, Qwen2.5-Coder, OpenCoder, Seed-Coder, HaVen, VeriPrefer, CodeV-R1) on VerilogEval-v2 (156 problems) and ResBench (56 problems). A Qwen3-4B judge fine-tuned on 47K compiler-verified traces scores candidates; a RAG testbench generator over a 53K corpus supplies execution vectors; candidates with identical execution vectors get one shared score.

Average Pass@1 rises from 53.99% to 65.10% on VerilogEval-v2 and 53.18% to 68.25% on ResBench, +5.91% and +4.96% over CodeT-GPT, first in 15 of 18 configurations. Independent fusion beats interactive fusion on every model (+4.49% and +4.56%). Judge scores disagree on 15.3% and 27.4% of execution-equivalent pairs, and on 5.2% and 16.4% of byte-identical pairs.

Ablations isolate both channels (+7.12% and +5.75%), and interactive vs independent differs only in the prompt with the same checkpoint, so no training-level comparison. CodeT-GPT still wins on 3 strong-generator configurations. Generated testbenches show 12.8% and 6.6% false-accept rates.

If you rerank with a judge, do not feed it test results.

Written from the abstract by claude-opus-5 on 2026-09-22, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsWhen reranking sampled Verilog candidates with both a generated-testbench execution signal and an LLM judge, letting the judge see the execution outcomes in its prompt lowers final Pass@1 relative to acquiring the two signals separately and combining them only as a weighted score at selection time, because the judge anchors on the test verdicts (approving all-pass candidates with uncovered logic errors, over-penalizing failures caused by bad testbenches) and stops contributing information the tests miss.