reranking-sampled-verilog-candidates-both-generated-testbench-execution-signal-llm CLM-0132When reranking sampled Verilog candidates with both a generated-testbench execution signal and an LLM judge, letting the judge see the execution outcomes in its prompt lowers final Pass@1 relative to acquiring the two signals separately and combining them only as a weighted score at selection time, because the judge anchors on the test verdicts (approving all-pass candidates with uncovered logic errors, over-penalizing failures caused by bad testbenches) and stops contributing information the tests miss.
Capability: Generating and editing working code
Observed on
Verilog code reranking over 10 sampled candidates per problem, two benchmarks (VerilogEval-v2, ResBench), nine generator models, with a 4B fine-tuned judge; the interactive arm dif.
Sources
- Independent fusion beat interactive fusion on every one of 9 models on both benchmarks (+4.49 and +4.56 average Pass@1); interactive fusion scored only slightly above the execution-only ablation, consistent with the reasoning signal being absorbed. Anchoring bias is supported by qualitative inspection of reasoning chains, not a quantitative measure. Only one judge checkpoint and one prompt-level interaction design tested, so it does not rule out better interactive designs.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
- When an LLM judge is given a prior evaluation score as context metadata on a revision, its rubric scores shift toward that uninformative prior — seven of eight tested models (API and open-weight, 0-5 scale, 20 fixed texts) show a negative total shift, and on a categorical compliance task the metadata blocks 48% of error corrections and flips about 10% of previously correct judgments to the anchored wrong label.Judging another's work as well as a careful reviewer would · reviewed by AI
- When comparing a weight-shared recursive model against a depth-matched plain transformer on code generation, the ranking read from a single checkpoint can invert during training, because the larger-parameter arm passes its validation optimum and overfits while the parameter-lean recursive arm keeps improving — so a late-epoch "recursion fits better" reading is a degradation artifact, not a capability difference.Whether the measurement made the finding · reviewed by AI
- For LLM agents that score their own actions during exploration in text-based games and then reuse that experience, the accuracy of those self-scores against the environment verifier does not predict later gains on held-out stricter configurations: block-level correlations between judgment agreement or calibration and the next checkpoint's score change sit near zero, so recognizing a good action is not the bottleneck — converting it into a transferable policy is.Improving its own scaffolding from failures · reviewed by AI
- Verifying a chain of thought by decomposing it into nodes and asking binary consistency questions per node localizes errors better than judging the whole chain at once (F1 up 10-25% over CoT-based verifiers, error-localization accuracy far above an 8-vote end-to-end judge) while using 6-58 times fewer tokens, because whole-chain judging dilutes attention and multi-sampling pays for long generations.Carrying a chain of inference through to its conclusion · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A judge given execution results as context — whether by prompt or trained to weigh feedback reliability — matching or beating decision-stage-only fusion on the same candidates and benchmarks. Proposed technique, not catalogued: Independent-channel execution/reasoning fusion at the decision stage.