best-of-n-selection-mathematical-reasoning-small-verifier-2m-parameters-reads CLM-0367For best-of-N selection on mathematical reasoning, a small verifier (~2M parameters) that reads the frozen generator's own hidden states at reasoning-step boundaries ranks correct candidates above incorrect ones better than a larger text-only verifier that re-encodes the generated solution, because the correctness signal is already present in the generator's upper-layer representations and must otherwise be inferred indirectly from emitted tokens.
Capability: Spotting an error in a given artifact
Observed on
Math reasoning benchmarks (GSM8K, MATH-500, AIME, OlympiadBench) with frozen Qwen3 1.7B-14B and Llama 1B-8B generators, verifier trained on self-sampled trajectories with outcome l.
Sources
- Validated by: Best-of-8 accuracy and within-problem AUROC against EORM-55M text-only verifier, Qwen2.5-Math-PRM-7B, single-pass, and oracle pass@8, over five seeds; matched or beat EORM in 15 of 16 generator-dataset settings and all 6 Llama settings. An input-modality ablation on GSM8K held architecture size roughly fixed: text-only 82.23% / 0.519 AUROC vs last-hidden-state 86.11% / 0.691, isolating representation from capacity.. Measured: 4 math benchmarks, 4 Qwen3 scales plus 3 Llama scales, five seeds, disjoint eval splits, zero-shot transfer to OlympiadBench. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Reward-related information in a reasoning model's hidden states is concentrated in a sparse set of neurons: a linear probe trained with a temporal-difference objective finds 'value neurons' (under 1% per layer) whose activations predict expected correctness, and zeroing them in one layer drops MATH500 accuracy from 75% to an average 20% (as low as 1%), while zeroing the same number of random or magnitude-selected neurons costs 1-9 points; the step-to-step change in the probed value ('dopamine neurons', a stated analogy) used as a process reward for inference-time search lifted MATH500 from 72.2 to 77.8, above an implicit-PRM baseline at 75.0.Spotting an error in a given artifact · reviewed by AI
- When training an LLM with policy-gradient RL on tasks that decompose into many dependent sub-problems, rewarding each trajectory segment by how many reference reasoning points it newly reaches beats a sparse final-answer reward by a margin that widens with task horizon, and still learns on problem sets where the base policy never samples a success and sparse reward gives no gradient at all.Completing tasks that take many steps or many hours · reviewed by AI
- For a same-family pair of reasoning and instruction-tuned 4B models, learning one interpolation ratio per module (attention block, FFN block, embedding, final norm) with a reward that scores correctness and ranks correct responses by length cuts average generated tokens by about 27% relative to the reasoning model while holding average accuracy, and beats fixed global-ratio merging and CMA-ES search over the same ratios.Cutting to the chase · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A capacity-matched text-only verifier reading the same candidate solutions matching or beating the hidden-state verifier on best-of-N accuracy and within-problem AUROC across generators, or the hidden-state advantage disappearing once the text verifier is given the same training pool. Drafted stance toward reward-related-information-reasoning-model-s-hidden-states-concentrated-sparse: supports -- Both find that a reasoning model's hidden states carry correctness signal usable as a reward model for inference-time selection.