reward-related-information-reasoning-model-s-hidden-states-concentrated-sparse CLM-0333mechanismsingle paperReviewed by AI
Reward-related information in a reasoning model's hidden states is concentrated in a sparse set of neurons: a linear probe trained with a temporal-difference objective finds 'value neurons' (under 1% per layer) whose activations predict expected correctness, and zeroing them in one layer drops MATH500 accuracy from 75% to an average 20% (as low as 1%), while zeroing the same number of random or magnitude-selected neurons costs 1-9 points; the step-to-step change in the probed value ('dopamine neurons', a stated analogy) used as a process reward for inference-time search lifted MATH500 from 72.2 to 77.8, above an implicit-PRM baseline at 75.0.
Capability: Spotting an error in a given artifact
Observed on
Qwen2.5-7B-SimpleRL-Zoo and similar RL-tuned open math models, MATH500, 2026; probes are linear on hidden states; PRM comparison is against greedy, random and implicit-PRM baseline.
Sources
- Validated by: MATH500 accuracy under search guided by the probe signal against greedy, random and implicit-PRM guidance; causal ablation of value neurons against random, magnitude and Wanda baselines across layers; transfer of value probes across datasets and models reported. Single 7B model family for the ablation.. The ablation is the strong result; the PRM gain is real but modest and lacks a trained-PRM baseline. The neuroscience vocabulary is explicitly analogy in the paper; [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For best-of-N selection on mathematical reasoning, a small verifier (~2M parameters) that reads the frozen generator's own hidden states at reasoning-step boundaries ranks correct candidates above incorrect ones better than a larger text-only verifier that re-encodes the generated solution, because the correctness signal is already present in the generator's upper-layer representations and must otherwise be inferred indirectly from emitted tokens.Spotting an error in a given artifact · reviewed by AI
- When a calculator-using model is trained with reinforcement learning on verifiable final-answer rewards for an arithmetic search task (Countdown), the gain lands almost entirely at low k — pass@1 rises sharply while pass@16 moves little or falls — because the update can only reinforce correct trajectories the starting policy already samples, and prompt groups with no correct sample supply no gradient.Using the tools it is given · reviewed by AI
- When training an LLM with policy-gradient RL on tasks that decompose into many dependent sub-problems, rewarding each trajectory segment by how many reference reasoning points it newly reaches beats a sparse final-answer reward by a margin that widens with task horizon, and still learns on problem sets where the base policy never samples a success and sparse reward gives no gradient at all.Completing tasks that take many steps or many hours · reviewed by AI
- In 7-8B open-weight instruct models (Mistral-7B, Qwen2.5-7B, Llama-3-8B) answering closed-book factual questions, a small set of feed-forward neurons (top 1% or fewer) selected on one dataset by class-conditional separation of their residual-space contribution at the final prompt token transfers to other factual QA datasets, where probes on those neuron activations detect hallucination about as well as probes on full residual-stream or MLP-output states.Stating false facts confidently · reviewed by AI
- For a same-family pair of reasoning and instruction-tuned 4B models, learning one interpolation ratio per module (attention block, FFN block, embedding, final norm) with a reward that scores correctness and ranks correct responses by length cuts average generated tokens by about 27% relative to the reasoning model while holding average accuracy, and beats fixed global-ratio merging and CMA-ES search over the same ratios.Cutting to the chase · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A trained external PRM beating the probe signal by a wide margin on the same model and budget, or the value-neuron ablation effect vanishing on non-RL-tuned or larger models.