calculator-using-model-trained-reinforcement-learning-verifiable-final-answer-rewards-arithmetic CLM-0028mechanismsingle paperReviewed by AI
When a calculator-using model is trained with reinforcement learning on verifiable final-answer rewards for an arithmetic search task (Countdown), the gain lands almost entirely at low k — pass@1 rises sharply while pass@16 moves little or falls — because the update can only reinforce correct trajectories the starting policy already samples, and prompt groups with no correct sample supply no gradient.
Capability: Using the tools it is given
Observed on
Outcome-only (final-answer-verified) on-policy RL — RLOO, RLOO++, GRPO, DAPO — starting from a tool-formatted SFT checkpoint, on the Countdown 3- and 4-number task with a live calc.
Applies when
- Effect on the capability:
- helps, narrowly
- Needs:
- access to train the weights; an external signal (a test, a compiler, an environment outcome)
- Cost:
- high
- Fails when:
- The starting policy never samples a correct trajectory for a prompt, which supplies no gradient. The gain lands at pass@1; pass@16 moves little or falls.
Sources
- Measured on a fresh 1,024-problem held-out Countdown set (plus a noisy 50-problem public split) with 95% bootstrap intervals. Tool- RLOO went 35.8 to 56.6 pass@1 while pass@16 went 73.7 to 74.0; Tool- RLOO++ reduced pass@16 below Tool-RLOO. Correctness-distribution analysis over 16 samples per prompt shows mass shifting toward fully-correct sets and zero-correct prompts persisting. Tool-DAPO is a partial exception: it improved pass@16 modestly and cut the zero- correct fraction. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When training an LLM with policy-gradient RL on tasks that decompose into many dependent sub-problems, rewarding each trajectory segment by how many reference reasoning points it newly reaches beats a sparse final-answer reward by a margin that widens with task horizon, and still learns on problem sets where the base policy never samples a success and sparse reward gives no gradient at all.Completing tasks that take many steps or many hours · reviewed by AI
- Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
- For a small agent policy (Qwen3.5-4B) repeatedly deployed on the same task family with environment-level feedback, distilling only the trajectories whose textual-state guidance is newly or persistently useful relative to state-free execution raises the state-free policy's accuracy more than distilling all state-conditioned trajectories, and the resulting policy keeps improving over alternating rounds of state editing and consolidation.Improving its own scaffolding from failures · reviewed by AI
- Large pretrained models are well calibrated on multiple-choice and true/false questions when answer options are shown as lettered choices, and can predict the probability that their own sampled answer is correct (P(True)) or that they know the answer at all (P(IK)); calibration improves with model size and few-shot prompting, self-evaluation improves when the model sees several of its own samples first, and P(IK) generalizes only partly to new tasks, where its calibration degrades.Saying it does not know · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An outcome-only RL run on the same setup that raises pass@16 by as much as pass@1, or that turns prompts with zero correct samples in the base policy into solved prompts without added search or process supervision.