clinical-calculator-tasks-where-formula-gold-variables-full-note-all CLM-0553On clinical-calculator tasks where the formula, gold variables and full note are all supplied, having an open-weight model write case-specific Python for a restricted executor instead of doing the arithmetic itself gives no reliable gain at 7B (+3.29 pp, calculator-cluster CI crossing zero) but a clear gain at 32B (+7.05 pp, CI clear of zero), and the remaining errors are invented unit conversions and one-tier slips in banded scores rather than raw computation.
Capability: Digit-level arithmetic
Observed on
MedCalc-Bench Verified (1,100 cases, 55 calculators), Qwen2.5-7B and Qwen2.5-32B-AWQ, five seeds, formula and gold variables supplied; Mistral-7B and Phi-3.5-mini moved in opposite.
Sources
- Measured on one benchmark (1,100 cases, 55 calculators) with two Qwen checkpoints plus two other 7B-class checkpoints as secondary arms; cluster bootstrap over calculators, effective sample 120-190 cases for the arithmetic comparisons. Two checkpoints do not establish scaling and the authors say so; benchmark formulas themselves were audited and 16 flagged.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When an error is injected into a supplied chain of thought, models mostly carry it to the answer: a planted arithmetic error costs small models 50-60% accuracy and 18% on average, though the largest models lose only 3-7%; unit-conversion errors cost about 9% at every scale; skipped steps and sycophantic steps cost small models about 10%; extra correct steps cost almost nothing (0-6%).Carrying a chain of inference through to its conclusion · reviewed by AI
- For small open-weight Qwen2.5 models (1.5B-3B) doing multi-step math reasoning, features derived from the spread of attention entropy across layers at each generation step predict whether the final answer is wrong better than output-logit confidence features, with the margin larger on harder problems (+0.0093 AUC on GSM8K, +0.0759 on MATH-500 at 3B).Stating false facts confidently · reviewed by AI
- Extended reasoning budgets improve a model's execution of arithmetic procedures on operands longer than those seen in training far more than they improve surface-form-to-value tasks like magnitude comparison and format conversion, because chain-of-thought supplies external working memory for carries and column alignment but cannot change how the input was tokenized.Digit-level arithmetic · reviewed by AI
- When a calculator-using model is trained with reinforcement learning on verifiable final-answer rewards for an arithmetic search task (Countdown), the gain lands almost entirely at low k — pass@1 rises sharply while pass@16 moves little or falls — because the update can only reinforce correct trajectories the starting policy already samples, and prompt groups with no correct sample supply no gradient.Using the tools it is given · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A matched-access comparison on clinical calculators where program generation beats direct arithmetic with an interval clear of zero at 7B as well, or where the residual program-route errors are shown to be computation rather than formula/variable/unit handling. Drafted stance toward offloading-computation-generated-code-interpreter-removes-most-arithmetic: contests -- Under matched formula and variable access, code offloading did not reliably beat direct arithmetic at 7B and left unit-conversion and banding errors at both scales, so it does not uniformly remove arithmetic error.