numeric-sequences-whose-first-differences-repeat-only-after-long CLM-0107On numeric sequences whose first differences repeat only after a long prefix, LLaMA 3.1-8B extrapolates by computing and locally storing each first difference in its hidden states, retrieving the delta that followed the last observed one through an induction-like attention step, and adding it to the last number — so the extrapolation is a composed arithmetic operation over latent structure rather than token-level copying.
Capability: Digit-level arithmetic
Observed on
One model (LLaMA 3.1-8B, whose tokenizer maps 0-999 to single tokens), synthetic integer sequences with deltas in -9..9, 29-element prompts; patching analyses run on 100 instances .
Sources
- Measured: MAE 4.27 and R^2 0.9958 over 10,000 sequences; layerwise linear probes (4,500 train / 500 test) show delta decodable from layer ~7 with R^2 above 0.8 while shuffled-label and wrong-position controls give negative R^2; activation patching from a clean sequence into a flat all-100s corrupt sequence raises probability of 100+delta sharply at token 27 from layer 14 onward; head-level patching ranks tokens 57, 33, 27, 55, 25 as most causal. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a text-pretrained LLM is prompted to forecast a numeric price series, rounding the input values to integers while keeping the original price scale gives lower forecast error than full-precision raw values, training-set standardization, or per-window instance normalization, because normalized values fall outside the numeric distributions the model saw in pretraining and per-window rescaling makes the same price appear differently across samples.Predicting future events · reviewed by AI
- In 7-8B open-weight instruct models (Llama-3.1, Mistral-v0.3, Qwen2.5), the drop in a token's probability when a contiguous mid-layer MLP span is zero-ablated is near zero for function words, elevated for knowledge-bearing tokens, and abnormally spiked for hallucinated tokens, so this internal attribution signal — approximable by a small probe from one forward pass — can be used in beam scoring to raise factuality on open-ended and long-form generation, while leaving strict short-answer EM/F1 unchanged or worse.Stating false facts confidently · reviewed by AI
- When a calculator-using model is trained with reinforcement learning on verifiable final-answer rewards for an arithmetic search task (Countdown), the gain lands almost entirely at low k — pass@1 rises sharply while pass@16 moves little or falls — because the update can only reinforce correct trajectories the starting policy already samples, and prompt groups with no correct sample supply no gradient.Using the tools it is given · reviewed by AI
- When post-training a speech LLM for ASR with GRPO, adding a reward term from a separately pretrained, frozen CTC acoustic judge that scores each sampled transcription against the audio cuts insertion errors on out-of-distribution meeting speech (22-28% fewer on AMI) relative to a schedule-matched text-only -WER reward, without raising word error rate and without needing the judge at inference.Stating false facts confidently · reviewed by AI
- In audio language models that feed encoder outputs into a text LLM, downstream understanding accuracy across speech, sound and music is set by how semantically text-aligned the encoder is rather than by whether its representation is discrete or continuous — reconstruction-oriented codecs with high acoustic fidelity underperform, semantically distilled tokenizers can beat their own continuous teacher, and scaling the LLM backbone does not recover the lost information.Understanding speech and audio · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Linear probes at the delta-bearing token position decoding no better than the position-shifted control, or patching the hidden state at that position (with the final query patched) producing no rise in probability of last-value-plus-retrieved-delta, would show the described circuit is not what drives the prediction.