arxiv-2608-18419 SRC-0095 · paper

Mechanistic Interpretability of Structure-Aware Numerical Reasoning in LLaMA 3.1 8B

Created: 2026-08-19 · Ingested: 2026-09-15

Recent work has shown that large language models (LLMs) exhibit strong numerical sequence modeling capabilities and show promise in time-series prediction. While LLMs display in-context learning capabilities, the mechanisms with which they accomplish time-series prediction remain unclear. Specifically, whether they truly understand the underlying structure, which at a minimum requires reasoning over first differences in the sequence of numbers. To study this, we investigate Llama 3.1-8B from a mechanistic interpretability point of view. Mechanistic interpretability is an emerging field concern

In brief

LLaMA 3.1-8B extrapolates numerical sequences by computing first differences internally, storing them at specific token positions, and retrieving the right one through an induction-like mechanism before adding it to the last value. This is structure-level induction, not token-level copying.

The task is built so copying cannot work: sequences of unique integers between 0 and 999 whose first differences (drawn from -9 to 9, excluding 0) are unique until position 17, after which the delta sequence repeats. The model sees 29 numbers and predicts the 30th. Analysis uses linear probes on hidden states plus activation patching from a clean sequence into a flat corrupt sequence of 100s, via NNsight/NDIF.

On 10,000 sequences the model reaches MAE 4.2748 and R2 0.9958. Probes trained on 4,500 samples decode the delta at token 27 from layer 7 onward with R2 above 0.8; control probes decoding the same delta from token 57 give negative R2. Patching (100 zero-MAE instances) shows the final token's query causally reads token 27 from layer 14-15 onward. Head-level attribution picks out tokens 57, 33, 27, 55, 25 as the structurally meaningful positions. Swapping keys 27 and 25 shifts probability toward the counterfactual but never overtakes the ground truth.

Evidence is causal and includes shuffled-label and position controls, but rests on 1 model, 1 synthetic task family, and 100 hand-selected clean instances for all patching results. No other model sizes, tokenizers, or real time-series data were tested.

Treat LLM time-series performance as partly mechanistic rather than memorized, at least for integer sequences with repeating deltas.

Written from the abstract by claude-opus-5 on 2026-09-15, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsOn numeric sequences whose first differences repeat only after a long prefix, LLaMA 3.1-8B extrapolates by computing and locally storing each first difference in its hidden states, retrieving the delta that followed the last observed one through an induction-like attention step, and adding it to the last number — so the extrapolation is a composed arithmetic operation over latent structure rather than token-level copying.