arxiv-2608-29376 · paper

Evaluating Tiny Recursive Models Across Training for Code Generation

Created: 2026-08-29 · Ingested: 2026-09-09

Code generation increasingly relies on large transformer models, whose capability advances with scale. Yet such a scale is costly, creating demand for small models, especially where data is limited. Recursive models address this by reusing a single block to add depth rather than stacking independent layers. Such models are typically evaluated by teacher-forced fit (next-token loss on ground-truth prefixes) or task accuracy, at a single checkpoint, whereas code is produced by free-running generation, where the model extends its own output. Whether a teacher-forced advantage survives free-runnin

In brief

Which architecture "wins" between a recursive weight-shared model and a plain transformer depends on the epoch you read it at: the fit ranking reverses twice inside one training budget, and the late reversal is the larger model overfitting, not the recursive one improving.

An autoregressive Tiny Recursive Model (TRM-AR, 27,830,784 parameters, effective depth 20 from a looped 2-layer block with deep supervision) was compared on natural-language-to-Python generation from a filtered tiny-codes corpus (47,649 records) against 2 controls: vanilla-2L at matched parameters and vanilla-20L at matched depth (46,713,600 parameters). All 3 arms ran 40 epochs, 3 seeds, 9 snapshots, scored on test perplexity, AST-parse rate, BLEU-4, and a Claude Haiku 4.5 judge.

At validation-selected checkpoints: vanilla-20L 1.154 val loss and 3.12 judge, TRM-AR 1.277 and 2.85, vanilla-2L 1.379 and 2.49. TRM-AR closes about 45% of the val-loss gap and about 57% of the judge gap, at roughly 175x the per-step time of vanilla-2L. Generalization gap at epoch 40: 0.13, 0.27, 1.04.

Effects are measured with matched-parameter and matched-depth controls and seed-sign consistency, but nothing is compute-matched, recursion and deep supervision are not ablated, and 3 seeds preclude significance tests.

No execution-based correctness (no unit tests, no pass@k), 1 synthetic templated corpus, ~28M scale, repeated-data regime, and 2 arms right-censored at epoch 40.

If you rank small architectures from a single checkpoint, expect the ranking to be an artifact of when you looked.

Written from the abstract by claude-opus-5 on 2026-09-09, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsWhen comparing a weight-shared recursive model against a depth-matched plain transformer on code generation, the ranking read from a single checkpoint can invert during training, because the larger-parameter arm passes its validation optimum and overfits while the parameter-lean recursive arm keeps improving — so a late-epoch "recursion fits better" reading is a degradation artifact, not a capability difference.