comparing-weight-shared-recursive-model-against-depth-matched-plain-transformer-codeWhen comparing a weight-shared recursive model against a depth-matched plain transformer on code generation, the ranking read from a single checkpoint can invert during training, because the larger-parameter arm passes its validation optimum and overfits while the parameter-lean recursive arm keeps improving — so a late-epoch "recursion fits better" reading is a degradation artifact, not a capability difference.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Whether the measurement made the finding
Observed on
~28M-parameter models, instruction-to-Python corpus of ~38k training records, many passes over a fixed corpus (data-constrained repeated-data regime), 40-epoch budget, three seeds;.
Sources
- Measured: three arms (recursive 20-effective-depth shared 2-layer block, iso-parameter 2-layer, iso-depth 20-layer) trained 40 epochs, 3 seeds, snapshots at 9 epochs. Fit ranking between recursive and iso-depth arms reversed at epochs 8 and 40; iso-depth validation loss bottomed at epoch 20 (1.15) and rose to 1.45 by epoch 40 while training loss fell. Cross-seed spread tight. Only one architecture family, one synthetic corpus, no execution-based correctness metric; [truncated]
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- When training small instruction-tuned models with reinforcement learning plus factual supervision, routing each atomic fact's verification score only to the tokens that produced it, and down-weighting verifier judgements that do not change when their key evidence is removed, improves factuality benchmark scores over trajectory-level or reasoning- step-level factual rewards.Checking claims against evidence · unreviewed
- On multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.Whether the measurement made the finding · unreviewed
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · unreviewed
- For data-science Python tasks where several correct solutions differ by orders of magnitude in runtime, a model's correctness rate does not predict the efficiency of the code it writes: rankings invert between the two, code-specialized open models are less efficient than general open models of similar size class, and the fastest model rotates by library (NumPy vs Pandas vs PyTorch vs Polars).Generating and editing working code · unreviewed
- When a video-understanding instruction contains several conditional branches and the model must pick the branch matching what the video shows, both proprietary and open multimodal models pick correctly far more often when the correct branch is listed first, and accuracy falls as the correct branch moves later in the list.Following an unfamiliar procedure · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Tracking the same matched arms across training on other corpora or scales and finding the fit ranking stable across all checkpoints, so a single arbitrary checkpoint reproduces the validation-optimum ordering. Proposed technique, not catalogued: Trajectory-wide evaluation with per-arm validation-optimum checkpoint selection.