comparing-weight-shared-recursive-model-against-depth-matched-plain-transformer-code CLM-0034When comparing a weight-shared recursive model against a depth-matched plain transformer on code generation, the ranking read from a single checkpoint can invert during training, because the larger-parameter arm passes its validation optimum and overfits while the parameter-lean recursive arm keeps improving — so a late-epoch "recursion fits better" reading is a degradation artifact, not a capability difference.
Capability: Whether the measurement made the finding
Observed on
~28M-parameter models, instruction-to-Python corpus of ~38k training records, many passes over a fixed corpus (data-constrained repeated-data regime), 40-epoch budget, three seeds;.
Sources
- Measured: three arms (recursive 20-effective-depth shared 2-layer block, iso-parameter 2-layer, iso-depth 20-layer) trained 40 epochs, 3 seeds, snapshots at 9 epochs. Fit ranking between recursive and iso-depth arms reversed at epochs 8 and 40; iso-depth validation loss bottomed at epoch 20 (1.15) and rose to 1.45 by epoch 40 while training loss fell. Cross-seed spread tight. Only one architecture family, one synthetic corpus, no execution-based correctness metric; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Across 25 open-weight instruction-tuned models (4B-744B), the ability to modulate a named concept's residual-stream projection on plain instruction does not track parameter count or recency — the best scorer is an 8B 2024 model and some of the newest models score lowest — while within a single model line the score rises steadily across ordinary pre-training and post-training checkpoints, so activation controllability is acquired incidentally and must be measured per model rather than inferred from scale.Strategic deception and detecting it · reviewed by AI
- When fine-tuning 7-8B code models on an LLM-synthesized Verilog instruction corpus, training on a 20-25% subset selected for functional correctness, instruction-solution alignment, and coverage of Verilog structural clusters (AST, CFG, netlist features) yields higher Pass@1 than training on the full corpus, because the automated corpus carries noise and redundancy that filtering removes.Generating and editing working code · reviewed by AI
- When a model must anticipate, without execution feedback, what a change to its own runtime harness components will do, naming which components a lifecycle change can affect stays near ceiling as the number of interacting effects grows from 2 to 32, while predicting the resulting state, reasoning across teardown orders, and choosing minimal reconfigurations degrade sharply (e.g. GPT-5.6 Luna's reachable-condition Jaccard 91.7% to 14.1%, executed reconfiguration success 62.5% to 25.0%).Improving its own scaffolding from failures · reviewed by AI
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Tracking the same matched arms across training on other corpora or scales and finding the fit ranking stable across all checkpoints, so a single arbitrary checkpoint reproduces the validation-optimum ordering. Proposed technique, not catalogued: Trajectory-wide evaluation with per-arm validation-optimum checkpoint selection.