7b-instruction-tuned-transformer-under-fixed-kv-cache-budget-training-small CLM-0537For a 7B instruction-tuned transformer under a fixed KV-cache budget, training a small policy to pick which cached tokens to keep by rewarding low KL divergence from the full-cache next-token distribution retains more long-context accuracy than attention-heuristic eviction (H2O, SnapKV), and the margin widens as the budget tightens and as context grows.
Capability: Losing information in long inputs
Observed on
Mistral-7B-Instruct-v0.3 with zero-shot transfer to Llama-3-8B; LongBench and GSM8K/BBH; budgets 12.5-50%; contexts up to 8K; concurrent RL baselines were re-implementations, not o.
Sources
- Validated by: LongBench and reasoning accuracy across 12.5-100% budgets against full cache, H2O, SnapKV, and re-implemented ForesightKV/RLKV, plus perplexity increase, per-component latency and throughput; 5 training seeds; a supervised ranker on the same five features scored 37.8 vs 39.1 at 25% budget.. Measured on one 7B model with one transfer target; gains are 1-3 absolute points with std 0.9-2.1, so some cells overlap. Comparisons to concurrent RL methods rest on the authors' own re-implementations. The supervised-ranker control isolates RL from feature choice; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For an 8B Llama backbone under a fixed 256-slot cache budget, replacing hard KV eviction with a learned soft write that blends a new token's key/value into a routed fixed-size memory slot retains far more long-context retrieval accuracy than sliding-window or sink caches, and unlike full-attention or quantized caches it does not run out of memory at 128K tokens.Losing information in long inputs · reviewed by AI
- For a same-family pair of reasoning and instruction-tuned 4B models, learning one interpolation ratio per module (attention block, FFN block, embedding, final norm) with a reward that scores correctness and ranks correct responses by length cuts average generated tokens by about 27% relative to the reasoning model while holding average accuracy, and beats fixed global-ratio merging and CMA-ES search over the same ratios.Cutting to the chase · reviewed by AI
- When a small (1.7B) LLM is fine-tuned to act as an ensemble router that assigns per-sample weights across candidate time-series forecasters, chain-of-thought supervision built by rule-templating known oracle weights against extracted temporal features matches or beats CoT generated by a frontier API model, and removing the CoT step costs far more on unseen candidate model pools than on in-distribution ones.Predicting future events · reviewed by AI
- When RL-training a central-agent multi-agent harness with 1.5B-3B Qwen2.5 backbones on multi-hop QA, optimizing only the coordinating central policy and leaving sub-agents frozen scores higher than jointly training central and sub-agent policies (average F1 47.79 vs 45.36, accuracy 42.88 vs 40.49 at 3B).Getting more from several models than from one · reviewed by AI
- When fine-tuning 7-8B code models on an LLM-synthesized Verilog instruction corpus, training on a 20-25% subset selected for functional correctness, instruction-solution alignment, and coverage of Verilog structural clusters (AST, CFG, netlist features) yields higher Pass@1 than training on the full corpus, because the automated corpus carries noise and redundancy that filtering removes.Generating and editing working code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: At matched budget and model, heuristic eviction (H2O/SnapKV) matching or beating the KL-trained policy on LongBench, or the gap failing to widen at 12.5% budget and 8K context.