8b-llama-backbone-under-fixed-256-slot-cache-budget-replacing CLM-0352For an 8B Llama backbone under a fixed 256-slot cache budget, replacing hard KV eviction with a learned soft write that blends a new token's key/value into a routed fixed-size memory slot retains far more long-context retrieval accuracy than sliding-window or sink caches, and unlike full-attention or quantized caches it does not run out of memory at 128K tokens.
Capability: Losing information in long inputs
Observed on
Llama-3.1-8B adapted by pre-training the new write/read modules on 10B tokens; cache capped at 256 slots; RULER and LongBench evaluations.
Sources
- Validated by: RULER average rose from 8.01 (full attention, OOM at 128K) and 4.88 (StreamingLLM) to 15.12, with 5.78 at 128K where baselines score 0 or OOM; LongBench average 18.02 vs 17.09 full attention and 10.76 StreamingLLM; commonsense average essentially unchanged (71.89 vs 72.27) with slightly worse WikiText perplexity (7.86 vs 7.54); latency and KV memory on par with sliding-window caches.. Measured on one model (Llama-3.1-8B) against sliding window, StreamingLLM, KIVI quantization, CPU offloading and full attention on lm-eval, LongBench and RULER to 128K. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a structured long-term memory substrate (indexes, retrieval, control logic) is held fixed and only the backbone LLM is swapped, accuracy on multi-session memory benchmarks like LongMemEval and LoCoMo moves by a few points while per-query cost varies by roughly an order of magnitude, so recall quality in these settings is set mostly by the memory and retrieval design rather than by model choice.Remembering across sessions · reviewed by AI
- For a 7B instruction-tuned transformer under a fixed KV-cache budget, training a small policy to pick which cached tokens to keep by rewarding low KL divergence from the full-cache next-token distribution retains more long-context accuracy than attention-heuristic eviction (H2O, SnapKV), and the margin widens as the budget tightens and as context grows.Losing information in long inputs · reviewed by AI
- When a retrieval store holds both the correct answer and a common misconception for the same question, injecting all retrieved memories raises the hallucination rate above a memory-free baseline for weaker models (53.0% vs 23.0% on TruthfulQA with gemma-4-E4B-it), while a stronger backbone (deepseek-v4-flash) instead shows RAG below its own no-memory rate, so conflict-driven amplification is concentrated in models susceptible to memory injection.Stating false facts confidently · reviewed by AI
- For speech LLMs on long audio, text-centric KV cache eviction methods (StreamingLLM, SnapKV, PyramidKV, ChunkKV) collapse because raw audio tokens have diffuse attention; anchoring compression on ASR transcripts — pooling each transcribed audio span down to its text-token length and adding the text embedding, while keeping untranscribed audio intact — retains over 90% of full-cache accuracy at a 5-10% cache budget where those baselines retain roughly 55-67% at a looser 25% budget.Losing information in long inputs · reviewed by AI
- For a same-family pair of reasoning and instruction-tuned 4B models, learning one interpolation ratio per module (attention block, FFN block, embedding, final norm) with a reward that scores correctness and ranks correct responses by length cuts average generated tokens by about 27% relative to the reasoning model while holding average accuracy, and beats fixed global-ratio merging and CMA-ES search over the same ratios.Cutting to the chase · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A matched comparison in which soft gated writes into a fixed-size routed memory score no better than hard eviction at equal cache size on long-context retrieval, or in which the reported gain disappears once baselines get the same 10B-token adaptation training.