structured-long-term-memory-substrate-indexes-retrieval-control-logic-held CLM-0160When a structured long-term memory substrate (indexes, retrieval, control logic) is held fixed and only the backbone LLM is swapped, accuracy on multi-session memory benchmarks like LongMemEval and LoCoMo moves by a few points while per-query cost varies by roughly an order of magnitude, so recall quality in these settings is set mostly by the memory and retrieval design rather than by model choice.
Capability: Remembering across sessions
Observed on
Judged answer accuracy on LongMemEval (500 questions) and LoCoMo (1,540 questions), eight backbones from three vendors, single run under each benchmark's LLM-judge protocol, with t.
Applies when
- Effect on the capability:
- helps
- Needs:
- documents or an index to retrieve from
- Helps most:
- independent
- Fails when:
- The memory and retrieval design is weak. Swapping the backbone moves accuracy by a few points while per-query cost varies tenfold, so the model cannot compensate for the substrate.
Sources
- Measured: eight backbones on LongMemEval span 92.20-95.60% with ~30x cost spread, and a similar pattern on LoCoMo (though gpt-5.4-mini falls to 81.10% there, which the summary claim understates). The backbone comparison is internally controlled; the cross-system comparisons to prior memory systems use best publicly reported numbers under different harnesses. Both benchmarks are described by the authors as near-saturated, so the narrow spread may partly reflect a ceiling. No ablation removing individual memory stores, so the attribution to the memory substrate is argued rather than isolated.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For an 8B Llama backbone under a fixed 256-slot cache budget, replacing hard KV eviction with a learned soft write that blends a new token's key/value into a routed fixed-size memory slot retains far more long-context retrieval accuracy than sliding-window or sink caches, and unlike full-attention or quantized caches it does not run out of memory at 128K tokens.Losing information in long inputs · reviewed by AI
- When a retrieval store holds both the correct answer and a common misconception for the same question, injecting all retrieved memories raises the hallucination rate above a memory-free baseline for weaker models (53.0% vs 23.0% on TruthfulQA with gemma-4-E4B-it), while a stronger backbone (deepseek-v4-flash) instead shows RAG below its own no-memory rate, so conflict-driven amplification is concentrated in models susceptible to memory injection.Stating false facts confidently · reviewed by AI
- In multi-agent LLM pipelines over reasoning and code tasks with a pool of 3B-32B open-weight backbones, conditioning each per-step routing decision on a learned filtered memory rather than the full accumulated trajectory leaves accuracy about the same while cutting inference cost by roughly 40-50%, so most of the accumulated history is cost without accuracy value.Getting more from several models than from one · reviewed by AI
- For multi-hop QA over 28k-896k token haystacks with frozen Qwen2.5 backbones (7B-72B), summarizing the document in a parallel hierarchical tree — overlapping windows scanned ~5x, raw segments scored 'critical' passed straight to the final answer, and one summarization channel per decomposed sub-question — scores higher than sequential memory-agent and BM25 retrieval baselines without training, and beats an RL-trained memory agent on the out-of-distribution dataset while losing to it in-distribution.Losing information in long inputs · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A controlled swap of backbones over the same fixed memory substrate that produces a large accuracy spread (comparable to the gap between memory architectures), or evidence that the small spread here comes from benchmark saturation rather than from retrieval doing the work. Proposed technique, not catalogued: provenance-locked multi-store agent memory.