structured-long-term-memory-substrate-indexes-retrieval-control-logic-heldWhen a structured long-term memory substrate (indexes, retrieval, control logic) is held fixed and only the backbone LLM is swapped, accuracy on multi-session memory benchmarks like LongMemEval and LoCoMo moves by a few points while per-query cost varies by roughly an order of magnitude, so recall quality in these settings is set mostly by the memory and retrieval design rather than by model choice.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Remembering across sessions
Observed on
Judged answer accuracy on LongMemEval (500 questions) and LoCoMo (1,540 questions), eight backbones from three vendors, single run under each benchmark's LLM-judge protocol, with t.
Sources
- Measured: eight backbones on LongMemEval span 92.20-95.60% with ~30x cost spread, and a similar pattern on LoCoMo (though gpt-5.4-mini falls to 81.10% there, which the summary claim understates). The backbone comparison is internally controlled; the cross-system comparisons to prior memory systems use best publicly reported numbers under different harnesses. Both benchmarks are described by the authors as near-saturated, so the narrow spread may partly reflect a ceiling. No ablation removing individual memory stores, so the attribution to the memory substrate is argued rather than isolated.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · unreviewed
- For persistent-memory agents, any memory-write rule that decides using only recency and provenance is stuck on a single tradeoff — accepting more genuine preference updates means admitting more poisoned ones — because an adversary who can launder a claim through the user's own channel matches the statistics a genuine revision produces; conditioning the write decision on the inferred authenticity of the claim given the interaction history moves both axes at once.Remembering across sessions · unreviewed
- On multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.Whether the measurement made the finding · unreviewed
- In partially observable text environments (ALFWorld, ScienceWorld), letting an LLM agent query an externally maintained state estimate that keeps an explicit distribution over unobserved object locations raises task success more than querying a deterministic memory of what has already been observed, and the gain shrinks to near zero on a frontier model that already nearly saturates the benchmark.Tracking state through a long task · unreviewed
- When fine-tuning a base model on instruction data, removing from the training targets any factual claim the base model cannot consistently recall raises the supported-claim rate of later long-form generations, but the gain comes largely from a more conservative response policy — more refusals and fewer supported claims per answer — not from generating more correct facts at equal coverage.Stating false facts confidently · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A controlled swap of backbones over the same fixed memory substrate that produces a large accuracy spread (comparable to the gap between memory architectures), or evidence that the small spread here comes from benchmark saturation rather than from retrieval doing the work. Proposed technique, not catalogued: provenance-locked multi-store agent memory.