multi-hop-qa-over-28k-896k-token-haystacks-frozen-qwen2-5 CLM-0448For multi-hop QA over 28k-896k token haystacks with frozen Qwen2.5 backbones (7B-72B), summarizing the document in a parallel hierarchical tree — overlapping windows scanned ~5x, raw segments scored 'critical' passed straight to the final answer, and one summarization channel per decomposed sub-question — scores higher than sequential memory-agent and BM25 retrieval baselines without training, and beats an RL-trained memory agent on the out-of-distribution dataset while losing to it in-distribution.
Capability: Losing information in long inputs
Observed on
RULER-style needle-in-haystack multi-hop QA (HotpotQA, 2WikiMultiHopQA) with Qwen2.5-Instruct backbones; costs more total tokens than linear scanning and depends on correct query d.
Sources
- Validated by: F1 and Sub-EM on RULER-HotpotQA and RULER-2WikiMultiHopQA at six context lengths against vanilla Qwen2.5-32B, RAG-BM25, MemAgent-w/o-RL, Mem-alpha-w/o-RL, and RL-trained MemAgent and Mem-alpha; ablations on stride, skip connections, channel count, kernel size.. Measured on two RULER-derived synthetic benchmarks only, single model family, one paper, no significance testing; the OOD claim rests partly on two anecdotal cases and the OOD set was synthesized by the authors using the same protocol. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For 1.5B-7B Qwen2.5 backbones trained with RL on multi-hop retrieval QA, giving the policy reversible memory operations (summary, backtrack over a stack) with per-action process rewards raises F1 over trajectory-reward agentic-RAG baselines, and removing the two memory actions costs more than removing the planning action.Carrying a chain of inference through to its conclusion · reviewed by AI
- For conversational long-term memory with 8-14B backbones and plain top-K dense retrieval, writing each dialogue span into several complementary low-overlap semantic views before indexing retrieves gold evidence for more queries than structured extraction into a single view, and the advantage is largest at small retrieval budgets.Remembering across sessions · reviewed by AI
- On knowledge, math and medical QA benchmarks with 8B open backbones (Llama-3.1-8B-Instruct, Qwen3-8B), the accuracy advantage of multi-agent debate pipelines over a single agent can be matched by a single-turn single agent whose hidden state is steered by a query-composed mixture of role vectors, at roughly a twentieth of the token cost, so the benefit of these pipelines comes from multi-perspective specialization rather than from inter-agent text exchange.Getting more from several models than from one · reviewed by AI
- On repository-grounded LLM-infrastructure engineering tasks, frontier agents score far below ceiling (best 36.53% overall), and every one of eight models scores lower on long-horizon multi-file implementation than on single-file kernel completion, even though the full repository and test cases are visible.Completing tasks that take many steps or many hours · reviewed by AI
- When an agent builds a whole Python repository from natural-language requirements alone, refining component boundaries with explicit cohesion and coupling thresholds until no split or merge is triggered produces higher functionality coverage and test pass rate than letting the LLM decide restructuring actions for a fixed budget of rounds, because unconstrained LLM-decided actions keep over-decomposing past the point of a coherent modular structure.Generating and editing working code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: On the same benchmarks and backbones, the hierarchical parallel scheme scores no better than single-pass sequential memory agents, or its OOD advantage over the RL-trained agent disappears when the RL agent is trained on a broader mix.