knowledge-math-medical-qa-benchmarks-8b-open-backbones-llama-3 CLM-0408On knowledge, math and medical QA benchmarks with 8B open backbones (Llama-3.1-8B-Instruct, Qwen3-8B), the accuracy advantage of multi-agent debate pipelines over a single agent can be matched by a single-turn single agent whose hidden state is steered by a query-composed mixture of role vectors, at roughly a twentieth of the token cost, so the benefit of these pipelines comes from multi-perspective specialization rather than from inter-agent text exchange.
Capability: Getting more from several models than from one
Observed on
Two 8B instruct backbones with accessible hidden states; five short-answer reasoning benchmarks plus a personality benchmark; MAS baselines LLM Debate, AutoGen, AgentVerse, DyLAN; .
Sources
- Validated by: Average accuracy and average rank over MMLU, TriviaQA, MATH, GSM8K, MedQA against vanilla, text-role, CAA, ActAdd, Prompt FT, Billy single-agent baselines and four MAS baselines, plus PersonalityBench scores; token counts compared; ablations on router, codebook and steering position.. Measured on two 8B backbones and five reasoning benchmarks; MoRe averaged 69.32 vs AgentVerse 69.28 on Llama and 71.48 vs 72.40 on Qwen, with about 1/20 the tokens. Margins over the best single-agent baseline are under one point, so the single-agent-beating part is thin; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For medical multiple-choice QA with a multi-agent pipeline, mining natural-language lessons from its own past trajectories (both correct and failed) into a reusable library gives a larger accuracy gain than the multi-agent role decomposition itself, and the gain comes mostly from the most abstract tier: removing principle-level entries costs more than removing pattern- or case-level entries.Improving its own scaffolding from failures · reviewed by AI
- On four diagram-grounded geometry benchmarks a multi-agent pipeline consistently helps open-source vision-language models (Qwen-2.5-VL 7B +6.8 and 32B +3.3 points on Geometry3K, with further gains on OlympiadBench and We-Math) but a strong closed model (Gemini 2.0 Flash) generally did better as a single agent on the classic benchmarks, gaining only modestly on the newest one, so agentic decomposition pays for weaker models and less familiar tasks rather than universally.Getting more from several models than from one · reviewed by AI
- For multi-agent pipelines on knowledge-intensive QA with an 8B backbone, deciding the collaboration topology after retrieval and conditioning it on the retrieved evidence yields higher accuracy than generating the topology from query semantics alone with the same retrieved evidence available at execution.Getting more from several models than from one · reviewed by AI
- For LLM agents that must compose and then execute domain tool workflows, training separate adapters for the two roles splits the gains cleanly: supervised fine-tuning of the planner on reference tool sequences mainly improves workflow-order fidelity, while node-level RL on argument-match rewards mainly improves tool-argument grounding and final-answer quality, and combining both beats either on a 9B backbone.Planning a sequence of actions before acting · reviewed by AI
- For multi-hop QA over 28k-896k token haystacks with frozen Qwen2.5 backbones (7B-72B), summarizing the document in a parallel hierarchical tree — overlapping windows scanned ~5x, raw segments scored 'critical' passed straight to the final answer, and one summarization channel per decomposed sub-question — scores higher than sequential memory-agent and BM25 retrieval baselines without training, and beats an RL-trained memory agent on the out-of-distribution dataset while losing to it in-distribution.Losing information in long inputs · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A single-agent steered variant that stays well below multi-agent debate and AgentVerse accuracy on the same benchmarks and backbones, or a setting where the MAS gain survives only with actual inter-agent text exchange and no activation-space composition reproduces it.