multi-agent-debate-3-open-weight-agents-over-3-rounds-giving CLM-0611In multi-agent debate with 3 open-weight agents over 3 rounds, giving each agent a memory of past debates helps only if retrieval is conditioned on the current consensus ratio and peer responses are annotated with memory-derived reliability labels; randomly inserting past cases is the weakest retrieval policy and plain task-similarity retrieval trails every consensus-conditioned variant.
Capability: Getting more from several models than from one
Observed on
3 agents, 3 rounds, Qwen2.5-7B-Instruct, Qwen3-8B, Gemma-3-4B-IT on MATH500 / MMLU-Pro Economics and Engineering / TruthfulQA; memory built offline from a disjoint training split w.
Sources
- supportsRemember and Reweight: Enhancing Multi-Agent Debate with Experience Memory and Confidence EstimationValidated by: Accuracy on four benchmarks against CoT, Self-Consistency, MAD and MAD-M2; retrieval-policy ablation against random, similarity-only, positive-only, diversity-only and three fixed-lambda variants (0.663 avg vs 0.635 random, 0.647 similarity, 0.657 best fixed); component ablation; and stance-transition rates on the shared-misconception subset (Engineering correct-to-wrong flips 0.410 to 0.255).. Measured across 3 small open models and 4 benchmarks, plus two larger models in an appendix. Gains are small in aggregate (e.g. 0.607 vs 0.571 MAD on Qwen2.5-7B; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For agentic retrieval-augmented multi-hop QA with 7B-class policy models, scoring each intermediate step for both logical validity and grounding in the retrieved passages — rather than scoring steps by the final answer — yields higher EM/F1 than outcome-reward and final-answer-based process-reward training, because it removes preference pairs where flawed retrieval happened to reach the right answer.Carrying a chain of inference through to its conclusion · reviewed by AI
- On multimodal cultural-interpretation multiple-choice questions, pipelines that pool several same-backbone evidence streams (perception, retrieval, web verification) and weight them by scored relevance beat majority voting, self-consistency and agent debate mainly on the items where the streams disagree, and the margin shrinks sharply as the backbone gets stronger.Getting more from several models than from one · reviewed by AI
- In fully connected networks of LLM agents (Llama 3.2-8B, 50 agents, 10 rounds) where a fixed minority holds an unchanging extreme stance, neutral agents' textual justifications converge on the biased agents' vocabulary faster and more completely than their numeric stances do, so monitoring only the text overstates how much the group actually agrees.Getting more from several models than from one · reviewed by AI
- When one agent's context carries planted misinformation, the wrong answer persists through multi-agent debate and other agents often adopt it, yet debate still degrades less than a single misinformed agent (relevant misinformation cut single-agent accuracy by 16-25% on CWQ), especially when most agents are unexposed; consensus protocols were more stable than voting under peer pressure, and a misinformed majority can steer the group.Getting more from several models than from one · reviewed by AI
- In label-free RL post-training of reasoning models, deriving each model's reward from a separately-trained peer model's majority vote — rather than from its own majority vote — avoids the reward collapse and length degeneration that self-rewarding runs show, and the benefit grows as the paired models differ in family or size because their errors overlap less.Getting more from several models than from one · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A controlled comparison in which plain task-similarity or random retrieval of past debate cases matches consensus-conditioned retrieval at equal retrieval budget, or in which removing the confidence annotations leaves correct-to-wrong flip rates unchanged.