medical-multiple-choice-qa-multi-agent-pipeline-mining-natural-language-lessons-own CLM-0438observationsingle paperReviewed by AI
For medical multiple-choice QA with a multi-agent pipeline, mining natural-language lessons from its own past trajectories (both correct and failed) into a reusable library gives a larger accuracy gain than the multi-agent role decomposition itself, and the gain comes mostly from the most abstract tier: removing principle-level entries costs more than removing pattern- or case-level entries.
Capability: Improving its own scaffolding from failures
Observed on
Text-only medical multiple-choice benchmarks (MedQA, MedMCQA, PubMedQA, MMLU/MMLU-Pro medical subsets) with Qwen3-8B and GPT-4o; ground-truth labels available to tag trajectories a.
Sources
- Validated by: Accuracy over nine medical QA sets against single-LLM, static multi-agent (MedAgents, MDAgents) and experience-based baselines (ExpeL, ReasoningBank, G-Memory, MDTeamGPT, EvolveR); average accuracy 81.1 vs 77.8 for MDAgents on Qwen3-8B and 93.9 vs 92.4 on GPT-4o; ablations remove experience (-6.7 on MedQA) versus removing planner and department agents (-2.9), and remove each tier (principles -4.6, patterns -2.5, cases -1.2).. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On knowledge, math and medical QA benchmarks with 8B open backbones (Llama-3.1-8B-Instruct, Qwen3-8B), the accuracy advantage of multi-agent debate pipelines over a single agent can be matched by a single-turn single agent whose hidden state is steered by a query-composed mixture of role vectors, at roughly a twentieth of the token cost, so the benefit of these pipelines comes from multi-perspective specialization rather than from inter-agent text exchange.Getting more from several models than from one · reviewed by AI
- In a hierarchical multi-agent system where a controller gates acceptance of worker outputs on semantic entropy and semantic density computed over resampled completions, the intervention helps only where failures come from semantic instability (vague or contradictory outputs) and hurts where they come from evidence selection or reasoning-chain errors \u2014 gaining on ambiguity/hallucination-prone benchmarks while losing 10-14 points on multi-hop QA.Getting more from several models than from one · reviewed by AI
- For agentic retrieval-augmented multi-hop QA with 7B-class policy models, scoring each intermediate step for both logical validity and grounding in the retrieved passages — rather than scoring steps by the final answer — yields higher EM/F1 than outcome-reward and final-answer-based process-reward training, because it removes preference pairs where flawed retrieval happened to reach the right answer.Carrying a chain of inference through to its conclusion · reviewed by AI
- When an agent builds a whole Python repository from natural-language requirements alone, refining component boundaries with explicit cohesion and coupling thresholds until no split or merge is triggered produces higher functionality coverage and test pass rate than letting the LLM decide restructuring actions for a fixed budget of rounds, because unconstrained LLM-decided actions keep over-decomposing past the point of a coherent modular structure.Generating and editing working code · reviewed by AI
- For small open-weight backbones (Qwen3-4B) driving long-horizon remote-sensing tool chains, inserting a step-level verifier that checks each intermediate tool output against schema, sub-goal and domain constraints and triggers replanning is the single largest architectural gain, while hierarchical decomposition without domain-aligned tool grouping adds little and can lower final accuracy relative to a flat agent with the same verifier.Completing tasks that take many steps or many hours · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: An ablation on the same benchmarks where removing the role decomposition costs more accuracy than removing the experience library, or where removing case-level entries costs as much as removing principle-level entries.