hierarchical-multi-agent-system-where-controller-gates-acceptance-worker-outputs CLM-0575In a hierarchical multi-agent system where a controller gates acceptance of worker outputs on semantic entropy and semantic density computed over resampled completions, the intervention helps only where failures come from semantic instability (vague or contradictory outputs) and hurts where they come from evidence selection or reasoning-chain errors \u2014 gaining on ambiguity/hallucination-prone benchmarks while losing 10-14 points on multi-hop QA.
Capability: Getting more from several models than from one
Observed on
Hierarchical controller-plus-worker systems with small local workers (3-8B) and a frontier controller; benchmark subsets of 100-500 items, mostly single runs..
Sources
- Validated by: Accuracy against the same system with metrics disabled across nine benchmarks: StrategyQA 78% vs 70%, TruthfulQA 41% vs 19%, JailbreakBench 61% vs 25%, but MuSiQue 45% vs 55% and HotpotQA 54% vs 68%; metric-isolation ablation showed either metric alone fell below the no-metric baseline.. Measured on nine benchmarks with bootstrapped intervals, but subsets of 100-500 items, mostly one seed, one base architecture; the failure-mode attribution rests on trace inspection rather than a controlled manipulation of failure type.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When fine-tuning a 14B model on knowledge-graph-derived multiple-choice QA for a single medical domain, attaching supporting triples from the same source text chunk to each target triple during supervision, and repairing residual one-hop failures before reinforcement learning, gives higher accuracy on held-out 3-, 4- and 5-hop questions than training on isolated triples or running RL from an unrepaired checkpoint.Carrying a chain of inference through to its conclusion · reviewed by AI
- For long-horizon repository-level coding agents, a small (4B) critic trained to diagnose each just-completed step with a severity label, error type and causal rationale, and to redirect the next action when severity is critical, repairs more initially-failing tasks with a single branch (14.1% fail-to-pass) than five full resamples of the base agent (13.7%), at about one-fifth the base-agent output tokens.Judging another's work as well as a careful reviewer would · reviewed by AI
- When RL-training a central-agent multi-agent harness with 1.5B-3B Qwen2.5 backbones on multi-hop QA, optimizing only the coordinating central policy and leaving sub-agents frozen scores higher than jointly training central and sub-agent policies (average F1 47.79 vs 45.36, accuracy 42.88 vs 40.49 at 3B).Getting more from several models than from one · reviewed by AI
- When a tool-using agent is trained with outcome-only RL and a surface prompt cue co-occurs with one tool in most training examples, the agent invokes that tool on tasks where it is useless, but only for tools it has already learned to use well and mainly when the cue is semantically related to the tool.Using the tools it is given · reviewed by AI
- When an agent system evolves its own harness from failure traces, batching failures across tasks before editing buys efficiency but not better failure attribution: pooled across three agent benchmarks, the share of harness edits that merely accommodate the current model rose from 60.0% under per-failure serial editing to 75.6% under cross-task aggregation, and only fell to 45.5% when a separate analyst/critic/engineer refinement stage attributed each recurring pattern before the edit was written.Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A run showing the same gating raises accuracy on multi-hop QA or evidence-retrieval tasks, or that gains on TruthfulQA/JailbreakBench disappear at full benchmark scale with multiple seeds. Automatic check flagged: figures not in the source: 014.