adding-more-models-llm-ensemble-yields-diminishing-marginal-gains CLM-0355Adding more models to an LLM ensemble yields diminishing marginal gains and past some ensemble size the marginal gain turns negative, so aggregate performance degrades with further additions.
Capability: Getting more from several models than from one
Observed on
Argued formally for cross-model ensembles/mixture-of-agents that aggregate outputs; no empirical measurement of the threshold is reported in the source text..
Sources
- Validated by: Not measured; the paper defines the benefit yield function and implosion threshold formally without reporting measured curves.. Argued, not measured. The source text is a framework proposal: it formalizes a marginal-gain function and a zero-crossing threshold and notes that prior ensemble work does not study performance versus ensemble size. No benchmark numbers or thresholds are given.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a small (1.7B) LLM is fine-tuned to act as an ensemble router that assigns per-sample weights across candidate time-series forecasters, chain-of-thought supervision built by rule-templating known oracle weights against extracted temporal features matches or beats CoT generated by a frontier API model, and removing the CoT step costs far more on unseen candidate model pools than on in-distribution ones.Predicting future events · reviewed by AI
- On repository-grounded LLM-infrastructure engineering tasks, frontier agents score far below ceiling (best 36.53% overall), and every one of eight models scores lower on long-horizon multi-file implementation than on single-file kernel completion, even though the full repository and test cases are visible.Completing tasks that take many steps or many hours · reviewed by AI
- For an LLM acting as orchestrator of dynamically created subagents, summarizing its own past trajectories into reusable workflow skills, short operational tricks, and capability cards for the candidate subagent backbones, then injecting those into the orchestrator's context, raises task scores on delegation- and aggregation-heavy tasks, and the subagent model cards are the component whose removal costs most.Getting more from several models than from one · reviewed by AI
- When an agent builds a whole Python repository from natural-language requirements alone, refining component boundaries with explicit cohesion and coupling thresholds until no split or merge is triggered produces higher functionality coverage and test pass rate than letting the LLM decide restructuring actions for a fixed budget of rounds, because unconstrained LLM-decided actions keep over-decomposing past the point of a coherent modular structure.Generating and editing working code · reviewed by AI
- For LLM data-preparation pipelines in specialist domains (math, finance, law, medicine), diagnosing a small pilot set of generated samples against domain-selected quality criteria and rewriting the individual pipeline components (chunking, instruction builder, response generator) before full-scale generation yields training data with higher downstream fine-tuning utility than post-hoc filtering or one-shot agent/skill generators using the same backbone — but the gain is non-monotonic in rounds, peaking at two and degrading after.Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Measuring accuracy as a function of ensemble size across many models and tasks and finding gains that stay non-negative up to the full model universe, with no degradation point.