frozen-backbone-multi-agent-scientific-discovery-pipeline-hypobench-tasks-distilling-reusable CLM-0393observationsingle paperReviewed by AI
For a frozen-backbone multi-agent scientific discovery pipeline on HypoBench tasks, distilling reusable procedural skills separately per stage — from discriminator critique of a whole hypothesis batch for generation, and from execution outcomes compared to ground-truth results for testing — beats no skill, an LLM-written skill, and a hand-written skill, and removing each stage's feedback signal costs most of the gain.
Capability: Improving its own scaffolding from failures
Observed on
13 HypoBench tasks split 1:1 into train/test, DeepSeek-V4-Flash backbone, skill updates kept only when the training objective improves; data-driven discovery with executable statis.
Sources
- Validated by: On held-out HypoBench tasks: hypothesis quality 0.785 and Hit@K 0.648 versus 0.724/0.409 with no skill, 0.738/0.579 for an AI-generated skill, 0.761/0.497 for a human-designed skill, 0.719/0.472 for HypoGeniC; test pass rate 0.659 versus 0.562 no-skill, 0.612 human-designed, 0.589 ReAct, 0.417 POPPER. Ablations: removing discriminator feedback drops Hit@K to 0.491; removing execution outcomes drops pass rate to 0.565. Three runs. The generation quality metric is the same discriminator used as the training signal, so that number is not an independent check.. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM data-preparation pipelines in specialist domains (math, finance, law, medicine), diagnosing a small pilot set of generated samples against domain-selected quality criteria and rewriting the individual pipeline components (chunking, instruction builder, response generator) before full-scale generation yields training data with higher downstream fine-tuning utility than post-hoc filtering or one-shot agent/skill generators using the same backbone — but the gain is non-monotonic in rounds, peaking at two and degrading after.Improving its own scaffolding from failures · reviewed by AI
- Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
- In multi-agent LLM workflows with a mixed-size model zoo, choosing which model instantiates each agent step-by-step from an online estimate of task-progress gain plus remaining time and cost budget keeps task quality near the best single-model policy at lower energy than one-shot query-level routing or stall-triggered escalation.Getting more from several models than from one · reviewed by AI
- For LLM agents doing active clinical diagnosis in interactive patient environments, wrapping the model in a scaffold of role-specialized agents, evidence tools and an explicit evidence state, with a runtime observer that tracks uncertainty and evidence coverage to decide when to stop acquiring evidence, raises both diagnostic accuracy and execution stability over static case-to-answer prompting, with the size of the gain depending on the backbone model.Using the tools it is given · reviewed by AI
- For agentic retrieval-augmented multi-hop QA with 7B-class policy models, scoring each intermediate step for both logical validity and grounding in the retrieved passages — rather than scoring steps by the final answer — yields higher EM/F1 than outcome-reward and final-answer-based process-reward training, because it removes preference pairs where flawed retrieval happened to reach the right answer.Carrying a chain of inference through to its conclusion · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: On a fresh held-out task split with an independently built hypothesis evaluator, feedback-refined skills score no better than a fixed human-designed or LLM-written skill, or removing per-stage feedback leaves the test pass rate unchanged.