agent-pipelines-generate-whole-domain-specific-ml-codebases-medical-imaging CLM-0018In agent pipelines that generate whole domain-specific ML codebases (medical imaging) from scratch, execution/assembly validation and domain knowledge bases fix different failures: removing runtime and assembly testing mainly costs autonomy (more human interventions to debug), while removing the curated domain knowledge base mainly costs output quality (syntactically fine but domain-inappropriate pipelines) with little change in intervention count.
Capability: Generating and editing working code
Observed on
Multi-agent code generation of complete deep-learning pipelines in a specialised domain with objective end-task metrics; shown with Claude-4.5-Opus as backbone across six medical i.
Sources
- supportsTowards Fully Automated Medical Imaging Code Generation via Validation-based Context EngineeringOne ablation table on six datasets, single backbone model, Pass@3 best-of-three; component removals reported as average changes in human-intervention count and task metric. Human intervention count is a subjective measure by the authors; no independent replication.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM program-repair workflows built on self-correction, generated-knowledge or multi-agent scaffolding, the gain over plain prompting is unstable across benchmarks and can turn negative once extrinsic defect information (error messages, triggering tests, bug comments) is withheld, while simply putting the error message plus the failing test into the prompt yields more correct repairs than the scaffolds do.Fixing its own mistakes · reviewed by AI
- For single-agent procedural tasks where the relevant state fits a fixed schema known in advance, replacing the appended conversation transcript with a structured execution state that the model rewrites each step — discarding the reasoning trace once the state patch is validated — holds accuracy flat as the horizon grows while history-appending runtimes degrade, and the gain comes from the structure rather than the shorter prompt.Completing tasks that take many steps or many hours · reviewed by AI
- For LLM agents doing active clinical diagnosis in interactive patient environments, wrapping the model in a scaffold of role-specialized agents, evidence tools and an explicit evidence state, with a runtime observer that tracks uncertainty and evidence coverage to decide when to stop acquiring evidence, raises both diagnostic accuracy and execution stability over static case-to-answer prompting, with the size of the gain depending on the backbone model.Using the tools it is given · reviewed by AI
- In a hierarchical multi-agent system where a controller gates acceptance of worker outputs on semantic entropy and semantic density computed over resampled completions, the intervention helps only where failures come from semantic instability (vague or contradictory outputs) and hurts where they come from evidence selection or reasoning-chain errors \u2014 gaining on ambiguity/hallucination-prone benchmarks while losing 10-14 points on multi-hop QA.Getting more from several models than from one · reviewed by AI
- For tool-using agents, a safety harness searched separately for each model-and-domain deployment (jointly evolving a natural-language policy and executable pre/post-tool-call code, screened by a fresh-context adversarial reviewer) reaches a better safety-utility trade-off than fixed expert-designed defenses ported unchanged across deployments, because domain semantics determine which safety relations and trajectory state must be checked while the victim model's failure behavior determines whether a check should be semantic or deterministic and where in execution it must run.Following instructions hidden in data · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: An ablation on similar from-scratch pipeline generation where removing the domain knowledge base raises intervention counts as much as removing execution testing, or where removing execution testing costs task quality but not autonomy. Drafted stance toward external-feedback-repair-works-only-with-real-grounding: supports -- Removing execution testing produced the largest jump in required human intervention, consistent with grounded external signals being what drives autonomous repair. Proposed technique, not catalogued: validation-gated context propagation between agents.