agents-working-over-heterogeneous-data-exposing-schema-content-tool CLM-0332observationsingle paperReviewed by AI
For agents working over heterogeneous data, exposing a schema/content/tool ontology as an MCP server the agent queries, and refining it by a loop of diagnose, attribute-to-level, typed patch, and a gate that accepts an edit only if a backbone-conditional paired evaluation shows a margin, raised trajectory accuracy on DDR-Bench by an average 17.8 points across six backbones and text-to-SQL EX on BIRD by about 7 points; injecting the same semantic content as a static prompt did not help consistently and cost 15 points on one backbone, and in ablation removing the acceptance gate was the largest loss (-11.2), then attribution (-6.3), diagnosis (-4.8), typed patches (-1.7).
Capability: Improving its own scaffolding from failures
Observed on
DDR-Bench, BIRD and one further data-agent benchmark, six backbones including GPT-5.5, Claude Sonnet 4.5/5, DeepSeek-V3/V4-Flash and Qwen3, 2026; ontology built by a builder agent,.
Sources
- Validated by: Trajectory and message accuracy on DDR-Bench across six backbones against a no-ontology ReAct baseline and a static-prompt semantic layer; EX/VES on BIRD; step-wise and level-wise ablations averaged over four backbones. Developer-reported.. Two findings worth separating: the query-vs-inject result (a static prompt of the same content hurt) and the gate result (accepting every edit was the largest ablation loss). The second is the regression-gated-loop claim already in the catalog, replicated in a different harness; [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For single-agent procedural tasks where the relevant state fits a fixed schema known in advance, replacing the appended conversation transcript with a structured execution state that the model rewrites each step — discarding the reasoning trace once the state patch is validated — holds accuracy flat as the horizon grows while history-appending runtimes degrade, and the gain comes from the structure rather than the shorter prompt.Completing tasks that take many steps or many hours · reviewed by AI
- In a hierarchical multi-agent system where a controller gates acceptance of worker outputs on semantic entropy and semantic density computed over resampled completions, the intervention helps only where failures come from semantic instability (vague or contradictory outputs) and hurts where they come from evidence selection or reasoning-chain errors \u2014 gaining on ambiguity/hallucination-prone benchmarks while losing 10-14 points on multi-hop QA.Getting more from several models than from one · reviewed by AI
- For LLM agents under a fixed ReAct harness answering questions over executable database-backed APIs plus document retrieval, accuracy falls sharply as the required reasoning depth grows from 1 to 5 hops (most models lose over half their accuracy), and the surviving failures concentrate in language-mediated steps — entity disambiguation, cross-source grounding, schema alignment — rather than in emitting syntactically valid tool calls.Carrying a chain of inference through to its conclusion · reviewed by AI
- When automatically rewriting the prompts of a multi-agent LLM system from its failures, locating the responsible agent by injecting a hint into one agent at a time and using that agent's hint-corrected intermediate output as the supervision signal produces prompt updates that generalize better than critique derived only from the system-level output.Improving its own scaffolding from failures · reviewed by AI
- For locally deployed open-weight models driving a stateful, dependency-ordered MCP tool server, cutting tool descriptions from full specifications (purpose, parameter semantics, constraints, failure conditions) down to one sentence each raises the fraction of calls the server rejects for every model tested, while its effect on task coverage is less consistent.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A static semantic layer that matches the MCP-queried one when both are refined by the same loop, which would locate the gain in the evolution rather than the query interface; or the gate failing to prevent regressions once the edit store grows large.