llm-data-preparation-pipelines-specialist-domains-math-finance-law-medicine CLM-0590For LLM data-preparation pipelines in specialist domains (math, finance, law, medicine), diagnosing a small pilot set of generated samples against domain-selected quality criteria and rewriting the individual pipeline components (chunking, instruction builder, response generator) before full-scale generation yields training data with higher downstream fine-tuning utility than post-hoc filtering or one-shot agent/skill generators using the same backbone — but the gain is non-monotonic in rounds, peaking at two and degrading after.
Capability: Improving its own scaffolding from failures
Observed on
GPT-5.6 Sol as the coding-agent backbone; two RSI rounds; downstream SFT of Qwen2.5-7B and Llama-3.1-8B on domain data mixed with Dolly-15k; four domains with 53-154 source documen.
Sources
- Validated by: Downstream benchmark averages after SFT on the produced data, versus DataFlow filtering pipelines, direct LLM generators, agent generators and skill generators including same-backbone controls: +3.3 Math, +1.3 Finance, +1.4 Law, +2.2 Medical over the strongest competitor on Qwen2.5-7B, and +3.8/+6.2/+7.9/+5.9 over the same-backbone skill generator; round sweep shows Math 25.6 to 28.4 to 28.8 then 26.1.. Measured on DataPrep-Bench across four domains with two downstream base models and same-backbone agent/skill controls, which isolates framework from backbone. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a small (1.7B) LLM is fine-tuned to act as an ensemble router that assigns per-sample weights across candidate time-series forecasters, chain-of-thought supervision built by rule-templating known oracle weights against extracted temporal features matches or beats CoT generated by a frontier API model, and removing the CoT step costs far more on unseen candidate model pools than on in-distribution ones.Predicting future events · reviewed by AI
- On EEG and spike neuroscience classification benchmarks, a multi-agent LLM pipeline with planning, code generation and closed-loop retry does not by itself beat a single-shot LLM coding agent; the measured gain comes from making domain-specific pre-trained neural models addressable through one interface and choosing among them per dataset.Competence in a specialist domain · reviewed by AI
- When asked to generate a full procedural execution graph for an unseen task (recipe) in a specific creator's style, few-shot LLMs assign ingredients to steps better than a statistical two-stage pipeline but predict the number of steps worse, and swapping in a larger LLM backbone does not close that step-count gap.Planning a sequence of actions before acting · reviewed by AI
- For LLM agents that must compose and then execute domain tool workflows, training separate adapters for the two roles splits the gains cleanly: supervised fine-tuning of the planner on reference tool sequences mainly improves workflow-order fidelity, while node-level RL on argument-match rewards mainly improves tool-argument grounding and final-answer quality, and combining both beats either on a 9B backbone.Planning a sequence of actions before acting · reviewed by AI
- For a frozen-backbone multi-agent scientific discovery pipeline on HypoBench tasks, distilling reusable procedural skills separately per stage — from discriminator critique of a whole hypothesis batch for generation, and from execution outcomes compared to ground-truth results for testing — beats no skill, an LLM-written skill, and a hand-written skill, and removing each stage's feedback signal costs most of the gain.Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A matched comparison in which post-hoc filtering of a larger candidate pool, or the same backbone given equal compute without pipeline evolution, produces data with equal or better downstream domain scores; or rounds beyond two improving monotonically.