automated-unit-test-generation-domain-specific-java-libraries-nlp-toolkits CLM-0364mechanismsingle paperReviewed by AI
For automated unit test generation on domain-specific Java libraries (NLP toolkits), injecting LLM-written test snippets into a search-based generator's population when the evolutionary search stagnates yields higher branch and line coverage than either the search tool alone or the LLM alone, with the LLM contributing object-construction sequences and domain-shaped string inputs that random search does not reach.
Capability: Generating and editing working code
Observed on
100 classes from five Java NLP libraries; GPT-4o-class LLM; EvoSuite/DynaMOSA as the search backbone; 900s budget, 5 runs per class. Gains are smaller on mutation score and absent .
Sources
- Validated by: Median branch coverage 68.60% vs 58.03% (CodaMosa-J), 53.33% (EvoSuite), 32.64% (LLM-only); Wilcoxon p<<0.05, A12 0.72-0.85; mutation score 63.0% vs 52.0/58.0/38.26. Ablations show class-level context, self-refinement, and repeated calls each contribute; LLM-generated strings alone gave -1.4%.. Measured: 100 classes, 5 runs each, three baselines including a re-implemented CodaMOSA, with Wilcoxon, Friedman/Conover, A12 and bootstrap CIs. Per-class wins 36-59 with 2-3 losses. Single LLM family for the main configuration; [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When grading LLM-generated unit tests by pass rate and line coverage alone, a share of passing test suites never directly call the function under test — they reach it through wrappers, mocks, or unrelated public APIs — so pass rate overstates test quality, and this gap widens under agentic repository-access workflows and richer file-level context.Whether the measurement made the finding · reviewed by AI
- When an agent builds a whole Python repository from natural-language requirements alone, refining component boundaries with explicit cohesion and coupling thresholds until no split or merge is triggered produces higher functionality coverage and test pass rate than letting the LLM decide restructuring actions for a fixed budget of rounds, because unconstrained LLM-decided actions keep over-decomposing past the point of a coherent modular structure.Generating and editing working code · reviewed by AI
- When fine-tuning 7-8B code models on an LLM-synthesized Verilog instruction corpus, training on a 20-25% subset selected for functional correctness, instruction-solution alignment, and coverage of Verilog structural clusters (AST, CFG, netlist features) yields higher Pass@1 than training on the full corpus, because the automated corpus carries noise and redundancy that filtering removes.Generating and editing working code · reviewed by AI
- For LLM agents on long-horizon web penetration testing, holding the backbone model fixed and moving global state out of the context window onto a persistent graph of verified facts and fact-conditioned intent edges raises task success most on the hardest tasks, where success depends on facts obtained dozens of steps earlier (88.2% overall and 75.0% hard, vs 44.1% and 25.0% for the strongest shared-plain-text-memory baseline).Completing tasks that take many steps or many hours · reviewed by AI
- For LLM data-preparation pipelines in specialist domains (math, finance, law, medicine), diagnosing a small pilot set of generated samples against domain-selected quality criteria and rewriting the individual pipeline components (chunking, instruction builder, response generator) before full-scale generation yields training data with higher downstream fine-tuning utility than post-hoc filtering or one-shot agent/skill generators using the same backbone — but the gain is non-monotonic in rounds, peaking at two and degrading after.Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A replication on the same or comparable domain-specific library corpus in which the hybrid injection gives no coverage gain over the search-based tool alone, or in which an LLM-only baseline with equal compute matches it.