grading-llm-generated-unit-tests-pass-rate-line-coverage-alone CLM-0064When grading LLM-generated unit tests by pass rate and line coverage alone, a share of passing test suites never directly call the function under test — they reach it through wrappers, mocks, or unrelated public APIs — so pass rate overstates test quality, and this gap widens under agentic repository-access workflows and richer file-level context.
Capability: Whether the measurement made the finding
Observed on
Repository-level unit test generation for focal functions in Rust, Go, Julia, PHP and Ruby; 14 models including Claude 4.5, GPT-5.2, DeepSeek V4-pro; effect size varies strongly by.
Sources
- supportsXREPOTEST: Benchmarking Multilingual Repository-Level Unit Test Generation for Large Language ModelsMeasured on a 3,642-function benchmark with containerized execution. Reported 9.7% of pass-rate-passing suites failing invocation checks overall, concentrated in Rust (28.3%) and Ruby (19.2%) and near zero in Go/PHP. File-level context on Ruby dropped invocation rate from 85.6% to 53.2% across three models; a headless Claude Code agent on PHP raised pass rate while invocation rate fell to 27.6%. Invocation is computed by AST call-identifier matching, so it can miss indirect-but-intentional invocation and cannot itself judge assertion quality; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When an agent builds a whole Python repository from natural-language requirements alone, refining component boundaries with explicit cohesion and coupling thresholds until no split or merge is triggered produces higher functionality coverage and test pass rate than letting the LLM decide restructuring actions for a fixed budget of rounds, because unconstrained LLM-decided actions keep over-decomposing past the point of a coherent modular structure.Generating and editing working code · reviewed by AI
- For LLM agents on long-horizon web penetration testing, holding the backbone model fixed and moving global state out of the context window onto a persistent graph of verified facts and fact-conditioned intent edges raises task success most on the hardest tasks, where success depends on facts obtained dozens of steps earlier (88.2% overall and 75.0% hard, vs 44.1% and 25.0% for the strongest shared-plain-text-memory baseline).Completing tasks that take many steps or many hours · reviewed by AI
- For automated unit test generation on domain-specific Java libraries (NLP toolkits), injecting LLM-written test snippets into a search-based generator's population when the evolutionary search stagnates yields higher branch and line coverage than either the search tool alone or the LLM alone, with the LLM contributing object-construction sequences and domain-shaped string inputs that random search does not reach.Generating and editing working code · reviewed by AI
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- Auditing AI-generated code with static smell detection at code, structure and architecture level, from single-file tasks up to agent-built systems, more capable models produced more bloated and more coupled code, code volume was a near-perfect predictor of structural decay, and neither functional correctness nor more detailed prompting reduced it, so passing tests does not indicate maintainable design.Making design decisions that hold up as a system grows · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Show that on the same benchmark, tests scored as passing almost always contain a direct call to the focal function, and that agentic or file-context settings do not lower direct-invocation rates while raising pass rate. Proposed technique, not catalogued: invocation-rate metric for generated tests. Automatic check flagged: figures not in the source: 53.2%.