grading-llm-generated-unit-tests-pass-rate-line-coverage-aloneWhen grading LLM-generated unit tests by pass rate and line coverage alone, a share of passing test suites never directly call the function under test — they reach it through wrappers, mocks, or unrelated public APIs — so pass rate overstates test quality, and this gap widens under agentic repository-access workflows and richer file-level context.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Whether the measurement made the finding
Observed on
Repository-level unit test generation for focal functions in Rust, Go, Julia, PHP and Ruby; 14 models including Claude 4.5, GPT-5.2, DeepSeek V4-pro; effect size varies strongly by.
Sources
- supportsXREPOTEST: Benchmarking Multilingual Repository-Level Unit Test Generation for Large Language ModelsMeasured on a 3,642-function benchmark with containerized execution. Reported 9.7% of pass-rate-passing suites failing invocation checks overall, concentrated in Rust (28.3%) and Ruby (19.2%) and near zero in Go/PHP. File-level context on Ruby dropped invocation rate from 85.6% to 53.2% across three models; a headless Claude Code agent on PHP raised pass rate while invocation rate fell to 27.6%. Invocation is computed by AST call-identifier matching, so it can miss indirect-but-intentional invocation and cannot itself judge assertion quality; [truncated]
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · unreviewed
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · unreviewed
- For locally deployed open-weight models driving a stateful, dependency-ordered MCP tool server, cutting tool descriptions from full specifications (purpose, parameter semantics, constraints, failure conditions) down to one sentence each raises the fraction of calls the server rejects for every model tested, while its effect on task coverage is less consistent.Using the tools it is given · unreviewed
- On multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.Whether the measurement made the finding · unreviewed
- When a calculator-using model is trained with reinforcement learning on verifiable final-answer rewards for an arithmetic search task (Countdown), the gain lands almost entirely at low k — pass@1 rises sharply while pass@16 moves little or falls — because the update can only reinforce correct trajectories the starting policy already samples, and prompt groups with no correct sample supply no gradient.Using the tools it is given · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Show that on the same benchmark, tests scored as passing almost always contain a direct call to the focal function, and that agentic or file-context settings do not lower direct-invocation rates while raising pass rate. Proposed technique, not catalogued: invocation-rate metric for generated tests. Automatic check flagged: figures not in the source: 53.2%.