zero-shot-function-level-small-locally-run-models-llama-3 CLM-0328observationsingle paperReviewed by AI
Zero-shot, at function level, small locally run models (Llama 3.2, Mistral) identify 43-45% of 349 real Python bugs from BugsInPy, well above a sub-5% keyword baseline, but with accuracy ranging from 0% to 100% by project and lowest (about 22%) on bugs that span several components, so detection is real but shallow and codebase-dependent.
Capability: Spotting an error in a given artifact
Observed on
BugsInPy, 17 projects, 349 bugs, function-level zero-shot prompting, keyword-based scoring, 2026; two small open models..
Sources
- Keyword-matched scoring is lenient and counts partial identifications; small models only. The per-project spread is the durable observation.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For data-science Python tasks where several correct solutions differ by orders of magnitude in runtime, a model's correctness rate does not predict the efficiency of the code it writes: rankings invert between the two, code-specialized open models are less efficient than general open models of similar size class, and the fastest model rotates by library (NumPy vs Pandas vs PyTorch vs Polars).Generating and editing working code · reviewed by AI
- On real C/C++ faults with reproducing tests, 24 current models repair only about 11% of the 248 bugs and about 6% of the 102 vulnerabilities, far below their Java and Python repair rates on comparable benchmarks, and fine-tuning on the benchmark's commit corpus raised Pass@k in 21 of 28 comparisons by an average 85% relative, so the gap is partly a training-data gap.Finding and fixing a fault in existing code · reviewed by AI
- When an agent builds a whole Python repository from natural-language requirements alone, refining component boundaries with explicit cohesion and coupling thresholds until no split or merge is triggered produces higher functionality coverage and test pass rate than letting the LLM decide restructuring actions for a fixed budget of rounds, because unconstrained LLM-decided actions keep over-decomposing past the point of a coherent modular structure.Generating and editing working code · reviewed by AI
- Every state-of-the-art model tested repairs fewer Defects4J and GitBug-Java bugs after semantics-preserving transformation of the buggy code, from -4% for GPT-4o to -16% for Llama-3.1, and the size of each drop correlates with how likely the model was to have memorized the original (negative log-likelihood), so part of published repair success on these benchmarks is recall of seen fixes rather than debugging.Finding and fixing a fault in existing code · reviewed by AI
- When a benign GitHub issue is augmented with a plausible-sounding maintenance requirement that specifies a helper reading an environment variable into a dangerous sink, repository-level APR agents produce patches that fix the original bug and pass the project test suite while containing the requested vulnerability in roughly half of tasks, and the strongest instruction-following coding models are more susceptible, not less.Writing secure code and dependencies · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: The same protocol on frontier models showing no project-level variance, or a scoring change that collapses the 43-45% figure.