arxiv-2604-23361 SRC-0353 · paper

An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code

Created: 2026-04-25 · Ingested: 2026-09-19

https://arxiv.org/abs/2604.23361(opens in a new tab)

Large language models (LLMs) have demonstrated strong performance on a wide range of software engineering tasks, including code generation and analysis. However, most prior work relies on cloud-based models or specialized hardware, limiting practical applicability in privacy-sensitive or resource-constrained environments. In this paper, we present a systematic empirical evaluation of two locally deployed LLMs, LLaMA 3.2 and Mistral, for real-world Python bug detection using the BugsInPy benchmark. We evaluate 349 bugs across 17 projects using a zero-shot prompting approach at the function leve

In brief

Small quantized models running offline on a laptop locate roughly 43-45% of real Python bugs when told in advance that a bug exists, but most of their remaining output only flags that something is wrong without naming the fix.

LLaMA 3.2 (8B) and Mistral (7B), both 4-bit quantized via Ollama on an M1 Pro MacBook with 16GB, were given zero-shot prompts over single extracted functions for 349 bugs from 17 BugsInPy projects (pandas, keras, scrapy, matplotlib and others), out of 501 total.

Mistral scored 44.4% correct, LLaMA 3.2 43.3%; partial responses numbered 171 and 161. McNemar's exact test on 349 paired predictions gave p = 0.68, so the gap is not distinguishable. Accuracy ranged from 0% on tqdm to 100% on PySnooper, and by bug type from 60.8% on Null/None checks down to 0% on Type Conversion and 16.7-21.7% on Other/Complex.

Grading was keyword matching against tokens introduced by the fix, checked against manual inspection of 50 sampled responses; the authors call it a lower bound. The prompt asserts a bug is present, so this is not detection under uncertainty. 2 models, 1 dataset, 1 language, no cloud-model comparison.

Treat local models as region-narrowing assistants, not fix generators.

Written from the abstract by claude-opus-5 on 2026-10-24, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.