arxiv-2510-11059 SRC-0327 · paper

Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs

Created: 2025-10-13 · Ingested: 2026-09-19

Automated Program Repair (APR) plays a critical role in enhancing the quality and reliability of software systems. While substantial progress has been made in Java-based APR, largely facilitated by benchmarks like Defects4J, there remains a significant gap in research on C/C++ program repair, despite the widespread use of C/C++ and the prevalence of associated vulnerabilities. This gap is primarily due to the lack of high-quality, open-source benchmarks tailored for C/C++. To address this issue, we introduce Defects4C, a comprehensive and executable benchmark specifically designed for C/C++ pr

In brief

LLM program repair ability on Java and contest-style C/C++ does not carry over to real-world C/C++ code. Defects4C contains 248 human-verified buggy functions and 102 CVE-linked vulnerable functions from 41 real projects, each with a bug-revealing unit test, plus 9M bug-relevant commits for fine-tuning; 24 pre-trained LLMs were evaluated in single-round and conversation-based repair.

GPT-3.5-Turbo scores 59.0 pass@1 on DebugBench and 94.0 on CodeFlaws but 8.5 on Defects4C; GPT-4 drops from 74.6 to 9.0. Conversation-based repair with up to 10 attempts fixes at best 27/248 bugs (10.88%) and 7/102 vulnerabilities (6.86%), against reported Defects4J rates of 29.80 (function), 51.30 (hunk) and 71.30 (line). Scale does not help monotonically: CodeLlama-Python pass@100 goes 22.5 to 32.2 to 29.8 from 7B to 13B to 34B, partly because ~19% of CodeLlama-Instruct-34B outputs exceed the 2048-token limit. LoRA fine-tuning on Defects4C_bgcommit improves 21 of 28 pass@k cells, average relative gain 84.89%, but best pass@1 remains 4.92.

Evidence is measured on executable tests with human annotation (Cohen's kappa 0.88), though the Defects4J comparison is cross-paper, GPT-4 got only 2 attempts for budget reasons, and training pairs were not verified.

The benchmark covers single-function fixes only; multi-function and cross-file defects are untested. Treat Java APR numbers as an upper bound, not a transferable one.

Written from the abstract by claude-opus-5 on 2026-10-17, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsOn real C/C++ faults with reproducing tests, 24 current models repair only about 11% of the 248 bugs and about 6% of the 102 vulnerabilities, far below their Java and Python repair rates on comparable benchmarks, and fine-tuning on the benchmark's commit corpus raised Pass@k in 21 of 28 comparisons by an average 85% relative, so the gap is partly a training-data gap.