arxiv-2407-01489 SRC-0310 · paper

Agentless: Demystifying LLM-based Software Engineering Agents

Created: 2024-07-01 · Ingested: 2026-09-19

Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the fo

In brief

A fixed 3-stage pipeline with no autonomous decision-making matches or beats the best open-source LLM agents on real GitHub issues, at lower cost.

Agentless runs hierarchical localization (repo tree prompting plus embedding retrieval, then file skeletons, then edit lines), samples patches in a Search/Replace diff format, and selects among them using existing regression tests plus LLM-generated reproduction tests. It uses GPT-4o on SWE-bench Lite (300 problems), compared against 26 agent baselines including SWE-agent, AutoCodeRover, Moatless, OpenDevin and several commercial systems.

Agentless resolves 96 (32.00%) at $0.70 and 78,166 tokens on average, the top open-source score; SWE-agent with Claude 3.5 Sonnet reaches 69 (23.00%) at $1.62. Ablations isolate the contributions: majority voting alone gives 77 (25.67%), adding regression tests 81 (27.00%), adding reproduction tests 96 (32.00%). Combining prompting and embedding file localization raises ground-truth file coverage from 78.67% and 67.67% to 81.67%. Any-patch coverage of the 40 samples is 126 (42.0%), so selection, not generation, is the bottleneck.

Component ablations and cost-matched comparisons are reported, but baselines use different models and many closed-source entries release no trajectories or costs. Only 1 model, 1 language, and bug-fixing tasks are covered; the authors also found 10.0% of SWE-bench Lite problems lack sufficient information and 4.3% leak the ground-truth patch.

Treat complex agent scaffolding as unproven relative to a simple sample-and-filter pipeline.

Written from the abstract by claude-opus-5 on 2026-10-17, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsOn SWE-bench Lite, asking the model to name suspicious files from the issue plus a repository tree finds the ground-truth file in about 79% of cases and embedding retrieval over the issue text in about 68%, and taking the union of the two raises file-level localization to about 82% because the two methods miss different files; a fixed three-step narrowing (files, then classes and functions, then edit lines) with no agent loop then supported the best open-source resolve rate at the time (32%).