arxiv-2306-14898 SRC-0035 · paper

InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback

John Yang, Akshara Prabhakar, Karthik Narasimhan, Shunyu Yao

Created: 2023-06-26 · Ingested: 2026-09-14

Humans write code in a fundamentally interactive manner and rely on constant execution feedback to correct errors, resolve ambiguities, and decompose tasks. While LLMs have recently exhibited promising coding capabilities, current coding benchmarks mostly consider a static instruction-to-code sequence transduction process, which has the potential for error propagation and a disconnect between the generated code and its final execution environment. To address this gap, we introduce InterCode, a lightweight, flexible, and easy-to-use framework of interactive coding as a standard reinforcement learning (RL) environment, with code as actions and execution feedback as observations. Our framework is language and platform agnostic, uses self-contained Docker environments to provide safe and reproducible execution, and is compatible out-of-the-box with traditional seq2seq coding methods, while e…

In brief

Letting a code model execute and retry raises success rates far more than better single-shot prompting does. On text-to-SQL, GPT-4 goes from 9.1% in one shot to 73.7% when allowed up to 10 interaction turns with execution feedback.

InterCode wraps coding as a POMDP with code as actions and execution output as observations, in Docker containers. Three environments were built: Bash (200 filtered NL2Bash instructions across 4 file systems), SQL (1034 Spider dev instances in MySQL), and Python (117 MBPP tasks). Models tested: text-davinci-003, gpt-3.5-turbo, gpt-4, text-bison-001, chat-bison-001, Vicuna-13B, StarChat-16B, under Single Turn, Try Again, ReAct, and Plan & Solve.

Every model improves in the interactive setting. On Bash, gpt-4 goes 34.0% to 48.5%; gpt-3.5-turbo 34.5% to 46.5%. With gpt-3.5-turbo, ReAct beats Try Again on SQL (47.3% to 58.7%) with fewer turns (5.30 vs 7.25) and lower error rate (6.94% vs 46.4%), but is much worse on Bash (20.5% vs 46.5%). Success rate plateaus as turns accumulate.

The interactive gain is measured against a matched single-turn baseline on the same instances, but not against equal token budget, and failure analysis is manual trajectory inspection rather than controlled ablation.

Boundaries: 3 languages, adapted static datasets, and the CTF extension is described as 100 picoCTF objectives while the limitations section says 4 task instances.

Use this if you want evidence that feedback loops, not prompt structure, carry most of the gain.

Written from the abstract by claude-opus-5 on 2026-09-15, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

No claims cite this yet.