arxiv-2211-10435 · paperPAL: Program-aided Language Models
Luyu Gao, Aman Madaan, Shuyan Zhou, et al.
Created: 2022 · Ingested: 2026-09-02
This source: PAL: Program-aided Language ModelsHow much the field cites it — very heavily cited in the last 12 months268 in the last 12 months · 824 totalpublished 2022checked 2026-09-04268 citations in the last 12 months · 824 total · checked 2026-09-04
https://arxiv.org/abs/2211.10435(opens in a new tab)In brief
Having a language model write Python and letting an interpreter run it beats having the model both decompose and compute. PAL prompts the LLM to emit interleaved natural-language comments and code as its reasoning chain, withholds final answers from in-context examples, and takes the interpreter's output as the answer.
Evaluation covers 13 tasks: 8 math word-problem sets (gsm8k, svamp, asdiv, mawps subsets), 3 BIG-Bench Hard symbolic tasks (Colored Objects, Penguins, Date), and 2 algorithmic tasks (Object Counting, Repeat Copy), with Codex code-davinci-002 as the backend and shared in-context examples with prior chain-of-thought work.
PAL reaches 72.0 on gsm8k versus 65.6 for CoT with Codex and 56.9 for PaLM-540b, 95.1 vs 86.3 on Colored Objects, 93.3 vs 79.2 on Penguins, 96.7 vs 73.0 on Object Counting. On gsm-hard, built by swapping in integers of up to 7 digits, CoT falls from 65.6% to 20.1% while PAL holds at 61.5%.
The cause is isolated reasonably well. Prompting the model to generate Python but self-execute it scores 23.2 on gsm8k, only 4.5 above Direct, so the interpreter, not the code-style prompt, carries the gain. Ablations show removing comments costs little but replacing meaningful variable names with random characters drops below CoT.
Boundaries: 1 model family plus text-davinci variants, where PAL loses to CoT at text-davinci-001; tasks are all programmatically expressible.
If your task has a deterministic solution procedure, spend prompt effort on code, not prose.
Written from the abstract by claude-opus-5 on 2026-09-05, with every figure checked against it. Not a substitute for the paper.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
- supportsWriting out reasoning steps improves how a problem is decomposed but does not fix the arithmetic inside a step — the model can decompose correctly and still get the calculation wrong, at the same confident register as when it is right. For exact arithmetic the fix is to move the computation out of the text, not to add more steps.
- supportsOffloading the computation to generated code and an interpreter removes most arithmetic errors on math word problems.
- supportsOffloading the calculation to generated code removes the arithmetic as a source of error but does not remove error — what remains is the translation from problem to program, which is now the only thing the model is doing. Verification has to move there with it.