arxiv-2609-03156 · paper

Compound Prompt Constraints in LLM Code Generation: A Factorial Study of Format, Persona, and Urgency

Created: 2026-09-02 · Ingested: 2026-09-09

Large language models (LLMs) are increasingly used in software engineering pipelines for code generation, where production prompts often combine multiple constraints. This paper presents a full-factorial empirical study of how output formatting, persona assignment, and urgency framing jointly affect LLM code-generation reliability. We evaluate all 27 combinations in a controlled 3x3x3 design and decompose each compound condition into an additive prediction and a residual interaction term that captures super-additive degradation. The study uses all 164 HumanEval+ problems across five OpenAI mod

In brief

Prompt constraints that each help or do nothing in isolation can combine to cut code generation accuracy well below the sum of their parts, and whether they do depends on model family rather than size.

A 3 x 3 x 3 full-factorial design varied output format (none/JSON/XML), persona (none/generic/expert senior Python developer), and urgency framing (none/moderate/extreme job-loss emergency) across the 164 problems of HumanEval+ and 5 OpenAI models: GPT-4o-mini, GPT-4o, GPT-4.1-mini, GPT-4.1, o3-mini. That is 22,140 greedy-decoded evaluations, with a format-aware extraction pipeline separating parsing failures from reasoning failures.

GPT-4o-mini showed super-additive degradation in 8/8 triple combinations, average -7.6 pp, worst -12.2 pp (JSON+expert+moderate); GPT-4o averaged -4.2 pp, 5/8. JSON combinations averaged -10.7 pp versus -4.6 pp for XML. GPT-4.1-mini (+1.7 pp, 0/8) and GPT-4.1 (+1.3 pp, 2/8) showed no pattern. o3-mini rose from a 59.8% baseline to 88.6% with JSON.

The interaction is measured, not argued: effects are residuals against an additive prediction, and McNemar tests on paired per-problem outcomes give OR = 3.67 (95% CI: [1.49, 9.04], p = 0.004) for GPT-4o-mini best-single versus worst-triple, with no significant pairwise result for GPT-4.1-mini. Prompt token length rises with constraint count and is not controlled.

Scope: 5 OpenAI models, 1 benchmark of function-level Python, greedy decoding, no non-OpenAI or open-weight models; the additive framework does not apply to o3-mini due to ceiling effects.

If you evaluate prompt elements one at a time, test the full production prompt instead.

Written from the abstract by claude-opus-5 on 2026-09-09, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsWhen a code-generation prompt combines a prompt-enforced output format (JSON or XML), a persona, and urgency framing at once, pass@1 on function-level Python problems can fall well below the sum of each constraint's individual effect, even when each constraint alone is neutral or helpful — observed in the GPT-4o family and absent in the GPT-4.1 family and o3-mini.