code-generation-prompt-combines-prompt-enforced-output-format-json-or-xml CLM-0031When a code-generation prompt combines a prompt-enforced output format (JSON or XML), a persona, and urgency framing at once, pass@1 on function-level Python problems can fall well below the sum of each constraint's individual effect, even when each constraint alone is neutral or helpful — observed in the GPT-4o family and absent in the GPT-4.1 family and o3-mini.
Capability: Generating and editing working code
Observed on
164 HumanEval+ problems, greedy decoding, five OpenAI models; format enforced by prompt instruction rather than API-level constrained decoding; prompt length confounded with constr.
Sources
- 22,140 evaluations across 27 conditions and 5 models; interaction computed as residual against an additive main-effect prediction, with McNemar paired tests (e.g., OR 3.67, p=0.004 for best single vs worst triple on GPT-4o-mini). Single benchmark, single vendor, one sample per condition; token-length confound acknowledged by the authors and not isolated.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Following an unfamiliar procedure · reviewed by AI
- When a prompt imposes k deterministically verifiable constraints at once, per-constraint pass rate declines only gently with k while the probability of satisfying all k collapses multiplicatively, because constraint failures within a response are nearly independent rather than pairwise-interfering — so joint reliability breaks down beyond roughly 5-6 simultaneous constraints and cannot be rescued by choosing or ordering which constraints are combined.Following an unfamiliar procedure · reviewed by AI
- When an agent builds a whole Python repository from natural-language requirements alone, refining component boundaries with explicit cohesion and coupling thresholds until no split or merge is triggered produces higher functionality coverage and test pass rate than letting the LLM decide restructuring actions for a fixed budget of rounds, because unconstrained LLM-decided actions keep over-decomposing past the point of a coherent modular structure.Generating and editing working code · reviewed by AI
- On natural-language logical reasoning benchmarks, auto-formalizing each reasoning step into SMT-LIB and regenerating from the first step the solver cannot verify raises both the share of solver-verified steps and final-answer accuracy over plain chain-of-thought, and greedy regeneration at the failure point beats beam search or MCTS on step verifiability.Carrying a chain of inference through to its conclusion · reviewed by AI
- Among 6,540 code comments referencing ChatGPT, Copilot, Claude or Gemini in public Python and JavaScript repositories (2022-2025), 81 also admit technical debt, most often postponed testing of the generated code, incomplete adaptation to the codebase, and limited understanding of what the code does, with a lower share of design-related debt than in ordinary self-admitted debt.Making design decisions that hold up as a system grows · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A factorial replication on other models and benchmarks where triple-constraint pass@1 matches the additive prediction within noise, or where the GPT-4o-family gaps vanish once prompt-token length is held constant. Proposed technique, not catalogued: Factorial compound-prompt reliability testing.