code-generation-prompt-combines-prompt-enforced-output-format-json-or-xmlWhen a code-generation prompt combines a prompt-enforced output format (JSON or XML), a persona, and urgency framing at once, pass@1 on function-level Python problems can fall well below the sum of each constraint's individual effect, even when each constraint alone is neutral or helpful — observed in the GPT-4o family and absent in the GPT-4.1 family and o3-mini.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Generating and editing working code
Observed on
164 HumanEval+ problems, greedy decoding, five OpenAI models; format enforced by prompt instruction rather than API-level constrained decoding; prompt length confounded with constr.
Sources
- 22,140 evaluations across 27 conditions and 5 models; interaction computed as residual against an additive main-effect prediction, with McNemar paired tests (e.g., OR 3.67, p=0.004 for best single vs worst triple on GPT-4o-mini). Single benchmark, single vendor, one sample per condition; token-length confound acknowledged by the authors and not isolated.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- On scientific tasks where a domain-specific constraint and a general presentation constraint (letter case, output format, structure) are both stated in the same prompt, multimodal models satisfy the domain-specific constraint more often than the general formatting one, so domain competence does not imply full instruction compliance.Following an unfamiliar procedure · unreviewed
- For data-science Python tasks where several correct solutions differ by orders of magnitude in runtime, a model's correctness rate does not predict the efficiency of the code it writes: rankings invert between the two, code-specialized open models are less efficient than general open models of similar size class, and the fastest model rotates by library (NumPy vs Pandas vs PyTorch vs Polars).Generating and editing working code · unreviewed
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · unreviewed
- Handing an agent context-editing tools (delete, summarize, note, fold) by prompt alone makes mid-size models worse than leaving their context untouched — Qwen3-8B, Qwen3-14B and Gemma4-E4B all scored lower on long-context QA with the tools than without them; only after fine-tuning on trajectories that use the tools does the toolset become a gain.Keeping its own context clean · unreviewed
- On code prompts that are unsatisfiable by design, open-weight code and reasoning models refuse based on how suspicious the request looks rather than how deeply impossible it is: they refuse famous theoretical impossibilities far more often than requests for plausible-sounding nonexistent packages, and the stronger models in the set close the gap on theory while barely improving on fabricated ecosystem entities.Stating false facts confidently · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A factorial replication on other models and benchmarks where triple-constraint pass@1 matches the additive prediction within noise, or where the GPT-4o-family gaps vanish once prompt-token length is held constant. Proposed technique, not catalogued: Factorial compound-prompt reliability testing.