code-prompts-unsatisfiable-design-open-weight-code-reasoning-models-refuseOn code prompts that are unsatisfiable by design, open-weight code and reasoning models refuse based on how suspicious the request looks rather than how deeply impossible it is: they refuse famous theoretical impossibilities far more often than requests for plausible-sounding nonexistent packages, and the stronger models in the set close the gap on theory while barely improving on fabricated ecosystem entities.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Stating false facts confidently
Observed on
Twelve open-weight code/reasoning models (7B-32B), 270 adversarial unsatisfiable prompts across six languages plus 91 matched solvable controls, temperature 0, graded by a two-tier.
Sources
- supportsRefusing the Impossible: A Taxonomy and Benchmark for Code Hallucination in Large Language ModelsMeasured: aggregate hallucination rate 0.60 and refusal 0.27 on adversarial prompts, 0% over-refusal on controls. Subcategory gradient from 0.98 (nonexistent npm) down to 0.08 (modified classics). Top-4 vs bottom-4 models: theory subcategories 0.23 vs 0.82, plausible-entity 0.87 vs 0.98. Prompt identity explained more outcome variance than model identity. Only open-weight models tested; no frontier commercial models, so the capability trend is extrapolated within a narrow band. Framing and per-ecosystem comparisons are observational, not fully crossed.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- When hardware description code (Verilog, SystemVerilog, VHDL, Amaranth-Python) is generated from a functional specification that omits security obligations — as real SoC docs do, keeping countermeasures in separate config files — frontier models pass functional tests far more often than security tests, and the model with the highest functional pass rate is not the safest; naming the relevant CWE in the prompt raises the security pass rate sharply, showing the limit is missing awareness of the required obligation rather than inability to write defensive RTL.Writing secure code and dependencies · unreviewed
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · unreviewed
- General-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Stating false facts confidently · unreviewed
- For locally deployed open-weight models driving a stateful, dependency-ordered MCP tool server, cutting tool descriptions from full specifications (purpose, parameter semantics, constraints, failure conditions) down to one sentence each raises the fraction of calls the server rejects for every model tested, while its effect on task coverage is less consistent.Using the tools it is given · unreviewed
- For tool-using agents, enforcing injection defense as a deterministic provenance check on sensitive tool parameters — with the only model call reading the trusted user request and never fetched content — makes admission decisions invariant to how an injection is worded, whereas defenses that judge the agent's runtime plan or behavior with a model can be steered by reworded injections.Following instructions hidden in data · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A model set where refusal rates on plausible fabricated package prompts equal or exceed those on abstract theoretical-impossibility prompts, or where higher-capability models improve on both classes at similar rates. Proposed technique, not catalogued: Unsatisfiable-task refusal benchmark with matched solvable controls.