nl-to-pddl-problem-generation-scored-operational-acceptance-parses-conforms-domain CLM-0459When NL-to-PDDL problem generation is scored by operational acceptance (parses, conforms to domain, a planner finds a plan, VAL validates it, an LLM critic accepts), the score overstates faithfulness to the described task, and the gap widens for PDDL 2.1 temporal and numeric problems where operationally accepted instances can almost never reconstruct the curated reference encoding.
Capability: Whether the measurement made the finding
Observed on
gpt-4.1-mini generating problem.pddl given a correct domain file, up to three attempts, on Planetarium, AutoPlanBench and six curated IPC-2002-derived PDDL 2.1 variants; reference .
Sources
- Validated by: Measured on 60-instance runs per setting: on PDDL 2.1, final operational success 0.500 (baseline) and 0.733 (few-shot) against final structural match 0.000 and 0.067, with 0.500 and 0.667 of examples operationally accepted but structurally unmatched; on Planetarium+FS, 0.583 operational versus 0.417 semantic and structural. A manual audit of 24 outputs found the LLM critic agreed with human faithfulness judgment in 13/24 cases.. Measured across three benchmark families with one main model (gpt-4.1-mini; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Translating natural-language planning problems into PDDL can be learned from solver feedback alone, with one model acting as actor, judge and editor: on PlanBench average success rose from 35.5% (LLM+P) to 70.8%, with 66.3% of successes judged faithful to the described task and semantic drift, where a solvable but wrong problem is produced, cut to 6.4%.Planning a sequence of actions before acting · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- On 102 IPC Blocksworld instances with a 180-second budget, a classical planner (Fast Downward) solved 85%, a model planning directly solved 64%, and the same model choosing one action at a time against a PDDL simulator with observe, reset and retry solved 67%, a consistent but small three-point gain at about 5.7 times the token cost, so interactive simulation helps a little and classical search still wins where a formal model exists.Planning a sequence of actions before acting · reviewed by AI
- When a language model must judge whether a specific vehicle can perform a described mission, it is overconfident about partially-capable configurations — declaring the mission fully possible despite missing hardware — and supplying the required-capability list plus the vehicle's inferred capabilities from an ontology is what corrects that specific error, not the harder fully/impossible cases which the models already get mostly right.Planning a sequence of actions before acting · reviewed by AI
- On text-to-SQL benchmarks, aggregate accuracy hides a steep difficulty gradient: pre-trained-language-model-based parsers resolved roughly 80% of Spider cases overall but only about 50% of its extra-hard cases, and as natural-language understanding improved the residual errors shifted toward database-side problems — large schemas, ambiguous columns, dirty cell values — rather than language parsing.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A setting where operationally accepted PDDL problems match the reference encoding (or a semantics-preserving equivalent, checked by plan transfer) at essentially the same rate as operational acceptance, including on temporal and numeric PDDL 2.1 domains.