prompting-task-method-knowledge-model-which-spells-out-what-do-how CLM-0320observationsingle paperReviewed by AI
Prompting with a Task-Method-Knowledge model, which spells out what to do, how, and why in a hierarchical structure, lifted o1 on the obfuscated Random Blocksworld set of PlanBench from about 32% to 97%, and o1-mini from about 9% to 27%, gains far beyond the roughly 1% that zero-shot chain-of-thought gave on PlanBench in earlier work; the smaller model did not benefit on the Mystery variant.
Capability: Planning a sequence of actions before acting
Observed on
PlanBench Blocksworld variants (standard, Mystery, Random), o1 and o1-mini among others, 2026; prompt-only intervention..
Model cards: o1
Sources
- Validated by: Plan validity on PlanBench Blocksworld variants with and without TMK across several models; gains up to 65.8 points on Random Blocksworld for o1; no gain for o1-mini on Mystery.. Large effect on one benchmark family; the TMK model is hand-authored per domain, which is the cost the paper does not price.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Prompting a model to behave as if its knowledge ended at an earlier date works for facts asked about directly (about 82% unlearning success) and for word-sense shifts (about 70%), but fails when the post-cutoff knowledge is only causally connected to the question (about 19%), so a simulated cutoff cannot be trusted to remove contamination from temporal prediction tests.Recalling facts it was trained on · reviewed by AI
- Translating natural-language planning problems into PDDL can be learned from solver feedback alone, with one model acting as actor, judge and editor: on PlanBench average success rose from 35.5% (LLM+P) to 70.8%, with 66.3% of successes judged faithful to the described task and semantic drift, where a solvable but wrong problem is produced, cut to 6.4%.Planning a sequence of actions before acting · reviewed by AI
- On 102 IPC Blocksworld instances with a 180-second budget, a classical planner (Fast Downward) solved 85%, a model planning directly solved 64%, and the same model choosing one action at a time against a PDDL simulator with observe, reset and retry solved 67%, a consistent but small three-point gain at about 5.7 times the token cost, so interactive simulation helps a little and classical search still wins where a formal model exists.Planning a sequence of actions before acting · reviewed by AI
- On SelfAware, a set of unanswerable questions (no scientific consensus, imagination, philosophical, too many variables, subjective) paired with answerable ones, models' ability to flag the unanswerable ones rises with parameter count, with instruction tuning and with in-context examples, yet the best model tested (GPT-4, F1 about 75) stays well below the human reference of about 85, so self-knowledge is present but incomplete.Saying it does not know · reviewed by AI
- Instruction tuning that forces an answer to every question teaches models to fabricate when the question lies outside their parametric knowledge; splitting the tuning data by whether the base model already answers correctly and appending "I am unsure" to the rest (refusal-aware tuning) raises average precision on known-versus-unknown separation over vanilla tuning, transfers to out-of-domain datasets, and yields better calibrated confidence than post-hoc uncertainty scoring.Saying it does not know · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Domains where writing the TMK model is as hard as solving the task, or where the gain vanishes for reasoning models released after o1.