language-model-must-judge-whether-specific-vehicle-perform-described CLM-0409When a language model must judge whether a specific vehicle can perform a described mission, it is overconfident about partially-capable configurations — declaring the mission fully possible despite missing hardware — and supplying the required-capability list plus the vehicle's inferred capabilities from an ontology is what corrects that specific error, not the harder fully/impossible cases which the models already get mostly right.
Capability: Planning a sequence of actions before acting
Observed on
Locally hosted Llama3, Qwen2.5 and GPT-OSS judging one inspection/anode-test mission description against three simulated UUV configurations, 5 runs per condition..
Sources
- Validated by: Feasibility-verdict accuracy over 5 runs per vehicle configuration per model, compared against the same prompt with the ontology requirements removed; with requirements GPT-OSS reached 100% on all three configurations, and without them the models mislabelled the partially-possible configuration and named the wrong missing component.. Measured but very thin: one mission description, three vehicle configurations, 5 runs each, three small models, simulation only. The ablation does isolate the ontology requirements from the rest of the prompt. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
- On code prompts that are unsatisfiable by design, open-weight code and reasoning models refuse based on how suspicious the request looks rather than how deeply impossible it is: they refuse famous theoretical impossibilities far more often than requests for plausible-sounding nonexistent packages, and the stronger models in the set close the gap on theory while barely improving on fabricated ecosystem entities.Stating false facts confidently · reviewed by AI
- For LLM agents under a fixed ReAct harness answering questions over executable database-backed APIs plus document retrieval, accuracy falls sharply as the required reasoning depth grows from 1 to 5 hops (most models lose over half their accuracy), and the surviving failures concentrate in language-mediated steps — entity disambiguation, cross-source grounding, schema alignment — rather than in emitting syntactically valid tool calls.Carrying a chain of inference through to its conclusion · reviewed by AI
- On 2,154 expert-curated multiple-choice questions across eight architectural categories, 11 current models all score above 90% zero-shot (best about 94%), with quality-attribute and pattern questions near ceiling, but accuracy varies by category and drops for smaller models on the harder tiers (to about 85-87%), so declarative architectural knowledge is largely present while the benchmark cannot show whether it is applied in design.Making design decisions that hold up as a system grows · reviewed by AI
- For a 7B code model fine-tuned on competitive-programming tasks, training it to emit a fixed-schema requirement specification (goals, input/output constraints, examples, edge cases, dependencies) before the code raises Pass@1 more than training it to emit free-form reasoning or a stepwise implementation plan of similar length, with the largest margin on an out-of-distribution benchmark.Generating and editing working code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Running the same feasibility judgements with and without ontology-supplied capability requirements and finding the partially-possible configuration no better with them, or finding the gain spread evenly across fully-possible and impossible cases rather than concentrated on partial ones.