llm-generated-multi-day-travel-itineraries-high-qualitative-coherence-scores-spatial CLM-0593For LLM-generated multi-day travel itineraries, high qualitative coherence scores (spatial, persona, meal, ordering) coexist with near-zero robustness to realistic disruption: across GPT-5, Qwen3, Mistral-7B and Phi-4-mini, transport-delay buffer scores stay under 12% and buffer-adequacy scores under ~41%, so coherence and static constraint satisfaction do not indicate that a plan survives empirical transit delays or crowd peaks.
Capability: Planning a sequence of actions before acting
Observed on
1,000 3/5/7-day India travel queries with human gold itineraries, empirical delay and crowd-density data, direct sole-planning prompting that already describes the uncertainty dist.
Sources
- Validated by: Applied to four LLMs and an agentic baseline over 3/5/7-day plans; the metrics separate models that binary final-pass rate collapses to near zero for, and expose model-specific patterns (Qwen3 degrading with horizon, Phi-4 strong crowd timing but weak buffering); no human-agreement study of the metrics is reported.. Measured on a new benchmark with the authors' own metrics, so the headline gap is partly a property of the metric's acceptable-buffer parameterisation; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On month-level egocentric recordings (20-120 days per participant) with human-written multiple-choice questions, current multimodal models degrade as the question moves from inferring recurring habits, to retrieving one specific episode, to composing evidence across days — the best model reached 71.8% macro-average against a 94.2% human baseline, with several models near or below the 25% chance level on route reasoning, cross-view spatial reasoning and direction judgement.Reasoning about time in video · reviewed by AI
- For LLM agents on dependent multi-step tasks, success decays geometrically with the number of steps under a single per-step reliability parameter that rises with model scale but stays below 1, so long horizons eventually collapse even for the strongest deployed systems — and the decay tracks step count, not context length, since bounding the context window steepens the decay rather than easing it.Completing tasks that take many steps or many hours · reviewed by AI
- In multi-agent LLM workflows with a mixed-size model zoo, choosing which model instantiates each agent step-by-step from an online estimate of task-progress gain plus remaining time and cost budget keeps task quality near the best single-model policy at lower energy than one-shot query-level routing or stall-triggered escalation.Getting more from several models than from one · reviewed by AI
- On cross-system urban tasks where several services must be chained under matched tools and budgets, the task-success gap between orchestration strategies is not explained by required-tool coverage, call ordering, or asking for missing inputs — those are near-saturated across strategies — but by whether the agent goes beyond the minimal toolchain to refine coarse results and verify each candidate against the stated constraints before answering.Using the tools it is given · reviewed by AI
- For LLM agents that score their own actions during exploration in text-based games and then reuse that experience, the accuracy of those self-scores against the environment verifier does not predict later gains on held-out stricter configurations: block-level correlations between judgment agreement or calibration and the next checkpoint's score change sit near zero, so recognizing a good action is not the bottleneck — converting it into a transferable policy is.Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A model (or prompting setup) that scores high on these qualitative coherence metrics and also allocates buffers within the empirical delay-derived ranges, or evidence that low BAS/TDAS plans in fact execute without cascading failure under simulated delays.