vision-language-action-world-action-manipulation-policies-evaluated-libero-style-tasks-near-ceiling-one-shot CLM-0230For vision-language-action and world-action manipulation policies evaluated on LIBERO-style tasks, near-ceiling one-shot task success does not predict the ability to resume a task from a failure state that arose during the model's own execution: all six tested policies lost over half their success rate on recovery scenarios, rankings inverted between the two settings, and recovery success fell steeply as the failure altered object or environment state rather than just the last action.
Capability: Turning plans into physical action
Observed on
Simulated tabletop manipulation (LIBERO suites), six VLA/WAM policies, failure states harvested from real rollouts and labelled into four recovery levels by an LLM judge..
Sources
- Measured: 2178 recovery scenarios from 130 subtasks, 10 trials each, six policies; recovery success rates per level reported (e.g. 0.0% at L4 on LIBERO-100 for most models). Failure levels are assigned by an LLM from rendered video, so level difficulty is not independently validated, and results are simulation-only.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For VLA and world-action manipulation policies that score near-ceiling on standard LIBERO, recovery success after a real execution failure falls sharply as recovery requires deeper state reasoning \u2014 retrying or adapting the immediate action works far more often than restoring a displaced task object or clearing an environmental obstruction \u2014 and rankings on the standard benchmark do not predict recovery rankings.Turning plans into physical action · reviewed by AI
- When a pretrained vision-language-action policy is fine-tuned sequentially on a stream of real-robot manipulation tasks, the choice of action-normalization statistics is itself a determinant of continual-learning success: statistics that are task-derived, first-task-derived, or recomputed each stage produce forgetting or poor transfer, while statistics estimated once from a small task-independent calibration set covering the embodiment's motion range and then frozen give the best acquisition, retention and task-order stability.Turning plans into physical action · reviewed by AI
- For LLM agents that score their own actions during exploration in text-based games and then reuse that experience, the accuracy of those self-scores against the environment verifier does not predict later gains on held-out stricter configurations: block-level correlations between judgment agreement or calibration and the next checkpoint's score change sit near zero, so recognizing a good action is not the bottleneck — converting it into a transferable policy is.Improving its own scaffolding from failures · reviewed by AI
- When vision-language models are given an explicit abstention option plus a stated payoff for correct, wrong and declined answers, quadrupling the penalty for a wrong answer barely changes how often they answer, so their abstention rate is largely insensitive to the stated cost of error across text, synthetic video and natural video recall tasks.Saying it does not know · reviewed by AI
- When adapting a pretrained VLA manipulation policy with only a handful of target demonstrations, retrieving prior demonstrations at the level of LLM-decomposed subtasks (then reranking by behavior-feature similarity) raises success more than retrieval by visual, optical-flow, or state-action similarity, and the margin is concentrated on long-horizon composite tasks rather than single-skill pick-and-place.Turning plans into physical action · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A policy set whose standard LIBERO ranking preserves its ranking on recovery scenarios, with recovery success rates close to one-shot success and no drop from level-1 to level-4 failures. Proposed technique, not catalogued: Failure-recovery benchmark built from harvested execution failures.