vision-language-action-models-pretrained-multi-embodiment-video-latent-actions-learned-only CLM-0505For vision-language-action models pretrained on multi-embodiment video, latent actions learned only from RGB frame pairs miss finger-level end-effector articulation, and adding a shared discrete latent over 3D end-effector point-cloud transitions improves fine-grained motion probing and downstream manipulation success, but only when the geometric latent is forced to encode motion rather than morphology (shared bimanual code, pair-consistent augmentation, bidirectional transitions).
Capability: Turning plans into physical action
Observed on
Manipulation policies co-trained across human hands, dexterous robot hands and parallel-jaw grippers, where end-effector geometry is recoverable from hand reconstruction or URDF fo.
Sources
- Validated by: Frozen-latent MLP probing of end-effector translation/rotation/finger angle error on a held-out XHand split; cross-embodiment grasp/hold/release retrieval (R@1 45.67 vs 33.37 without point clouds, 38.69 without the unification designs); RoboCasa GR-1 24-task success (55.7% vs 50.9% image-only, 54.4% without unification; 68.3% under multi-embodiment co-training); 4 real tasks, 50 trials each (75.5% vs 68.5% without unification, 71.5% HARP-VLA).. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For pretrained 2D vision-language-action manipulation policies, adding 3D geometry only on the perception side is not enough: conditioning the action denoiser directly on a raw 3D semantic point field — at the specific diffusion blocks where linear probes show end-effector-coordinate error rising — raises bimanual task success beyond perception-side 3D fusion alone, and encoding the point cloud with a PointNet++-style encoder instead of keeping raw points lowers success.Turning plans into physical action · reviewed by AI
- For post-training a pretrained vision-language-action manipulation policy under a fixed per-round frame budget, collecting handheld demonstrations only at states where the human action chunk diverges from the policy's sampled action distribution, and weighting them by a learned task-progress advantage, raises task progress more than collecting the same number of frames of ordinary demonstrations and fine-tuning on them.Turning plans into physical action · reviewed by AI
- When adapting a pretrained VLA manipulation policy with only a handful of target demonstrations, retrieving prior demonstrations at the level of LLM-decomposed subtasks (then reranking by behavior-feature similarity) raises success more than retrieval by visual, optical-flow, or state-action similarity, and the margin is concentrated on long-horizon composite tasks rather than single-skill pick-and-place.Turning plans into physical action · reviewed by AI
- When a pretrained vision-language-action policy is fine-tuned sequentially on a stream of real-robot manipulation tasks, the choice of action-normalization statistics is itself a determinant of continual-learning success: statistics that are task-derived, first-task-derived, or recomputed each stage produce forgetting or poor transfer, while statistics estimated once from a small task-independent calibration set covering the embodiment's motion range and then frozen give the best acquisition, retention and task-order stability.Turning plans into physical action · reviewed by AI
- For 7B multimodal models on long-video QA, a multi-turn agentic exploration harness (adaptive viewing speed, cross-turn textual notes, rewatch) lowers accuracy below plain uniform-frame single-pass answering unless the model is trained to use it, because the untrained model commits to an answer before gathering evidence; with cold-start SFT on verified exploration turns plus trajectory-level RL, the same harness beats the single-pass baseline.Reasoning about time in video · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An image-only latent action model matching or beating the geometry-augmented one on finger-articulation probing and downstream success under identical data and architecture, or the unification designs (shared bimanual code, augmentation, bidirectional training) showing no effect when ablated.