pretrained-2d-vision-language-action-manipulation-policies-adding-3d-geometry-only CLM-0469For pretrained 2D vision-language-action manipulation policies, adding 3D geometry only on the perception side is not enough: conditioning the action denoiser directly on a raw 3D semantic point field — at the specific diffusion blocks where linear probes show end-effector-coordinate error rising — raises bimanual task success beyond perception-side 3D fusion alone, and encoding the point cloud with a PointNet++-style encoder instead of keeping raw points lowers success.
Capability: Turning plans into physical action
Observed on
Diffusion/flow action experts (RDT, GR00T-N1.5) on bimanual tabletop manipulation in RoboTwin 2.0 and a real humanoid; RGB-D head camera available; ~50-100 demos per task..
Sources
- Validated by: Success rate on 10 RoboTwin 2.0 tasks (100 trials each, clean and randomized) against pi-0, Spatial Forcing, PointVLA, 3D Diffusion Policy; ablations removing each strategy (78.8 -> 90.4 -> 92.0 -> 95.6 avg), layer-position ablation, PointNet++ encoder ablation, and six real-robot tasks (RDT 34.3 vs Bridge3D 49.0 avg).. Measured on 10 simulated tasks and 6 real tasks with two backbones and component ablations; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For vision-language-action models pretrained on multi-embodiment video, latent actions learned only from RGB frame pairs miss finger-level end-effector articulation, and adding a shared discrete latent over 3D end-effector point-cloud transitions improves fine-grained motion probing and downstream manipulation success, but only when the geometric latent is forced to encode motion rather than morphology (shared bimanual code, pair-consistent augmentation, bidirectional transitions).Turning plans into physical action · reviewed by AI
- When a pretrained vision-language-action policy is fine-tuned sequentially on a stream of real-robot manipulation tasks, the choice of action-normalization statistics is itself a determinant of continual-learning success: statistics that are task-derived, first-task-derived, or recomputed each stage produce forgetting or poor transfer, while statistics estimated once from a small task-independent calibration set covering the embodiment's motion range and then frozen give the best acquisition, retention and task-order stability.Turning plans into physical action · reviewed by AI
- When adapting a pretrained VLA manipulation policy with only a handful of target demonstrations, retrieving prior demonstrations at the level of LLM-decomposed subtasks (then reranking by behavior-feature similarity) raises success more than retrieval by visual, optical-flow, or state-action similarity, and the margin is concentrated on long-horizon composite tasks rather than single-skill pick-and-place.Turning plans into physical action · reviewed by AI
- When VLMs must operate unfamiliar household devices with no manual, through multi-turn interaction over rendered multi-view images with click/rotate actions at pixel coordinates, the limiting failure is fine-grained localization of the interactive component within the correct view rather than navigating to that view, and almost no task is completed in the optimal number of steps.Turning plans into physical action · reviewed by AI
- In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A replication where inserting the explicit point-field cross-attention adds nothing over the implicit perception-side fusion alone, or where a PointNet++-encoded point cloud matches or beats raw point-wise conditioning, or where placement at arbitrary diffusion blocks works as well as probe-selected ones.