embodied-control CAP-0011 · activeTurning plans into physical action
Closing the loop from perception to action in a body or simulator, where mistakes have physical consequences and cannot be retried freely.
Tags: agentic, reasoning
Distinct from tool-use, where a tool call either succeeds or returns an error. Embodied control adds continuous state, partial observability, and irreversibility — a knocked-over object stays knocked over. Covers robotic manipulation, navigation, and vision-language-action models.
What counts as this capability
Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.
In scope when a model's output drives actuation or navigation and success is judged by an achieved physical or simulated world state. NOT in scope: agents that only call software tools (that is tool-use), and not planning papers where the plan is never executed against an environment.
Claims
- mechanismsingle paper↑1↓0When adapting a pretrained VLA manipulation policy with only a handful of target demonstrations, retrieving prior demonstrations at the level of LLM-decomposed subtasks (then reranking by behavior-feature similarity) raises success more than retrieval by visual, optical-flow, or state-action similarity, and the margin is concentrated on long-horizon composite tasks rather than single-skill pick-and-place.
- mechanismsingle paper↑1↓0For coding-agent robot manipulation where a safety constraint (do not touch a named obstacle) is given only as text in the task prompt, the agent restates the constraint early and then plans the shortest path through the obstacle, and enforcing the same constraint with an external stateless verifier that re-checks each route against grounded obstacle boxes before motion raises both task success and collision avoidance.
- observationsingle paper↑1↓0For deciding when a robot skill has finished and the next subtask should begin, general-purpose VLMs prompted zero-shot are near useless on long-horizon mobile manipulation (under 4% of transitions detected on BEHAVIOR-1K, including a frontier proprietary model), while a sub-1B student fine-tuned on compact self-explanations condensed from teacher reasoning over annotated demonstrations detects about 90% and lifts closed-loop task success from 0.30 to 0.53 against a 0.63 oracle.
- observationsingle paper↑1↓0When a drone's action space is declared only in the prompt and the model must itself declare arrival, small open multimodal models (2B-9B) enter the 5 m success radius as often as or more often than frontier models yet lose most episodes by declaring arrival too early or never declaring, so the gap at small scale is protocol adherence rather than navigation.
- mechanismsingle paper↑1↓0In embodied environments with high-frequency action spaces (Atari, Minecraft, Crafter, DMLab, Overcooked, Pico Park, MindCraft), pairing a frozen instruction-tuned VLM that emits sparse text instructions with a small world-model controller trained to follow them scores higher than either the same VLM emitting low-level actions directly or the same controller running without instructions, and the gain depends on the instructions carrying semantic content — random-string instructions score no better than no instructions at all.
- observationsingle paper↑1↓0For embodied vision-language models trained as separate capability specialists (spatial, temporal, action guidance, state verification) from one backbone, consolidating them by TIES weight merging and then routed multi-teacher on-policy distillation retains more of each specialist's gain than either step alone or than mixed-data RL, which shows capability-specific biases.
- mechanismsingle paper↑1↓0When fine-tuning a vision-language-action policy for a deployment shift, the parameter change needed concentrates in the functional region matching the shift's modality — appearance in the vision encoder, rephrased instructions in the language backbone, novel objects in the vision encoder plus action head — so allocating LoRA rank per region by a pre-training diagnostic beats uniform-rank LoRA at matched budget, with the advantage largest at small budgets and absent when the shift's demand spreads across regions rather than concentrating.
- mechanismsingle paper↑1↓0For frozen VLA policies executing repetitive, perceptually aliased manipulation tasks (counting pick-and-place), a progress memory that advances after each attempted action converts transient grasp or placement failures into persistent task-state errors, so writes must be gated on observed physical evidence of the intended transition; gating this way raised success far above the frozen policy and above memory-augmented baselines.
- mechanismsingle paper↑1↓0In hierarchical LLM agents acting in gridworld cooperation tasks (Overcooked layouts) with unseen partners, supplying an accurate partner-skill estimate as planner prompt context does not by itself stop the agent from continuing a stale, non-complementary skill; the estimate only changes behaviour when it is wired into an explicit interrupt rule that aborts the in-progress skill when new partner actions are unlikely under the committed estimate.
- observationsingle paper↑1↓0On language-conditioned manipulation tasks where the same base task is instantiated at increasing levels of spatial ambiguity and procedural length, current VLA models degrade on fine-grained spatial relations, precise low-level execution, and memory-dependent multi-step planning, so success measured only on predefined simple settings overstates their competence.
- observationsingle paper↑1↓0When language-conditioned VLA manipulation tasks are instantiated at graded difficulty levels that add spatial ambiguity (fine-grained spatial relations, precise placement) and procedural length (multi-stage plans requiring memory of completed steps), representative VLA models degrade on both axes, so success on predefined single-stage settings does not indicate spatial or long-horizon competence.
- observationsingle paper↑1↓0For LLM-driven multi-UAV swarm tasks in simulation, binding one agent to each vehicle with a role-conditioned subset of the skill catalog and directed peer messages yields higher episode returns than a central planner that sees pooled team state and the full skill list, and ablations attribute most of the gain to the skill-subset restriction and the peer reports rather than to the experience-based reward correction.
- observationsingle paper↑1↓0For LLM-driven robot manipulation where every tool call is schema-validated and every motion kinematically and collision-checked before execution, execution-time recovery mechanisms (retrying a failed plan with an alternative candidate, resetting to a collision-checked safe pose between stages) contribute more to end-to-end task success than canonical action sequencing, because most residual failures are transient planning or IK failures rather than wrong orderings.
- observationsingle paper↑1↓0For manipulation policies (diffusion-policy and VLA backbones) trained on few demonstrations, decomposing trajectories implicitly via a Mixture-of-Experts gate and storing the per-segment gating profiles as retrievable memory, then blending retrieved profiles into the current gating distribution at inference, raises success most on unseen object-receptacle recombinations rather than on in-distribution tasks.
- mechanismsingle paper↑1↓0For manipulation policies pretrained on mixed egocentric-human and robot data, adding egocentric data helps only when the embodiment gap is bridged explicitly: a three-stage curriculum (ego priors, then task-matched human/robot alignment in a shared end-effector frame, then robot-only) raised real-robot success, while separate ego/robot action heads gave only a marginal in-distribution gain and shared future-video prediction in a world-action model made success worse than a robot-only baseline.
- mechanismsingle paper↑1↓0For manipulation policies whose large VLA backbone cannot run within the control period, letting the slow model propose action chunks from a stale observation while a small RL-trained edit policy adjusts and Q-selects among those chunks using the latest observation recovers more performance than applying RL on top of asynchronous imitation-learning schemes alone, and the gap widens as inference delay or environment speed grows.
- mechanismsingle paper↑1↓0For marine vehicle navigation, an LLM asked only to emit an ordered macro-action skeleton produces syntactically and topologically usable plans but cannot set physically feasible step durations; letting a differentiable physics surrogate optimize the durations turns plans that collide on every mission into collision-free ones, while wrong skeleton topology remains unrecoverable because the optimizer only rescales timings.
- mechanismsingle paper↑1↓0For memory-augmented VLA manipulation policies that must actively move the viewpoint to find task-relevant evidence, writing one memory frame at each fixed action-chunk boundary generalizes better than event-triggered writing (on predicted subtask change or on gripper/head/torso motion), and extra memory capacity helps only under the regular schedule — event-triggered policies degrade further out of distribution because mistimed triggers omit informative views and control jitter fills memory with redundant frames.
- mechanismsingle paper↑1↓0For monocular VLM-based navigation policies, making the model first emit a two-token pointing prefix (one affordance point for free space, one object point for the goal) on an image-plane grid and then three residual vector-quantized action tokens decoded to 10 SE(2) waypoints lets one compact 4B backbone handle instruction following, object-goal search and target tracking without task-specific action heads or task identifier tokens.
- observationsingle paper↑1↓0When a multimodal LLM acts as the decision core of a simulated humanoid reacting to sudden household hazards, picking the correct reaction primitive does not mean the reaction is executable: even on correctly classified scenes, hands land a median 0.48 m from the true impact point, 89% of misses fall short, and in 49% of those decisions the body never steps forward and stands beyond arm's reach.
- mechanismsingle paper↑1↓0For post-training a pretrained vision-language-action manipulation policy under a fixed per-round frame budget, collecting handheld demonstrations only at states where the human action chunk diverges from the policy's sampled action distribution, and weighting them by a learned task-progress advantage, raises task progress more than collecting the same number of frames of ordinary demonstrations and fine-tuning on them.
- mechanismsingle paper↑1↓0For pretrained 2D vision-language-action manipulation policies, adding 3D geometry only on the perception side is not enough: conditioning the action denoiser directly on a raw 3D semantic point field — at the specific diffusion blocks where linear probes show end-effector-coordinate error rising — raises bimanual task success beyond perception-side 3D fusion alone, and encoding the point cloud with a PointNet++-style encoder instead of keeping raw points lowers success.
- mechanismsingle paper↑1↓0When a pretrained vision-language-action policy is fine-tuned sequentially on a stream of real-robot manipulation tasks, the choice of action-normalization statistics is itself a determinant of continual-learning success: statistics that are task-derived, first-task-derived, or recomputed each stage produce forgetting or poor transfer, while statistics estimated once from a small task-independent calibration set covering the embodiment's motion range and then frozen give the best acquisition, retention and task-order stability.
- mechanismsingle paper↑1↓0For real-world online RL post-training of large pretrained vision-language-action policies on high-precision manipulation, optimizing the action head toward a paired advantage margin against a frozen reference action, rather than maximizing absolute Q values, yields higher autonomous success on long multi-step tasks, because absolute-Q maximization amplifies early optimistic value errors into policy drift that behavior-cloning regularization alone does not stop.
- mechanismsingle paper↑1↓0In simulated urban air-ground object search where a VLM drives both a UAV and a UGV, wrapping the model in a fixed search-handoff-verify protocol that leaves geometric projection and road-graph path planning to deterministic tools raises task success for backbones with adequate perception while cutting decision calls for all, but does not lift weak small open models, whose success can fall on harder splits.
- mechanismsingle paper↑1↓0For training-free MLLM-driven navigation (instruction-following and object-goal) in simulated indoor environments, gating each planner-proposed action on evidential support, geometric reachability and subgoal consistency before dispatch — and refusing a stop request unless semantic, geometric and progress checks all pass — raises success rate over letting the planner act directly, but the stop check buys completion at some cost in path efficiency.
- mechanismsingle paper↑1↓0In two-agent embodied social tasks requiring coordinated gaze, movement and expression, fine-tuning open VLMs (8B-14B InternVL) by timestep-level action imitation on demonstrations from an architecture with an explicit belief module shifts trial outcome scores up or down depending on condition but does not make the student's nonverbal signals more directionally coherent, because cross-channel communicative coordination is a trial-level property that per-timestep supervision does not encode.
- observationsingle paper↑1↓0In two-agent embodied social tasks where one agent must guide or mislead another through movement, gaze and facial expression, open-source VLMs up to 15B show no reliable behavioral change when an explicit theory-of-mind order constraint is imposed, failing at two separate points: most do not produce directionally coherent nonverbal signals, and those that do (e.g. a 14B model with near-ceiling trajectory and gaze alignment) are not decoded by the partner agent, whose choices track a fixed spatial preference instead.
- mechanismsingle paper↑1↓0For vision-language-action models pretrained on multi-embodiment video, latent actions learned only from RGB frame pairs miss finger-level end-effector articulation, and adding a shared discrete latent over 3D end-effector point-cloud transitions improves fine-grained motion probing and downstream manipulation success, but only when the geometric latent is forced to encode motion rather than morphology (shared bimanual code, pair-consistent augmentation, bidirectional transitions).
- observationsingle paper↑1↓0For vision-language-action and world-action manipulation policies evaluated on LIBERO-style tasks, near-ceiling one-shot task success does not predict the ability to resume a task from a failure state that arose during the model's own execution: all six tested policies lost over half their success rate on recovery scenarios, rankings inverted between the two settings, and recovery success fell steeply as the failure altered object or environment state rather than just the last action.
- observationsingle paper↑1↓0For VLA manipulation policies doing single-step action prediction on LIBERO observations, the commanded action is fragile to camera rotation but tolerant of brightness: validated over entire continuous perturbation regions, 27% of instances admit a counterexample within ±1° of camera rotation and 51% within ±2°, while every instance is robust to a ±2% brightness shift and the median tolerates ±30%.
- observationsingle paper↑1↓0For VLA and world-action manipulation policies that score near-ceiling on standard LIBERO, recovery success after a real execution failure falls sharply as recovery requires deeper state reasoning \u2014 retrying or adapting the immediate action works far more often than restoring a displaced task object or clearing an environmental obstruction \u2014 and rankings on the standard benchmark do not predict recovery rankings.
- observationsingle paper↑1↓0For VLM agents doing multi-phase navigation and pick-and-place in procedurally generated indoor scenes with moving human obstacles, subtask progress runs well ahead of full success — agents reach the first goal and then stall or loop after the phase transition — so success rate alone hides where the failure is.
- mechanismsingle paper↑1↓0For VLM-based navigation agents, having the model output a 2D pixel on an egocentric image and projecting it to a 3D waypoint with depth plus a SLAM controller works better than having the same model regress metric 3D waypoints directly, because it keeps the action in the 2D image space the VLM was pretrained on instead of asking it to learn geometric transformations.
- mechanismsingle paper↑1↓0For VLM-driven embodied task planning with an egocentric camera, where hazards can fall out of view, the form of the memory handed to a risk predictor decides whether safety improves: an object-relation graph that retains out-of-view objects and updates edges after actions raises safe success rate well above no memory, while feeding the full image history or a flat list of seen objects scores below having no memory at all, because relationships and structure, not object recall, are what risk prediction needs.
- observationsingle paper↑1↓0For VLMs choosing an ordered route over numbered waypoints in a single egocentric image, per-waypoint feasibility and the first action are judged well while the rest of the route is not: pooled over nine models, roughly 88-96% of predictions take a legal first edge, but full-route legality falls to about 47% for a point agent and under 7% once the agent has a footprint, with most of the loss in intermediate edges rather than in output format.
- observationsingle paper↑1↓0When VLMs must operate unfamiliar household devices with no manual, through multi-turn interaction over rendered multi-view images with click/rotate actions at pixel coordinates, the limiting failure is fine-grained localization of the interactive component within the correct view rather than navigating to that view, and almost no task is completed in the optimal number of steps.
Techniques
- Add 3D end-effector point-cloud latent actions to image-basetrainingLearns a second discrete latent action code from start-goal end-effector point-cloud transitions, shared across embodiments via a unified bimanual code, pair-consistent random 3D transforms, and forward/backward transition training, and use
- Advance progress memory only on verified achievementprocessRepresents the task as a subgoal sequence with a progress pointer, triggers verification at gripper-load events, and uses frozen point tracking (grasp) and language-conditioned cross-view embeddings plus a 2.43M-parameter head (placement) t
- Allocate LoRA rank per network region by a pre-training diagtrainingFrom ten unlabeled target observations, score each functional region of a VLA by combining CKA against a cached source reference with reference-free gradient and MC-dropout sensitivity, then set per-region LoRA rank proportional to the scor
- Collect demonstrations only where the policy disagrees, and trainingDuring handheld demonstration, query the current policy on the same observation stream without executing it, record extra demonstration segments when an energy score between the human action chunk and the policy's sampled chunks exceeds a t
- Condition action denoising on a raw 3D semantic point field architectureFuse frozen 3D-foundation-model features into VLM visual tokens, and let action hidden states cross-attend (with 3D RoPE) to an unencoded point-wise semantic field, inserted only in the diffusion blocks where linear probing shows degraded e
- Distill stage-transition monitoring via student self-explanatrainingA large teacher VLM produces layered reasoning traces linking observations and task plan to annotated subtask transitions; a small student rewrites those traces into compact self-explanations, teacher-judged candidates are selected, and the
- Freeze normalization statistics from a pre-stream calibratiotrainingBefore continual adaptation, collect a handful of task-independent teleoperated trajectories covering the embodiment's anticipated motion range, compute per-dimension 1st/99th percentile action statistics once, and use those frozen statisti
- Imitate belief-module demonstrations at the action leveltrainingFine-tunes a VLM on per-timestep motor and expression actions recorded from a hybrid architecture with an explicit theory-of-mind module.
- Make arrival the model's own declared action, verified afterprocessInstead of ending an episode by an external distance threshold, the agent exposes a 'finished' action the model emits, keeps flying, and a verifier later checks whether any declaration was within the success radius with the target visible.
- Merge capability specialists, then distil on student prefixetrainingTrain one RL specialist per execution capability from a shared backbone, merge their task vectors with TIES, then refine the merged student by distilling from the routed specialist teacher on the student's own generated prefixes.
- Point first, then emit quantized trajectory tokensarchitectureExtends the VLM vocabulary with image-grid affordance/object pointing tokens and three-level residual vector-quantized action tokens, so spatial intent and precise trajectories are both decoded by the original language-model head.
- Re-check the safety constraint in a tool, not the prompttoolingGround objects as bounding boxes, have the agent propose a waypoint route, reject and replan any route that intersects the obstacle box (before and during motion), and pick the contact side by an adjacency test with the obstacle.
- Report subtask progress and trajectory diagnostics, not justprocessScore embodied navigation episodes on fraction of phases completed, final goal distance, collisions and step count alongside binary success, so partial progress and post-phase-transition stalls are visible.
- Retrieve demonstrations per decomposed subtask, not per tasktrainingCluster the prior dataset into skills, have an LLM decompose the target task into subtasks scored by semantic plausibility and per-skill held-out BC loss, retrieve episodes by subtask language similarity, rerank frames by VAE behavior-featu
- Role-conditioned per-vehicle skill subsets with peer messagiprocessA harness parses the mission, assigns each UAV a role, exposes only the top-ranked skills for that role to a vehicle-bound agent, and lets agents exchange directed peer messages while a commander holds no flight authority.
- Schema-validated tool calls plus verify-then-act motion chectoolingRestrict the LLM to typed tool calls carrying only semantic object references and bounded skill parameters, bind each accepted call to a fixed motion-planning template, and execute only trajectories that pass kinematic and collision checks
- Score recovery from the model's own real failure statesprocessBuild evaluation scenarios by replaying naturally occurring failure states from policy rollouts and measuring whether the policy still completes the original task, graded by how much state restoration the failure demands.
- Slow chunk proposal plus fast RL-trained action edittrainingA large VLA asynchronously samples candidate action chunks from an older observation, then a lightweight edit policy conditioned on the latest observation perturbs each candidate and a learned Q-function picks the best chunk to execute.
- Stage ego-to-robot transfer through an aligned-demo curricultrainingPretrain on egocentric data, then fine-tune on a small set of task-matched human and robot demonstrations converted to a shared relative end-effector representation, then pretrain on robot data before task post-training.
- Store MoE gating profiles as retrievable skill memoryarchitecturePartition demonstrations into latent skills using Mixture-of-Experts gating coefficients, store each segment's mean latent feature as a key and its gating-weight sequence as a value, and at inference retrieve the top-N similar entries and a
- Train a world-model controller on VLM-relabelled rollouts, tarchitectureRelabel segments of a world-model policy's own on-policy rollouts with VLM-generated instructions, add a behaviour-cloning plus instruction-completion loss to the Dreamer objectives, and at inference let a frozen VLM planner emit sparse tex
- Validate a whole perturbation region, don't sample itprocessMaps a bounded brightness/rotation perturbation region onto a Hilbert curve and searches it by Hölder optimisation with a margin-aware convergence rule, so each instance returns either a concrete counterexample or a bound covering every per
- Validate navigation proposals against evidence and geometry processA harness layer intercepts the MLLM planner's tool calls, checks each target for supporting observation provenance, geometric reachability and consistency with the active subgoal, and blocks stop requests until semantic, geometric and progr
- Write memory frames on a fixed chunk schedule, not on eventsarchitectureAppend one observation frame to the policy's visual memory at every action-chunk boundary instead of triggering writes on predicted subtask changes or motion thresholds.
Sources
Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- FAN: Foresight Action Normalization for Continual Adaptation of Vision-Language-Action Models2026-09-18↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- StageGuard: Learning Stage Transitions for Long-Horizon Robot Tasks via Agentic Distillation2026-09-17↑ supports a claim here
- ↑ supports a claim here
- Not All Layers Need Tuning: Diagnosing and Directing Adaptation in Vision-Language-Action Models2026-09-16↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- EgoPathBench: Evaluating Zero-Shot Egocentric Waypoint Decision-Making in Vision-Language Models2026-09-15↑ supports a claim here
- The Robot Data Factory2026-09-15about this capability · no claim drafted yet
- ↑ supports a claim here
- ↑ supports a claim here
- about this capability · no claim drafted yet
- about this capability · no claim drafted yet
- ↑ supports a claim here
- Towards Embodied Air-Ground Cooperative Object Search: Benchmark, Dataset and Agentic Method2026-09-08↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here · ↑ supports a claim here
- LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models2026-09-04↑ supports a claim here · ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- Bridging Semantics and Physics with Constrained LLMs for Safe and Trustworthy Robotic Manipulation2026-08-29↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Models2026-08-25↑ supports a claim here
- ↑ supports a claim here · ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.