embodied-control CAP-0011 · active

Turning plans into physical action

Closing the loop from perception to action in a body or simulator, where mistakes have physical consequences and cannot be retried freely.

Tags: agentic, reasoning

Distinct from tool-use, where a tool call either succeeds or returns an error. Embodied control adds continuous state, partial observability, and irreversibility — a knocked-over object stays knocked over. Covers robotic manipulation, navigation, and vision-language-action models.

What counts as this capability

Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.

In scope when a model's output drives actuation or navigation and success is judged by an achieved physical or simulated world state. NOT in scope: agents that only call software tools (that is tool-use), and not planning papers where the plan is never executed against an environment.

Claims

Techniques

  • Add 3D end-effector point-cloud latent actions to image-basetraining
    Learns a second discrete latent action code from start-goal end-effector point-cloud transitions, shared across embodiments via a unified bimanual code, pair-consistent random 3D transforms, and forward/backward transition training, and use
  • Advance progress memory only on verified achievementprocess
    Represents the task as a subgoal sequence with a progress pointer, triggers verification at gripper-load events, and uses frozen point tracking (grasp) and language-conditioned cross-view embeddings plus a 2.43M-parameter head (placement) t
  • Allocate LoRA rank per network region by a pre-training diagtraining
    From ten unlabeled target observations, score each functional region of a VLA by combining CKA against a cached source reference with reference-free gradient and MC-dropout sensitivity, then set per-region LoRA rank proportional to the scor
  • Collect demonstrations only where the policy disagrees, and training
    During handheld demonstration, query the current policy on the same observation stream without executing it, record extra demonstration segments when an energy score between the human action chunk and the policy's sampled chunks exceeds a t
  • Condition action denoising on a raw 3D semantic point field architecture
    Fuse frozen 3D-foundation-model features into VLM visual tokens, and let action hidden states cross-attend (with 3D RoPE) to an unencoded point-wise semantic field, inserted only in the diffusion blocks where linear probing shows degraded e
  • Distill stage-transition monitoring via student self-explanatraining
    A large teacher VLM produces layered reasoning traces linking observations and task plan to annotated subtask transitions; a small student rewrites those traces into compact self-explanations, teacher-judged candidates are selected, and the
  • Freeze normalization statistics from a pre-stream calibratiotraining
    Before continual adaptation, collect a handful of task-independent teleoperated trajectories covering the embodiment's anticipated motion range, compute per-dimension 1st/99th percentile action statistics once, and use those frozen statisti
  • Imitate belief-module demonstrations at the action leveltraining
    Fine-tunes a VLM on per-timestep motor and expression actions recorded from a hybrid architecture with an explicit theory-of-mind module.
  • Make arrival the model's own declared action, verified afterprocess
    Instead of ending an episode by an external distance threshold, the agent exposes a 'finished' action the model emits, keeps flying, and a verifier later checks whether any declaration was within the success radius with the target visible.
  • Merge capability specialists, then distil on student prefixetraining
    Train one RL specialist per execution capability from a shared backbone, merge their task vectors with TIES, then refine the merged student by distilling from the routed specialist teacher on the student's own generated prefixes.
  • Point first, then emit quantized trajectory tokensarchitecture
    Extends the VLM vocabulary with image-grid affordance/object pointing tokens and three-level residual vector-quantized action tokens, so spatial intent and precise trajectories are both decoded by the original language-model head.
  • Re-check the safety constraint in a tool, not the prompttooling
    Ground objects as bounding boxes, have the agent propose a waypoint route, reject and replan any route that intersects the obstacle box (before and during motion), and pick the contact side by an adjacency test with the obstacle.
  • Report subtask progress and trajectory diagnostics, not justprocess
    Score embodied navigation episodes on fraction of phases completed, final goal distance, collisions and step count alongside binary success, so partial progress and post-phase-transition stalls are visible.
  • Retrieve demonstrations per decomposed subtask, not per tasktraining
    Cluster the prior dataset into skills, have an LLM decompose the target task into subtasks scored by semantic plausibility and per-skill held-out BC loss, retrieve episodes by subtask language similarity, rerank frames by VAE behavior-featu
  • Role-conditioned per-vehicle skill subsets with peer messagiprocess
    A harness parses the mission, assigns each UAV a role, exposes only the top-ranked skills for that role to a vehicle-bound agent, and lets agents exchange directed peer messages while a commander holds no flight authority.
  • Schema-validated tool calls plus verify-then-act motion chectooling
    Restrict the LLM to typed tool calls carrying only semantic object references and bounded skill parameters, bind each accepted call to a fixed motion-planning template, and execute only trajectories that pass kinematic and collision checks
  • Score recovery from the model's own real failure statesprocess
    Build evaluation scenarios by replaying naturally occurring failure states from policy rollouts and measuring whether the policy still completes the original task, graded by how much state restoration the failure demands.
  • Slow chunk proposal plus fast RL-trained action edittraining
    A large VLA asynchronously samples candidate action chunks from an older observation, then a lightweight edit policy conditioned on the latest observation perturbs each candidate and a learned Q-function picks the best chunk to execute.
  • Stage ego-to-robot transfer through an aligned-demo curricultraining
    Pretrain on egocentric data, then fine-tune on a small set of task-matched human and robot demonstrations converted to a shared relative end-effector representation, then pretrain on robot data before task post-training.
  • Store MoE gating profiles as retrievable skill memoryarchitecture
    Partition demonstrations into latent skills using Mixture-of-Experts gating coefficients, store each segment's mean latent feature as a key and its gating-weight sequence as a value, and at inference retrieve the top-N similar entries and a
  • Train a world-model controller on VLM-relabelled rollouts, tarchitecture
    Relabel segments of a world-model policy's own on-policy rollouts with VLM-generated instructions, add a behaviour-cloning plus instruction-completion loss to the Dreamer objectives, and at inference let a frozen VLM planner emit sparse tex
  • Validate a whole perturbation region, don't sample itprocess
    Maps a bounded brightness/rotation perturbation region onto a Hilbert curve and searches it by Hölder optimisation with a margin-aware convergence rule, so each instance returns either a concrete counterexample or a bound covering every per
  • Validate navigation proposals against evidence and geometry process
    A harness layer intercepts the MLLM planner's tool calls, checks each target for supporting observation provenance, geometric reachability and consistency with the active subgoal, and blocks stop requests until semantic, geometric and progr
  • Write memory frames on a fixed chunk schedule, not on eventsarchitecture
    Append one observation frame to the policy's visual memory at every action-chunk boundary instead of triggering writes on predicted subtask changes or motion thresholds.

Sources

Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.

Related: Using the tools it is given, Tracking state through a long task
Suggest a change

Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.