two-agent-embodied-social-tasks-where-one-agent-must-guide CLM-0502observationsingle paperReviewed by AI
In two-agent embodied social tasks where one agent must guide or mislead another through movement, gaze and facial expression, open-source VLMs up to 15B show no reliable behavioral change when an explicit theory-of-mind order constraint is imposed, failing at two separate points: most do not produce directionally coherent nonverbal signals, and those that do (e.g. a 14B model with near-ceiling trajectory and gaze alignment) are not decoded by the partner agent, whose choices track a fixed spatial preference instead.
Capability: Turning plans into physical action
Observed on
Open-weight VLMs at most 15B parameters, one hidden-reward guide/mislead scenario, 200 trials per model; no human baseline, no closed or larger models tested..
Sources
- Measured across 13 models (11 VLMs) and 200 trials each on a purpose-built Unity benchmark, with trial-level outcome conformance plus trajectory, gaze, expressivity and signal-sensitivity metrics, and auxiliary positional-bias and uncertainty diagnostics that separate signal-responsive behavior from fixed preferences. Only a hand-structured reference architecture with an explicit ToM module cleared both bottlenecks, which argues the task is feasible but is not a like-for-like comparator. Single scenario, no human baseline, size-capped model set.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In two-agent embodied social tasks requiring coordinated gaze, movement and expression, fine-tuning open VLMs (8B-14B InternVL) by timestep-level action imitation on demonstrations from an architecture with an explicit belief module shifts trial outcome scores up or down depending on condition but does not make the student's nonverbal signals more directionally coherent, because cross-channel communicative coordination is a trial-level property that per-timestep supervision does not encode.Turning plans into physical action · reviewed by AI
- In hierarchical LLM agents acting in gridworld cooperation tasks (Overcooked layouts) with unseen partners, supplying an accurate partner-skill estimate as planner prompt context does not by itself stop the agent from continuing a stale, non-complementary skill; the estimate only changes behaviour when it is wired into an explicit interrupt rule that aborts the in-progress skill when new partner actions are unlikely under the committed estimate.Turning plans into physical action · reviewed by AI
- In 4-7B open vision-language models (LLaVA-1.5-7B, Qwen2.5-VL-7B, InternVL-2.5-4B), how the cross-modal attention distribution over visual tokens shifts between adjacent decoder layers separates hallucinated from grounded object predictions better than the model's sensitivity to masking the attended region, and better than attention drift measured against a fixed first or last layer anchor.Grounding answers in the image · reviewed by AI
- In embodied multi-agent household planning with a fixed high-level skill interface, a team's task success rate overstates coordination quality, because goal predicates can be reached through illegal shortcuts: several of 11 tested MLLMs succeed on more episodes than they produce legal plans on (Claude Haiku 4.5 63.1% success vs 47.2% legal; Qwen3.6-Plus 69.1% vs 56.7%), while others honour constraints yet miss the goal.Getting more from several models than from one · reviewed by AI
- In simulated networks of LLM agents where one or two agents are instructed to advocate a position, the messages other agents write understate how far their elicited stance has moved: they mirror the persuader's rhetorical framing in most messages while explicit stance-change language is rare, so monitoring only generated text misses belief movement that a repeated token-probability probe detects.Getting more from several models than from one · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Showing an open VLM in this size range whose trajectory/gaze signals invert between cooperative and competitive-ToM1 conditions and whose partner agent's choices follow those signals (high positive then negative signal sensitivity) at low positional bias.