long-cooking-videos-where-question-depends-ingredient-changes-no CLM-0208For long cooking videos where a question depends on ingredient changes no longer visible, vision-language models fail to carry observed events forward into later answers, and adding generic retrieved-context RAG makes it worse, while splitting memory into a temporally anchored event log plus an explicitly updated inventory ledger recovers much of the gap.
Capability: Tracking state through a long task
Observed on
Multiple-choice and numeric QA over 160 long cooking videos; Qwen3-VL 4B/8B backbones for the agent; proprietary and open VLMs as zero-shot baselines; observational only, no actuat.
Sources
- Single benchmark of 1,500 human-verified QA pairs; humans ~92% average vs GPT-4o 47.7% and Gemini 2.5 Pro 61.1%. Nutri-Vgent raises Qwen3-VL-8B from 53.4% to 62.3% while generic Vgent lowers it to 31.7%. Ablation shows either memory alone lowers overall score below the visual-only baseline; the combination is highest. Easy splits dip slightly, attributed to retrieval overhead. No isolation of prompt/function-call differences from the memory structure itself.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For streaming video question answering with a frozen video-LLM, appending retrieved historical frames as extra visual context helps less than optimizing a small set of latent memory tokens at test time and then dropping the retrieved tokens before decoding; the gain is largest on questions requiring backward tracing over past history.Reasoning about time in video · reviewed by AI
- For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Reasoning about time in video · reviewed by AI
- On hour-scale (mean 88.8 min) open-ended streaming video QA with causal access, every streaming memory design degrades as the evidence-to-query gap grows, and they fail in different ways: recent-window systems cannot recover distant events (AURA 25.4% within 5 minutes falling to 10.5% beyond 30 minutes), text-summary systems keep temporal coverage but discard visual evidence, and model-internal compression loses fine-grained detail (StreamForest 14.4%, ThinkStream 7.5% on retrospection), while human accuracy without rewatching is 63.4%.Reasoning about time in video · reviewed by AI
- For long-video QA with a frozen VLM and fixed answering interface, an outer loop that proposes, executes and retains only development-accuracy-improving rewrites of the executable context-construction program can advance past a strong hand-crafted keyframe-selection harness, and the gain survives a matched cumulative visual-token control and direct reuse on other long-video benchmarks.Improving its own scaffolding from failures · reviewed by AI
- For memory-augmented VLA manipulation policies that must actively move the viewpoint to find task-relevant evidence, writing one memory frame at each fixed action-chunk boundary generalizes better than event-triggered writing (on predicted subtask change or on gripper/head/torso motion), and extra memory capacity helps only under the regular schedule — event-triggered policies degrade further out of distribution because mistimed triggers omit informative views and control jitter fills memory with redundant frames.Turning plans into physical action · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafter linked technique "hierarchical-memory", which does not list this capability in addresses -- recorded, not asserted. Drafted confidence: medium. Falsifier as drafted: A generic flat-retrieval long-video RAG pipeline matching the structured episodic+food-state memory on the same hard state-tracking splits, or the structured memory failing to beat its own visual-only backbone. Drafted stance toward only-models-substantial-code-pretraining-track-entity-state: supports -- Baseline VLMs degrade sharply on the harder long-range regime of ingredient-state questions (e.g. Qwen2.5-VL-7B from 30.7% to 3.3% on Ingredient Entry). Proposed technique, not catalogued: Separate event-log and inventory-ledger memories.