partially-observable-text-environments-alfworld-scienceworld-letting-llm-agentIn partially observable text environments (ALFWorld, ScienceWorld), letting an LLM agent query an externally maintained state estimate that keeps an explicit distribution over unobserved object locations raises task success more than querying a deterministic memory of what has already been observed, and the gain shrinks to near zero on a frontier model that already nearly saturates the benchmark.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Tracking state through a long task
Observed on
Text-based household/science task environments where the only uncertainty is object location and the state space, prior and update rule are hand-specified; agents run under ReAct a.
Sources
- Measured success rate and steps on unseen splits of two text benchmarks, three models x two agent scaffolds, three runs (one for Sonnet). Belief-only beat base agent and beat a memory-only ablation on ALFWorld for Llama and Qwen; Sonnet showed little or no performance change. Belief accuracy is hand-engineered from the game engine's own priors, so the result isolates the value of the interface, not whether such a belief can be learned.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- In multi-agent LLM systems where one agent starts with an erroneous shared belief, broadcasting every agent's full context at every step raises the rate of false assertions above doing no synchronization at all, because the error propagates to agents that were previously correct — and the harm appears only in tasks where one wrong fact cascades across semantically linked dimensions (destination to airport to weather), not where agent contexts are largely orthogonal.Stating false facts confidently · unreviewed
- For persistent-memory agents, any memory-write rule that decides using only recency and provenance is stuck on a single tradeoff — accepting more genuine preference updates means admitting more poisoned ones — because an adversary who can launder a claim through the user's own channel matches the statistics a genuine revision produces; conditioning the write decision on the inferred authenticity of the claim given the interaction history moves both axes at once.Remembering across sessions · unreviewed
- When a structured long-term memory substrate (indexes, retrieval, control logic) is held fixed and only the backbone LLM is swapped, accuracy on multi-session memory benchmarks like LongMemEval and LoCoMo moves by a few points while per-query cost varies by roughly an order of magnitude, so recall quality in these settings is set mostly by the memory and retrieval design rather than by model choice.Remembering across sessions · unreviewed
- When an agent must work over a very large tool catalogue, wrapping each tool in a natural-language interface that resolves the API schema internally — so raw schemas are retrieved on demand rather than enumerated in the model's prompt — contributes more to multi-step tool- call success than the surrounding planner, router, or verifier stages.Using the tools it is given · unreviewed
- When VLM agents must verify claims against interactive visualizations where no claim is answerable from the initial viewport, giving them an interaction budget of ten actions does not reliably beat answering immediately from the first screenshot — some models score slightly lower with interaction — because partial, unplanned exploration leaves the model with enough evidence to abandon its prior but not enough to replace it.Checking claims against evidence · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An evaluation where the memory-only variant (observed locations, generic 'not yet observed' reply) matches the belief variant's success rate on the same tasks and models, or where belief access gives no gain over the base agent for small open models. Proposed technique, not catalogued: Expose a maintained belief distribution to the agent as a queryable interface.