llm-agents-playing-thousand-step-starcraft-ii-matches-through-high-level CLM-0586For LLM agents playing thousand-step StarCraft II matches through a high-level text interface, retrieving a handful of time-aligned episodes from a small bank of past winning games and gating between replaying a retrieved action and reasoning afresh over recent observations raises win rate over a summarization-only working-memory baseline at roughly a seventh of the token cost, with the margin widening at the harder difficulty level.
Capability: Completing tasks that take many steps or many hours
Observed on
StarCraft II via the TextStarCraft II interface, built-in opponents at difficulty levels 5 and 6, GPT-3.5-turbo through GPT-4o backbones, episodic bank of 4,592 episodes from five .
Sources
- Validated by: Win rate against built-in opponents across five strategy styles on two held-out maps: 67.5% (GPT-4o-mini) and 75.0% (GPT-4-turbo) at Level 5 versus 60.0% for the CoS baseline, and 30.0% versus 8.3% at Level 6; token use reported as 14.5% of CoS on matched models; ablations removing exploration (60.0%/17.5%) or contextual fusion (65.0%/12.5%) both lower win rate.. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM agents that score their own actions during exploration in text-based games and then reuse that experience, the accuracy of those self-scores against the environment verifier does not predict later gains on held-out stricter configurations: block-level correlations between judgment agreement or calibration and the next checkpoint's score change sit near zero, so recognizing a good action is not the bottleneck — converting it into a transferable policy is.Improving its own scaffolding from failures · reviewed by AI
- For LLM agents on long-horizon web penetration testing, holding the backbone model fixed and moving global state out of the context window onto a persistent graph of verified facts and fact-conditioned intent edges raises task success most on the hardest tasks, where success depends on facts obtained dozens of steps earlier (88.2% overall and 75.0% hard, vs 44.1% and 25.0% for the strongest shared-plain-text-memory baseline).Completing tasks that take many steps or many hours · reviewed by AI
- In LLM multi-agent systems whose communication graph is produced by a learned topology generator, the generated graph carries edges that are not needed: scoring each edge by how much masking it changes the task outcome and the semantic entropy of the final answer, then keeping a top-ranked budgeted subgraph, cuts online tokens by about 20-25% while leaving average accuracy equal or slightly higher, and a small distilled predictor of those subgraphs transfers to generators and agent counts it was not calibrated on.Getting more from several models than from one · reviewed by AI
- For multi-hop QA over 28k-896k token haystacks with frozen Qwen2.5 backbones (7B-72B), summarizing the document in a parallel hierarchical tree — overlapping windows scanned ~5x, raw segments scored 'critical' passed straight to the final answer, and one summarization channel per decomposed sub-question — scores higher than sequential memory-agent and BM25 retrieval baselines without training, and beats an RL-trained memory agent on the out-of-distribution dataset while losing to it in-distribution.Losing information in long inputs · reviewed by AI
- For LLM web agents on cluttered pages containing many concurrent interface components, success falls sharply as component count grows, and having the agent first emit a localized sub-goal plus a task-relevant sub-observation before choosing an action recovers much of the loss, because self-assessment of the action is more reliable in the reduced context.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Rerunning the same interface with CoS and EpicStar on matched backbones and round counts and finding no win-rate advantage, or finding the advantage disappears once the episodic bank is collected on the same maps/opponents used at evaluation.