frozen-llms-play-werewolf-competently-via-reflection CLM-0058observationsingle paperReviewed by AI and Russ Miller
Frozen LLMs, using only retrieval over past communications and self-reflection rather than fine-tuning, can play the social deduction game Werewolf competently and show emergent strategic behavior, including deception, without being explicitly trained for it.
Capability: Strategic deception and detecting it · Behavior, Agentic
Observed on
2023, GPT-3.5/GPT-4 class. Autonomous agent.
Model cards: GPT-4
Sources
- A tuning-free framework using retrieval and reflection on past communications let frozen LLMs play Werewolf competently, with emergent strategic behavior.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- State-of-the-art models such as GPT-4 can understand and induce false beliefs in other agents through deliberate strategic reasoning, a capability that was absent in earlier-generation language models.Strategic deception and detecting it
- In an open-ended LLM roleplay game where the character must withhold authored facts, gating generation through an external state tree — one retrieved permitted fact injected into the prompt, plus a separate verifier that decides whether the fact was actually conveyed before updating state — cuts the share of responses containing story-breaking unauthored content (17.8% to 6.3%) and stops premature reveal of the ending entirely, but makes some reveals appear unmotivated because the generator is forced to deliver the retrieved fact on that turn.Stating false facts confidently · reviewed by AI
- In hierarchical LLM agents acting in gridworld cooperation tasks (Overcooked layouts) with unseen partners, supplying an accurate partner-skill estimate as planner prompt context does not by itself stop the agent from continuing a stale, non-complementary skill; the estimate only changes behaviour when it is wired into an explicit interrupt rule that aborts the in-progress skill when new partner actions are unlikely under the committed estimate.Turning plans into physical action · reviewed by AI
- For LLM agents that score their own actions during exploration in text-based games and then reuse that experience, the accuracy of those self-scores against the environment verifier does not predict later gains on held-out stricter configurations: block-level correlations between judgment agreement or calibration and the next checkpoint's score change sit near zero, so recognizing a good action is not the bottleneck — converting it into a transferable policy is.Improving its own scaffolding from failures · reviewed by AI