arxiv-2608-04009 SRC-0591 · paper

SocietyBench: Forecasting Counterfactual Social-World Evolution

Zhenran Wang, Zhonghan Bian, Jinsong Li, Zhangyang Qi

Created: 2026-08-04 · Ingested: 2026-10-24

Large language models (LLMs), and the agents built on top of them, are now benchmarked heavily on whether they can finish a task -- fix a bug, drive a browser, operate a GUI. A complementary social ability, namely how well a model understands and forecasts the way real social events unfold, has barely been measured. We introduce SocietyBench, an end-to-end benchmark that takes a one-line event topic, collects Web news and social-media posts across five platforms, distills them into a date-indexed timeline that keeps factual events and a public-opinion layer separate, and then turns every cutof

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsOn anonymized retrospective social-event forecasting (entities replaced, dates shifted), frontier LLMs score much better at assigning low probability to events that did not happen than at assigning high probability to events that did, with a mean false-minus-true calibration gap of about 15.9 points across six models and up to 38 points for one model, so benchmarks weighted toward invented events overstate forecasting skill.