forecasting-underperforms-humans-without-scaffolding CLM-0057observationsingle paperReviewed by AI and Russ Miller
Without added retrieval infrastructure, language models underperform human experts at forecasting real-world events from a benchmark of actual forecasting-tournament questions, though accuracy improves with model scale and access to relevant news context.
Capability: Predicting future events · Reasoning
Observed on
2022, pre-GPT-4-era models. General.
Model cards: GPT-4
Sources
- Introduced the Autocast forecasting benchmark; models underperform human experts, but accuracy scales with model size and access to relevant news.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On natural-language stock questions that require judging a future condition (e.g. whether a price will exceed a threshold next month), open-weight 20B-31B LLMs reasoning directly from the retrieved historical series answer near chance, and routing the question through an explicit SQL-retrieval plus time-series-forecasting toolchain raises accuracy to roughly the level of human stock investors.Predicting future events · reviewed by AI
- For LLM-based forecasting systems, reported benchmark gains often cannot be attributed to temporal reasoning, because test periods predate the training cutoff, ablations show the LLM component sometimes adds nothing over the statistical or retrieval parts, and accuracy shifts under small input perturbations.Predicting future events · reviewed by AI
- On anonymized retrospective social-event forecasting (entities replaced, dates shifted), frontier LLMs score much better at assigning low probability to events that did not happen than at assigning high probability to events that did, with a mean false-minus-true calibration gap of about 15.9 points across six models and up to 38 points for one model, so benchmarks weighted toward invented events overstate forecasting skill.Predicting future events · reviewed by AI
- Across LLM-based forecasting systems, reported accuracy gains frequently cannot be attributed to the language model's temporal reasoning, because evaluation windows predate training cutoffs, component ablations show the LLM sometimes adds nothing over the statistical or retrieval parts, and results shift under small input perturbations.Predicting future events · reviewed by AI
- On forecasting next-year music festival lineups (link prediction between artists and festivals), zero-shot LLMs prompted for ranked candidate lists score far below specialized temporal-knowledge-graph forecasting models trained on the domain graph, and their errors are dominated by popularity bias toward globally famous artists plus duplicate entries in the ranked list.Predicting future events · reviewed by AI
Notes
Not contested by the 2024 scaffolded result above — different eras and setups, showing a real progression over time rather than a disagreement. A clean example of why claims need a date and an era, not a single timeless verdict.