forecasting-scaffolded-system-approaches-human-crowd-accuracy CLM-0056observationsingle paperReviewed by AI and Russ Miller
A retrieval-augmented forecasting system built on a GPT-4-class model — not the base model alone — approaches, and sometimes exceeds, the accuracy of competitive human forecasters on real prediction-market questions dated after the model's training cutoff.
Capability: Predicting future events · Reasoning
Observed on
2024, GPT-4 class with retrieval scaffolding. General.
Model cards: GPT-4
Sources
- Achieved accuracy comparable to, and sometimes exceeding, human crowd forecasts when tested against real forecasting-tournament questions with dates after training cutoff.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On natural-language stock questions that require judging a future condition (e.g. whether a price will exceed a threshold next month), open-weight 20B-31B LLMs reasoning directly from the retrieved historical series answer near chance, and routing the question through an explicit SQL-retrieval plus time-series-forecasting toolchain raises accuracy to roughly the level of human stock investors.Predicting future events · reviewed by AI
- When a search agent is asked to forecast a future distribution rather than report the current state over the same cumulative literature archive, the forward-looking target changes what it retrieves — it allocates a smaller share of retrieved documents to the most recent window, which is the more predictive evidence — so carrying the agent's own state estimate forward beats its explicit forecast.Predicting future events · reviewed by AI
- On county-level US knowledge (14,782 questions over 526 counties), the best of 13 models reaches only about 57% on narrative questions and no model exceeds 13% on numeric Census facts, larger models are not reliably better, and web search helped Gemini (+14 points) while hurting GPT models (about -11), so hyper-local recall is thin and retrieval does not fix it uniformly.Recalling facts it was trained on · reviewed by AI
- Without added retrieval infrastructure, language models underperform human experts at forecasting real-world events from a benchmark of actual forecasting-tournament questions, though accuracy improves with model scale and access to relevant news context.Predicting future events
- In prompt-pool LLM time series forecasting where top-K semantic anchors are retrieved by cosine similarity, the harmful redundancy is concentration of anchor usage accumulated across training batches, not redundancy among the anchors picked for a single forecast: diversifying within a retrieval (MMR) never beat relevance-only selection on four dataset-horizon cells, while an EMA usage-tracking coverage penalty in the training loss lowered MSE on most of six long-term benchmarks.Predicting future events · reviewed by AI
Notes
Requires real infrastructure (search, retrieval, aggregation) built around the model — not an out-of-the-box capability. Worth reading alongside the earlier, lower-scaffolding result below to see the progression.