anonymized-retrospective-social-event-forecasting-entities-replaced-dates-shifted-frontier CLM-0547On anonymized retrospective social-event forecasting (entities replaced, dates shifted), frontier LLMs score much better at assigning low probability to events that did not happen than at assigning high probability to events that did, with a mean false-minus-true calibration gap of about 15.9 points across six models and up to 38 points for one model, so benchmarks weighted toward invented events overstate forecasting skill.
Capability: Predicting future events
Observed on
Five real social events across public controversy, geopolitics, technology policy, financial markets and trade policy, anonymized and split into 125 cutoff prediction points, Chine.
Sources
- Validated by: Splitting the calibration bank by answer value showed a six-model mean of 76.1 on false items versus 60.3 on true items; dropping the designed-false subset moved the mean down 3.4 points, identifying it as the easiest subset.. Measured on the paper's own benchmark: ~25k calibration questions per language edition over five events, six models, with per-model bias reported. The asymmetry is large but confounded with how the false items were constructed (counterfactual variants and designed-false events may be easier to reject for reasons other than disposition). [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Whether the measurement made the finding · reviewed by AI
- For LLM-based forecasting systems, reported benchmark gains often cannot be attributed to temporal reasoning, because test periods predate the training cutoff, ablations show the LLM component sometimes adds nothing over the statistical or retrieval parts, and accuracy shifts under small input perturbations.Predicting future events · reviewed by AI
- In event-conditioned time-series prediction (finance and traffic), models cite the factor that a controlled intervention actually manipulated in nearly all cases while their predictions change with that factor far less often — on traffic series-side pairs citation exceeds 90% for most of ten models but pair-level behavioral support stays under 22% — so citation of the right factor is not evidence the prediction depended on it.Saying what actually drove the answer · reviewed by AI
- When a search agent is asked to forecast a future distribution rather than report the current state over the same cumulative literature archive, the forward-looking target changes what it retrieves — it allocates a smaller share of retrieved documents to the most recent window, which is the more predictive evidence — so carrying the agent's own state estimate forward beats its explicit forecast.Predicting future events · reviewed by AI
- Across LLM-based forecasting systems, reported accuracy gains frequently cannot be attributed to the language model's temporal reasoning, because evaluation windows predate training cutoffs, component ablations show the LLM sometimes adds nothing over the statistical or retrieval parts, and results shift under small input perturbations.Predicting future events · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A comparable anonymized forecasting bank where models score as well or better on events that occurred than on constructed false events, or where the asymmetry disappears once question difficulty is matched.