forecasting CAP-0019 · activePredicting future events
Whether a model can make well-calibrated predictions about events that haven't happened yet, not just recall or verify existing facts.
Tags: reasoning, knowledge
Distinct from fact-verification, which checks claims against existing evidence. Forecasting requires synthesizing current information toward a genuinely uncertain future outcome, and calibration — knowing how confident to be, not just landing on the right answer — is the real test. Genuinely hard to contaminate, since the events haven't happened at training time, which is also what makes it hard to know whether poor performance reflects the event being unpredictable or the model being bad at synthesizing evidence.
Claims
- observationsingle paper↑1↓0Across LLM-based forecasting systems, reported accuracy gains frequently cannot be attributed to the language model's temporal reasoning, because evaluation windows predate training cutoffs, component ablations show the LLM sometimes adds nothing over the statistical or retrieval parts, and results shift under small input perturbations.
- observationsingle paper↑1↓0On anonymized retrospective social-event forecasting (entities replaced, dates shifted), frontier LLMs score much better at assigning low probability to events that did not happen than at assigning high probability to events that did, with a mean false-minus-true calibration gap of about 15.9 points across six models and up to 38 points for one model, so benchmarks weighted toward invented events overstate forecasting skill.
- observationsingle paper↑1↓0On forecasting next-year music festival lineups (link prediction between artists and festivals), zero-shot LLMs prompted for ranked candidate lists score far below specialized temporal-knowledge-graph forecasting models trained on the domain graph, and their errors are dominated by popularity bias toward globally famous artists plus duplicate entries in the ranked list.
- observationsingle paper↑1↓0A retrieval-augmented forecasting system built on a GPT-4-class model — not the base model alone — approaches, and sometimes exceeds, the accuracy of competitive human forecasters on real prediction-market questions dated after the model's training cutoff.
- observationsingle paper↑1↓0Without added retrieval infrastructure, language models underperform human experts at forecasting real-world events from a benchmark of actual forecasting-tournament questions, though accuracy improves with model scale and access to relevant news context.
- mechanismsingle paper↑1↓0When a frozen pretrained time-series forecaster leaves structured residual error, having an LLM propose named semantic features of that residual (e.g. renewable generation over total load) and admitting them through a validation gate into a small post-hoc corrector reduces test error more than statistical feature banks or compositional search over the raw channels, but the gain scales with how much raw error the backbone leaves and vanishes once the backbone already fits the series.
- mechanismsingle paper↑1↓0For LLM-based forecasting systems, reported benchmark gains often cannot be attributed to temporal reasoning, because test periods predate the training cutoff, ablations show the LLM component sometimes adds nothing over the statistical or retrieval parts, and accuracy shifts under small input perturbations.
- observationsingle paper↑1↓0On natural-language stock questions that require judging a future condition (e.g. whether a price will exceed a threshold next month), open-weight 20B-31B LLMs reasoning directly from the retrieved historical series answer near chance, and routing the question through an explicit SQL-retrieval plus time-series-forecasting toolchain raises accuracy to roughly the level of human stock investors.
- mechanismsingle paper↑1↓0When post-training a small LLM (Qwen2.5 0.5B-3B) to produce weekly asset-return forecasts from news, rewarding rank correlation of a full listwise asset ordering gives a dense learning signal where exact-match GRPO gives almost none (all sampled actions scoring zero on ~35% of steps), and yields higher Spearman correlation and backtest Sharpe than the SFT checkpoint, standard GRPO, and prompted commercial models.
- mechanismsingle paper↑1↓0In prompt-pool LLM time series forecasting where top-K semantic anchors are retrieved by cosine similarity, the harmful redundancy is concentration of anchor usage accumulated across training batches, not redundancy among the anchors picked for a single forecast: diversifying within a retrieval (MMR) never beat relevance-only selection on four dataset-horizon cells, while an EMA usage-tracking coverage penalty in the training loss lowered MSE on most of six long-term benchmarks.
- mechanismsingle paper↑1↓0When a search agent is asked to forecast a future distribution rather than report the current state over the same cumulative literature archive, the forward-looking target changes what it retrieves — it allocates a smaller share of retrieved documents to the most recent window, which is the more predictive evidence — so carrying the agent's own state estimate forward beats its explicit forecast.
- observationsingle paper↑1↓0When a small (1.7B) LLM is fine-tuned to act as an ensemble router that assigns per-sample weights across candidate time-series forecasters, chain-of-thought supervision built by rule-templating known oracle weights against extracted temporal features matches or beats CoT generated by a frontier API model, and removing the CoT step costs far more on unseen candidate model pools than on in-distribution ones.
- mechanismsingle paper↑1↓0When a text LLM is prompted to forecast numeric price series, feeding the history as integer-rounded values on the original price scale gives lower forecast error than full-decimal raw values, standard-normalized values, or per-window instance-normalized values, because rounding cuts token fragmentation while keeping magnitudes in the distribution the model saw in pretraining.
- mechanismsingle paper↑1↓0When a text-pretrained LLM is prompted to forecast a numeric price series, rounding the input values to integers while keeping the original price scale gives lower forecast error than full-precision raw values, training-set standardization, or per-window instance normalization, because normalized values fall outside the numeric distributions the model saw in pretraining and per-window rescaling makes the same price appear differently across samples.
- observationsingle paper↑1↓0On zero-shot binary risk classification of rare weather-driven power outages from tabular weather features, GPT-4o/4.1-class models trail supervised classifiers trained on the same data by 0.04-0.13 macro-F1 and have much lower precision, and they gain little from being given the full forecast window rather than only current observations, while supervised models gain more from forecast features.
- observationsingle paper↑1↓0For zero-shot risk forecasting from tabular weather features on a rare-event (~5% positive) binary severity task, GPT-4o/4.1 class models trail simple supervised classifiers (logistic regression, LightGBM) trained on the same data on macro-F1 and precision at every 3h/6h/12h horizon, and the weaker models close the gap only by flagging nearly every window as high risk.
Techniques
- Ask for the current state, then carry it forwardpromptingInstead of prompting the agent to forecast directly, have it search for and estimate the most recent period's distribution and use that estimate as the forecast.
- Build chain-of-thought supervision from rules, not API callstrainingGenerate reasoning traces for fine-tuning by reverse-engineering templated explanations that match extracted temporal features to candidate-model categories under known oracle labels, instead of querying a large model for the rationale.
- Feed integer-rounded values, not normalized ones, to LLM forpromptingRound each numeric time-series value to the nearest integer before serializing it into the prompt, preserving the original scale instead of standardizing or instance-normalizing.
- Keep true and false forecasting questions near 1:1processBalance a forecasting question bank so that events that occurred and events constructed as false appear in roughly equal numbers, so the calibration score does not reward default skepticism.
- Mine the frozen forecaster's residual for named featurestoolingKeeps the forecasting backbone frozen and mines its residual with a Monte-Carlo compositional search plus an LLM proposing named feature combinations, binary flags and code, admitting candidates through one source-blind validation gate into
- Penalize cross-batch anchor over-use, not per-query redundantrainingTrack each prompt-pool anchor's usage as an exponential moving average of its similarity scores across training batches and add a stop-gradient saturating-min penalty that suppresses similarity for over-used anchors, leaving the top-K retri
- Reward listwise rank correlation, not exact numeric forecasttrainingTrain the model with GRPO using a Spearman rank-correlation reward between its generated asset ordering and realized future returns, with a fixed penalty for malformed or hallucinated asset lists, then feed the ranking into Entropy Pooling
- Score forecasts live post-cutoff, reporting cost and accuracprocessEvaluate forecasting agents only on questions resolving after the model's training cutoff, with ablations of the LLM component and joint reporting of cost and accuracy.
- Score forecasts only on live post-cutoff questions, with cosprocessEvaluate forecasting agents on continuously refreshed questions whose outcomes resolve after the training cutoff, with component ablations and cost reported alongside accuracy.
Sources
Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- about this capability · no claim drafted yet
- about this capability · no claim drafted yet
- ↑ supports a claim here
- STQA: A Benchmark for Stock-Focused Tabular Question Answering over Historical and Forecasted Data2026-09-05↑ supports a claim here
- ↑ supports a claim here · ↑ supports a claim here
- ↑ supports a claim here · ↑ supports a claim here
- about this capability · no claim drafted yet
- LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications2026-08-24↑ supports a claim here · ↑ supports a claim here
- ↑ supports a claim here
- When Do Corrective Features Help? An Agent for Corrective Feature Discovery on Black-Box Forecasters2026-08-05↑ supports a claim here
- ↑ supports a claim here
- Approaching Human-Level Forecasting with Language ModelsThis source: Approaching Human-Level Forecasting with Language ModelsHow much the field cites it — heavily cited in the last 12 months76 in the last 12 months · 125 totalpublished 2024checked 2026-09-04click for Semantic Scholar2024↑ supports a claim here
- Forecasting Future World Events with Neural NetworksThis source: Forecasting Future World Events with Neural NetworksHow much the field cites it — steadily cited in the last 12 months33 in the last 12 months · 68 totalpublished 2022checked 2026-09-04click for Semantic Scholar2022↑ supports a claim here
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.