across-llm-based-forecasting-systems-reported-accuracy-gains-frequently-cannot CLM-0542Across LLM-based forecasting systems, reported accuracy gains frequently cannot be attributed to the language model's temporal reasoning, because evaluation windows predate training cutoffs, component ablations show the LLM sometimes adds nothing over the statistical or retrieval parts, and results shift under small input perturbations.
Capability: Predicting future events
Observed on
LLM-based forecasting agents evaluated on static benchmarks rather than live post-cutoff questions; survey-level synthesis, not a controlled experiment.
Sources
- supportsLLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and ApplicationsValidated by: Not measured; argued from collected negative evidence across the surveyed literature. Survey synthesis of positive and negative evidence across finance, weather, health, energy and operations; no new measurement, no single baseline, and the counts of null ablations are not given in the text available.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM-based forecasting systems, reported benchmark gains often cannot be attributed to temporal reasoning, because test periods predate the training cutoff, ablations show the LLM component sometimes adds nothing over the statistical or retrieval parts, and accuracy shifts under small input perturbations.Predicting future events · reviewed by AI
- When a frozen pretrained time-series forecaster leaves structured residual error, having an LLM propose named semantic features of that residual (e.g. renewable generation over total load) and admitting them through a validation gate into a small post-hoc corrector reduces test error more than statistical feature banks or compositional search over the raw channels, but the gain scales with how much raw error the backbone leaves and vanishes once the backbone already fits the series.Predicting future events · reviewed by AI
- When a small (1.7B) LLM is fine-tuned to act as an ensemble router that assigns per-sample weights across candidate time-series forecasters, chain-of-thought supervision built by rule-templating known oracle weights against extracted temporal features matches or beats CoT generated by a frontier API model, and removing the CoT step costs far more on unseen candidate model pools than on in-distribution ones.Predicting future events · reviewed by AI
- On anonymized retrospective social-event forecasting (entities replaced, dates shifted), frontier LLMs score much better at assigning low probability to events that did not happen than at assigning high probability to events that did, with a mean false-minus-true calibration gap of about 15.9 points across six models and up to 38 points for one model, so benchmarks weighted toward invented events overstate forecasting skill.Predicting future events · reviewed by AI
- In prompt-pool LLM time series forecasting where top-K semantic anchors are retrieved by cosine similarity, the harmful redundancy is concentration of anchor usage accumulated across training batches, not redundancy among the anchors picked for a single forecast: diversifying within a retrieval (MMR) never beat relevance-only selection on four dataset-horizon cells, while an EMA usage-tracking coverage penalty in the training loss lowered MSE on most of six long-term benchmarks.Predicting future events · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A set of LLM forecasting systems whose benchmark gains persist on strictly post-cutoff live questions, survive LLM-component ablation, and are stable under small input perturbations. Drafted stance toward llm-based-forecasting-systems-reported-benchmark-gains-often-cannot-be: supports -- This survey makes the same attribution argument from contamination, null ablations and perturbation sensitivity.