search-agent-asked-forecast-future-distribution-rather-than-report CLM-0481When a search agent is asked to forecast a future distribution rather than report the current state over the same cumulative literature archive, the forward-looking target changes what it retrieves — it allocates a smaller share of retrieved documents to the most recent window, which is the more predictive evidence — so carrying the agent's own state estimate forward beats its explicit forecast.
Capability: Predicting future events
Observed on
Agents searching a temporally restricted arXiv corpus with a BM25 interface and full pre-cutoff (expanding-history) access; measured on four models (GPT-5.5, Qwen3.6-27B, GPT-OSS-1.
Sources
- Validated by: State-as-Forecast beat direct Forecast on episode-level Spearman agreement for all four diagnostic models under expanding-history access; frozen-evidence replay with byte-identical query traces and matched search-call budgets attributed a shared part of the gap to retrieval rather than readout.. Measured on 1,055 held-out episodes across 211 fields with five rolling origins; four diagnostic models; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Whether the measurement made the finding · reviewed by AI
- For agentic retrieval-augmented multi-hop QA with 7B-class policy models, scoring each intermediate step for both logical validity and grounding in the retrieved passages — rather than scoring steps by the final answer — yields higher EM/F1 than outcome-reward and final-answer-based process-reward training, because it removes preference pairs where flawed retrieval happened to reach the right answer.Carrying a chain of inference through to its conclusion · reviewed by AI
- For LLM-based forecasting systems, reported benchmark gains often cannot be attributed to temporal reasoning, because test periods predate the training cutoff, ablations show the LLM component sometimes adds nothing over the statistical or retrieval parts, and accuracy shifts under small input perturbations.Predicting future events · reviewed by AI
- On natural-language stock questions that require judging a future condition (e.g. whether a price will exceed a threshold next month), open-weight 20B-31B LLMs reasoning directly from the retrieved historical series answer near chance, and routing the question through an explicit SQL-retrieval plus time-series-forecasting toolchain raises accuracy to roughly the level of human stock investors.Predicting future events · reviewed by AI
- In partially observable text environments (ALFWorld, ScienceWorld), letting an LLM agent query an externally maintained state estimate that keeps an explicit distribution over unobserved object locations raises task success more than querying a deterministic memory of what has already been observed, and the gain shrinks to near zero on a frontier model that already nearly saturates the benchmark.Tracking state through a long task · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A setup where forecast-conditioned queries retrieve as large or larger a share of recent evidence as state-conditioned queries, or where direct forecasting matches or beats state carry-forward under matched search budgets on cumulative-history access.