natural-language-stock-questions-require-judging-future-condition-e-g CLM-0214On natural-language stock questions that require judging a future condition (e.g. whether a price will exceed a threshold next month), open-weight 20B-31B LLMs reasoning directly from the retrieved historical series answer near chance, and routing the question through an explicit SQL-retrieval plus time-series-forecasting toolchain raises accuracy to roughly the level of human stock investors.
Capability: Predicting future events
Observed on
Templated and human-written stock QA over daily OHLC/volume tables for 4,417 tickers; four open-weight backbones (Qwen3-30B, GPT-oss-20B/120B, Gemma 4-31B); forecaster held fixed (.
Sources
- supportsSTQA: A Benchmark for Stock-Focused Tabular Question Answering over Historical and Forecasted DataMeasured: direct-reasoning FR accuracy 50.88-56.67% versus 59.17-68.33% with the SQFRS toolchain, against 69.17% (investors) and 63.33% (non-investors) on a 120-question set. Small human set and only six annotators. Ground-truth-forecast injection gave the largest single FR gain (+19.85%) yet FR still reached only 86.14%, so forecast quality dominates but a reasoning gap remains. The tool arm also gets extra structure (SLU labels, fixed workflow), so tool access is not fully isolated.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM-based forecasting systems, reported benchmark gains often cannot be attributed to temporal reasoning, because test periods predate the training cutoff, ablations show the LLM component sometimes adds nothing over the statistical or retrieval parts, and accuracy shifts under small input perturbations.Predicting future events · reviewed by AI
- When a search agent is asked to forecast a future distribution rather than report the current state over the same cumulative literature archive, the forward-looking target changes what it retrieves — it allocates a smaller share of retrieved documents to the most recent window, which is the more predictive evidence — so carrying the agent's own state estimate forward beats its explicit forecast.Predicting future events · reviewed by AI
- When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Whether the measurement made the finding · reviewed by AI
- Across LLM-based forecasting systems, reported accuracy gains frequently cannot be attributed to the language model's temporal reasoning, because evaluation windows predate training cutoffs, component ablations show the LLM sometimes adds nothing over the statistical or retrieval parts, and results shift under small input perturbations.Predicting future events · reviewed by AI
- Without added retrieval infrastructure, language models underperform human experts at forecasting real-world events from a benchmark of actual forecasting-tournament questions, though accuracy improves with model scale and access to relevant news context.Predicting future events
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A study where the same models answering forecast-threshold questions directly from historical tables match or beat the toolchain pipeline and human investors, or where the direct-reasoning accuracy is well above chance. Drafted stance toward forecasting-underperforms-humans-without-scaffolding: supports -- Without forecasting scaffolding the models sit near chance on forecast-based reasoning, below human investors, and only the tool pipeline closes the gap. Proposed technique, not catalogued: Route forecast questions through a forecaster tool instead of reasoning from history.