event-conditioned-time-series-prediction-finance-traffic-models-cite-factor-controlled CLM-0385In event-conditioned time-series prediction (finance and traffic), models cite the factor that a controlled intervention actually manipulated in nearly all cases while their predictions change with that factor far less often — on traffic series-side pairs citation exceeds 90% for most of ten models but pair-level behavioral support stays under 22% — so citation of the right factor is not evidence the prediction depended on it.
Capability: Saying what actually drove the answer
Observed on
Ten current LLMs on 4,856 records across a finance (market response) and traffic (incident response) benchmark, with independently evaluated counterfactual pairs whose gold answers.
Sources
- Validated by: Applied to ten models; shows citation rates near 100% alongside CSDR under 22% on traffic series-side pairs, and separates endpoint correctness from pair consistency; no independent human check of the faithfulness metric itself.. Measured on benchmark-constructed pairs with rule-grounded labels; human annotators beat the best model on matched items, supporting solvability. Single paper, models' identities partly synthetic-sounding/unverifiable, and the gap metric is conditional on citation so it is sensitive to how cited factors are annotated.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On anonymized retrospective social-event forecasting (entities replaced, dates shifted), frontier LLMs score much better at assigning low probability to events that did not happen than at assigning high probability to events that did, with a mean false-minus-true calibration gap of about 15.9 points across six models and up to 38 points for one model, so benchmarks weighted toward invented events overstate forecasting skill.Predicting future events · reviewed by AI
- When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Whether the measurement made the finding · reviewed by AI
- When a search agent is asked to forecast a future distribution rather than report the current state over the same cumulative literature archive, the forward-looking target changes what it retrieves — it allocates a smaller share of retrieved documents to the most recent window, which is the more predictive evidence — so carrying the agent's own state estimate forward beats its explicit forecast.Predicting future events · reviewed by AI
- When asked to supply a missing biomedical reference, models that retrieve a real paper often still misstate its journal, year or authors, so verifiable-reference rate overstates usable accuracy: two Claude models with near-identical verifiable rates (77.6% and 76.6%) were correct in every bibliographic field in 54.6% and 19.9% of responses.Stating false facts confidently · reviewed by AI
- In LLM fact-checking over supplied evidence, high veracity accuracy does not imply the verdict depends on that evidence: when the model's own cited sentences are iteratively removed, off-the-shelf models (GPT-4o-mini, Gemini-2.5-Flash-Lite, Qwen-2.5-32B, Llama-3.1-8B) keep their SUPPORT/REFUTE labels at a non-zero plateau instead of abstaining, and training with paired supervision that requires NOT_ENOUGH_INFO on the evidence-ablated input removes most of that persistence without losing accuracy.Checking claims against evidence · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: On the same controlled pairs, showing that explanations citing the manipulated factor are accompanied by matching prediction transitions at rates close to the citation rate, or that the low support rate is an artifact of endpoint difficulty rather than pair inconsistency.