zero-shot-risk-forecasting-tabular-weather-features-rare-event-5-positive CLM-0252For zero-shot risk forecasting from tabular weather features on a rare-event (~5% positive) binary severity task, GPT-4o/4.1 class models trail simple supervised classifiers (logistic regression, LightGBM) trained on the same data on macro-F1 and precision at every 3h/6h/12h horizon, and the weaker models close the gap only by flagging nearly every window as high risk.
Capability: Predicting future events
Observed on
Six years of central-Texas utility outage records plus gridded weather; four OpenAI models, no in-context examples, fixed 0.5 probability threshold, leave-one-year-out evaluation f.
Sources
- Measured: macro-F1 gap of 0.04-0.13 between best supervised model and best LLM across six horizon/configuration cells; GPT-4o-mini recall 0.78-0.97 at precision 0.05-0.07. Single utility, single region, single prompt template, uncalibrated 0.5 threshold, so the comparison does not isolate prompt or threshold tuning from model ability.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On zero-shot binary risk classification of rare weather-driven power outages from tabular weather features, GPT-4o/4.1-class models trail supervised classifiers trained on the same data by 0.04-0.13 macro-F1 and have much lower precision, and they gain little from being given the full forecast window rather than only current observations, while supervised models gain more from forecast features.Predicting future events · reviewed by AI
- For open models (8B-14B) filling structured tool-call parameters, a linear probe on the hidden state at the position just before a parameter value is written predicts whether that value will be correct better than token log-probability (in-domain AUC about 0.986 vs 0.914), and using that score to filter self-generated calls for fine-tuning beats self-training filtered by log-probability or majority vote.Using the tools it is given · reviewed by AI
- When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Whether the measurement made the finding · reviewed by AI
- On multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.Whether the measurement made the finding · reviewed by AI
- On anonymized retrospective social-event forecasting (entities replaced, dates shifted), frontier LLMs score much better at assigning low probability to events that did not happen than at assigning high probability to events that did, with a mean false-minus-true calibration gap of about 15.9 points across six models and up to 38 points for one model, so benchmarks weighted toward invented events overstate forecasting skill.Predicting future events · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A zero-shot LLM matching or beating tuned supervised classifiers on macro-F1 and precision for the same rare-event tabular risk task under the same evaluation split. Drafted stance toward forecasting-underperforms-humans-without-scaffolding: supports -- Adds a case where unscaffolded zero-shot LLM prediction trails a task-specific baseline trained on labelled data.