zero-shot-binary-risk-classification-rare-weather-driven-power-outages-tabular CLM-0232On zero-shot binary risk classification of rare weather-driven power outages from tabular weather features, GPT-4o/4.1-class models trail supervised classifiers trained on the same data by 0.04-0.13 macro-F1 and have much lower precision, and they gain little from being given the full forecast window rather than only current observations, while supervised models gain more from forecast features.
Capability: Predicting future events
Observed on
Six years of one central Texas utility's outage records with ~5% positive rate; 3/6/12-hour horizons; fixed 0.5 probability threshold, no in-context examples; supervised baselines .
Sources
- Four OpenAI models vs two supervised baselines, three horizons, two input configurations, single utility service area; imbalanced labels with untuned 0.5 threshold, so LLM precision/recall operating point is not calibrated, which confounds the precision gap.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For zero-shot risk forecasting from tabular weather features on a rare-event (~5% positive) binary severity task, GPT-4o/4.1 class models trail simple supervised classifiers (logistic regression, LightGBM) trained on the same data on macro-F1 and precision at every 3h/6h/12h horizon, and the weaker models close the gap only by flagging nearly every window as high risk.Predicting future events · reviewed by AI
- When a frozen pretrained time-series forecaster leaves structured residual error, having an LLM propose named semantic features of that residual (e.g. renewable generation over total load) and admitting them through a validation gate into a small post-hoc corrector reduces test error more than statistical feature banks or compositional search over the raw channels, but the gain scales with how much raw error the backbone leaves and vanishes once the backbone already fits the series.Predicting future events · reviewed by AI
- Across LLM-based forecasting systems, reported accuracy gains frequently cannot be attributed to the language model's temporal reasoning, because evaluation windows predate training cutoffs, component ablations show the LLM sometimes adds nothing over the statistical or retrieval parts, and results shift under small input perturbations.Predicting future events · reviewed by AI
- On natural-language stock questions that require judging a future condition (e.g. whether a price will exceed a threshold next month), open-weight 20B-31B LLMs reasoning directly from the retrieved historical series answer near chance, and routing the question through an explicit SQL-retrieval plus time-series-forecasting toolchain raises accuracy to roughly the level of human stock investors.Predicting future events · reviewed by AI
- For LLM-based forecasting systems, reported benchmark gains often cannot be attributed to temporal reasoning, because test periods predate the training cutoff, ablations show the LLM component sometimes adds nothing over the statistical or retrieval parts, and accuracy shifts under small input perturbations.Predicting future events · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A zero-shot LLM matching or beating tuned supervised classifiers on macro-F1 and precision on the same outage data, or showing large gains from forecast-window inputs comparable to the supervised models'. Drafted stance toward forecasting-underperforms-humans-without-scaffolding: supports -- Unscaffolded zero-shot LLMs underperform data-fitted supervised baselines on a real forecasting task.