forecasting-next-year-music-festival-lineups-link-prediction-between-artists CLM-0392On forecasting next-year music festival lineups (link prediction between artists and festivals), zero-shot LLMs prompted for ranked candidate lists score far below specialized temporal-knowledge-graph forecasting models trained on the domain graph, and their errors are dominated by popularity bias toward globally famous artists plus duplicate entries in the ranked list.
Capability: Predicting future events
Observed on
380 German festivals, 1971-2025 yearly TKG, single-step prediction of 2025; LLMs tested were Llama 3.1 8B, Llama 3.3 70B (quantized), Mixtral 8x7B, GPT-4o-mini, given no candidate .
Sources
- Measured on the paper's own benchmark: for artist prediction the best LLM reached 1.7% normalized 10-precision versus 11.9% for CognTKE; for festival prediction the best LLM reached 17.2% versus 41.4% for CountTRuCoLa. Popularity bias is evidenced by GPT-4o-mini naming Billie Eilish in the top 3 for 47.1% of queries and Llama 3.1 8B repeating one artist ~23 times per query. LLMs were not given the entity set, so part of the gap is the harder open-vocabulary setting rather than forecasting skill; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Whether the measurement made the finding · reviewed by AI
- In prompt-pool LLM time series forecasting where top-K semantic anchors are retrieved by cosine similarity, the harmful redundancy is concentration of anchor usage accumulated across training batches, not redundancy among the anchors picked for a single forecast: diversifying within a retrieval (MMR) never beat relevance-only selection on four dataset-horizon cells, while an EMA usage-tracking coverage penalty in the training loss lowered MSE on most of six long-term benchmarks.Predicting future events · reviewed by AI
- For LLM-based forecasting systems, reported benchmark gains often cannot be attributed to temporal reasoning, because test periods predate the training cutoff, ablations show the LLM component sometimes adds nothing over the statistical or retrieval parts, and accuracy shifts under small input perturbations.Predicting future events · reviewed by AI
- On anonymized retrospective social-event forecasting (entities replaced, dates shifted), frontier LLMs score much better at assigning low probability to events that did not happen than at assigning high probability to events that did, with a mean false-minus-true calibration gap of about 15.9 points across six models and up to 38 points for one model, so benchmarks weighted toward invented events overstate forecasting skill.Predicting future events · reviewed by AI
- When post-training a small LLM (Qwen2.5 0.5B-3B) to produce weekly asset-return forecasts from news, rewarding rank correlation of a full listwise asset ordering gives a dense learning signal where exact-match GRPO gives almost none (all sampled actions scoring zero on ~35% of steps), and yields higher Spearman correlation and backtest Sharpe than the SFT checkpoint, standard GRPO, and prompted commercial models.Predicting future events · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A zero-shot or retrieval-augmented LLM matching or exceeding the best TKG model's R-Precision and normalized 10-precision on the same 2025 test queries in both prediction directions, without popularity-biased or duplicated candidates.