research-idea-forecasts-scored-whether-later-paper-matches-them-under CLM-0133When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.
Capability: Whether the measurement made the finding
Observed on
Open-ended generation scored by retrieve-then-judge matching against a future document stream; shown for GPT-4.1 vs Qwen2.5 backbones on 624 topic-cutoff episodes of arXiv cs.ML li.
Model cards: GPT-4
Sources
- Measured: outcome-blind generality on 832 forecasts (one per topic per config), Qwen2.5-7B Summary generality 6.58 vs GPT-4.1 3.58; forecast-level match rate rises 0.205 to 0.406 from lowest to highest generality bin, correlation 0.17; passing candidates among ten retrieved 0.565 for GPT-4.1 vs 1.470/1.642 for Qwen2.5. Association only, no causal control; the authors say the S>=3 gate does not control intrinsic generality, and generality itself is rated by an LLM.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When an LLM judge is given a prior evaluation score as context metadata on a revision, its rubric scores shift toward that uninformative prior — seven of eight tested models (API and open-weight, 0-5 scale, 20 fixed texts) show a negative total shift, and on a categorical compliance task the metadata blocks 48% of error corrections and flips about 10% of previously correct judgments to the anchored wrong label.Judging another's work as well as a careful reviewer would · reviewed by AI
- On anonymized retrospective social-event forecasting (entities replaced, dates shifted), frontier LLMs score much better at assigning low probability to events that did not happen than at assigning high probability to events that did, with a mean false-minus-true calibration gap of about 15.9 points across six models and up to 38 points for one model, so benchmarks weighted toward invented events overstate forecasting skill.Predicting future events · reviewed by AI
- Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
- When a search agent is asked to forecast a future distribution rather than report the current state over the same cumulative literature archive, the forward-looking target changes what it retrieves — it allocates a smaller share of retrieved documents to the most recent window, which is the more predictive evidence — so carrying the agent's own state estimate forward beats its explicit forecast.Predicting future events · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An outcome-blind specificity rating that shows no association between forecast breadth and match rate, or a controlled intervention holding specificity fixed that reproduces the same score gap between backbones. Proposed technique, not catalogued: Outcome-blind generality rating alongside match-rate scoring.