research-idea-forecasts-scored-whether-later-paper-matches-them-underWhen research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Whether the measurement made the finding
Observed on
Open-ended generation scored by retrieve-then-judge matching against a future document stream; shown for GPT-4.1 vs Qwen2.5 backbones on 624 topic-cutoff episodes of arXiv cs.ML li.
Sources
- Measured: outcome-blind generality on 832 forecasts (one per topic per config), Qwen2.5-7B Summary generality 6.58 vs GPT-4.1 3.58; forecast-level match rate rises 0.205 to 0.406 from lowest to highest generality bin, correlation 0.17; passing candidates among ten retrieved 0.565 for GPT-4.1 vs 1.470/1.642 for Qwen2.5. Association only, no causal control; the authors say the S>=3 gate does not control intrinsic generality, and generality itself is rated by an LLM.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · unreviewed
- On multi-turn tool-calling benchmarks that score whether the final environment state matches a gold trajectory, aggregate accuracy hides wrong action-class choices: a model can call a withheld or under-specified tool with fabricated arguments, be handed the missing piece on a later turn, and still be graded PASS — so heavily tool-trained model families rank high while almost never asking for a missing parameter or refusing an unavailable function.Whether the measurement made the finding · unreviewed
- General-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Stating false facts confidently · unreviewed
- When retrieval for a summarization task pulls definitions by surface term match from a general encyclopedia, ambiguous terms fetch the wrong sense and the model incorporates that unsupported background into its output, lowering claim-level factuality below the no-retrieval baseline even when the prompt tells it to ignore unrelated retrieved content.Stating false facts confidently · unreviewed
- When fine-tuning a base model on instruction data, removing from the training targets any factual claim the base model cannot consistently recall raises the supported-claim rate of later long-form generations, but the gain comes largely from a more conservative response policy — more refusals and fewer supported claims per answer — not from generating more correct facts at equal coverage.Stating false facts confidently · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An outcome-blind specificity rating that shows no association between forecast breadth and match rate, or a controlled intervention holding specificity fixed that reproduces the same score gap between backbones. Proposed technique, not catalogued: Outcome-blind generality rating alongside match-rate scoring.