20-year-deterministic-football-management-simulation-hidden-state-adaptive-rivals-340-400 CLM-0340observationsingle paperReviewed by AI
In a 20-year deterministic football-management simulation with hidden state, adaptive rivals and 340-400 decision stops, final score across 15 frontier models is not ordered by model scale, price, vendor or token spend, and early standings barely predict the final order (rank correlation with the final order about 0.19 at year 5, rising to 0.78 by year 15), so short-horizon measurements would rank a different set of models.
Capability: Completing tasks that take many steps or many hours
Observed on
One benchmark (FM-Bench), 15 flagship models, three seeds on the solo track plus one shared-world Arena seed; scoring by a deterministic engine with no LLM judge..
Sources
- Measured: three seeds per model on the solo track, one stamped run per Arena seat, no run selection. Rank correlations reported per seed for behavioral axes; the scale/price null rests on observed orderings (open-weight kimi-k2.6 above both GPT seats) and p>0.38 for token spend at n=15. Single benchmark, single environment family, so generality beyond management-style simulations is untested.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In a 20-year deterministic football-management simulation with hidden state and roughly 340-400 decision stops, what separates frontier models' final scores is managerial behavior rather than deliberation volume: cutting slow-payoff investment as the horizon ends (Spearman -0.58 with late-minus-mid investment shift), keeping cash deployed rather than idle (-0.50), and opening contract renewals early (+0.45) track the score on every seed, while token spend is uncorrelated under every accounting (r_s -0.19, all p > 0.38).Completing tasks that take many steps or many hours · reviewed by AI
- In a deterministic 365-day e-commerce business simulation of 1,000-4,000 agent steps, end-of-year assets do not predict the six underlying operating competences: across 18 frontier and open-weight models the top earner ranked 16th of 18 on fraud avoidance and below another model on profit per tool call, and the best negotiator finished mid-pack on assets.Completing tasks that take many steps or many hours · reviewed by AI
- When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Whether the measurement made the finding · reviewed by AI
- On anonymized retrospective social-event forecasting (entities replaced, dates shifted), frontier LLMs score much better at assigning low probability to events that did not happen than at assigning high probability to events that did, with a mean false-minus-true calibration gap of about 15.9 points across six models and up to 38 points for one model, so benchmarks weighted toward invented events overstate forecasting skill.Predicting future events · reviewed by AI
- Asked which of two entities is larger on a numeric attribute, models often contradict numbers they can state correctly, following entity popularity, mention order and co-occurrence instead; for 7-8B models a logistic regression on those three surface cues predicts the model's choice better than the model's own numeric recall does, while 32B models selectively use the numbers when they are reliable.Recalling facts it was trained on · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A replication where year-5 standings or model scale/price correlate strongly with 20-year final score, or where the ranking matches standard short-horizon agent leaderboards.