deterministic-365-day-e-commerce-business-simulation-1-000-4-000-agent CLM-0560In a deterministic 365-day e-commerce business simulation of 1,000-4,000 agent steps, end-of-year assets do not predict the six underlying operating competences: across 18 frontier and open-weight models the top earner ranked 16th of 18 on fraud avoidance and below another model on profit per tool call, and the best negotiator finished mid-pack on assets.
Capability: Completing tasks that take many steps or many hours
Observed on
One benchmark, five episodes per model, fixed world and deterministic customer demand and supplier negotiation kernel; only renderer wording varies between runs..
Sources
- Validated by: Not measured as an intervention; demonstrated by showing rank inversions across the seven axes for 18 models over 90 episodes.. Measured over 18 models x 5 episodes (90 runs) in one deterministic environment; spreads are large (std exceeds mean on fraud share for 15 of 18 models), and the dimension set is the authors' own choice, so independence of the axes is shown by rank inversion rather than by a validated construct.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In a 20-year deterministic football-management simulation with hidden state and roughly 340-400 decision stops, what separates frontier models' final scores is managerial behavior rather than deliberation volume: cutting slow-payoff investment as the horizon ends (Spearman -0.58 with late-minus-mid investment shift), keeping cash deployed rather than idle (-0.50), and opening contract renewals early (+0.45) track the score on every seed, while token spend is uncorrelated under every accounting (r_s -0.19, all p > 0.38).Completing tasks that take many steps or many hours · reviewed by AI
- In a 20-year deterministic football-management simulation with hidden state, adaptive rivals and 340-400 decision stops, final score across 15 frontier models is not ordered by model scale, price, vendor or token spend, and early standings barely predict the final order (rank correlation with the final order about 0.19 at year 5, rising to 0.78 by year 15), so short-horizon measurements would rank a different set of models.Completing tasks that take many steps or many hours · reviewed by AI
- For LLM agents that score their own actions during exploration in text-based games and then reuse that experience, the accuracy of those self-scores against the environment verifier does not predict later gains on held-out stricter configurations: block-level correlations between judgment agreement or calibration and the next checkpoint's score change sit near zero, so recognizing a good action is not the bottleneck — converting it into a transferable policy is.Improving its own scaffolding from failures · reviewed by AI
- On 102 IPC Blocksworld instances with a 180-second budget, a classical planner (Fast Downward) solved 85%, a model planning directly solved 64%, and the same model choosing one action at a time against a PDDL simulator with observe, reset and retry solved 67%, a consistent but small three-point gain at about 5.7 times the token cost, so interactive simulation helps a little and classical search still wins where a formal model exists.Planning a sequence of actions before acting · reviewed by AI
- In hours-long ML development runs on Kaggle-style competitions, agent scaffolds each lock into one strategy policy — the single-loop CLI agent pivots far less than humans (9% vs 25% of transitions) and the evolutionary search agent far more (58%) with pivot gains cancelling out — and neither reopens an approach it had abandoned, even though it recovers scores after setbacks as well as top humans.Planning a sequence of actions before acting · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Re-running the benchmark with more episodes and finding that per-dimension rankings correlate strongly with the asset ranking, so the extra axes carry no independent information. Drafted stance toward 20-year-deterministic-football-management-simulation-hidden-state-adaptive-rivals-340-400: supports -- Both find that a single accumulated outcome in a long deterministic business simulation fails to order models by the competences that produce it.