realistic-web-tasks-explicit-goals-best-model-completedmechanismsingle paper
On realistic web tasks with explicit goals, the best model completed only a small fraction end to end, far below human performance.
Capability: Following an unfamiliar procedure
Sources
- On realistic web tasks with explicit goals, the best model completed only a small fraction end to end, far below human performance.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- When a model gets a real environment signal after acting — a test result, a tool error, a task-success indicator — reflecting on that signal in words and retrying substantially improves success rates on multi-step coding and decision-making tasks over a single attempt.Fixing its own mistakes
- Models reproduce common human misconceptions, and larger models were not more truthful on this benchmark.Stating false facts confidently