tool-use-reliability-in-customer-support-gpt-4o CLM-0171observationsingle paperReviewed by AI and Russ Miller
In simulated retail customer-support tasks with real tools and policy documents, GPT-4o — the best performer in the study — still completed well under two-thirds of tasks correctly, and its rate of passing the identical task eight times in a row (a proxy for production reliability) was far lower than its single-attempt success rate.
Capability: Using the tools it is given
Observed on
GPT. 2024, GPT-4o. Customer support.
Sources
- GPT-4o succeeded on well under two thirds of retail tasks, with a far lower rate of passing the same task eight times in a row.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In simulated retail and airline domains with tools and policy documents, frontier models succeeded on well under two thirds of tasks and success dropped sharply when the same task had to be passed several times in a row.Using the tools it is given
- On 430 tasks distilled from real professional-forum threads across seven domains, the best model (GPT-5) scores about 54%, under the 60% passing line the authors set, with computer science and applied technology lowest at roughly 40-45%, so long-tail practitioner knowledge remains a gap even where standard benchmarks are saturated.Competence in a specialist domain · reviewed by AI