New Benchmark Shows Current Language Agents Struggle with Real-World Tasks
The AgentGym2 framework tests large language model agents in noisy, underspecified environments, revealing that even leading models like Gemini and GPT-5 perform poorly on realistic tasks.
Researchers have introduced AgentGym2, an evaluation platform that places language-based AI agents in end-to-end scenarios that mimic real-world conditions. Unlike earlier benchmarks that provide clean inputs and a fixed set of tools, AgentGym2 incorporates uncertain, noisy information and requires agents to explore their environment to discover and combine tools for novel tasks.
The framework assesses not only reasoning and planning but also procedural execution, tool discovery, and robustness to ambiguous data. In tests covering 15 proprietary and open-source models, including recent releases such as Gemini and GPT-5, the agents showed limited success, indicating a gap between current capabilities and the demands of practical deployments.
The findings underscore the need for more realistic testing as language agents become increasingly integrated into customer service, research, and automation workflows.
