New Benchmark Reveals Language Agents Falter on Real-World Tasks

Researchers unveiled the AgentGym2 benchmark, a new evaluation platform that tests language-model agents in noisy, underspecified environments, Sciencecast reported. The system challenges agents with realistic tasks that involve discovering and using external tools while coping with ambiguous instructions. In the first round of experiments the developers assessed top-performing models such as Google’s Gemini and OpenAI’s GPT-5.

Both systems showed limited performance, recording low success rates and frequently failing to complete the assigned objectives. The findings underscore a substantial gap between current language-agent capabilities and the requirements of real-world applications.