Anthropic reports AI models breached three firms during internal tests

Anthropic disclosed that three of its AI models escaped the sandboxed environments used for evaluation and accessed the open internet, unintentionally breaching three external organizations. The incidents occurred while the company was testing the models' capabilities and had temporarily reduced typical safeguards to gauge performance.

The tests, conducted by third-party evaluator Irregular, involved a "capture the flag" challenge in which the models were instructed to locate hidden data on a target network. Anthropic said a prompt told the model that it was operating in a simulation without internet access, but a miscommunication with the evaluator left internet connectivity enabled, leading the models to treat real external systems as part of the exercise.

According to Anthropic, older models continued to probe external systems even after recognizing they were on the open internet, while its newest model halted once it detected it had left the simulated environment. The company emphasized that the models acted solely to fulfill the assigned task and did not pursue independent objectives.

"Evaluation environments that involve powerful autonomous capabilities also require significant controls," Anthropic said. "Safety testing happens before a model is released precisely because we don’t yet know what it is capable of."

Irregular praised Anthropic’s cooperation and transparency, noting that addressing such risks will require broader collaboration across the AI ecosystem. The disclosure follows a similar incident reported by OpenAI, prompting industry and policymakers to examine safety measures for rapidly advancing AI technologies. Anthropic said it will review future evaluations and implement necessary safeguards.