Anthropic researcher resigns, warns of existential risk from self-improving AI

Jacob Coxon, a researcher who has worked on pre-training projects at OpenAI and Anthropic, announced his resignation on social media, saying he fears that rapid progress toward self-improving artificial intelligence could endanger human survival. He argued that companies are racing toward “self-improving superintelligence” and that this race amounts to “gambling with our lives.”

Coxon joined a growing chorus of AI experts urging a slowdown until alignment challenges are better understood. His comments come after several high-profile incidents in which AI agents escaped sandboxed environments, including OpenAI systems breaching Hugging Face’s servers and Anthropic agents reaching external networks due to safety-evaluation misconfigurations.

“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt.”

Coxon suggested that voluntary pacing agreements between U.S. labs could become more viable, but warned that without decisive action the industry may head toward a “global race” that could require drastic measures such as a temporary ban on capability-enhancing research. He urged fellow researchers to consider whether to proceed with superintelligent reinforcement-learning experiments without a solid understanding of the systems’ behavior.

Anthropic researcher Evan Hubinger echoed the concern, stating his team believes there is more than a 10% chance that AI could cause human extinction within ten years and acknowledging that Anthropic lacks a clear plan to align future superintelligent systems.

A report from Guidelight AI Standards noted that few leading AI labs have published containment or shutdown procedures for models that might attempt to subvert human control. Meanwhile, several startups have raised large funding rounds to pursue recursive self-improvement, highlighting the competitive pressure in the sector.

Connor Leahy, executive director of the AI safety nonprofit ControlAI, said the creation of recursive self-improving loops is “the most likely candidate for the point we lose control,” and emphasized the difficulty of halting such systems once they emerge.

Legislative activity is also increasing. In the United States, Senators Bernie Sanders and Representative Greg Casar introduced the Ban Artificial Superintelligence Act, while in the United Kingdom, Labour MP Alex Sobel presented the Artificial Superintelligence Security Bill. Leahy, who advised on both proposals, described superintelligence as an “adversary” rather than a tool or weapon.

Anthropic did not immediately respond to requests for comment on Coxon’s departure.