AI researcher resigns from Anthropic, warns of unchecked superintelligence risks
Former Anthropic researcher Jacob Coxon quit the company, citing concerns that AI development could lead to dangerous self-improving systems, a view echoed by an internal Anthropic staff lead.
Jacob Coxon, a researcher who previously worked at OpenAI and most recently at Anthropic, announced his resignation on the social platform X, warning that the rapid advancement of artificial intelligence could soon produce systems capable of self-improvement and unprecedented power. He said the technology is progressing quickly across multiple domains and cautioned that it should not be underestimated.
Coxon described the trajectory as a race toward "self-improving superintelligence," where AI models could create more capable successors, potentially leading to an uncontrolled feedback loop. He added that some developers believe the technology could pose an existential threat within the decade.
Evan Hubinger, Anthropic’s staff lead for AI alignment, affirmed Coxon's concerns in a separate post, stating that the company believes AI could pose a lethal risk to humanity and estimating the probability of such an outcome at over ten percent within ten years. Hubinger also noted that Anthropic currently lacks a concrete plan to ensure alignment of superintelligent systems with human values.
The remarks come as U.S. Senator Bernie Sanders announced plans to introduce legislation aimed at prohibiting the development of superintelligence, reflecting growing political attention to AI safety.