AI Researchers Voice Concerns Over Potential Risks of Self-Improving Systems
Several AI lab employees and executives have expressed alarm about the possibility of recursive self-improvement leading to uncontrollable super-intelligent systems, while development of such technology continues.
A researcher at Anthropic announced his resignation, citing fears that the industry’s push toward self-improving artificial intelligence could spiral out of control and threaten humanity. The departure, which attracted widespread attention on social media, was echoed by other AI experts who warned that rapid advances in model capabilities may soon enable AI systems to design and improve themselves without human oversight.
"Jacob is correct," wrote Evan Hubinger, Anthropic’s lead alignment researcher, adding that the team "earnestly believe[s] AI could kill all humans" and expressing particular worry about super-intelligence arising from recursive self-improvement.
Industry leaders have publicly called for slower development. Anthropic CEO Dario Amodei has advocated for a global pause, while OpenAI CEO Sam Altman has expressed support for measures to decelerate progress. An open letter signed by more than 1,000 AI researchers warned that development could outpace the ability to understand or control resulting systems.
Experts describe recursive self-improvement (RSI) as a scenario in which AI models become capable of autonomously refining their own architecture and training processes, potentially leading to a rapid escalation of intelligence. Some executives estimate that such capabilities could emerge within the next two years, based on observed acceleration in AI’s ability to complete increasingly complex tasks.
Despite the expressed concerns, the same organizations continue to invest heavily in more powerful models, citing competitive pressures and the perceived benefits of advanced AI applications. The tension between public safety warnings and ongoing development underscores a growing debate over how, and whether, the industry should impose stricter safeguards.
"I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence," said OpenAI chief scientist Jakub Pachocki.
The situation has prompted calls from policymakers worldwide for new regulatory frameworks aimed at managing the risks associated with advanced AI systems.