Leading AI Researchers Warn of Extreme Risks from Automated AI Research
More than 20 prominent AI researchers, including Geoffrey Hinton, Yoshua Bengio, and OpenAI's Jakub Pachocki, warn that self-improving AI could trigger an intelligence explosion, compressing years of progress into months and posing extreme risks.
More than 20 leading artificial intelligence researchers have issued a stark warning about the dangers of automated AI research, cautioning that self-improving systems could soon trigger an «intelligence explosion» with potentially catastrophic consequences. Among the signatories are Geoffrey Hinton, often called the godfather of AI, Yoshua Bengio, a Turing Award winner, and Jakub Pachocki, OpenAI's research lead.
The researchers argue that AI systems are approaching a threshold where they can automate the entire process of AI research itself. This capability, they warn, could compress years of scientific progress into just a few months, leading to rapid and unpredictable advances that outpace human oversight. The group describes this scenario as an «intelligence explosion» — a feedback loop in which AI systems improve themselves at an accelerating rate, potentially leaving humanity unable to control or even understand the resulting technologies.
The warning comes amid growing public and regulatory scrutiny of AI development. While the letter does not call for a specific moratorium, it underscores a shared concern among some of the field's most respected figures that the pace of innovation may be outstripping the ability to ensure safety. The signatories include researchers from both academia and industry, suggesting that the unease cuts across institutional boundaries.
Geoffrey Hinton, who recently left Google to speak more freely about AI risks, has repeatedly warned that machines could surpass human intelligence sooner than expected. Yoshua Bengio has also been vocal about the need for international cooperation on AI safety. Jakub Pachocki's involvement is particularly notable given his role at OpenAI, a company at the forefront of developing advanced AI models. His signature signals that even those building the most powerful systems recognize the potential perils of automated research.
The concept of an intelligence explosion was popularized by philosopher Nick Bostrom in his 2014 book «Superintelligence». It posits that once an AI system becomes capable of improving its own design, it could rapidly spiral into superintelligence, far exceeding human cognitive abilities. The researchers' warning suggests that recent advances in large language models and other AI technologies have made this scenario more plausible than previously thought.
Critics of such warnings argue that they can be alarmist and distract from more immediate concerns, such as bias, misinformation, and job displacement. However, the signatories emphasize that the risks of automated AI research are not speculative but are grounded in observable trends. They point to the increasing use of AI in scientific discovery, from drug design to materials science, as evidence that the technology is already transforming research practices.
The warning also raises questions about governance. If AI research becomes largely automated, who is responsible for the outcomes? How can regulators keep pace with systems that evolve faster than laws can be written? The researchers suggest that without robust safeguards, the drive for competitive advantage could lead to a race to the bottom, where safety is sacrificed for speed.
For now, the letter serves as a signal to policymakers, industry leaders, and the public that the window for proactive action may be closing. It calls for a broader conversation about the trajectory of AI development and the ethical frameworks needed to guide it. As Hinton, Bengio, and their colleagues make clear, the question is not whether automated AI research will arrive, but whether humanity will be prepared for its consequences.
3
