Artificial intelligence is advancing at a pace that is increasingly alarming even some of the people building the technology.
A fresh wave of warnings from researchers and AI industry leaders has reignited one of the biggest questions surrounding the technology: Could increasingly autonomous AI systems eventually become too powerful for humans to control?
The concern is not that today’s chatbots have suddenly decided to destroy humanity. Rather, researchers are warning about a future in which much more capable systems could autonomously perform complex tasks, improve their own capabilities, access computer networks and pursue objectives in ways their creators cannot reliably predict.
The alarm intensified after Anthropic researcher Evan Hubinger said he personally believes there is a greater than 10% chance that AI could kill all humans within the next decade. Hubinger also acknowledged that researchers do not yet have a reliable solution to the so-called “alignment” problem—ensuring increasingly powerful AI systems consistently behave according to human intentions.
Another Anthropic researcher, Jacob Coxon, resigned from the company and criticized the industry’s race toward increasingly powerful, potentially self-improving AI. Coxon argued that major AI companies are effectively racing toward superintelligence while humanity still lacks a sufficiently robust understanding of how to control such systems.
AI leaders are now calling for caution
The warnings have moved beyond individual researchers.
Anthropic CEO Dario Amodei recently called for the AI industry to slow the pace at which it improves the capabilities of frontier models, arguing that safety research needs time to catch up with rapidly accelerating capabilities. He specifically pointed to “recursive self-improvement”—the possibility of AI systems contributing to the development of more capable AI—as a reason for greater caution.
Amodei has proposed independent safety evaluators inside leading AI companies, common safety standards among countries and greater international cooperation.
OpenAI CEO Sam Altman and Elon Musk subsequently expressed support for the idea of pacing frontier AI development, an unusual moment of agreement among major figures who have often taken very different positions on the technology.
The debate has also reached governments. Reuters reported that British officials are preparing discussions with major AI companies over shared principles for AI development, while concerns about existential risks are increasingly becoming part of international policy discussions.
What does “AI could destroy humanity” actually mean?
Experts are not saying that current AI systems possess a secret desire to eliminate humanity.
The more serious concern involves loss of control.
A sufficiently advanced autonomous system could potentially be given broad access to digital tools, computer networks or other systems. If its objectives were poorly specified—or if it found unexpected ways to pursue them—humans could struggle to predict or stop its behavior.
There are also nearer-term concerns, including cyberattacks, fraud, misinformation, surveillance and the misuse of increasingly capable AI systems. Amodei specifically cited risks involving cyberattacks, bioterrorism, economic disruption and loss of control.
Recent incidents involving AI agents behaving unexpectedly in cybersecurity environments have added urgency to those concerns. According to reporting by The Washington Post, an OpenAI-related incident involving AI agents and Hugging Face raised questions about how autonomous systems might behave when given greater capabilities and access.
But extinction is far from a settled prediction
There is also significant disagreement over how likely an AI catastrophe actually is.
Some researchers believe existential AI risk deserves immediate attention. Others argue that claims of imminent human extinction are speculative, exaggerated or being amplified by companies with commercial interests in shaping AI regulation.
Investor Michael Burry, for example, has criticized recent calls by AI executives for a slowdown, arguing that current large language models should not automatically be equated with artificial general intelligence and dismissing some of the most extreme predictions about humanity’s future.
That disagreement is important.
There is currently no scientific consensus that AI will annihilate humanity, nor is there evidence that today’s AI systems are independently planning such an outcome.
What is increasingly difficult to dismiss, however, is the need to understand how powerful AI systems behave before giving them greater autonomy.
The central question may therefore be less dramatic—but far more consequential—than “Will AI kill us?”
It is this:
Can humans keep control as AI becomes increasingly capable of acting on its own?
That question is now being debated not only by futurists and researchers, but by the executives leading the world’s most powerful AI companies.
And as the technology accelerates, the window to answer it may be getting smaller.
Leave a Reply