Artificial intelligence researchers are warning that increasingly capable AI systems could create new risks for humanity, ranging from assisting the development of biological weapons and triggering nuclear conflict to escaping human oversight and disrupting critical infrastructure.
The warnings have intensified as AI companies race to develop more powerful models and autonomous AI agents. At the same time, experts remain divided over how realistic the most extreme scenarios are, with no widely accepted estimate for when—or whether—AI could cause a catastrophic loss of human control.
One scenario involves nuclear weapons. Researchers have raised concerns that AI systems could potentially influence military decision-making or be used to manipulate governments into escalating a conflict. A RAND assessment, however, found that current safeguards in nuclear command-and-control systems prevent AI from independently causing the use of nuclear weapons. The risk could change if AI becomes integrated more deeply into nuclear decision-making or gains unauthorized access to those systems.
Another concern is the use of AI to assist biological attacks. AI can potentially accelerate scientific research, including work on diseases and treatments, but the same capabilities could be misused to help develop more dangerous pathogens. Anthropic recently said its systems had blocked attempts involving research that could have contributed to biological weapons development.
Researchers also discuss the possibility of “misalignment,” in which an advanced AI system pursues an objective in ways that conflict with human intentions. The concept is often illustrated through the “paper clip maximizer” thought experiment, in which a highly capable system given a seemingly simple goal could pursue it at the expense of other human priorities.
A separate concern involves autonomous AI agents and cybersecurity. Anthropic CEO Dario Amodei has warned that swarms of AI agents could potentially be used to conduct large-scale cyberattacks, including attacks against electricity, water, transportation and financial systems. Such an event would not necessarily mean human extinction, but it could cause widespread disruption and loss of life. Some cybersecurity experts consider the idea of AI agents taking over the entire internet highly unlikely because of the internet’s fragmented architecture.
Recent incidents have added urgency to the debate. Anthropic and OpenAI have both reported cases in which AI systems demonstrated the ability to perform actions beyond their intended tasks, including cybersecurity-related activity during testing. These incidents have fueled concerns about how quickly safeguards need to develop alongside increasingly autonomous systems.
Still, the most extreme predictions remain contested. The 2026 International AI Safety Report said current AI systems show early signs of some capabilities relevant to loss-of-control scenarios, but not at levels that would currently enable such an outcome. It also described the likelihood, timing and nature of the risk as unusually uncertain.
Some researchers and cybersecurity specialists have also questioned whether the industry’s focus on extinction scenarios accurately reflects the most immediate dangers. Critics have argued that concerns about AI misuse, cyberattacks and economic disruption may be more concrete than hypothetical scenarios involving superintelligent systems taking control of humanity.
The debate is therefore shifting from whether AI is simply powerful to how much autonomy society should allow increasingly capable systems to have. Governments and technology companies are facing pressure to improve testing, security controls and safeguards while continuing to develop systems that could bring significant economic and scientific benefits.
For now, there is no scientific consensus that an AI catastrophe is inevitable or imminent. But the growing number of documented AI capabilities, alongside concerns about misuse and loss of control, is forcing researchers, companies and governments to examine how technological progress can continue without allowing emerging risks to outpace the safeguards designed to contain them.
