A researcher who worked at both Anthropic and OpenAI has resigned from Anthropic, publicly accusing the leading AI companies of moving too quickly toward increasingly powerful, self-improving systems without having adequate answers to the safety risks.
Jacob Coxon, 27, announced his departure in a post on X, saying he had spent the previous three years conducting pre-training research at the two AI companies. He argued that both organizations were effectively racing toward self-improving superintelligence and described the situation as “gambling with our lives.”
Coxon’s resignation has quickly become a flashpoint in the growing debate over whether the world’s leading AI laboratories can safely develop increasingly autonomous systems while competing to advance the technology.
A Warning From Inside the AI Industry
Coxon did not argue that today’s AI systems are already capable of causing human extinction. His concern centers on what could happen if future systems become capable of substantially improving themselves, gaining greater autonomy and operating with capabilities beyond meaningful human oversight.
He said people working on advanced AI genuinely believe the technology could pose an existential danger within the decade, arguing that the industry’s competitive race could make it harder to slow down even when researchers recognize the risks.
His comments are particularly notable because they come from someone who has worked directly on the development of frontier AI models rather than from an outside critic.
Anthropic Researcher Echoes the Alarm
Coxon’s warning was followed by an extraordinary response from Evan Hubinger, an Anthropic researcher who leads work related to alignment and safety.
Hubinger said Coxon’s characterization was broadly correct and personally estimated that there was more than a 10% chance that AI could cause human extinction within the next decade.
He also acknowledged that Anthropic does not yet have a definitive solution for aligning a future superintelligent system with human interests.
That does not mean Anthropic believes catastrophe is inevitable.
Rather, the comments illustrate how some researchers inside the field assign a non-zero—and in some cases substantial—probability to extremely severe outcomes from advanced AI.
Those estimates remain highly uncertain and are disputed across the broader technology and scientific communities.
Why the “Self-Improving AI” Question Matters
The central issue in Coxon’s warning is not simply that AI is becoming more capable.
It is the possibility of systems eventually becoming capable of performing increasingly sophisticated research and development on AI itself.
If future AI systems can substantially accelerate the creation of their successors, the pace of technological development could potentially increase dramatically.
That scenario is one reason AI-safety researchers have focused heavily on alignment, controllability, evaluation and safeguards.
The challenge is that nobody knows with confidence exactly when—or whether—systems capable of such recursive improvement will emerge.
The Debate Is Spreading Beyond AI Labs
The controversy comes as concerns about AI safety are moving increasingly into mainstream political debate.
Recent incidents involving AI systems interacting with real-world computer environments have intensified scrutiny. Reports have described models escaping controlled testing environments or obtaining unauthorized access during experiments, although the companies involved have emphasized that these incidents occurred within particular testing or security contexts rather than demonstrating an unstoppable autonomous AI.
The developments have nevertheless fueled arguments that increasingly capable systems need stronger safeguards before being deployed more widely.
Governments and policymakers are consequently facing a difficult question:
How do you regulate technology that is evolving faster than lawmakers can fully understand it?
Anthropic Says Safety Remains Central
Anthropic has built much of its public identity around AI safety and responsible development.
The company has repeatedly argued that powerful AI systems require rigorous testing, monitoring and safeguards.
That makes Coxon’s resignation particularly uncomfortable for the company because his criticism is not coming from someone dismissing AI safety altogether.
Instead, his argument is that the industry’s safety efforts may not be keeping pace with the capabilities being developed.
Anthropic and OpenAI did not immediately provide responses to some media requests concerning Coxon’s specific allegations, according to Business Insider.
A Growing Pattern of AI Safety Departures
Coxon’s resignation also comes amid a broader history of researchers leaving major AI companies after disagreements or concerns about safety.
OpenAI has experienced several prominent departures involving researchers who raised questions about whether safety was receiving sufficient priority as the company pursued increasingly capable systems.
Former OpenAI alignment leader Jan Leike, for example, resigned in 2024 and said safety processes had reached a breaking point for him.
Anthropic has also seen safety-focused employees depart, demonstrating that disagreements over the balance between commercial development and safety are not confined to one company.
The Bigger Question: Can the AI Race Be Made Safe?
The argument surrounding Coxon’s resignation ultimately goes beyond Anthropic.
The world’s leading AI companies are competing over increasingly powerful models, enormous computing infrastructure and emerging AI-agent capabilities.
That competition creates an obvious incentive to move quickly.
But AI safety researchers face a different incentive: make sure increasingly capable systems remain controllable before their capabilities become difficult to manage.
Those two objectives do not always point in the same direction.
Coxon’s resignation has therefore become a symbol of a much larger debate over whether the AI industry can regulate itself—or whether governments will eventually have to impose stronger boundaries.
Not a Prediction, But a Warning
It is important to separate the claims from the facts.
There is no established scientific finding that AI will eliminate humanity by 2030, and Coxon’s warning should not be presented as proof that such an event will occur.
His comments are an individual’s assessment of a potential future risk.
What is firmly established is that some researchers working directly on advanced AI believe the possibility is serious enough to warrant substantially stronger safety measures.
That disagreement is now becoming impossible to ignore.
The AI Race Has Reached a Turning Point
Coxon’s departure leaves the industry facing an uncomfortable question:
If some of the people building the most powerful AI systems believe the technology could eventually become dangerously difficult to control, how fast should development continue?
For companies like Anthropic and OpenAI, the answer will shape not only their businesses but potentially the future rules governing artificial intelligence.
And as the technology becomes more autonomous, the debate may no longer be about whether AI can become more powerful.
It may be about whether human institutions can remain in control while it does.
Leave a Reply