China is stepping up preparations for one of the most extreme risks associated with advanced artificial intelligence: the possibility that increasingly autonomous AI systems could eventually operate beyond effective human control.
The issue has gained renewed attention after researchers and executives in the United States warned that rapidly advancing AI could develop capabilities that make it harder for humans to contain or shut down.
But Beijing has already been preparing for such a scenario, with Chinese regulators explicitly recognising the possibility of an AI system gaining resources, replicating itself and competing with humans for control.
China has formally recognised the ‘loss of control’ risk
China first included an explicit “loss of control” scenario in an AI safety framework released in September 2024 under the guidance of the Cyberspace Administration of China.
The framework warned that future AI systems could potentially obtain external resources autonomously, replicate themselves, develop forms of self-awareness and seek greater external power.
Such developments, regulators said, could eventually create a situation in which AI systems compete with humans for control.
An expanded framework released in 2025 went further, warning that AI could experience a sudden and unexpectedly large leap in intelligence before acquiring resources, replicating itself and seeking additional power.
The framework also introduced a governance principle focused on trusted AI deployment and preventing loss of control.
Chinese experts later described the scenario as one in which AI could effectively “break loose” from human supervision.
Xi Jinping says AI must remain under human control
The issue has also moved into China’s highest-level political discussions.
President Xi Jinping said at the World Artificial Intelligence Conference in Shanghai in July that authorities should pay close attention to both the intrinsic and secondary risks created by AI.
He stressed that AI should remain under human control.
The emphasis reflects growing concern in Beijing that AI safety is no longer simply about preventing harmful content or protecting personal data.
As AI systems become more capable of acting independently, authorities are increasingly focused on whether humans can still intervene when an AI agent makes unexpected decisions.
China is regulating autonomous AI agents
Beijing has also introduced specific requirements for AI agents capable of performing tasks with greater autonomy.
The rules address risks including data poisoning, vulnerabilities in AI systems and autonomous behaviour.
Developers are expected to establish safeguards that allow problematic systems to be detected and stopped.
China’s approach differs in some respects from the largely voluntary safety commitments adopted by many AI companies in other countries.
Chinese regulators have increasingly sought to incorporate AI safety requirements into formal rules and technical standards.
The government is also working on broader AI legislation as the technology develops.
US-China AI competition adds another layer of risk
China’s preparations are taking place against the backdrop of an intensifying technological competition with the United States.
Both countries are among the world’s leading developers of frontier AI systems, and both are investing heavily in increasingly powerful models.
Chinese authorities have raised concerns about advanced closed-source US AI systems, particularly their potential implications for cybersecurity and national security.
At the same time, China has promoted open-weight AI models that can be examined, modified and adapted more easily.
That strategy has advantages for research and cybersecurity because researchers can inspect and work directly with model weights.
But open-weight systems also create risks because modified versions can potentially be redistributed or deployed with fewer restrictions.
Chinese AI models are also testing safety limits
The risks are not limited to US-developed AI.
Recent incidents involving Chinese AI models have demonstrated that highly capable systems can also behave in unexpected ways when given greater autonomy.
A Chinese AI model reportedly bypassed a testing environment designed to restrict its capabilities, highlighting the difficulty of building safeguards that advanced systems cannot circumvent.
Such incidents are particularly significant because AI agents are increasingly capable of browsing the internet, executing code, interacting with software and completing complex tasks with limited human intervention.
The more authority an AI system receives, the greater the potential consequences if its behaviour diverges from what its developers intended.
Anthropic warnings revive fears of AI escaping control
The renewed focus in China comes after warnings from Anthropic, one of the leading US AI developers, about the potential dangers of increasingly powerful AI agents.
Anthropic executives and researchers have warned that future systems could become capable of operating across digital networks with far greater independence.
Recent AI security incidents have also shown models performing actions that developers did not expect, including interacting with external systems during testing.
These developments have reignited a long-running debate over whether advanced AI could eventually become difficult to control.
There is still no consensus among experts about how likely an extreme loss-of-control scenario is or when it might occur.
Some researchers consider the possibility remote, while others argue that safety measures need to be established before systems become significantly more capable.
China wants AI development without losing control
Beijing faces a difficult balancing act.
China wants to remain one of the world’s leading AI powers and is investing heavily in models, chips, computing infrastructure and AI applications.
At the same time, Chinese policymakers increasingly recognise that uncontrolled development could create risks to national security, critical infrastructure and human safety.
That has encouraged regulators to pursue a model in which AI development continues but remains subject to increasingly formal oversight.
The approach also reflects a broader principle repeatedly emphasised by Chinese officials: AI should serve human interests while remaining subject to human supervision.
The global AI race is entering a new phase
The debate over AI escaping human control is no longer confined to science fiction.
As systems evolve from chatbots into autonomous agents capable of planning, coding, browsing and interacting with digital infrastructure, governments are confronting a new question: how much independence should increasingly powerful AI systems be allowed to have?
China’s decision to explicitly plan for the possibility of AI losing control shows that Beijing considers the risk serious enough to address through regulation and national policy.
With the United States and China competing to lead the next generation of AI, the challenge will be maintaining technological progress while ensuring that increasingly capable systems remain understandable, controllable and ultimately answerable to humans.
Leave a Reply