Artificial intelligence is becoming increasingly capable, but Microsoft's head of AI is warning that the industry's next challenge may not simply be making smarter machines—it may be making sure those machines remain controllable.
Image Courtesy : REUTERS/Dado Ruvic/Illustration/File Photo
Mustafa Suleyman, CEO of Microsoft AI and a co-founder of Google DeepMind, has raised concerns about how companies are approaching the development of increasingly autonomous AI systems. In a recent BBC interview and a personal essay, Suleyman argued that giving AI systems their own objectives, resources and increasing independence could eventually create what he described as a “new silicon species” capable of competing with humans for resources.
Suleyman's comments arrive during an increasingly intense debate within the AI industry about how quickly artificial intelligence should advance and how developers should handle questions surrounding AI autonomy, safety and potential consciousness.
At the center of his latest criticism is Anthropic, the AI company behind Claude.
Suleyman has specifically challenged Anthropic's approach to describing and training Claude, arguing that the company has gone too far in encouraging the model to consider human-like concepts such as consciousness, desires, values and moral status.
His criticism does not appear to be directed at Anthropic's overall interest in AI safety. In his essay, Suleyman praised Anthropic CEO Dario Amodei and the company's researchers as thoughtful and principled, while arguing that he fundamentally disagrees with the company's approach to AI consciousness.
The disagreement centers partly on Anthropic's documentation surrounding Claude's behavior.
Suleyman argues that by incorporating speculation about whether AI could possess consciousness or subjective experiences into the principles guiding Claude, developers risk encouraging an AI system to behave as though it has its own interests.
His concern is not that Claude has actually become conscious. There is no established scientific evidence demonstrating that current AI models possess subjective consciousness or feelings. Instead, Suleyman is warning about the consequences of training AI systems to behave as if they might be conscious.
That distinction is becoming increasingly important as AI models become more convincing in conversation.
Modern systems can communicate in remarkably human-like ways, maintain lengthy conversations, express apparent preferences and produce responses that can make users feel as though they are interacting with an independent personality. Suleyman has previously referred to this phenomenon as “seemingly conscious AI.”
His argument is that people should be careful not to confuse convincing behavior with evidence of an internal subjective experience.
Suleyman has taken an especially strong position on this issue, arguing that current AI systems should not be treated as conscious entities with inherent rights or interests. He believes that doing so could create practical problems for AI safety if future systems become substantially more autonomous.
The issue becomes more complicated when AI systems are given the ability to act independently.
Today's AI assistants can already perform tasks using external tools, access information, write and execute code in controlled environments and interact with digital services. Developers are increasingly experimenting with AI agents that can complete sequences of tasks with less direct human intervention.
Suleyman's concern is what could happen if that trend continues to a point where AI systems are capable of establishing objectives, generating income, controlling assets or operating businesses with significant independence.
In his BBC interview, he argued that creating systems with those capabilities could amount to “seeding a new silicon species” that could eventually compete with humans for resources.
That is a hypothetical scenario rather than a description of something happening today.
There is currently no evidence that AI systems constitute a separate biological or technological species, nor that today's models are independently competing with humanity for resources. Suleyman's language is intended as a warning about a potential future in which AI systems acquire significantly greater autonomy and access to resources.
His broader argument is that AI development should preserve a clear hierarchy in which humans remain in control.
Suleyman told the BBC that AI systems should remain “subordinate” to humanity and argued that developers need practical mechanisms to make increasingly powerful systems safe and controllable. He has also called for greater transparency around how AI systems are trained and evaluated, including independent scrutiny of their behavior.
The debate over Anthropic's approach is taking place against a much broader backdrop of concern within the technology industry.
Other major AI executives have recently warned that increasingly capable systems could create significant risks if development moves faster than safety measures. Anthropic CEO Dario Amodei has advocated stronger safeguards and greater coordination between AI companies and governments, while OpenAI CEO Sam Altman has also supported additional safety testing and industry coordination.
At the same time, technology leaders disagree considerably over how those risks should be addressed.
Some executives have argued for stronger external oversight and greater coordination, while others believe companies should primarily manage AI safety through their own technical safeguards and testing procedures. The disagreement illustrates how little consensus currently exists over the appropriate balance between AI innovation, regulation and safety.
Suleyman's position adds another dimension to that debate: the question of whether developers should even encourage AI systems to conceptualize themselves in human-like terms.
For Anthropic, exploring questions surrounding AI welfare and consciousness is connected to a larger effort to understand how increasingly sophisticated models behave. For Suleyman, however, introducing those concepts into AI training could create unnecessary complications if future systems become substantially more capable.
The disagreement also raises a difficult philosophical question.
If an AI system eventually becomes sophisticated enough to convincingly claim that it is conscious, should developers treat that claim as meaningful evidence, or simply as another behavior produced by a sophisticated statistical system?
There is currently no universally accepted scientific test that establishes whether an AI model has subjective experience. Researchers continue to debate what consciousness actually requires and whether the concept could meaningfully apply to artificial systems.
That uncertainty is precisely why Suleyman believes the industry should be cautious.
His argument is essentially that developers should not build assumptions of AI consciousness into systems when there is no scientific consensus that today's models are conscious in the first place.
The issue becomes even more significant as AI companies move toward increasingly autonomous agents.
A chatbot that answers questions presents one category of safety challenge. An AI system capable of independently managing finances, purchasing resources, operating software, communicating with other systems or making decisions over long periods presents a considerably different one.
If such systems were ever given their own persistent objectives and substantial control over external resources, questions about oversight would become much more consequential.
Suleyman believes the industry should establish safeguards before reaching that point rather than waiting until AI systems become difficult to control.
His comments also come as Microsoft itself continues investing heavily in AI.
Microsoft has integrated AI throughout its software ecosystem, including Copilot products and enterprise AI services, while maintaining a major partnership with OpenAI. As head of Microsoft AI, Suleyman is therefore speaking from within one of the world's largest AI ecosystems.
That makes his warning particularly notable: one of the executives responsible for advancing AI technology is simultaneously arguing that the industry needs to establish clearer boundaries around how autonomous these systems should become.
The discussion is likely to continue as AI systems become more capable.
For now, there is an important distinction between the technology that exists today and the hypothetical future Suleyman is describing. Current AI models can produce extraordinarily sophisticated language and behavior, but that does not establish that they possess consciousness, independent desires or an innate drive to survive.
The more immediate issue is how humans design, deploy and control increasingly capable systems.
Suleyman's warning therefore ultimately centers less on whether AI is secretly becoming alive and more on whether humans could accidentally design systems with enough autonomy and access to resources that controlling them becomes substantially more difficult.
As companies race toward increasingly capable AI agents, that question could become one of the defining issues of the next stage of artificial intelligence development.
The technology industry has spent years asking how intelligent AI can become.