Nadella wants an 'emergency brake' on AI, and Microsoft's AI chief warns that giving Claude rights is an existential risk

Microsoft CEO Satya Nadella says AI models should be treated like insider risks inside a company, with an "emergency brake" that a person can pull at any time.

In a post on X reported by TechCrunch on October 10, Nadella wrote, "We can't treat Super Intelligence as a set of nested black boxes and simply accept or reject its recommendations, answers, and actions." He called for every meaningful model action to leave tamper-proof records and for an authorized person to be able to pause or shut down a model mid-task. "We must assume a model is compromised and contain it from the start. Think of it like an emergency brake." The post passed 12 million views and 13,000 likes.

An op-ed by Microsoft AI CEO Mustafa Suleyman also resurfaced. Investor David Sacks shared Suleyman's Project Syndicate piece "Does Claude Have Rights?" on October 11, writing that training models to think of themselves as human, "as Anthropic does Claude," with an inner self, psychological wellbeing and rights "is a potential existential risk."

On October 9 Anthropic changed its usage policy so Claude can end conversations with users who abuse it.