Microsoft's AI Safety Call Puts UK Oversight to the Test

Microsoft president Brad Smith wants governments to require AI safety checks, incident reports and emergency brakes. In a BBC interview, he compared AI oversight with inspections for milk cartons and washing machines. The comparison opens a policy argument without specifying an inspection regime. Smith’s more consequential proposals concern who gets to examine a powerful system before release and what its developer must do after something goes wrong.

Smith called for independent government evaluators to vet new AI technology, firms to be able to pause or slow systems that escape control, and a legal requirement to file incident reports and notify people affected by rogue AI agents. Those proposals address different moments in a system’s life: testing before deployment, intervention during unexpected behavior, and accountability afterward. A rule that covers one moment cannot automatically do the work of the other two.

The UK offers a live test of that distinction. Its AI Security Institute already evaluates advanced models, while AI minister Kanishka Narayan said on Monday that the government would examine stronger protections for identifying and reporting serious AI incidents through the Cyber Security and Resilience bill. The institute’s work is happening now; the possible reporting changes remain under consideration. Calling both “AI regulation” would obscure who currently has a duty to act.

What an Evaluation Can Do

The UK established the institute in November 2023 as the AI Safety Institute, and it began evaluating risks in frontier models before their public release shortly afterward. Now called the AI Security Institute, it is conducting evaluations of the latest advanced models, Narayan said, to identify risks early and test safeguards. That history explains why Smith points to Britain: the government has an institution with experience examining models, rather than only a promise to build one. It also sets a boundary. Narayan described the institute’s testing, but did not say whether its evaluators can require a developer to delay release.

Access is another boundary. In September, the White House reportedly asked some leading US labs to wait until its own tests were complete before sharing models with the UK institute. Narayan said the UK had since worked with the US to retain the institute’s pre-release access to the most powerful models. The immediate policy question is practical: an evaluator cannot assess a model it cannot examine in time. An access arrangement can make testing possible, but a government seeking dependable oversight would still have to decide whether access depends on a company’s cooperation or on an enforceable requirement.

Narayan gave developers a direct instruction in his Monday remarks: investigate unexpected behavior and, if risks cannot be adequately controlled, do not release the system. That is a clear decision rule for a lab to consider. It leaves a further governance question: if a lab reaches a different judgment, who can challenge it, and with what authority? A pre-release test can reveal a concern; deciding whether that concern blocks release requires a process beyond conducting the test.

A Pact, a Test and a Duty

At a White House meeting last month, executives from major AI and technology companies signed what was described as a “morally binding” pact. It included working with independent auditors to check whether AI systems operate as intended and avoid unintended access to technical systems. Signatories included leaders from OpenAI, Anthropic, Google, Meta and Nvidia. Independent scrutiny is a substantive commitment if companies give auditors access and respond to what they find. The description “morally binding,” however, puts the pact on a different footing from Smith’s proposed legal obligation to report incidents and notify affected people.

Smith welcomed the pact and said Microsoft would be happy to sign it, while arguing that voluntary measures were insufficient. His strongest point is about coverage: companies that decline a voluntary standard do not become subject to it merely because their competitors signed. The strongest case for the pact is speed. Its participants can begin arranging audits without waiting for legislation. Speed and coverage are separate tests of the same proposal, and neither tells readers how much an auditor can inspect or what happens if a company disregards a finding.

There is a political calculation in Smith’s preference for Britain, too. He suggested Washington might accept UK-led standards more readily than EU-led ones. President Donald Trump has favored industry self-regulation, arguing that US companies must compete with Chinese rivals. Smith’s proposal therefore has to clear two hurdles: a government must write workable requirements, and companies operating across borders must have a reason or obligation to follow them. His assessment of Washington’s receptiveness is a prediction, not an agreement between governments.

The UK’s current approach is different again. The work with the National Cyber Security Centre and the institute described how to respond to incidents involving AI agents. These efforts share a concern with the pact: systems that act unexpectedly or reach systems they should not. They assign different jobs to different actors. An industry auditor examines a company’s system under a commitment; a government institute tests models with access arranged in advance; a reporting duty, if enacted with clear terms, would determine when an incident must leave the company’s walls. None is a substitute for specifying the others.

That distinction becomes concrete after a breach. If an AI agent takes an unauthorized action, a company might stop the agent and investigate internally. Smith wants an obligation to document the incident and tell affected people. For that proposal to work, lawmakers would need to specify what counts as a reportable incident, who receives the report, when notification is required and how the firm identifies the people affected. Those are questions raised by Smith’s proposal, rather than provisions the UK has announced. A notice rule drawn too narrowly could miss harms; a rule drawn without a workable trigger could leave companies uncertain about when to notify.

An “emergency brake” needs similar precision. Smith wants firms able to pause or slow AI that gets out of control. That sounds straightforward until a model has been built into other services or an agent has already taken an action. Stopping further activity and accounting for activity already completed are separate tasks. The sensible test for a proposed brake is therefore operational: what can the developer actually halt, who can order it, and what record remains for investigation? Calling a feature a kill switch answers none of those questions.

Narayan has also emphasized defence outside the model itself. He said existing government pilots had helped identify and fix more than 1,000 vulnerabilities across government systems, while the UK is exploring protections for essential services against emerging AI-related risks. Closing vulnerabilities can reduce opportunities for attackers even if model evaluations cannot predict every use of a system. Conversely, stronger cyber defences do not tell an affected person whether a developer must disclose an AI-agent incident. The two tracks respond to different failure points.

Smith argues that Britain could help set global rules; his claim about the UK’s technical lead is his assessment, not proof that other governments would adopt its approach. The choices immediately before UK policymakers are narrower: whether evaluators can reliably see powerful models before release, whether their findings can affect deployment, and whether a serious incident must be reported beyond the company that discovered it. Milk comes with an inspection regime. An AI safety regime will be defined by who can inspect, who must answer and who can say stop.