Newsom’s AI Kill Switch Order Is Mostly a

California Gov. Gavin Newsom signed an executive order on September 18 that asks experts to recommend stronger oversight of frontier AI within two months, including a possible “kill switch” for powerful models.

The phrase evokes a red button and an immediate shutdown. The order creates neither. It directs the Government Operations Agency and Governor’s Office of Emergency Services to convene experts and develop recommendations. Those proposals could eventually require companies to build shutdown capabilities, submit to outside evaluation and report incidents in which they lose control of a model.

California has moved quickly to frame the problem, but not yet to settle the solution. Newsom's order starts an accelerated policy process while leaving the legal authority, technical standards and enforcement structure largely unresolved. The Governor's executive order points toward measures such as independent third parties writing safety plans for frontier AI companies and mandatory emergency shutdown systems, but for now those remain proposals rather than operating rules.

The “kill switch” supplies the headline. The quieter provisions concerning auditors, incident reports and implementation deadlines may determine whether California can supervise companies before an emergency occurs.

What the Order Can Produce

The order’s quoted language calls for assessing whether California should require a kill switch whose effectiveness would undergo continuing verification by an independent organization. That wording, reported alongside the agencies’ assignment, contains three separate policy choices: a developer must create the mechanism, an outside body must test it, and somebody must decide when its use is justified.

Only the first two appear in the proposal described so far. The available account does not identify who could activate a switch, what conduct would trigger activation or whether a shutdown would cover training, public access, deployed copies or all three. Those omissions are understandable in an instruction to prepare recommendations. They also prevent the phrase “kill switch” from describing a settled regulatory mechanism.

The panel will consider a broader oversight structure. Details attributed to the executive order include placing designated independent verification organizations inside AI laboratories for periodic audits and evaluations. Companies could also have to file risk assessments with a third party and disclose “loss-of-control incidents.”

Those measures address the period before anyone reaches for an emergency control. Auditors need access to models and internal information, incident reports need definitions and disclosure deadlines, and evaluators need enough computing resources to reproduce demanding tests. A switch can fail spectacularly once. Oversight can fail every week through weak access, vague reporting rules or an auditor dependent on the company paying its bill.

Anthropic CEO Dario Amodei has proposed giving third-party evaluators “ongoing, employee-like access” for safety and compliance work. That overlap could make outside evaluation easier to negotiate with at least some companies. Voluntary acceptance by one chief executive, however, does not answer who selects evaluators, what they may publish or how California would handle a company that refuses access.

Newsom Previously Vetoed a Shutdown Requirement

California debated a firmer version of this idea two years ago. SB 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, would have required covered developers to retain the ability to enact a full shutdown that stopped operation and further training.

Newsom vetoed SB 1047 on September 29, 2024. He argued that its thresholds focused on model size and development cost rather than the uses and risks of a system. The history narrows any claim of a simple reversal. The 2024 bill would have imposed a statutory requirement on covered developers. The 2026 order asks experts to reconsider a shutdown mechanism alongside independent verification and other safeguards. Newsom’s earlier objection concerned how the law selected covered models, based on the account of his veto, rather than an objection to every possible shutdown capability.

Still, the sequence matters. California had legislative text establishing who was covered and what capability developers needed. Two years later, the state has returned to expert recommendations. That gives officials room to redesign the thresholds Newsom rejected, but it also moves the disputed questions back into a process whose membership had not been announced in the available reporting.

The political timetable adds another constraint. Newsom’s term ends January 4, 2027. A report delivered within two months of the September 18 order should arrive before then, but legislation, detailed regulations and implementation could pass to the next administration. The order can set priorities and accelerate administrative work; it cannot guarantee that a successor will preserve every recommendation.

Reporting on the order says it also shortens by one year the implementation schedules for two existing California laws that create regulatory entities. Newsom had already signed a law establishing a framework for third-party AI auditors earlier in September. Compared with the speculative shutdown provision, moving existing oversight bodies faster is a more immediate exercise of executive control.

Europe Put the Power Somewhere Else

The European Union offers a useful comparison because its AI Act assigns different forms of stopping power to different actors. Since August 2, the European Commission has had authority to restrict, withdraw or recall a general-purpose AI model from the EU market. The regulator holds that power. The Commission does not need to rely on a developer having built a self-destruct button.

The AI Act separately provides for human overseers to interrupt or halt certain high-risk systems through a stop mechanism, although those obligations are not yet in application. That rule concerns deployed high-risk systems rather than the underlying general-purpose model. California’s discussion combines these concepts under the looser language of a kill switch, even though a company-controlled circuit breaker, a regulator-ordered market recall and a stop button for a deployed system solve different governance problems.

The comparison has limits. A market recall can prohibit distribution or use within a jurisdiction, but the source does not establish that it can erase model weights or halt every copy operating elsewhere. California, meanwhile, has only requested recommendations. Comparing the two systems shows where authority could sit; it does not establish which mechanism would contain a model in every technical setting.

Independent evaluation exposes a shared implementation problem. California’s SB 813 gives the state until January 2028 to certify organizations capable of testing frontier models. Europe appointed a 60-member scientific panel on June 1, but the available reporting says neither system has settled who pays for the computing power needed for evaluations. If the tested company pays, rules governing conflicts, evaluator selection and publication rights become central to whether “independent” describes more than the letterhead.

Urgency Does Not Settle Design

Newsom framed federal inaction as a reason for California to move. The order also arrived amid warnings from prominent researchers and AI executives. Geoffrey Hinton told lawmakers that Congress might have “maybe a year, but not much more than a year” to regulate before advances make intervention harder, according to reporting on his closed-door briefing. Amodei has called for development to slow enough for safety work to catch up, while executives from several companies have discussed greater coordination on testing.

Those warnings support the case for faster institutional preparation. They do not determine the correct trigger for shutting down a model, the evidence required or the official empowered to act. President Donald Trump has instead called fears of AI catastrophe a “hoax,” illustrating the distance between federal political camps but supplying no substitute oversight design.

California’s next decisions will reveal what its dramatic label conceals. Officials must choose whether shutdown authority belongs to developers or government, define which models qualify, give evaluators access and independence, and decide who bears the cost. Until those choices appear in law or enforceable rules, Newsom’s kill switch remains a policy blueprint with a two-month drafting clock.