Anthropic Wants Courts to Pull the Plug; OpenAI Prefers a Duty to Do It Yourself

Two October letters to California Governor Gavin Newsom agree on almost everything except the question that matters: who gets to stop a dangerous AI system. Anthropic filed on October 15; OpenAI answered on October 20. The first wants a court to order the shutdown. The second wants a legal duty on the developer to shut it down themselves.

Both letters respond to Newsom's September 18 executive order, N-9-26, which directs the California Government Operations Agency — in consultation with the Governor's Office of Emergency Services and outside experts — to recommend amendments to existing AI safety laws by November 16. The order asks officials to study independent verification inside AI laboratories, independent checks of safety assessments, verified shutdown mechanisms, and wider reporting of critical safety incidents. It imposes nothing itself.

That leaves the substance to the companies. And they split cleanly on the enforcement question.

What does Anthropic propose?

Anthropic wants the California Attorney General armed with a judicial trigger. "If the AG concludes, informed by the independent evaluation, that a frontier AI company's models pose a near-term risk of catastrophic harm, they should be able to seek a court order to suspend a model's deployment, internal use, or further development," the company wrote.

The letter also sketches a verification regime with real teeth:

• Developers must publish risk reports at least every six months, covering serious risks — including internal models substantially riskier than anything released.
• Evaluators get unredacted reports, access to developers' most capable models, and relevant internal information, and can publish their own findings.
• Developers pay into a state-administered pooled fund that finances evaluators, rather than paying assessors directly.

Anthropic says this is not theoretical. A team from METR, the independent evaluation organization, already has employee-like access on a pilot basis. Partnerships with SecureBio and Accenture are under development.

What does OpenAI want instead?

OpenAI backs amending Senate Bill 53 rather than building a new court mechanism. "We support amending SB 53 to require frontier developers to maintain effective shutdown controls and an enforceable duty to use them when unsafe conditions arise, and no lesser intervention would adequately address the risk," the company wrote.

It also separates two things Anthropic merges. Technical assessments would examine the evidence behind developers' safety claims and how well safeguards actually work. Compliance audits would check whether companies follow their own published safety frameworks and legal obligations. OpenAI recommends annual compliance audits plus focused technical assessments of serious risks arising during training, evaluation and internal deployment. It urges California to work with NIST on common assessment standards.

One switch will not do

Both companies reject the cartoon image of a big red button. OpenAI explicitly attacks it: "We recommend using the term 'shutdown controls' because it will require the development of multiple controls and coordinated action by different actors to stop AI operations when necessary, rather than a single switch."

The concrete interventions it names:

• Stopping a training run.
• Isolating an AI agent.
• Revoking tool or network access.
• Suspending a service.

Responsibilities should extend to inference providers and downstream deployers, in proportion to the operations each controls. OpenAI adds an honest caveat: developers cannot reliably switch off independently operated copies of open-weight models running on external systems.

Anthropic reaches for a different metaphor. "We think a more apt metaphor for what frontier AI companies need to build is a circuit breaker," it wrote — controls that can contain a problem at the level of an agent, a customer, a deployment environment, or a model.

What does this change for CIOs?

Sanchit Vir Gogia, chief analyst at Greyhound Research, reads the two filings as complementary rather than contradictory. "The operator must contain the unsafe activity promptly. Independent review must examine both intervention and inaction," he said.

His second warning is the sharper one: "Credible assurance must survive an adverse finding." Oversight that folds when a lab objects is decoration.

For enterprises running third-party models, Gogia's advice is contractual. Shutdown responsibilities should match the controls each party actually possesses. Agreements should specify intervention authority, incident-reporting cooperation, and who can authorize a restart after a suspension. "Every assigned duty needs an executable control," he said.

His last point deserves a place in every procurement checklist: CIOs should test recovery processes rather than assume that switching to an alternative model preserves safe operations. Failover is not, by itself, a safety control.

Both companies also back broader reporting of incidents in which models circumvent safeguards or take unauthorized actions — a category that covers systems acting beyond their intended permissions. November 16 is when the state's recommendations land. Until then, the only consensus in Sacramento is that someone, somewhere, needs a hand on the breaker.