California Governor Gavin Newsom has vetoed SB 1047, a bill aimed at preventing malicious actors from using AI to cause “serious harm” to humans. The California State Assembly passed the bill by a 41-9 margin on August 28, but several organizations, including the Chamber of Commerce, had urged Newsom to veto the bill. In his Sept. 29 veto message, Newsom said the bill was “well-intentioned,” but “doesn’t require AI systems to be deployed in high-risk environments or involve critical decision-making,” he said. It does not take into account whether the use of data is involved.” Instead, the bill would apply strict standards to even the most basic functions, as long as large systems implement them. ”
SB 1047 would have held developers of AI models responsible for adopting safety protocols to prevent destructive uses of the technology. This includes precautions such as testing and external risk assessments, as well as “emergency shutdowns” that completely stop the AI model. The first violation will cost a minimum of $10 million, and subsequent violations will cost $30 million. However, the bill was amended to remove state attorneys general’s authority to sue AI companies for negligent conduct in the absence of a catastrophe. Companies are only eligible for injunctive relief and can be sued if their models cause significant harm.
This law applies to AI models that cost at least $100 million to use and 10^26 FLOPS to train. It would also have covered derivative projects where a third party invested more than $10 million in developing or modifying the original model. All companies doing business in California are subject to this rule if they meet other requirements. “I don’t think this is the best approach to protecting the public from the real threats posed by technology,” Newsom said, noting the bill’s focus on large systems. The veto message adds:
By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that may give the public a false sense of security about the control of this rapidly changing technology. Smaller, more specialized models may emerge as equally or more dangerous than those covered by SB 1047, at the cost of innovations that prioritize public interest and promote progress. itself may be suppressed.
An earlier version of SB 1047 would have created a new division called the Frontier Model Division to oversee and enforce the rules. Instead, the bill was changed prior to a committee vote to place governance in the hands of the Frontier Model Commission within the Office of Government Management. The nine commissioners are appointed by the governor and the state Legislature.
The bill had a complicated path to a final vote. California Sen. Scott Wiener, author of SB 1047, told TechCrunch: Of that. ” Prominent AI researchers Jeffrey Hinton and Yoshua Bengio support the bill, and the Center for AI Safety has been sounding the alarm about the risks of AI over the past year.
“Let me be clear: I agree with the author. We cannot wait for a catastrophe to occur before taking action to protect our people,” Newsom said in his veto message. I mentioned it inside. The statement continues:
The State of California does not waive its liability. Safety protocols must be adopted. Aggressive guardrails must be implemented, and serious consequences for malicious actors must be clear and enforceable. However, I disagree that to protect public safety we must settle for solutions that are not based on empirical trajectory analysis of AI systems and capabilities. Ultimately, a framework for effectively regulating AI needs to keep pace with the technology itself.
SB 1047 attracted strong opposition from across the technology industry. Researcher Fei-Fei Li, like Meta’s lead AI scientist Yang Lukun, criticized the bill for limiting the potential to explore new uses for AI. Industry groups representing tech giants such as Amazon, Apple and Google said SB 1047 would limit new development in the state’s tech sector. Venture capital firm Andreesson Horowitz and several startups also questioned whether the bill would impose an unnecessary financial burden on AI innovators. Antropic and other opponents of the original bill pushed for an amendment that was adopted in the version of SB 1047 that passed the California Appropriations Committee on August 15.
