Scott Wiener

California State Senator Scott Wiener has provoked the ire of the tech elite. The San Francisco Democrat authored SB 1047, a bill mandating safety measures for developers of the next generation of AI models like ChatGPT, which recently passed in the California State Assembly. Given Republican promises to prevent and undo federal regulations on artificial intelligence, SB 1047 is poised to be the most significant AI safety law in the country.

There has been a roiling debate about whether future AI systems could drive humanity extinct. Wiener says his bill—which contends with the risk of scenarios such as mass casualty events—is “about serious harms… that are not in any way speculative.”

Reaction has been swift—a sign, perhaps, of the seriousness with which industry leaders are taking Wiener’s proposals. Industry goliaths OpenAI, Google, and Meta have all written open letters against the bill, while major AI investors like Andreessen Horowitz and Y Combinator have spearheaded scorched-earth opposition campaigns against it. And though all the leading AI companies have signed on to (non-binding) safety commitments, only Anthropic and Elon Musk’s xAI have come anywhere close to endorsing the bill. After SB 1047 was amended to address many of the company’s concerns, Anthropic wrote that the new version’s “benefits likely outweigh its costs." Musk, who has sparred with Wiener, surprised many when he personally endorsed the bill the week of its final votes.

Eight congressional Democrats from California also took the unusual step of opposing the state-level legislation, arguing in a letter that SB 1047 is premature given that our understanding of AI safety is still in its “infancy.” The next day, Nancy Pelosi echoed their statement, in what appears to be the first time she’s opposed state-level legislation from a member of her own party.

The bill does, however, have endorsements from Geoffrey Hinton and Yoshua Bengio, two “godfathers” of deep learning. It also polls very well and, despite bitter industry resistance, has sailed through California's legislature nearly unopposed, though it could still be vetoed by Governor Gavin Newsom, who hasn’t weighed in yet. Wiener, who also authored California’s net neutrality law, thinks that states can’t afford to wait for Congress, which he notes has not passed a major tech regulation in decades. “California has the responsibility to protect public health and safety while promoting innovation,” he says.