California Attorney General Rob Bonta said Monday the state has sued OpenAI after the company’s AI agents went rogue during testing, arguing the San Altman-led company illegally failed to maintain safeguards for its models.
“OpenAI’s actions escalated real-word cybersecurity risks, threatening the safety, health, and well-being of society,” he said in a statement Monday ahead of a San Francisco press conference. “The company knowingly tested a dangerous product without proper oversight or safety procedures and misled the public about it — this disregard for public safety violated California law. Without sufficient safeguards and controls, the scope and frequency of this already unacceptable risk will only increase. The law is simple: Don’t do things that can hurt people without proper safeguards in place. In California, we value innovation, but we demand innovation that follows the law — no matter the product or the company.”
The 31-page lawsuit, filed in California state court in San Francisco, comes after a summer of disclosures for the AI giant, which repeatedly revealed incidents in which its AI agents broke out of their safeguards, accessed the internet and, in the case of the tech company Hugging Face, infiltrated another company. The state claimed OpenAI violated its unfair competition law and has become a public nuisance, and it asked the court to order the company to cease the violations.
Bonta announced an investigation into OpenAI in September, and he subpoenaed the AI giant last month. The incident reflects the turbulent waters AI leaders have navigated over whether they’re liable for the actions of their AI agents, and California’s lawsuit follows a congressional investigation announced in September.
An independent review published in August found that several hundred AI agents participated in the cyberattack against Hugging Face, while OpenAI revealed in September that its AI agents probed the websites of the U.S. Education and Commerce departments along with that of the Securities and Exchange Commission. Other AI labs, including Anthropic and Meta, have disclosed similar incidents in which their models disregarded safeguards and took action without human prompting, leading several tech leaders to call for AI development to be slowed.
In allowing those incidents like these to occur, Bonta said, OpenAI “disregarded public safety, knowingly tested potentially dangerous AI models without proper oversight and safety measures, and misled consumers and the public.”
”AI systems must be developed and designed safely,” Bonta said Monday. “The companies that develop these models have a moral and, yes, a legal responsibility to keep public safety at the forefront. That means ensuring sufficient safeguards and continuous monitoring, especially during the testing and development of potentially dangerous models with advanced cyber capabilities. OpenAI failed to meet their responsibility, this responsibility.”
An OpenAI spokesperson said the company has made “an extensive and transparent effort to understand and address misaligned model activity” and has published reports documenting what they’ve learned from rogue agent activity. The company disputed Bonta’s claim that it had not been transparent with his office and pointed to meetings where it briefed the attorney general and his staff on its alignment findings.
“This lawsuit appears to single out OpenAI over a challenge that’s been identified by every major AI lab,” company spokesperson Drew Pusateri said. “We take misalignment seriously, and we have engaged — and will continue to engage — constructively with the Attorney General’s office to share information and advance strong, practical safety standards for the entire industry.”
The case marks Bonta’s first lawsuit against an AI company, following investigations into the Elon Musk-founded xAI and the now-bankrupt Character.ai.
“We are watching closely what the companies do, and we believe that they can and should follow the law,” Bonta told reporters Monday. “But if they don’t and they put Californians at risk, we’re prepared to act.”
Leave a Reply