Safety guardrails loosen as AI rivalries grows
Add Axios as your preferred source to
see more of our stories on Google.
Illustration: Sarah Grillo/Axios
As large language models grow more powerful and less predictable, AI companies are loosening safety guardrails in the race to be first — a shift that some warn could lead to catastrophe.
Why it matters: Even Google DeepMind CEO Demis Hassabis has warned that "race conditions" — pressure to outpace rivals or rival nations — can drive reckless decisions as the world nears superhuman AI.
Zoom in: Anthropic, long viewed as the most safety-focused major AI lab, last week revised a key safeguard — narrowing the conditions under which it would delay developing or releasing a model that could pose catastrophic risk.
- "We will delay AI development and deployment as needed to achieve this, until and unless we no longer believe we have a significant lead," the revised policy says.
Zoom out: Anthropic's recalibration comes amid a dispute with the Trump administration.
- The company refused to allow its models to be used for autonomous weapons or domestic surveillance. The Defense Department responded by cutting use of Claude and labeling the firm a supply chain risk.
Between the lines: That highlights another problem with competition. Even if one company refuses on safety grounds, another is likely to step in.
- Hours later, OpenAI announced a deal to provide models for classified networks.
- OpenAI says it shares many of Anthropic's safety concerns. Critics note its agreement leaves broad room for military use, including surveillance of U.S. citizens.
The big picture: Competitive pressure — among AI labs and between nations, particularly the U.S. and China — is intensifying.
- History shows such pressure can push companies toward decisions they might otherwise avoid, as executives fear restraint means losing the lead.
- Hassabis has repeatedly argued that the closer the world gets to superhuman AI, the more essential global cooperation becomes.
- "It's going to require everybody to come together — hopefully, in time," he said in early 2025.
Yes, but: The trajectory appears to be moving in the opposite direction.
- Capabilities are advancing, while global AI summits increasingly focus on commercialization over guardrails.
What they're saying: Future of Life Institute founder Max Tegmark, who has long warned of the risks of leaving AI unregulated, says that the AI companies for years have stalled any broad and meaningful regulation of potential harms.
- "It's their fault that we have the race condition in the first place," Tegmark told Axios.
- He argues that if companies had pushed to turn their voluntary commitments into law, the race dynamic might not have escalated.
- "All of [the AI labs] succumb to the incentives," he said. "It's just maybe Anthropic is the most striking one because they were the ones who always talk such a big game about safety."
The other side: Choosing competitive power over safety may not be a winning strategy with consumers.
- Anthropic has surged to the top of Apple's App Store download charts in the days since it stood up to the Pentagon.
What we're watching: Even Tegmark says he is seeing some signs that regulation may soon be possible.
- Mounting evidence of children and teen chatbot risks has sparked rare bipartisan concern — what he calls a "Bernie to Bannon" coalition.
What's next: Laws mandating companies test models before they release them, at least to make sure they aren't pushing people toward self-harm.
- That, Tegmark says, could open the door to broader testing requirements.
- "It breaks the taboo that AI must always be unregulated."
