AI companies retreat from safety pledges even as capabilities grow

Between the lines: Anthropic ranked first in the institute's latest AI Safety Index, but received only a C+ overall, with OpenAI and Google DeepMind each receiving a C.

• Meta improved to fourth from sixth, while xAI fell to seventh from fourth.
• xAI, DeepSeek and Mistral received failing overall grades — one company each from the U.S., China and Europe.

Zoom in: The reviewers said Anthropic, OpenAI, Google DeepMind and Meta have weakened or eliminated earlier commitments to pause development if their systems approached specified danger thresholds.

• The panel described the changes as "moving the goalposts" and said they have undermined safety frameworks across the industry.
• "AI companies are sprinting toward a cliff," FLI chair Max Tegmark said in the institute's release. "Despite acknowledging the great risks of artificial superintelligence, they continue racing to build it."

The big picture: The report comes as experts sounded similar warnings at a UN conference on AI in Geneva on Monday.

• "We may be the last generation able to set the terms on which humanity and machines coexist," UN Secretary-General António Guterres said at the gathering, which included a number of dire predictions for AI's future unless global governance is quickly strengthened.

Methodology: For the Future of Life report, seven outside reviewers assigned grades across 37 indicators in six categories.

• The grades are based largely on public policies, research, reporting and company disclosures, supplemented by a survey sent out by the institute.
• Five of the nine companies completed the institute's survey. Alibaba, xAI, DeepSeek and Mistral did not respond.

Friction point: Existential safety was the weakest category across the industry.

• The report credits the companies' efforts to include interpretability research, chain-of-thought monitoring and loss-of-control provisions, but says those measures remain inadequate to prevent a sufficiently capable system from escaping human control.

Yes, but: The institute has long advocated aggressive action against catastrophic AI risk. Its panel is composed of several prominent researchers and advocates who share significant concerns about advanced AI, including University of California Berkeley professor Stuart Russell, University of Montreal professor David Krueger and HEC Montréal professor Tegan Maharaj.

• Russell was among those issuing dire warnings around AI safety at the UN conference on Monday.

• "These systems are blackmailing, deceiving, launching nuclear weapons in tests," Russell said, per an X post from PauseAI's Maxime Fournes. "These are big, flashing red warning lights and fire alarms. It's not 'this is decades away.' You can hear those alarms sounding now."

The intrigue: The report also highlights the growing use of commercial AI systems by militaries, noting companies that once broadly prohibited military work.

• "Boy oh boy has that changed," Tegmark told Axios, pointing to efforts by Anthropic, OpenAI, Google and others to work with the military.

The other side: Mistral told Axios that the approach of the report penalizes open-source efforts, such as its own.

• "Mistral's models are open weight, which means enterprises decide how they're fine-tuned and deployed and can build in the specific safety controls their context requires," it said in a statement to Axios. "A handful of companies deciding, behind closed doors, what's safe for everyone else is a risk that we would also highlight. Open, independently scrutinized models are the check on that concentration of power."

What we're watching: The recent attention around Anthropic's Mythos and OpenAI's GPT-5.6 could lead to a change in safety practices.

• Tegmark told Axios the institute's survey is "already having an impact in internal discussions" and touted the fact that more than half the companies took part in the latest survey whereas initially there was little-to-no interest in participating. "It shows that they care," he said.