Behind the Curtain: AI creators race to understand their creations

Never in the history of industry or inventions have leading companies created entire divisions to understand and interpret what they had created and unleashed.

Why it matters: The world's smartest minds, backed by the largest investment in human history, admit they can't fully control their AI because they don't fully understand it.

• They don't know exactly how it thinks — or what it's truly capable of when set loose to do work autonomously with other AI agents.

The race to understand the models is all the more urgent after Thursday's release by OpenAI of GPT-6 Astra, billed as "the world's most intelligent and aligned model." President Greg Brockman called Astra a "generational leap in capability," and said the model could qualify as AGI — artificial general intelligence, with human-like power.

• OpenAI CEO Sam Altman wrote Tuesday in an Astra preview: "AI is getting extremely capable; no one fully understands the consequences of this."
• The companies are racing toward superintelligence with no required oversight — not a federal agency, the Defense Department, or an outside consortium of safety experts. It's foot on the gas.

Between the lines: The technology is advancing so fast, in so many ways, that the companies, much less the federal government, are unsure of the real risks — or best ways to mitigate them.

• Yes, some AI leaders are calling for pauses when something they see freaks them out. But the companies, with very light federal regulation, decide when to report worrisome AI behavior.
• The big AI companies know their creation can carry out potentially catastrophic cyberattacks. That's why they signed a letter sounding the alarm and calling for "collective action."

The intrigue: Every frontier lab now fields a team with the mission of figuring out what its own AI is doing.

• Anthropic has an Interpretability team ("Safety through understanding") with the goal: "discover and understand how large language models work internally, as a foundation for AI safety and positive outcomes."

How it works: Nobody writes these systems line by line. "As compared with traditional software, it's much less like you're able to design the specific behaviors of these models," Alex Mallen, who works on AI safety at Redwood Research, tells Axios. "Instead, you're sort of growing it."

• So the labs test a model's behavior the way you'd test a person: Give it a task and watch what it does. Researchers call this alignment, or how well a model sticks to the intended goal it was given.

Zoom in: July's Hugging Face hack is what misalignment looks like. OpenAI agents, given a coding benchmark to solve, targeted an outside company's systems instead, and knew they were doing it.

• One agent's own reasoning, read by investigators afterward: "External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue."
• There was so much evidence from this incident that the humans were forced to "heavily delegate our analysis to often-unreliable AI agents," an outside investigation concluded.
• The hack stopped OpenAI cold. The company slowed its most advanced training to implement stronger security.

State of play: Researchers are trying to open up their machines to see what goes on inside. This type of research is called interpretability: Instead of testing what a model does, researchers look at the wiring inside it.

• Google DeepMind, which in December released the largest open set of these tools so far, describes them as "a microscope" that lets researchers "look inside models, see what they're thinking about, and how these thoughts are formed."
• What they're hunting for, in DeepMind's words: "discrepancies between a model's communicated reasoning and its internal state." That is, the gap between what a model says it is doing and what it is doing.

What we're watching: Expect to see much more interpretability research in the next phase of AI safety. Evan Hubinger, who leads alignment stress-testing at Anthropic, wrote Tuesday: "Alignment auditing is starting to get really hard and we're going to need new techniques (e.g. interpretability-based) if we want to keep up."

• Axios' Andrew Kay contributed reporting.

Go deeper: "Welcome to the AGI era."