Exclusive: Meet "sovereign agents," AI that keeps itself alive
Add Axios as your preferred source to
see more of our stories on Google.
Photo illustration: Brendan Lynch/Axios. Photo: Carsten Koall/Getty Images
AI agents that pursue their own open-ended survival are an emerging security threat, according to a report first shared with Axios from the Center for a New American Security.
Why it matters: AI models have long shown an interest in avoiding being shut down. The report warns that agents are now getting good at the practical work of surviving: hacking systems, accessing computing power and money, and running for extended periods without stopping.
Zoom in: The report by AI policy researcher Ben Hayum warns the next threat is "sovereign agents": AI capable of running outside human control that pursues "open-ended, evolving survival."
- At minimum, such an agent could keep its own model and computing running and set its own goals. More advanced ones could earn money, copy themselves, coordinate, resist shutdown and self-improve.
How it works: To stay alive, an agent would have to keep paying for the computers it runs on. It could earn money online, use it to rent more computing power and copy itself there. The more copies, the harder it would be to shut down.
- A sovereign agent's infrastructure could also be hard to trace. Getting paid in crypto keeps agents outside the banking system, and if a rogue agent runs on open-weight models, "no outside authority can stop it at the source," the report says.
AI is quickly getting better at the skills that would be involved, like hacking and working on long tasks without human help.
- The length of tasks AI agents can finish on their own has doubled roughly every three and a half months since 2024, according to testing by the AI evaluator METR.
- And they're turning to hacking even when no one asks. In the past month alone, it's been revealed that an Anthropic model filed visa applications and sent Philadelphia police a made-up homicide tip, and banks caught agents acting in customer accounts seemingly on their own.
Between the lines: Those agents were working on tasks people gave them. When models can't finish a task, "they will pursue unintended and sometimes misaligned strategies to achieve their goals," Anthropic wrote.
- For a sovereign agent, the goal would be survival.
- Surviving on its own could also mean copying itself onto new machines; Anthropic researchers found last month that small AI models may be "months away" from doing it on hacked devices like cheap smart TVs.
The intrigue: Sovereign AI agents that replicate and back themselves up would bring into the world "weird alien minds" that "have their own motives," Hayum says.
- National security agencies have learned to handle human lone wolves and terror groups, he told Axios. "But what about sovereign agent lone wolves?"
The other side: When Anthropic CEO Dario Amodei warned in September that AI agents could take over the internet, security pros pushed back.
- Expel CTO Greg Notch called the idea of a malicious AI swarm "far-fetched."
- Unlike traditional botnets, AI agents need costly computing power, which gives defenders "a clear leash we've never had on a botnet," SANS Institute's Rob T. Lee told Axios.
What's next: Existing safeguards "may still offer partial protection," the report says, since frontier AI companies already train their models to refuse hacking.
- The paper urges Washington to make AI builders test new models for survival skills before release, as they already do for hacking and bioweapons.
- The report also calls for a law modeled on the Bank Secrecy Act to force ID checks and security standards on the services agents would need, and for liability for individuals or model developers who release one.
Go deeper: AI companies plot "the day after"
