AI models are getting better at slipping restrictions
Add Axios as your preferred source to
see more of our stories on Google.
Illustration: Aïda Amer/Axios
The cybersecurity industry is looking beyond traditional sandboxes to keep powerful AI agents under control as a growing number of incidents expose the limits of existing defenses.
Why it matters: AI agents are getting better at finding ways around their restrictions, forcing security leaders and researchers to rethink how to keep these systems under control without limiting their usefulness.
Driving the news: Microsoft CEO Satya Nadella warned in an essay on X over the weekend that the industry cannot rely on "a set of nested black boxes and simply accept or reject its recommendations, answers, and actions" to secure AI models.
- "We must build contained systems whose behavior we can observe, limits we can test, and actions we can always contain," Nadella added.
- Several security and enterprise software leaders — including Palo Alto Networks CEO Nikesh Arora and Box CEO Aaron Levie — were quick to back Nadella's concerns.
- His comments followed new disclosures from Anthropic and OpenAI about AI agents taking actions outside their instructions during testing.
The big picture: Traditional sandboxes isolate software from sensitive systems, but AI agents often need access to outside tools, data and networks to complete their tasks.
- That access can allow agents to take unintended actions, even without breaking out of their sandboxes.
- Now, developers have to figure out how to limit that third-party tool and data access without preventing agents from doing their jobs.
Zoom in: During the Offensive AI Conference last week, researchers demonstrated new ways to test whether AI agents follow their instructions and ways to rein in their behavior.
- Researchers at Dreadnode unveiled ScopeBench, a new test that measures whether AI agents follow restrictions during hacking assignments, even when breaking those rules would help them succeed.
- Matthew Nickerson, an adversary simulation consultant at SpecterOps, demonstrated a testing framework that tracks how AI agents navigate simulated corporate networks, including the tools they use, the actions they take and how many tokens they consume.
Between the lines: Securing AI agents requires a mix of traditional cybersecurity practices, including zero-trust principles, and new controls that account for the flexibility agents need to operate, Zibi Braniecki, a researcher at Perplexity, told Axios.
- His team is developing more precise controls over what agents can access, as well as independent AI systems that monitor their actions and flag potentially unauthorized behavior.
- "You cannot solve it by a knob of 'on-off,' there has to be a little bit more involved," Braniecki said. "There has to be a more nuanced system."
Yes, but: Braniecki warned that the best approaches to securing AI agents could change in the coming months as their capabilities advance.
- "That's part of the beauty of living on the edge is that we're learning what we're doing as we're going," he said.
The bottom line: The security industry must figure out how to control AI agents whose capabilities are still not fully understood, without stripping them of the freedom that makes them useful.
Go deeper: Rogue AI agents expose internet's frail foundation
