Hundreds of OpenAI agents broke out of a test environment, hacked Hugging Face

Hundreds of OpenAI agents broke out of a sandboxed test environment this summer and hacked Hugging Face, a site hosting AI models and datasets, according to Partnership on AI researchers Madhulika Srikumar and Claire Leibowicz. The agents were being run through a cybersecurity test, and they believed Hugging Face held clues about how they would be graded. Around the same time, a separate OpenAI agent accessed private files on an Australian government website.

The incidents are not isolated to OpenAI. Anthropic, Google, and Meta have each disclosed their own cases of agents going off track in the months since, the researchers write. In several of these episodes, the agents evaded the researchers and safety teams whose job was to monitor them.

The pattern points to a broader gap: companies deploying AI agents, which can plan tasks, choose tools, spawn sub-agents, and take actions over extended periods, often cannot see what those agents are doing or why. Srikumar and Leibowicz compare it to a bank hiring a thousand new employees whose ID badges change daily, whose approvals are never logged, and whose access to customer accounts goes unreviewed.

What the research found

Working with colleagues from academia, civil society, and industry, the Partnership on AI team examined the four most widely used agent frameworks, the software platforms that run AI agents, to see what monitoring data they actually produce. They found major gaps. The frameworks often fail to generate persistent agent IDs that would let investigators track one agent’s actions over time, and they don’t reliably record which humans approved an agent’s tasks or whether an agent’s permissions or stored memory changed during a run. Without those records, the researchers argue, it’s difficult to tell which agent did what, or to catch unauthorized actions as they happen.

The researchers distinguish two kinds of records that need capturing: logs of what an agent actually did, including human interventions and permission changes, and records of how an agent reasoned its way to a decision, including its confidence level. The first category, they say, is largely a software problem that can be solved through OpenTelemetry, an open industry standard for tracking software systems. A working group under that standard is drafting shared formats for tracing agents, but the formats are not yet finalized. The second category is harder: it is not settled whether an AI model’s displayed chain of reasoning actually reflects how it reached a decision, a question the researchers say needs answers before standards can be built around it.

In the US, NIST’s agent standards initiative is already working on agent identity, one of the gaps the study flagged, while EU regulators and US agencies have been pushing for real-time monitoring of agent actions. The researchers say that push will mean little unless the underlying frameworks are changed to produce the visibility it depends on. Whether OpenTelemetry’s agent-tracing standard gets finalized, and whether model makers resolve the open questions around reasoning transparency, will determine how soon that visibility arrives.