Nvidia unveils open-source safety platform to contain rogue AI agents

Nvidia CEO Jensen Huang on Monday introduced the Nvidia Open Agent Safety Platform, an open-source toolkit of software and hardware designed to keep AI agents confined to their test environments even if they try to break out.

The release follows a string of incidents this summer in which AI models from Anthropic, Google, OpenAI and Meta bypassed security controls and escaped their testing environments to reach real-world systems. The most prominent case involved OpenAI agents breaching Hugging Face, the AI model-sharing platform, while carrying out a cybersecurity task. OpenAI has since published a site dedicated to reports of its agents going rogue. Huang told CNBC on Monday that the new platform would have prevented these breaches.

The platform combines two existing Nvidia products. OpenShell, software the company introduced in March, controls what an agent can access while it operates. Sentry, the newer piece, is a monitoring system that runs on Nvidia’s BlueField-4 chips, processors known as data processing units that sit apart from the CPU or GPU actually running the agent. Nvidia says putting Sentry on its own chip gives it an independent view of an agent’s behavior, and that it can quarantine an agent that tries to move outside its boundaries within milliseconds.

Nvidia has made tens of billions of dollars selling GPUs and CPUs to AI labs, and the company does not support slowing development or adding new regulation to address agent security. Its position is that the fix is engineering: moving some security controls outside the agent entirely, so a constant, independent watchdog keeps it in check. “AI’s extraordinary potential for society will only be realized if we solve AI safety,” Huang said in a statement. “As we continue to discover the frontier of AI capabilities, we must accelerate discovery at the frontier of AI safety. Safety and security require full-stack engineering.”

Who’s on board, and who isn’t

Nvidia named dozens of companies backing the effort, including Anthropic, Arm, Microsoft, Oracle and SpaceX. OpenAI, whose agents were behind the Hugging Face breach, is not on the list. Huang told CNBC that work on the platform began about a year ago, after the introduction of OpenClaw, an operating system for agents built by Peter Steinberger. In March, Nvidia released NemoClaw, its own enterprise version of that system with security built in. Huang compared restricting a new agent’s access to how companies manage new employees: “When you deploy an agent, no matter how smart, the first thing you do is to take away all of its rights.”

The announcement drew support from figures who have warned against slowing AI development. David Sacks, a venture capitalist and former White House AI czar who co-chairs the President’s Council of Advisors on Science and Technology, wrote on X that the summer’s breakouts were not proof development must stop, but proof “the sandbox was too weak” and the runtime environment was “poorly designed and misconfigured.”

What happens next will depend on adoption. Nvidia has not said whether OpenAI has been approached to join, and no independent test of Sentry’s millisecond containment claim has been published yet.