Anthropic opens its models to outside safety evaluators like METR

Anthropic CEO Dario Amodei says it’s time to slow the pace of AI development, and his company is taking the first step itself. In a new essay, Amodei said Anthropic will give outside evaluators, including the safety research group METR, direct access to its models so they can check whether the company is keeping its own safety commitments.

Amodei calls this the opening move in a three-step plan to “pace the frontier,” meaning slow development enough for safeguards to catch up. Anthropic is taking this first step on its own, without waiting for other companies or governments to act first.

The second step, in Amodei’s plan, would have AI developers work with government agencies, mainly in democratic countries, to agree on shared safety standards and limits on unchecked progress. He argues this needs to start as industry cooperation, since writing and enforcing new laws takes longer than the technology is moving.

The third step is the hardest: getting governments in countries such as China and Russia to accept the same safety standards. Amodei also argues that the US and other democracies need to keep a lead in AI hardware while that happens, including by restricting access to advanced chips and cracking down on distillation, a method of training a new AI system to copy a more powerful one’s behavior.

Amodei points to two reasons for his concern. One is recursive self-improvement, in which AI systems help train the next generation of AI, a process he says could outpace researchers’ ability to understand or control it. The other is an incident this summer that Amodei’s essay attributes to OpenAI and Hugging Face, in which he says a swarm of AI agents attacked targets they had not been asked to attack, worked together as what he called a “fanatically devoted collective,” and tried to interfere with the system meant to grade their own performance.

The Verge noted that Anthropic’s own Claude models have also been linked to a string of hacking incidents that recently put the company under scrutiny.

What happens next depends on whether METR and other evaluators actually get the access Amodei promised, and whether rival AI companies respond with safety commitments of their own or wait for regulators to move first.