OpenAI, Google, Microsoft and Anthropic adopt shared AI scraping opt-out

OpenAI, Google, Microsoft and Anthropic say they will honor a new shared signal that lets publishers opt out of having their content scraped for AI training, applying one preference across multiple companies’ crawlers instead of setting it separately with each. The signal, called a “policy token,” was proposed by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force (IETF), the body that sets technical standards for the internet.

The token will sit inside a website’s robots.txt file, the text file that has long told web crawlers which parts of a site they can and cannot visit for search indexing, archiving or AI training. Publishers currently have to tell each AI company separately whether they want their content used for training. The move comes as publishers continue to deal with AI crawlers pulling content from their sites to feed AI models. Other efforts have tried to address the same problem: the Really Simple Licensing (RSL) standard lets publishers charge AI crawlers that visit their sites, and Cloudflare now blocks AI crawlers by default unless a site allows them.

Meta signals support, without a commitment

Meta’s communications head, Andy Stone, also voiced support for the shared signal, though he did not say whether Meta will actually honor it. “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” Stone said, adding that Meta looks forward to working with other AI companies on the standard and its rollout across the industry.

Microsoft says its version of the control will let publishers separately opt out of AI training and of AI grounding, the practice of pointing an AI model to a verified source to cut down on hallucinations. The company says sites will still show up in Bing search results regardless of which preference a publisher picks. Google has already rolled out similar controls that let publishers decide whether their content appears in AI search features or gets used for training, without affecting whether it shows up in regular search results.

The four companies are not moving at the same pace. Anthropic says it has already adopted the signal, while Google and Microsoft plan to support it next year and OpenAI says it will implement the standard within the “coming months.” None of the companies has given an exact date. What remains unclear is whether Meta, or AI companies beyond the four named here, will actually honor the token. Whether publishers come to treat the new control as a real alternative to standards like RSL should become clearer as Google, Microsoft and OpenAI move from stated intent to actual rollout over the next year.