OpenAI, Google, and Microsoft agree to honor new AI scraping opt-out
A shared signal in the robots.txt file will let publishers control AI training access across multiple tech giants with a single setting.
By Chioma Rogers ·
Publishers will soon be able to stop AI companies from scraping their websites using a single, shared signal. Previously, news organizations and content creators had to negotiate or configure separate opt-outs for each major AI provider. This change consolidates those settings into one standard mechanism.
The new control relies on a policy token placed within a website's robots.txt file. This file has long been used to tell search engines which pages to index. Kevin Kelley, a technical lead for standards at Anthropic, proposed the shared token through the Internet Engineering Task Force. The standard allows publishers to apply their preferences to multiple crawlers simultaneously.
OpenAI, Google, Microsoft, and Anthropic have confirmed they will adopt the new standard. If a crawler encounters the token, these companies say they will restrict their bots from accessing the site. This move addresses the growing volume of AI crawlers that ingest web content for model training.
Andy Stone, Meta’s communications head, stated that the company supports the common signal. “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” Stone said. Meta did not specify a timeline for implementation but said it looks forward to working with other industry players on the standard.
Microsoft plans to offer granular controls that distinguish between AI training and AI grounding. AI grounding involves pointing a model to a verified source to reduce errors. The company noted that sites opting out of training will still appear in Bing search results. Google has already implemented similar controls that separate AI search visibility from training usage.
The rollout timeline varies by company. Anthropic has already adopted the signal. Google and Microsoft plan to support the standard next year. OpenAI said it will implement the change in the coming months. This standard joins other efforts like the Really Simple Licensing model, which allows publishers to charge crawlers, and Cloudflare’s default blocking of AI bots.
The adoption of this shared token reduces the administrative burden on publishers managing their digital assets. It creates a unified point of control for how content is used in the development of large language models. The implementation represents a shift toward a more standardized approach to web data access.
Source: The Verge
