The Opt-Out Pivot: Big AI's New Standard for Scraping
AI-generated image · US National Wire
OpenAI, Google, Microsoft, and Anthropic are backing a shared 'policy token' to let publishers opt out of AI training—a move that shifts the burden of protection onto the creators.
In the creator economy, the battle over data ownership is moving toward a standardized 'opt-out' model. As first reported by The Verge, AI giants OpenAI, Google, Microsoft, and Anthropic have agreed to adopt a new standard that allows publishers to signal their AI training preferences across multiple crawlers simultaneously.
**The Technical Shift** The proposal comes from Kevin Kelley, a technical lead for standards at Anthropic, via the Internet Engineering Task Force (IETF). The mechanism involves a shared “policy token” located within a website’s robots.txt file. Previously, publishers had to signal their preferences to each AI company individually; this new standard allows a single signal to communicate preferences to multiple providers.
**The Corporate Rollout** While the companies are aligned on the concept, the timeline for implementation varies: * **Anthropic** has already adopted the signal. * **OpenAI** expects to implement the standard in the “coming months.” * **Google** and **Microsoft** plan to support the token next year.
Meta's communications head, Andy Stone, expressed support for creating a common signal, though he did not explicitly confirm if Meta would honor the setting.
**The Fine Print on Control** Microsoft noted that it will provide granular options, allowing publishers to opt out of either AI training or “AI grounding”—the process of using verified sources to minimize hallucinations. Microsoft clarified that these preferences will not affect a site's visibility in Bing search results. Google has similarly implemented controls that separate AI training and search feature visibility from general search indexing.
**The Broader Context** This pivot arrives as publishers seek ways to combat unauthorized scraping. The Verge notes other industry attempts to reclaim value, such as the Really Simple Licensing (RSL) standard, which enables publishers to charge crawlers, and Cloudflare's decision to block AI crawlers by default.
*Opinion: By shifting to an opt-out standard, AI platforms are effectively maintaining the status quo of mass ingestion while offering a veneer of partnership. Rather than asking for permission to use intellectual property, the burden is placed on the publisher to manually lock the door.*