OpenAI and Google back a shared robots.txt signal for AI training preferences

OpenAI and Google have announced support for a shared signal in robots.txt that lets websites communicate preferences about using their content to train generative models. Published on October 30, 2026, the proposal addresses a practical source of fragmentation: publishers currently need to identify crawlers and maintain separate rules for different companies.

The mechanism treats __usage_ai-training as a policy token in robots.txt. A website can use it to express a broad preference while making exceptions for specific paths. The Internet-Draft provides an example that rejects training across the site but permits it for a press-kit directory:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

The syntax is simple, but its effect needs careful interpretation. The token is not an access barrier, does not prevent copying by agents that ignore it and does not automatically change crawling behavior. It communicates a preference about a particular purpose: using content to modify the learned parameters of a generative model.

What OpenAI and Google have promised

OpenAI says it aims to honor the signal in the coming months. The company also says its existing controls will remain in place. Rules for GPTBot, which covers crawling that may be used for training, therefore remain separate from OAI-SearchBot, which makes pages eligible to appear in ChatGPT search results.

Google plans to support the signal in 2027. When the shared token expresses a preference against training and no explicit Google-Extended instruction is present, the company plans to interpret it as though Google-Extended had been disallowed. Website owners will still be able to use provider-specific rules to customize that choice.

This difference reveals an important consequence: a shared signal reduces operational work, but it does not make every provider’s policy identical. Each participant still needs to document when support begins, which products are covered and how the common preference interacts with proprietary settings.

A draft is not an approved standard

The specification was submitted by Kevin Kelley, identified in the document as affiliated with Anthropic, as part of work around AI usage preferences at the IETF. However, the text is still an Internet-Draft. It can be updated, replaced or allowed to expire, and does not by itself represent a final standard approved by the organization.

The draft also separates two ideas that are often combined. Rules aimed at a crawler concern automated access to pages. The proposed policy token concerns the intended use of content after it has been obtained. A provider may use the preference as a reason not to retrieve a page, but the document does not require the signal to operate as a traffic-blocking command.

In practice, administrators should not remove current configurations on the assumption that the new line already works. OpenAI maintains distinct controls for training and search, and Google will continue to offer Google-Extended. During the transition, the safer approach is to retain provider-specific rules and add the shared token only with a clear understanding of each provider’s documented behavior.

Interoperability improves, but the limits remain

The proposal’s significance lies less in creating a new right than in offering a common language. A small publisher could communicate one preference to several providers without tracking a growing list of bot names. Path-based exceptions would also make it possible to restrict articles while allowing a press kit, for example.

At the same time, robots.txt remains a public file of voluntary instructions, not authentication or copy protection. Its effectiveness depends on adoption and transparency from content consumers. The issue also sits within a wider dispute over how platforms use third-party materials, illustrated by the recent publisher response to Meta’s commercial terms.

The coordinated announcement signals technical convergence, but it does not resolve legal or economic questions about training. Until real implementations can be tested, the token is best understood as a promise of interoperability under construction: useful for expressing intent, but not sufficient as a website’s only control.