Quick Answer
OpenAI, Google, and Microsoft now support a shared robots.txt preference for AI training, but the control is not yet a finished standard and implementation dates differ. Websites can use User-Agent: __usage_ai-training with Disallow: / to express a whole-site opt-out. Keep existing crawler-specific rules, review permitted paths, and confirm each company’s rollout timetable before changing a publishing policy.
Key Takeaways
- The shared AI training opt-out signal uses
User-Agent: __usage_ai-training. - A
Disallow: /rule expresses a preference against whole-site foundational AI training. - The proposal supports path-specific preferences, including permitted directories such as
/press-kit/. - Google plans support in 2027, while Microsoft plans to honor the preferences beginning February 1, 2027.
- The Internet-Draft is not yet a completed IETF standard, so existing crawler-specific rules still matter.
What changed with the AI training opt-out rule?
The AI training opt-out rule gives website operators one shared robots.txt signal for expressing whether their content can be used for foundational generative AI training. OpenAI said on October 30, 2026, that it will honor the common signal, which is designed to reduce the need for publishers to maintain separate directives for every AI company. OpenAI’s announcement says the company aims to implement the preference in the coming months.
The new approach matters because website owners have faced a fragmented set of crawler-specific controls. A single shared preference can make a publishing policy easier to express, audit, and explain to staff responsible for editorial, legal, and technical decisions. The control does not force every AI developer to follow the signal, however, and it does not automatically replace existing robots.txt entries.
Website operators should treat the new directive as an additional policy tool rather than a complete enforcement mechanism. The practical response is to keep current crawler rules in place while tracking implementation details from each AI company that says it will recognize the shared preference.
How does the shared AI training preference work?
The shared AI training preference uses the token User-Agent: __usage_ai-training in a website’s robots.txt file. A site can pair that token with Disallow: / to express a whole-site preference against use of its content for AI training. The IETF Internet-Draft defines the proposed syntax and describes the signal as a preference for foundational generative-AI training.
The proposal also supports path-specific choices. A publisher could disallow training on most of a domain while allowing a directory such as /press-kit/, which may contain material intended for broad reuse. That flexibility matters because many organizations want different rules for reporting, support documentation, public announcements, and media assets.
The shared signal communicates a website owner’s stated preference, but robots.txt is not a technical barrier that prevents copying by itself. Website operators that need stronger access restrictions should evaluate separate technical, contractual, and legal controls with their hosting, security, and legal teams.
| Website policy goal | Shared robots.txt preference | Practical meaning |
|---|---|---|
| Block AI training across the domain | User-Agent: __usage_ai-trainingDisallow: / | Expresses a whole-site preference against foundational AI training. |
| Allow a limited public directory | Use path-specific rules for the permitted directory | Lets a site distinguish general content from material such as a press kit. |
| Maintain existing company-specific preferences | Keep current crawler entries | Preserves controls while company support for the shared signal rolls out. |
How does the shared signal differ from existing crawler rules?
The shared AI training opt-out signal is intended to complement, not immediately replace, crawler-specific robots.txt controls. OpenAI said its current crawler-specific controls will remain in place even as it works toward supporting the shared token. That limitation is important because companies may have different implementation dates and may interpret separate training and grounding preferences differently.
Existing crawler rules remain relevant for publishers that have already set policies for individual services. The distinction is especially important as AI tools become connected to more services and user data, including products that expand AI access across connected apps. A publisher’s web-crawling policy and an individual user’s connected-account permissions are separate issues, even though both involve control over data use.
Website operators should review robots.txt entries before deleting or consolidating any existing directives. The most sensible approach is to document the organization’s desired training and grounding policy, preserve current controls, and add the shared signal only after the technical team confirms that the syntax matches the intended scope.
When will OpenAI, Google, and Microsoft honor the preference?
OpenAI, Google, and Microsoft have announced different timelines for recognizing the shared AI training preference. OpenAI said it aims to implement the signal in the coming months, while Google said it plans to support the shared control in 2027. Microsoft said it will begin honoring the new AI Training and AI Grounding preferences when it crawls pages starting February 1, 2027.
Google’s planned handling includes an important default. Google Search Central says that when a site opts out through the shared token and does not explicitly name Google-Extended, Google plans to treat the shared opt-out as a Google-Extended disallow. That gives publishers a clearer default for Google’s AI training crawler when Google’s support arrives.
Different rollout dates mean a robots.txt update cannot produce an identical result across every AI company on the same day. Website owners should record the date of any policy change, keep a copy of the prior robots.txt file, and verify each company’s published behavior before treating the preference as fully operational.
What does the AI training preference mean for grounding?
The AI training preference addresses foundational model training, while grounding can involve using website content to answer a user’s current question in an AI product. Google said sites that explicitly allow Google-Extended remain eligible to appear in Gemini Apps grounding responses and receive clicks from those responses. That distinction matters because a publisher may want to limit training while still allowing current content to be surfaced to users.
Microsoft also separates training from grounding. Microsoft Bing’s implementation guide introduces a DisallowExtended grounding preference that permits use in Microsoft-operated AI experiences, including Copilot and Microsoft 365, while excluding third-party grounding services.
Grounding choices require a clear business decision because visibility and data-use preferences can point in different directions. Publishers that rely on discovery traffic may decide to permit selected grounding uses, while publishers handling sensitive, paid, or highly controlled material may prefer more restrictive rules. A website owner should involve editorial, product, and legal stakeholders before applying a broad policy.
Why is the AI training opt-out not yet a final standard?
The AI training opt-out specification is an individual Internet-Draft, not a completed IETF standard. The draft is authored by Kevin Kelley of Anthropic, has intended Standards Track status, and expires on May 3, 2027. Those details mean the proposal has a defined technical process behind it, but the exact specification can still change before any final standard is published.
The proposal’s incomplete status matters because organizations should avoid treating the current syntax as permanent or universally adopted. A shared directive becomes more useful as more crawlers honor it consistently, but the stated support of three major companies does not establish universal compliance across the AI industry.
Website operators should monitor updates to the draft and implementation notices from the companies they care about. Publishers also need to distinguish policy signals from broader AI governance questions, especially as regulators examine consumer harms and data practices in AI systems, including federal consumer-harm inquiries.
What should website owners do next?
Website owners should first decide whether they want to permit, restrict, or selectively allow foundational AI training on public content. The decision should cover specific content types, including reporting, product pages, documentation, images, media kits, and subscriber-only material. A written policy prevents an accidental robots.txt update from making a broader choice than the organization intended.
- Review the current robots.txt file and save a dated backup before making changes.
- List the existing crawler-specific directives and identify which services already have separate rules.
- Add the shared
__usage_ai-trainingpreference only if it reflects the organization’s documented policy. - Use path-specific rules when different areas of the site need different treatment.
- Check OpenAI, Google, and Microsoft implementation updates before removing any existing crawler-specific directive.
Website owners should stop and consult a web administrator or legal adviser if the site has complex redirects, multiple subdomains, licensed archives, membership content, or contractual content-sharing obligations. A robots.txt change can affect a public content policy, so the technical edit should not be treated as a routine marketing task. Organizations publishing AI-generated media should also keep separate records of their content labeling policy, particularly as services begin adding labels for generative AI content.
FAQ
What is the new AI training opt-out signal?
The new AI training opt-out signal is User-Agent: __usage_ai-training in robots.txt. A website can pair the token with Disallow: / to express a whole-site preference against foundational generative-AI training.
Does the shared robots.txt rule block all AI use of a website?
No, the shared robots.txt rule expresses a training preference rather than blocking every possible AI use. Training and grounding can be controlled differently, and individual companies may have separate implementation rules.
When will Google support the AI training opt-out?
Google plans to support the shared AI training control in 2027. Google says a shared opt-out without an explicit Google-Extended rule will be treated as a Google-Extended disallow.
When will Microsoft honor the AI training and grounding preferences?
Microsoft plans to begin honoring the new AI Training and AI Grounding preferences when it crawls pages starting February 1, 2027. Microsoft also provides a separate grounding preference for Microsoft-operated AI experiences and third-party services.
Should a website remove its existing crawler-specific rules?
No, website owners should keep existing crawler-specific rules while the shared preference rolls out. OpenAI says its current crawler-specific controls will remain in place, and support timelines differ across companies.