One robots.txt rule to block AI training: how the new __usage_ai-training signal works
Google, OpenAI, Microsoft and Anthropic will support one shared robots.txt signal, __usage_ai-training, to opt out of AI training. Here's the code, the timeline and what site owners should do now.
If you run a website, blocking AI companies from training on your content is about to get much simpler. Google, OpenAI, Microsoft and Anthropic have agreed to support one shared robots.txt signal, called __usage_ai-training, that tells AI companies whether they may use your content to train their models. Instead of writing separate rules for every AI crawler, you’ll be able to use one. Here’s how it works, when each company will support it, and what to add to your robots.txt today.
What was announced?
On October 30, 2026, the four companies said they will honor a shared AI training preference in robots.txt, according to Coywolf and H2S Media. The idea is described in an IETF Internet-Draft written by Anthropic’s Kevin Kelley.
Until now, website owners had to block each company separately, using names like GPTBot (OpenAI), Google-Extended (Google) and ClaudeBot (Anthropic). Every new AI company meant another line to add.
How to use it: the robots.txt code
Your robots.txt file sits at the root of your site (for example, yoursite.com/robots.txt). To block AI training on your whole site, add:
User-Agent: __usage_ai-training
Disallow: /You can also allow some parts. This example blocks training everywhere except a press kit folder:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Important: the wildcard User-Agent: * does not count as an AI training preference. You must name __usage_ai-training directly.
The new token controls how your content is used for AI training, not whether search engines can crawl and index your site.
Will it hurt my Google or Bing rankings?
No, according to the reports. The token is about training, not crawling. It does not block Googlebot or Bingbot from indexing your pages for normal search.
This is a big change for Bing: until now, allowing Bing to train on your site was tied to appearing in its search results. The new signal gives site owners a training opt-out without leaving Bing search.
When will each company support it?
- Anthropic: has started supporting it, but full coverage across all its services will take time.
- OpenAI: says it will honor the signal “in the coming months.” GPTBot rules still apply.
- Microsoft: starts honoring it on February 1, 2027, as Bingbot recrawls each site.
- Google: plans to support it in early 2027. If you use the shared token but don’t mention Google-Extended, Google will treat it as a Google-Extended block.
What should website owners do now?
Because support isn’t live everywhere yet, use both the new token and the old ones for now. A robots.txt that blocks AI training today could look like this:
# Shared AI training opt-out
User-Agent: __usage_ai-training
Disallow: /
# Company-specific rules (keep until each company confirms support)
User-Agent: GPTBot
Disallow: /
User-Agent: Google-Extended
Disallow: /
User-Agent: ClaudeBot
Disallow: /
# Normal search crawlers
User-Agent: *
Allow: /After editing, open yoursite.com/robots.txt in your browser to check the new rules are showing. On WordPress, SEO plugins like Rank Math and Yoast let you edit robots.txt without touching files.
Microsoft’s extra “AI grounding” control
Microsoft is also adding a page-level AI Grounding preference, set with a meta tag or X-Robots-Tag header. It controls whether your pages can be used to answer questions in AI tools. It has three settings: Allow (default), DisallowExtended, and Disallow. According to H2S Media:
- NoArchive and NoCache will stop working as AI preferences after March 1, 2027.
- A temporary grounding setting in Bing Webmaster Tools opens on December 15, 2026.
Limits you should know
- It’s voluntary. Robots.txt is a request, not a lock. Companies must choose to respect it. Bots that ignore robots.txt need a service like Cloudflare’s bot protection.
- Not every AI use is covered. Models used only for classification or ranking fall outside the training definition.
- It’s not a final standard yet. Other ideas, like a Content-Usage HTTP header and Cloudflare’s Content-Signal directive, are also being discussed. The IETF draft expires in May 2027 unless updated.
Should you block AI training?
It depends on your goals. Blocking protects your content from being used to train models. But AI search tools are becoming a bigger source of visitors, and some site owners want to stay visible there. Remember, this token is about training, so blocking it shouldn’t remove you from search. For more on how AI is changing website traffic, read our article on Reddit, Google AI Mode and the traffic drop.
Frequently asked questions
What is __usage_ai-training?
It’s a shared robots.txt user-agent token that tells AI companies whether your content can be used to train AI models. Google, OpenAI, Microsoft and Anthropic have said they will support it.
Does blocking AI training remove my site from Google search?
No. The token covers AI training only. It doesn’t block Googlebot or Bingbot from crawling and indexing your site for search.
When does the new robots.txt AI standard start?
Anthropic has begun supporting it, OpenAI plans to “in the coming months,” Microsoft starts on February 1, 2027, and Google plans to support it in early 2027.
Should I remove my GPTBot and Google-Extended rules?
Not yet. Keep your company-specific rules until each company confirms its support for the shared token is live.
Disclosure: AI tools, including Claude, helped write this article.
