What Changed
On October 30, 2026, OpenAI, Google, Microsoft and Anthropic announced support for one shared robots.txt token that states a preference against AI training: __usage_ai-training. Until now, opting out meant one rule per company, such as GPTBot, Google-Extended and ClaudeBot, and new crawlers kept appearing. The definitions are in AI training preference signal. This guide covers what to do about it.
Step 1: Decide Whether You Want to Opt Out at All
For most brand sites, the answer is no. Training is how a model learns your brand well enough to describe it when it doesn't search, which covers offline and local models, many API calls, and much of what assistants say before they cite anything. Opting out makes most sense when the content itself is what you sell:
| Site type | Typical choice |
|---|---|
| Product, pricing, docs, support pages | Allow training. You want models to know these facts |
| Publishers, paid research, datasets | Consider disallowing, and look at licensing instead |
| Mixed sites | Disallow only the paid or licensed paths |
Step 2: Write the Rule
Microsoft's Bing post gives the syntax. To opt the whole site out:
User-Agent: __usage_ai-training
Disallow: /To opt out one section only:
User-Agent: __usage_ai-training
Disallow: /research/
Disallow: /premium/Training is allowed by default, so you don't need a group at all if you are happy with that. Bing also accepts a page-level version: <meta name="Bingbot" content="ai-training:disallow"> or the matching X-Robots-Tag header, which overrides robots.txt for that page.
Step 3: Keep Your Existing Bot Rules
Don't delete your GPTBot, Google-Extended or ClaudeBot groups. OpenAI says the shared signal will not replace its existing controls, and Google says its company-specific controls keep working alongside it. Google also says that if you use the shared token and have no Google-Extended line, it reads that as disallowing Google-Extended. Make sure the two never contradict each other. A file that allows GPTBot training and disallows __usage_ai-training sends a mixed message nobody has yet said how to resolve.
Step 4: Don't Block Search by Accident
The shared token is about training. It should leave AI search alone, so leave OAI-SearchBot, Claude-SearchBot, ChatGPT-User and Bingbot allowed if you want to be cited. Microsoft says the token does not change eligibility for Bing results. Google has said nothing about Search or AI Overviews. Some coverage claims Google will also treat the token as blocking Gemini grounding. Google's post does not say that, so check Google's documentation before you rely on either reading.
Step 5: Know the Dates
| Company | When it honours the signal |
|---|---|
| Anthropic | Reported as live now (single report) |
| OpenAI | "In the coming months" |
| Microsoft | February 1, 2027, as pages are recrawled |
| "In 2027" |
Adding the rule now does no harm, but don't expect training crawlers to change their behaviour before these dates. The signal is voluntary, and a crawler that ignores it is not technically stopped.
Step 6: Check It in Your Logs
Once a company says it honours the token, compare its training-crawler traffic on disallowed paths before and after. Search crawlers should stay steady. If they drop, a rule is broader than you meant. Track your brand's mention rate in AI answers over the same period, because that is the outcome you care about. For the per-company differences, see how Google, Bing, OpenAI and Anthropic handle the signal.