What one robots.txt line actually does for AI training

On October 30, 2026, Google, Microsoft, OpenAI, and Anthropic each said they will honor one shared line in robots.txt for AI training. Write one group for the token __usage_ai-training, and every participating provider is supposed to read it as your training preference.

That sounds like a simple opt-out. It is not quite that. The token only covers training. It does not block crawling. Your existing User-agent: * rules do not count. And at Google, opting out of training through this token can also take your pages out of Gemini answers that cite the web.

This is a summary of what the four companies and the IETF draft actually say, and what to decide for your own site. Timelines and product behavior below are each company's own statements. None of this is live everywhere yet.

What changed?

There is now one shared signal for AI training preferences, instead of one rule per vendor.

Until now, a site that wanted to opt out of AI training had to name each company's crawler or control token separately. OpenAI's post says operators had to "identify individual crawlers and maintain separate rules for each and every provider." The list kept growing.

The new mechanism comes from an IETF Internet-Draft, draft-kelley-aipref-training-preference-signal-00, written by Kevin Kelley of Anthropic and published on October 30. It builds on the AI Preferences (AIPREF) working group's definition of AI training: "Using an asset to modify the learned parameters of a generative AI model."

Two limits are worth knowing up front:

• It is an individual draft. The datatracker page says it "is not endorsed by the IETF and has no formal standing in the IETF standards process." It expires on May 3, 2027 unless revised.
• Participation is voluntary. OpenAI calls it a "voluntary effort."

What does the new token look like?

It is an ordinary robots.txt group. The only new thing is the name on the User-Agent line. This is the draft's own example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

According to the draft, this says no to AI training for every path except /press-kit/, where it says yes. Path matching and longest-match rules work exactly as they do in RFC 9309, the existing robots.txt standard. No new directives, no new grammar.

The draft calls this a "policy token." It names a purpose, not a crawler. No bot sends __usage_ai-training in its HTTP requests. The double underscore at the front is there so older parsers that match loosely are unlikely to confuse it with a real bot name.

Why doesn't "Disallow all bots" count as an AI preference?

Because the draft says so directly. Section 4.2: "wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token."

It goes further. If no group in your file names __usage_ai-training, then "no preference has been expressed."

So a site with this:

User-agent: *
Disallow: /

has blocked crawling for bots that respect it. Under the draft, it has said nothing about AI training. The same applies in reverse. A vendor-specific rule, such as a Google-Extended block, keeps its existing meaning with that vendor, but the draft says consumers "MUST NOT treat it as expressing a preference" under this shared signal.

If you want a training preference that every participating provider reads, you have to write the group with the token name.

Is this a crawl block?

No. The draft says: "This policy token is not a directive for crawling behavior."

Usage preferences and crawl rules are "independent and orthogonal," in the draft's words. A crawler still checks the rules for its own name to decide what it may fetch. The token only states what you want done with the content after it is fetched.

The draft allows, but does not require, a provider to skip fetching pages it cannot use. If you actually want a bot to stay away, you still need a group for that bot's own name.

Microsoft says the same thing about its version: "These settings govern use of content, not crawl access."

When does each company start honoring it?

The dates are not the same, and only one is fixed.

Anthropic's commitment came from Kelley's message to the IETF mailing list, as reported by Engadget and PPC Land, not from a dated rollout notice.

Meta, Amazon, and Apple are not among the companies that committed on October 30.

How does Google handle the token?

Google maps it onto Google-Extended, the control it has offered since 2023.

From Google's post: "When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended."

That matters because Google-Extended covers more than training. Google's crawler documentation says it controls whether content may be used for training future Gemini models and "for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI."

Put those two documents together. A site that disallows the shared token and says nothing about Google-Extended would, under Google's stated plan, also be out of Gemini grounding. Google's post states the other side: "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."

Google says Google-Extended does not affect inclusion or ranking in Google Search.

How does Microsoft handle it?

Microsoft separates training from grounding.

• AI Training uses the shared robots.txt token for a site or path. Microsoft also adds a page-level override through a meta tag or HTTP header.
• AI Grounding is a separate, page-level preference that is Microsoft's own. It has three values: Allow (the default), Disallow, and a new DisallowExtended.

DisallowExtended keeps your content in Microsoft-operated experiences, which Microsoft lists as AI Mode, AI Overview, Copilot, and Microsoft 365, but out of grounding services Microsoft provides to third parties, such as Web IQ and grounding through API access in Foundry.

Microsoft's own example is a site that disallows training and allows grounding. Microsoft also warns that disallowing grounding "will significantly reduce your content's visibility in AI-generated experiences," and adds: "Neither choice guarantees inclusion, traffic, or payment."

There is a migration deadline. Microsoft says that after March 1, 2027, NoArchive and NoCache "will no longer be read as a preference for training or grounding." If it detects the new preferences on a site, it will disregard any remaining NoArchive and NoCache there.

What is the builder catch?

This is my read, not something the companies say.

The launch is framed as publishers getting control over training. For anyone building a product or publishing docs, the more useful split is training versus grounding.

• Training shapes what future models know.
• Grounding decides whether your page is pulled in, cited, and linked in an AI answer today.

Microsoft lets you set those separately. Google ties them together through Google-Extended. So one blanket line, __usage_ai-training with Disallow: /, means different things at each company. At Microsoft, it stops training and leaves grounding alone. At Google, if you have no Google-Extended group, it stops training and Gemini grounding.

There is no single Google setting for "do not train on my pages, but do cite them in Gemini." Allowing Google-Extended keeps grounding, but it also allows Gemini training. That is a tradeoff you should make on purpose, not by accident.

What does this not do?

• It does not undo past collection. As Engadget notes, if a page was already crawled, the company already has it. The draft ties a preference to the robots.txt in effect when the content was retrieved.
• It is not enforced. It is a stated preference. PPC Land cites TollBit data finding that about 15% of AI page fetchers in Europe reached URLs sites had disallowed.
• It does not change search. Google and Microsoft both say search eligibility is unaffected.
• It is not a license. The draft says private agreements may override it, and Microsoft says the preferences "do not establish licensing or payment arrangements."
• It is not final. The draft can change. OpenAI and Anthropic both say their handling may change as the standard evolves.

What should you decide for your own site?

Make two decisions, not one.

1. Training. Do you want your content used to train future generative models? If not, add a __usage_ai-training group with Disallow. Use Allow paths for anything you are happy to share, like docs or a press kit.
2. Grounding. Do you want your pages cited in AI answers now? At Google, that means deciding whether to add an explicit Google-Extended group, knowing it also covers training. At Microsoft, set page-level grounding preferences.

Then check three things:

• Do not rely on User-agent: *. It does not express a training preference under this draft.
• If you use NoArchive or NoCache for Bing, move to the new preferences before March 1, 2027.
• Keep your vendor-specific rules. Google, OpenAI, and Anthropic say their existing controls, such as Google-Extended and their crawler rules, still apply. The shared token adds to them. It does not replace them.

For a docs site or a product's marketing pages, being cited in answers often matters more than staying out of training. For paid or original content, the answer may be different. Either way, make the choice on purpose.

Source: Supporting a shared AI training control for websites, Google Search Central. See also the IETF draft and Microsoft's training and grounding post.