Google plans to honour a shared AI training opt-out in robots.txt from 2027

Google said on 30 October 2026 that it plans to honour a shared, cross-company AI training opt-out written in robots.txt, and that when a site uses it to opt out (without naming Google-Extended separately) Google will treat it as if the site had disallowed Google-Extended. The announcement, posted by Google Search Advocate John Mueller on the Google Search Central blog, gives 2027 as the target year and says Google aims to support the control "along with other AI providers".

The signal builds on work in the AI Preferences (AIPREF) working group at the Internet Engineering Task Force (IETF). It is a new robots.txt "policy token", __usage_ai-training, that any participating AI company can read. Below is what the token is, how a robots.txt file expresses it, how Google says it will map it onto its own controls, and what the announcement left unsaid.

What Google announced on 30 October 2026

The Search Central post is short. Its main points, in Google's own framing:

• Google plans to support "a shared AI training control in robots.txt" so site owners can state AI training preferences across multiple AI providers in one step.
• The control is based on work from the AIPREF working group at the IETF.
• Instead of separate instructions for each AI developer, site owners will be able to use a shared "policy token" inside robots.txt to set broad preferences across participating providers.
• If a site uses the token to opt out of training and Google-Extended is not explicitly specified, Google "plans to interpret that as if the publisher had disallowed Google-Extended".
• Company-specific controls keep working alongside the joint signal, and "websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps".
• Google calls it a "draft proposal" that reflects feedback from publishers, civil society groups, regulators and other stakeholders, and says it will keep working on standardisation through the IETF working group.
• Google says the proposal is consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act, which recognises robots.txt as a primary way for publishers to express machine-readable preferences.

On timing, the blog says Google aims to support the control in 2027 and will post progress updates. A note added to Google's crawler documentation for Google-Extended is a little more specific: it says Google intends to support the AIPREF signal, written as "user-agent: __usage_ai-training", in early 2027. Neither page gives a day or month.

What the __usage_ai-training token is

The syntax is defined in an IETF Internet-Draft, draft-kelley-aipref-training-preference-signal-00, dated 30 October 2026 and written by Kevin Kelley of Anthropic. The Google post links to it. Like every Internet-Draft it is a work in progress, and this version expires on 3 May 2027 unless it is updated or replaced.

The draft's core idea is that a site owner writes an ordinary robots.txt group whose User-Agent line names a purpose rather than a crawler. It introduces no new directive names and no new grammar. The path syntax, the longest-match rule and the group structure of RFC 9309, the robots.txt standard, all stay as they are. The draft's own example is three lines:

User-Agent: __usage_ai-training

Disallow: /

Allow: /press-kit/

According to the draft, that file expresses a preference against AI training for every path except those under /press-kit/, where it expresses a preference in favour of training. A site-wide opt-out is just the first two lines.

The meaning of "AI training" is taken from the AIPREF vocabulary draft (draft-ietf-aipref-vocab-08). It defines AI training as "using an asset to modify the learned parameters of a generative AI model". Training models used only for non-generative tasks is excluded, even when the model could generate content. A model used for classification, ranking or scoring is not counted as a generative AI model, even if it writes out a rationale for its output.

How a robots.txt lookup works under the draft

The Kelley draft spells out several rules that decide what a given file actually says. These matter more than the headline, because they decide whether an existing robots.txt already expresses a preference.

• The lookup uses normal RFC 9309 path matching. A disallowed path means a preference against AI training for that path. An allowed path means a preference in favour.
• Wildcard groups do not count. The draft says "User-Agent: *" MUST NOT be read as matching the policy token, so a blanket rule written for all crawlers says nothing about training.
• If no group in the file names __usage_ai-training, no preference has been expressed. Kelley's announcement email on the IETF ai-control list says this is true of every robots.txt published today.
• Consumers must match the token exactly, ignoring case. The double-underscore prefix is there to make accidental matches unlikely.
• One group can name several policy tokens, and policy tokens can be mixed with ordinary crawler names in the same group.
• The robots.txt file in effect when an asset was retrieved is the one whose preferences apply to it.

Usage preference, not crawl control

The draft keeps two things separate. Rules for a crawler's own name still decide whether that crawler may fetch a page, "exactly as they would in the absence of this document". The new token says how fetched content may be used. The draft says plainly that the token "is not a directive for crawling behavior". A company that crawls only for training MAY skip pages where a preference against training has been expressed, but nothing obliges it to. A site that wants to stop fetching still has to block crawlers by name.

The draft also sets out how it sits next to existing company controls. Specific expressions SHOULD take precedence over general ones, and a company MAY document that it treats its own proprietary token as more specific than the shared one. Private agreements MAY override preferences expressed this way. Google's plan follows that pattern: an explicit Google-Extended rule wins, and the shared token fills in only when Google-Extended is absent.

How Google will map the token onto Google-Extended

Google has offered Google-Extended as its AI training control since 2023. Its crawler documentation describes it as a standalone product token, with no separate HTTP user agent, that manages whether content Google crawls may be used to train future generations of Gemini models that power Gemini Apps and the Vertex AI API for Gemini. The same page says it also governs grounding in Gemini Apps and in Grounding with Google Search on Vertex AI, and that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search".

The table below sets out the robots.txt states the announcement and the draft describe, and what each one means. The right-hand column is taken from Google's post and documentation.

Analysis, not a Google statement: read together, the post and the crawler page mean a shared opt-out with no Google-Extended line reaches further than "training" might suggest. Google-Extended covers Gemini grounding as well as training, so a site that opts out through the shared token alone would, on Google's stated mapping, also step out of grounding in Gemini Apps. That may be why the post states explicitly sites that explicitly allow Google-Extended stay eligible for grounding and clicks there.

What the announcement did not say

Several questions a site owner will ask are not answered by the Search Central post. Each item below is a gap in the source, not a hint about Google's plans.

• AI Overviews and AI Mode are not mentioned. The post speaks only about training and about grounding in Gemini Apps. The Google-Extended documentation says the token does not affect inclusion in Google Search, but neither page addresses AI Overviews or AI Mode by name.
• The post does not say how Google will read an explicit Allow on the shared token, or a path-level mix of Allow and Disallow, beyond the general opt-out case.
• There is no launch date beyond "2027" in the blog and "early 2027" in the documentation.
• Nothing is said about reporting, such as whether Search Console will show which preference Google detected.
• The post does not name the "other AI providers" it expects to support the control.

Who else is behind the shared signal

Two other companies made public moves on 30 October 2026. Microsoft Bing announced on the same day that it is adopting the shared AI training preference and adding a separate page-level grounding control of its own; RA covers that in a companion article on Bing's AI grounding controls. In the email announcing the draft, Kelley wrote that Anthropic "has already started implementing support for this draft".

The draft's acknowledgements thank working group participants affiliated with Google, Microsoft, OpenAI, Meta, Cloudflare, Common Crawl, the BBC, the Financial Times, News Corp and others. The draft notes that affiliations are listed "for identification only, since IETF participants contribute as individuals", so the list shows who took part in the discussion, not which companies have committed to honour the token.

What a site owner should check in robots.txt

This section is RA analysis built on the sources above. Nothing needs to change today, since Google's support is planned for 2027, but the audit is quick.

• Find out whether the site already has Google-Extended rules. If it does, they keep working, and under Google's stated precedence they override the shared token for Google.
• Decide the training stance and the Gemini grounding stance separately. If the goal is to opt out of training across providers while staying citable in Gemini Apps, the file needs both a __usage_ai-training Disallow group and an explicit Google-Extended Allow.
• Do not count on "User-Agent: *" rules. Under the draft they say nothing about training, however strict they are.
• Keep crawl blocking and usage preferences apart. Blocking Googlebot to stop training would also take pages out of Search; the shared token is designed so that you do not have to.
• Check file size. The draft notes robots.txt files are size-limited under RFC 9309 and that the token was designed to avoid duplicating existing rules, so adding one short group should be enough.
• Remember the status. Google's own post calls this a draft proposal, the Internet-Draft is a work in progress, and the draft's author wrote in his announcement email that Anthropic's interpretation of these signals may change as standards evolve.

Brands that track whether their pages are cited in AI answers can fold this into the same review. RA's generative engine optimisation (GEO) service treats crawler and usage controls as part of AI visibility.

What this means for Thai marketers

The sources say nothing about Thailand. The mechanism is global because it lives in each site's robots.txt, so it applies to a .co.th or .th site the same way it applies anywhere else. For Thai publishers, retailers and brand sites that want to appear in Gemini answers, the practical point is the Google-Extended interaction: a broad training opt-out without an explicit Google-Extended Allow would, on Google's stated mapping, also cover Gemini grounding. Teams running AI SEO work should note the decision in their robots.txt change log now, so a later edit does not undo it without anyone noticing.

FAQ

When will Google start honouring the __usage_ai-training token?

Google says 2027, with no specific date. The Search Central blog post of 30 October 2026 says Google aims to support the control "in 2027", and the Google-Extended section of Google's crawler documentation says "early 2027".

Does opting out with the shared token remove a site from Google Search?

Google's documentation says Google-Extended, which is what the shared opt-out will map to, does not affect inclusion in Google Search and is not a ranking signal. The announcement itself does not mention AI Overviews or AI Mode, so it says nothing about them either way.

Can a site block AI training but stay eligible for Gemini citations?

Yes, according to Google, if the site explicitly allows Google-Extended. The post says company-specific controls continue to work and that sites which specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.

Does a "User-Agent: *" disallow already count as a training opt-out?

No. The IETF draft says wildcard groups must not be interpreted as matching the policy token, and that a file without a __usage_ai-training group expresses no preference at all.

Do I have to change anything now?

Not according to the sources. Google's support is planned for 2027 and the draft is still a work in progress. A site that already uses Google-Extended keeps that control; the shared token only matters for Google when Google-Extended is not specified.

The shared token gives site owners one line that several AI companies may read, while each company keeps its own controls. If you want help checking how your robots.txt, crawler rules and AI citations fit together before the 2027 rollout, RA's GEO team can review it with you.