Quick Answer
Separate the decision into three columns: training use, live grounding or answer use, and search discovery. For each provider, write down the exact documented control and the business outcome you want to preserve. Do not paste a universal “block AI” rule into robots.txt. In particular, a broad future training preference without an explicit Google-Extended choice could also close a Gemini grounding route under Google's announced interpretation. Get your legal, editorial and growth owners to agree on the tradeoff before an engineer changes a production policy file.
What Changed in October 2026?
On 30 October, Google announced planned support for a shared AI-training preference based on an IETF proposal. The draft uses an ordinary robots.txt group keyed to a policy token, __usage_ai-training; it is a use preference, not a new crawling directive. Google says it aims to support the mechanism with other providers in 2027.
The same announcement states a consequential exception. If a site uses the shared token to opt out of training and does not explicitly specify Google-Extended, Google plans to interpret that choice as though Google-Extended were disallowed. Google also says sites that specifically allow Google-Extended are eligible to appear in Gemini Apps responses and receive clicks through grounding. These are announced intentions as of this review, not proof that any site will get traffic by allowing it.
The IETF Internet-Draft is still a draft. It describes path-level preferences and says ordinary robots.txt matching mechanics stay in place. Do not implement a draft token today on the assumption that all AI providers already recognise it, or that they interpret it identically.
Training, Grounding and Discovery Are Three Different Jobs
Training means content may be used to develop a model. A training restriction can reflect licensing, competitive or creator-rights concerns. It does not necessarily prevent the page from being found in a live search.
Grounding means the system brings in information at answer time to improve the response. The page may be cited or linked, but whether it is selected depends on the system and query. Eligibility is not placement.
Discovery means a person can encounter your page in a search or AI-search result. It can create qualified visits even when a model's training use is restricted. A Search Console impression, a cited answer and a referred visit are different observations; keep them in separate reporting columns.
The practical consequence is easiest to see in a hypothetical B2B example. A specialist manufacturer may want product specifications discoverable so buyers can compare them, while reserving proprietary installation manuals for customers. The business question is not “allow AI?” It is which public information should be retrievable, under which rights, for which buyer task. That choice should inform the AI visibility audit, not be made after it.
The Control Map Site Owners Need
| Decision | Documented control as of November 2026 | Important limitation |
|---|---|---|
| OpenAI model-training crawl | GPTBot in robots.txt | Separate from OAI-SearchBot; a block does not make the page private |
| ChatGPT search inclusion | OAI-SearchBot in robots.txt | Allowing it does not ensure a citation or visit |
| Gemini training and Gemini Apps grounding | Google-Extended in robots.txt | One token covers both uses in Google's crawler documentation |
| Google Search and AI features | Google Search indexing and current Search Console generative-AI settings | Google-Extended does not determine ordinary Search inclusion; Search Console's generative-AI setting applies at the site level, unlike path-specific robots rules |
| Shared training preference | Proposed __usage_ai-training group | Provider support and interpretation are still evolving |
OpenAI's crawler documentation explicitly separates GPTBot from OAI-SearchBot and also distinguishes user-initiated ChatGPT-User visits. Google's crawler documentation says Google-Extended affects both training and grounding for specified Gemini products, but not Google Search inclusion or its ranking signals. Google Search Central's control guide describes a separate Search generative-AI control in Search Console.
Read those product definitions carefully. “AI answer” is not one channel, and the control labels do not mean the same thing across vendors. A single screenshot of robots.txt cannot verify the resulting visibility.
A Four-Decision Meeting Before Editing Robots.txt
1. Inventory the content, not just the domain
Separate public service pages, product data, editorial articles, licensed research, customer-only documents and user-generated content. Check whether sensitive material is actually access-controlled. Robots.txt is publicly readable and does not secure private content. A page that must remain private needs authentication or removal, not a crawler preference.
2. State the commercial goal for each class
Does a product page need to be discovered and cited? Would a paid research report lose value if used for training? Is a troubleshooting guide meant to reduce support volume? Write the intended user action next to each class. This prevents a “protect everything” rule from silently excluding the pages that support sales.
3. Choose a provider-specific policy with its side effects visible
For OpenAI, the documented search/training separation permits a clear combination. For Google, the Google-Extended training-and-grounding coupling deserves an explicit decision: there is no documented separate Google switch to deny Gemini training while allowing Gemini Apps grounding. For the proposed shared token, make no assumption of universal recognition before provider implementation is confirmed. Record the policy owner, date, files changed and reason.
4. Verify and monitor different outcomes separately
Have an engineer review the exact robots.txt served from the production host, check syntax, and test representative paths. After deployment, watch crawling and access logs where available. Review Google Search Console indexing and generative-AI performance separately from any referral data. If an answer engine cites a page, record the prompt and cited URL; do not infer training use from that citation. If visibility changes, check concurrent indexing, content and demand changes before assigning causation to one robots edit.
This is a governance decision, not an SEO trick. The ownership can sit with legal, marketing, engineering and product together; a unilateral growth-team change is easy to make and difficult to explain later.
The Tradeoff Most Guides Miss
The attractive story is “opt out of training, keep all AI referrals.” That is not uniformly possible under the documented controls. OpenAI separates its search and training bots. Google's Google-Extended currently joins Gemini training use with Gemini Apps grounding. The planned shared token would add a cross-provider training preference, but Google's announced default interpretation makes an explicit Google-Extended rule especially important for publishers who want Gemini grounding.
None of this tells a brand that it must allow training. A publisher may rationally prioritise rights over a speculative referral channel. The mistake is to make the tradeoff unintentionally. If your team wants more qualified visits, compare the value of the public pages with the rights you are reserving. If licensing is the main revenue stream, have counsel review the terms. The SEO strategy still has to serve people on your own site; no crawler setting compensates for an unhelpful page.
Tips for a Clean Rollout
- Keep a policy register. Record each user-agent or policy token, affected paths, intended use and approving owner.
- Test a small set of representative URLs. Check a service page, product page, article and protected page separately.
- Do not conflate
noindexwith training control.noindexaffects Search indexing when a crawler can read it; it is not a universal AI-training instruction. - Check Search Console before changing Google access. The newer generative-AI controls and reports are distinct from Google-Extended.
- Keep the timeline honest. The shared preference is planned for 2027. Recheck the final standard and provider documentation before deploying the draft syntax.
- Review with revenue owners. A lower crawl count is not a business outcome; qualified discovery, licensing and customer trust are.
What to Keep an Eye On
Monitor the IETF draft's status, Google's actual launch documentation and each provider's support statement. Check whether provider-specific rules override or coexist with the shared preference in the final implementations. Revisit the decision whenever a new AI-search channel becomes material to your buyers, a new licensed content product launches, or a Search Console control changes.
Avoid claiming that a rule “protects all content from AI.” A stated preference can influence participating systems; it is not proof of every downstream use. For private information, use access control.
FAQs
Can I block AI training and still appear in ChatGPT search?
OpenAI documents GPTBot for potential training use and OAI-SearchBot for ChatGPT search. A site can set different robots.txt rules for the two. Search inclusion still depends on other factors, so allow is not a promise of appearance.
Will blocking Google-Extended remove my pages from Google Search?
Google says Google-Extended does not affect inclusion or rankings in Google Search. It does cover training and grounding for the specified Gemini products. Review the current product documentation and Search Console controls before making a change.
Should I add the shared training token to robots.txt now?
Not as a universal fix. The proposal is an Internet-Draft and Google says it aims to support it in 2027. Decide your policy now, then implement against the final documentation of the providers you rely on.
Does an AI citation prove my content was used for training?
No. A citation can arise from live retrieval or a search index. It does not establish whether the page was in a training set. Keep training-policy records and answer-visibility reporting separate.
The Decision
Before 2027, create a one-page map of content classes, desired uses, provider controls and tradeoffs. Keep high-value public pages discoverable where that serves customers, protect private material with real access controls, and make training choices deliberately. Recheck the mechanism when the announced shared preference actually ships.
