Google, OpenAI and Microsoft Back a New Way to Stop AI Training on Website Content

Website owners may soon have a simpler way to tell several major artificial intelligence companies not to use their published content to train generative AI models.

Google, OpenAI and Microsoft announced on 30 October that they plan to support a shared instruction in a website’s robots.txt file. The proposed signal would let publishers communicate an AI-training preference through one common rule instead of maintaining separate instructions for every participating provider.

Anthropic is involved too. An Anthropic employee submitted the technical proposal to the Internet Engineering Task Force (IETF), and the company has reportedly started implementing support.

The change could make it easier for independent publishers, businesses and website owners to communicate how their content may be used. But it is not a universal block against AI companies accessing a website. The signal is a stated preference, its rollout is staggered, and Google’s interpretation could also affect whether content is available for certain Gemini services.

Here is what the new control means, when the companies expect to support it, and what website owners should know before changing their settings.

QUICK SUMMARY

The Story in 60 Seconds

Google, OpenAI and Microsoft are backing a shared way for website owners to express whether their content can be used to train generative AI models.

The five things to know

1. 01
   What changed?

   A proposed robots.txt signal called __usage_ai-training could let publishers communicate an AI-training preference to several participating providers through one rule.

2. 02
   Who is supporting it?

   Google, OpenAI and Microsoft announced support plans on 30 October 2026. Anthropic is also involved and has reportedly started implementing the proposal.

3. 03
   Does it block AI crawlers?

   No. It communicates a preference about AI training. It does not automatically stop crawling or remove content already used in previous training.

4. 04
   When will it work?

   Google is targeting early 2027, OpenAI says the coming months, and Microsoft plans to honour the preferences from 1 February 2027.

5. 05
   What should website owners know?

   The signal is not yet fully implemented. Google’s interpretation may also affect Gemini grounding, and a general robots.txt wildcard rule does not count as the shared preference.

i

This is a quick overview of the announcement. Continue reading for the technical details, provider timelines and practical guidance for website owners.

What is the new AI-training control?

The proposed control is a shared policy signal that website owners can place in their robots.txt file.

A robots.txt file is a public text file at the root of a website. It gives automated crawlers instructions about which parts of a site they are requested to access. Search engines and other automated services use it to read a website’s crawling preferences.

The new proposal introduces a policy token called __usage_ai-training. It is designed to communicate whether a website’s content may be used to train generative AI models.

A website that wants to express a preference against AI training across its entire site could use this example:

User-Agent: __usage_ai-training
Disallow: /

The first line identifies the type of use covered by the preference. The second expresses a preference against using content from all paths on the website for that purpose.

The proposal also allows more specific choices. For example, a publisher could disallow AI training across most of its site while allowing it for a particular folder:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Under the draft’s path-matching rules, this expresses a preference against training across the site, except for material under /press-kit/.

The technical mechanism is described in an IETF Internet-Draft called Machine-Readable Training Preference Signal, submitted on 30 October 2026.

The document is an individual Internet-Draft, not a final standard adopted by the IETF. The IETF Datatracker explicitly says that it has no formal standing in the standards process. The companies’ announcements therefore represent commitments to support a proposed mechanism, not evidence that a completed industry standard is already operating.

Why are major AI companies supporting one shared signal?

Website owners have traditionally needed to manage separate crawler instructions for different AI providers.

For example, OpenAI offers separate controls for its GPTBot and OAI-SearchBot crawlers. Google has its own Google-Extended control. Microsoft has also offered website owners ways to express preferences about how its crawlers use their content.

Managing these instructions individually can become complicated, particularly for smaller publishers that do not have dedicated technical teams.

The shared proposal is intended to reduce that work by giving participating providers one common signal to recognise.

In its 30 October announcement, Google said the policy token would allow websites to express broad AI-training preferences across multiple providers. Google also said its existing company-specific controls would continue to work.

OpenAI made a similar commitment in its announcement, Making it easier for websites to express AI training preferences. It described the initiative as a voluntary effort and said it aims to implement the signal in the coming months.

Microsoft’s announcement, Your content, your choice: clearer preferences for training and grounding, sets out a wider set of changes. These include support for the shared training preference and additional page-level controls for AI training and AI grounding.

The technical proposal was submitted by Kevin Kelley, whose IETF draft lists his affiliation as Anthropic. Reporting on his message to the IETF working group’s mailing list says Anthropic has started implementing support, although the company has not announced a firm completion date.

The aim is to make it easier for website owners to express a common preference without having to create a separate training rule for every participating AI provider.

When will the new control start working?

The announcements do not mean that every participating company is already recognising the shared token. Each provider has given a different implementation timeline.

Google: Early 2027

Google’s announcement says it aims to support the shared control in 2027. Its crawler documentation now describes the planned support as coming in early 2027.

Google has also explained how it intends to interpret the shared preference in relation to its existing Google-Extended control. That distinction matters because Google-Extended covers more than model training, as explained below.

OpenAI: Coming months

OpenAI says it aims to implement the common signal in the coming months. It has not provided a specific launch date in its announcement.

The company says its existing crawler controls will continue to operate alongside the shared signal.

Microsoft: 1 February 2027

Microsoft says it will begin honouring the new preferences as it crawls websites from 1 February 2027.

It is also introducing separate page-level controls for AI training and AI grounding. Microsoft says the new preferences will replace its older NoArchive and NoCache controls for these AI uses.

There is an important transition deadline: Microsoft says website owners relying on those older settings should publish the new AI preferences by 1 March 2027 if they want to maintain their chosen controls. Its announcement explains the steps required for both training and grounding preferences.

Anthropic: Implementation underway

Anthropic’s employee submitted the technical draft, and reporting on the IETF working-group discussion says the company has started implementing support.

However, there is no confirmed date for when support will be available across all Anthropic services.

These different timelines mean website owners should not assume that adding the shared token today will immediately change how every provider handles their content.

Does the new instruction stop AI companies from crawling a website?

No. This is one of the most important points to understand.

The proposed __usage_ai-training token communicates a preference about using website content to train generative AI models. It is not a general instruction to stop a crawler from visiting the website.

The IETF draft specifically says that the policy token is not a directive for crawling behaviour. Existing crawler-specific instructions continue to operate independently.

For example, OpenAI distinguishes between two important crawlers:

• GPTBot: Used to crawl content that may be used to train OpenAI’s generative AI foundation models.
• OAI-SearchBot: Used to help surface websites in ChatGPT search results.

OpenAI explains that website owners can allow one while disallowing the other. Its crawler documentation describes these controls separately.

This means a publisher could express a preference against AI training without automatically opting out of ChatGPT search visibility.

The shared signal should therefore not be described as a way to block all AI crawlers, prevent scraping or make website content inaccessible.

Does a normal robots.txt wildcard rule count as an AI-training opt-out?

No. This is an important technical detail that website owners could easily overlook.

Many websites already have a general robots.txt group such as:

User-Agent: *
Disallow:

That group applies to crawlers that do not have a more specific matching rule. However, under the proposed IETF draft, a wildcard User-Agent: * group does not count as expressing the shared AI-training preference.

The website needs a separate group that explicitly names:

User-Agent: __usage_ai-training

Without that policy-token group, the draft says no preference has been expressed through this mechanism.

This also means that existing rules for Google-Extended, GPTBot or other named crawlers do not automatically become the shared preference. Those company-specific instructions retain their own meanings.

The distinction matters because website owners should not assume that a general crawling restriction already communicates their choice about AI training.

What is the difference between AI training and AI grounding?

AI training and AI grounding are two different ways that a provider may use website content.

AI training involves using content to modify the learned parameters of a generative AI model. It is part of the process used to develop or update the model.

AI grounding involves providing relevant information to an AI system when it generates a response. The system can retrieve website content and use it as supporting information for an answer.

For example, imagine Secritic publishes a guide explaining how to secure a compromised email account.

An AI provider could use the article as training material when developing a future model. Alternatively, an AI search service could retrieve the article when someone asks how to secure an email account and use it as a source for its response.

Those are different uses of the same article.

Microsoft’s announcement makes this distinction especially important because it introduces separate preferences for AI training and AI grounding.

Google’s approach has a different consequence. Google says it plans to interpret a shared training opt-out as a Google-Extended disallow when the website has not explicitly specified a different Google-Extended preference.

Google-Extended covers both the use of content for training future Gemini models and certain grounding uses in Gemini Apps and Vertex AI. As a result, a shared opt-out could affect eligibility for those grounding experiences as well as training.

This is an important trade-off for publishers. A website owner may want to prevent model training but still want articles to appear as sources in AI-generated answers. The shared signal does not guarantee that every provider will separate those choices in the same way.

Could opting out affect Google Search rankings?

Google says Google-Extended does not affect a website’s inclusion in ordinary Google Search and is not used as a Search ranking signal.

That means a Google-Extended restriction should not, by itself, remove a website from normal Google Search results or lower its Search ranking.

However, this does not mean every AI-related restriction has no effect on visibility.

Google says Google-Extended also controls certain Gemini training and grounding uses. If a website’s shared opt-out is interpreted as a Google-Extended disallow, its content may no longer be eligible for those specified Gemini experiences.

Microsoft also distinguishes AI grounding from ordinary Bing search. Its announcement says that restricting grounding can reduce a website’s visibility in Microsoft’s AI-generated experiences, while the new preferences do not affect eligibility for Bing search results.

Publishers should therefore distinguish ordinary search visibility from the use of their content in AI-generated answers.

Can website owners control individual pages?

The shared IETF proposal allows website owners to express preferences for the whole site or particular paths through robots.txt.

Microsoft is introducing additional controls that go further.

According to its announcement, website owners will be able to set AI-training preferences for individual pages through a meta tag or HTTP response header. Microsoft is also adding a separate page-level AI-grounding preference.

These are Microsoft’s additional controls, not part of the shared robots.txt token itself.

This could help publishers that want different rules for different types of content. For example, a website might want to restrict training on certain pages while allowing other public pages to be used for that purpose.

The exact behaviour will depend on the provider’s implementation and the instructions it publishes.

Does the new signal remove articles already used to train AI models?

No. The proposal does not establish a way to remove content that AI companies have already collected or used in previous model training.

Its purpose is to communicate a preference about the use of content retrieved under the applicable instructions. It is not a mechanism for deleting historical datasets, reversing previous training or removing information from an existing model.

The draft also says that the preferences applying to an asset are those expressed in the robots.txt file in effect when that asset was retrieved.

That makes the timing important. Adding a preference later should not be presented as a way to withdraw material that was already collected.

The draft also recognises that private agreements may override preferences expressed through this mechanism. The signal is therefore not a universal legal or technical guarantee.

The existing Robots Exclusion Protocol, RFC 9309, makes another limitation clear: robots.txt rules are not a form of access authorisation. They are instructions that crawlers are requested to honour, not a security barrier that prevents every party from accessing a website.

What should WordPress website owners do now?

For now, website owners should understand the announcement and review their existing crawler settings before making changes.

The shared preference has been announced, but support is not yet fully active across the participating providers. Their implementation dates and the way they interpret the signal are different.

A sensible approach is to:

1. Review your current robots.txt file. Check which instructions already apply to Google, OpenAI, Microsoft and other crawlers.
2. Decide separately about training and grounding. Consider whether you want your content used to train models, appear in AI-generated answers, or both.
3. Do not rely on a wildcard rule. The proposed shared preference requires its own __usage_ai-training policy-token group.
4. Check each provider’s implementation guidance. Follow the official documentation when support becomes available rather than assuming the same rule has identical effects everywhere.
5. Back up your existing configuration. Do not replace your current robots.txt file with a short example copied from a news article.
6. Check the live file after making changes. Confirm that the intended rules appear at your website’s /robots.txt address.

WordPress websites may serve robots.txt instructions through WordPress itself, a plugin or server-level configuration. The correct method depends on the site’s setup. Editing the wrong file or replacing existing rules could affect crawler behaviour in ways the website owner did not intend.

Microsoft users should pay particular attention to the 1 March 2027 transition deadline if their websites currently rely on NoArchive or NoCache for AI-related preferences.

For Secritic and other independent publishers, the safest approach is to monitor the official provider documentation and make changes only after the relevant implementation details are clear.

Why does this matter for independent publishers?

The development addresses a growing concern among website owners: how to communicate their choices when AI companies collect and use material published on the web.

Large publishers may have technical teams and legal resources to manage individual crawler permissions, licensing arrangements and negotiations. Smaller websites often have fewer resources and may find it difficult to maintain separate instructions for every AI provider.

A shared signal could make that process easier.

But it does not settle wider questions about copyright, licensing, compensation or whether a company has a legal right to use particular material. A machine-readable preference is a way to communicate a website owner’s choice; it is not, by itself, a complete answer to those legal questions.

Its practical value will depend on how many AI providers adopt it, how consistently they honour it and whether publishers can understand the differences between training, crawling and grounding.

What happens next?

The next stage is implementation.

OpenAI aims to support the shared signal in the coming months. Google is targeting early 2027, while Microsoft says it will begin honouring its new preferences from 1 February 2027. Anthropic has reportedly started implementing support but has not announced a full rollout date.

The technical proposal remains an individual Internet-Draft within the IETF’s AI Preferences work. It has not yet become an adopted IETF standard.

For website owners, the important development is that major AI providers are moving towards a common way to express AI-training preferences. It could reduce the need to maintain separate training instructions for every participating company, but it will not give publishers complete control over every way their content can be accessed or used.

Until implementation is available, publishers should follow official provider guidance and avoid changing their live settings based on assumptions.

The shared signal could make AI-training preferences easier to communicate. Whether it gives publishers meaningful control will depend on how providers implement and honour it.