In this briefing
- 01A shared marker reuses existing syntax but expresses a content-use preference
- 02Three providers share a direction but publish different timetables
- 03Microsoft separates training from grounding for generated answers
- 04The shared signal still lacks standard status, enforcement and licensing meaning
- →What to watch next
- ↗Sources and verification
- An Internet-Draft dated 30 October defines `__usage_ai-training`, the complete marker formed from the `usage_ai-training` token and two leading underscores. Website operators can use the existing `robots.txt` syntax to express model-training preferences by path, but this policy marker does not automatically restrict crawling.
- OpenAI plans to support the shared signal in the coming months, Google plans to support the shared signal in 2027, and Microsoft will start applying the relevant settings on 1 February 2027. The shared syntax does not currently prove that the providers have implemented it in step.
- Microsoft also separates AI model training from grounding for generated answers as two independent preferences. Both preferences default to allow when they are not declared, and the preferences do not automatically establish licensing or payment arrangements.
A shared marker reuses existing syntax but expresses a content-use preference
On 30 October, the Internet Engineering Task Force datatracker published `draft-kelley-aipref-training-preference-signal-00`. The draft defines `__usage_ai-training`, the complete policy marker formed from the `usage_ai-training` token and two leading underscores. Website operators can use ordinary `User-Agent`, `Disallow` and `Allow` rules in `robots.txt` to bind an objection to or agreement with model training to an entire website or a specific path. The draft adds no new directive name, and it does not change the existing path-matching or longest-match rules.
The draft defines AI model training as the use of content to modify the learned parameters of a generative AI model. The draft also states explicitly that a `User-Agent: *` wildcard group cannot be interpreted as a training preference. The complete `__usage_ai-training` marker, whose token is `usage_ai-training`, is not a crawling directive. A website can place content-use preferences and traffic-management rules together in `robots.txt`, but the two types of rule remain independent.
For enterprise adoption, one shared marker can reduce the work required to maintain separate training rules for different model providers, but the marker cannot replace crawling access controls, authentication, network restrictions or contractual licences. An enterprise content catalogue still needs to distinguish public webpages, licensed materials, customer data and internal knowledge, and each content class still needs an explicit purpose of use and boundary of responsibility.
Three providers share a direction but publish different timetables
On 30 October, OpenAI announced that OpenAI would support the shared training-preference signal written in `robots.txt`. OpenAI plans to implement the support in the coming months, and OpenAI will continue to apply its existing provider-specific crawler controls. The OpenAI announcement does not provide a fixed activation date. The OpenAI announcement also does not explain how OpenAI will handle content that it has already crawled or content that has already entered existing training data.
On the same day, Google announced that Google plans to support the shared signal in 2027. If a webpage uses the shared signal to reject model training and the webpage does not explicitly set Google-Extended, Google plans to interpret the shared rejection signal as a rejection of Google-Extended. Google will also retain provider-specific settings such as Google-Extended. Google further states that a webpage that explicitly allows Google-Extended can become eligible for inclusion in grounding results in Gemini Apps and for related clicks, but that eligibility does not guarantee that the webpage will receive an appearance or traffic.
For enterprise adoption, a shared syntax can provide one policy entry point, but an enterprise still needs to check each provider's activation date, provider-specific rule priority and actual crawl records. A unified configuration can reduce operating differences only after providers apply the same meaning in practice; during the transition, shared rules and provider-specific rules may need to exist together.
Microsoft separates training from grounding for generated answers
On 30 October, Microsoft Bing announced that Microsoft would start applying new AI model-training preferences and AI grounding preferences when crawling on 1 February 2027. The training preference can cover a website or path in `robots.txt`, and Microsoft will also support a page-level training preference. The AI grounding preference is set only at page level, and the preference determines whether webpage content can help Bing, Copilot, Microsoft 365 or a third-party grounding service generate an answer.
Microsoft provides three page-level choices for grounding: `Allow`, `DisallowExtended` and `Disallow`. `DisallowExtended` permits only Microsoft's own experiences to use the webpage content, and the setting excludes grounding services that Microsoft provides to third parties. The training preference and grounding preference both default to allow when they are not declared. Microsoft also requires webpages that continue to use `NoArchive` or `NoCache` to migrate the relevant preferences by 1 March 2027. Microsoft states that publishing a setting does not mean that the setting has taken effect, because Microsoft must still crawl the webpage again.
For enterprise adoption, training and grounding need separate places in data governance, content authorisation and provider configuration. An enterprise can allow a model to retrieve a document while answering a question and refuse the provider permission to use the document to train a future model; an enterprise can also choose the reverse combination. A system can explain which use was permitted for a piece of content at a particular time only when the system records the content path, page metadata, provider crawl time and policy version together.
The shared signal still lacks standard status, enforcement and licensing meaning
The Internet Engineering Task Force datatracker marks the document as an individually submitted Internet-Draft. The datatracker states explicitly that the document has not received Internet Engineering Task Force endorsement. The datatracker also states that the document does not have formal standing in the standards process. This version will expire on 3 May 2027 unless a later version updates, replaces or advances the document. The draft also permits provider-specific preferences to be more specific than the shared preference, and it permits private agreements to override the public preference.
Microsoft states explicitly that training and grounding preferences do not establish licensing or payment arrangements. Microsoft also states that restricting grounding will reduce content visibility in AI-generated experiences, but the setting does not change a webpage's eligibility for traditional Bing search results. Google and OpenAI also keep their existing provider-specific controls outside the shared signal. The available materials therefore prove only that the three companies have announced an implementation direction; the available materials do not prove that every AI provider will follow the same signal.
For enterprise adoption, a machine-readable preference can become one item of evidence for content governance and provider audits, but a machine-readable preference cannot by itself prove that licensing, compliance or technical blocking has been established. An enterprise still needs to retain contractual rights, access permissions, server logs, provider commitments, and deletion or opt-out mechanisms separately; the shared signal can cover only participants that choose to recognise and apply it.
What to watch next
- Can the Internet Engineering Task Force AI Preferences working group advance the individual Internet-Draft into a formal standard with stable meanings and test requirements?
- Can OpenAI, Google and Microsoft publish queryable activation dates, crawl-discovery times, rule priorities and methods for verifying enforcement?
- Can more model providers adopt the same training preference while continuing to distinguish clearly between crawling, model training, grounding for generated answers, licensing and payment arrangements?
Sources and verification
Golden Data has edited this briefing from the public materials listed above. The original sources govern facts and figures. The enterprise relevance sections are Golden Data editorial analysis and do not constitute an endorsement of any third-party product.
← Back to AI Daily Briefing