One AI training opt-out, several different promises

One message for several companies

Instead of maintaining a separate instruction for each AI company, a website could publish one shared preference in robots.txt, the small file that tells automated visitors how the site wants to be treated. The proposed signal can cover the whole site or particular sections: a publisher could refuse training on its articles while allowing it on a press kit.

OpenAI’s announcement calls the effort voluntary and says its existing crawler controls will continue. The underlying proposal is an individual Internet-Draft by Anthropic’s Kevin Kelley, rather than an approved IETF standard. It expresses a preference about using content; the draft explicitly separates that from rules controlling whether a crawler fetches a page.

Support is arriving on different clocks

Kelley’s public message says Anthropic has started implementing support, with more work needed across its services. OpenAI targets the coming months. Google’s documentation specifies early 2027, while Microsoft sets 1 February 2027, as it crawls. These are implementation commitments, not proof that every service already follows the preference.

Meta communications executive Andy Stone also backed a common signal in his original statement. He said the company looked forward to work on standards and implementation; the post gives no activation date.

Training and being found are different decisions

Training changes what a model has learned. Grounding means using web content to help construct an answer when someone asks a question. Ordinary search indexing is another activity again. Search specialist Vanessa Fox’s updated reference separates those uses, which is more useful than treating every automated visit as the same thing.

OpenAI’s crawler documentation already separates training from search: a site can refuse GPTBot while allowing OAI-SearchBot for ChatGPT search. Google’s planned interpretation has a different consequence. Unless a site explicitly specifies Google-Extended, Google will treat the shared training refusal as a Google-Extended refusal. That existing control covers both Gemini training and grounding in Gemini Apps and relevant Vertex AI services. It does not affect inclusion or ranking in Google Search.

Microsoft is introducing independent training and answer-use preferences. Its new page-level grounding control can refuse answer use entirely or limit it to Microsoft-operated experiences. Those choices leave ordinary Bing search eligibility intact, subject to normal crawl and indexing requirements. Eligibility is not a promise of visits or payment.

Some existing restrictions need replacing

Microsoft’s transition deserves a diary entry for sites using NoArchive or NoCache to restrict AI use. It says those old settings stop serving that purpose on 1 March 2027. Its transition guide calls for new training preferences and, for continued grounding restrictions, page settings plus a temporary choice in Bing Webmaster Tools. The temporary grounding route opens on 15 December for qualifying existing users.

The shared syntax is progress toward an easier message, rather than a universal outcome. Travel Lemming founder Nate Hake, a participant acknowledged in the draft, stressed that the IETF work remains unfinished and disputed. The practical question now is whether each provider’s implementation preserves the choice a publisher intended. That needs checking against the live service, not just a new line in a file.