OpenAI, Google and Microsoft back a shared 'no AI training' signal: one robots.txt entry in place of per-company rules

OpenAI, Google and Microsoft each said on Oct. 30 (US time) that they will support a shared way for website operators to tell several AI companies, through robots.txt, that their content must not be used for AI training.

According to ITmedia News, the scheme comes from an Internet-Draft that Anthropic's standards lead Kevin Kelly filed with the IETF the same day. In robots.txt, the `User-Agent` line names the policy token `__usage_ai-training`; `Disallow` under it signals refusal of training use and `Allow` signals permission. Writing `Disallow: /` and `Allow: /press-kit/` asks that everything except press-kit not be used for training. The signal is independent of crawl controls, and omitting the token means no preference was stated. The draft was submitted in an individual capacity and has no formal IETF standing yet.

Timing differs. OpenAI aims to implement it within months and Google in 2027; if a site refuses training through the shared signal without explicitly setting Google-Extended, Google will treat Google-Extended as refused too. Microsoft will read and apply the new setting when crawling from Feb. 1, 2027, and will let sites set preferences per page with meta tags and HTTP headers that override robots.txt. Anthropic has begun implementing the draft but says extending it across its services will take time, Kelly told the AIPREF mailing list.

What works today are the existing per-company rules such as GPTBot, Google-Extended and ClaudeBot. ITmedia News's X post drew more than 2,070 likes, with Japanese users replying that "the AI has already finished learning, so only latecomers are blocked" and that it means nothing unless data is deleted retroactively. Amid a string of publisher lawsuits, Cloudflare opened controls in July that manage AI crawlers by search, agent and training purposes for all customers.