One robots.txt rule will opt your site out of AI training in 2027. Should your business use it?

What is the new robots.txt rule for AI training?

On 30 October 2026 an Internet-Draft called "Machine-Readable Training Preference Signal" was published at the IETF. Its author, Kevin Kelley, is listed with Anthropic. It gives robots.txt one shared name, __usage_ai-training. A group under that name tells AI companies whether your pages may be used to train their models.

The draft's own example:

• User-Agent: __usage_ai-training
• Disallow: /
• Allow: /press-kit/

That means no training on the whole site except the press kit. The draft adds no new commands, so anyone who has edited a robots.txt can write it. It is still an individual draft: the IETF page says it is not endorsed by the IETF and has no formal standing.

Which companies will honour it, and when?

Google's crawler documentation, updated on 30 October, gives the early 2027 date. Search Engine Journal reports that Google will treat a shared opt-out as if you had blocked Google-Extended, unless you set a Google-Extended rule yourself.

Does blocking AI training hurt my Google or Bing rankings?

Not according to either company. Google's crawler documentation says Google-Extended "does not impact a site's inclusion in Google Search" and is not a ranking signal. Microsoft says the new preferences do not change eligibility for traditional Bing results.

Two things can change, and they are easy to miss:

• Gemini Apps answers. Google-Extended also covers grounding with Google Search in Gemini Apps. If your opt-out counts as blocking Google-Extended, your pages may stop informing those answers. Search Engine Journal reports that sites which specifically allow Google-Extended stay eligible.
• Copilot and Bing AI answers. Microsoft treats grounding as a separate choice, set page by page with a meta tag or HTTP header. There is no site-wide robots.txt setting for it, and Microsoft warns that disallowing grounding will significantly reduce your visibility in AI answers.

What is the difference between AI training and AI grounding?

Training changes what a model has learned. Your page becomes part of its general knowledge, with no link back. Grounding means an AI assistant reads your page while writing an answer, usually with a citation. For a business that wants enquiries, grounding is the visibility that matters.

Should a small business opt out of AI training?

Our advice: for most studios, clinics, consultancies and SaaS companies, no. Service pages, pricing, case studies and FAQs exist to be found and quoted. The cases for opting out are narrower: paid courses, research you sell, or a member library. Even then, the draft is clear that the token "is not a directive for crawling behavior". A preference is not a lock, so private material needs a login.

If you want to know what your robots.txt, meta tags and structured data tell search engines and AI assistants today, our SEO, AEO and GEO service starts with a technical audit, and the free brand audit includes a quick AEO score.

Worked example: a 10-person Dubai consultancy with a paid report library

This is a hypothetical scenario, not a client.

1. 1List what each section is for. Service pages, case studies and the blog bring enquiries. A reports section holds paid industry reports.
2. 2Decide. Training stays allowed everywhere except the reports section.
3. 3Add one group at the end of robots.txt, separate from the others: User-Agent: __usage_ai-training, then Disallow: /reports/.
4. 4Leave the other groups alone. The draft says a User-Agent: * group does not count for this token, and the token works independently of crawler rules such as Googlebot or GPTBot.
5. 5Check old tags. The site once added NoArchive tags to its reports. Microsoft stops reading NoArchive and NoCache as AI preferences after 1 March 2027, so the team replaces them with Bing's new page-level training tag.
6. 6Move the full reports behind a login. The public pages keep a summary that can still be cited.
7. 7Test in a month. Ask ChatGPT, Perplexity and Copilot the questions buyers ask, and check whether the service pages are cited. Our free AEO score checker shows what a page is missing.

What should you do before 1 March 2027?

1. 1Open yoursite.com/robots.txt and read what is there. Templates and plugins sometimes block AI crawlers without anyone deciding to.
2. 2Decide section by section: pages that sell, and content that is paid or private.
3. 3Add a __usage_ai-training group only for the sections you want kept out of training.
4. 4If you use NoArchive or NoCache, replace them before 1 March 2027. Sites using them on 15 December 2026 can also set a temporary grounding preference in Bing Webmaster Tools from that date.
5. 5Do not block grounding on pages you want cited.
6. 6Keep vendor rules consistent. Search Engine Journal found that OpenAI's documentation does not yet say how a shared rule and a GPTBot rule interact, so our advice is to avoid rules that contradict each other.

For the other side of the question, how to be cited rather than how to opt out, see how AI answers now show more links and how to get recommended by ChatGPT.

Sources

• Machine-Readable Training Preference Signal, draft-kelley-aipref-training-preference-signal-00 (IETF Datatracker, 30 October 2026)
• Google's common crawlers (Google for Developers, updated 30 October 2026)
• Your content, your choice: clearer preferences for training and grounding (Microsoft Bing Blog, 30 October 2026)
• Google, Microsoft and OpenAI Plan To Honor Shared AI Training Opt-Out (Search Engine Journal, 1 November 2026)