In this guide · 8 sections
- 01Training crawlers vs search crawlers vs user fetches
- 02AI crawler user agents and what each controls
- 03Copy-ready robots.txt: block training, allow AI search
- 04Cloudflare AI bot settings: enforcement, not just a request
- 05Should a small business block AI crawlers at all?
- 06What robots.txt cannot do
- 07Do it yourself or get help?
- 08Frequently asked questions
Short answer
To opt out of AI training but stay visible in AI search, disallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot in robots.txt, and keep OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed. robots.txt is a request, so use your CDN's bot settings if you need enforcement.
Training crawlers vs search crawlers vs user fetches
Before you block anything, know that AI companies now publish separate user agents for separate jobs. Treating them all as "AI bots" is how businesses accidentally remove themselves from AI search.
| Type | What it does | Effect of blocking |
|---|---|---|
| Training | Collects pages that may be used to train future models | Your content is (by the vendor's word) left out of training |
| Search | Indexes pages so an AI assistant can cite and link them | You stop appearing as a cited source in that AI's answers |
| User fetch | Visits a page because a user asked the assistant to | The assistant can't read your page for that user; some vendors say robots.txt may not apply |
For most businesses that sell services or products, being cited in AI search is useful and training is the part they object to. So the usual choice is: block training, allow search.
AI crawler user agents and what each controls
These are the robots.txt tokens documented by each vendor, checked on 19 October 2026. Sources: OpenAI bots, Anthropic help centre, Google common crawlers, Perplexity bots, Apple and Common Crawl.
| Token | Company | Purpose | Block for training opt-out? |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train its models | Yes |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search answers | No, keep allowed |
| ChatGPT-User | OpenAI | Visits pages on a user's request; OpenAI says robots.txt may not apply | Not needed |
| ClaudeBot | Anthropic | Collects content that could contribute to training | Yes |
| Claude-SearchBot | Anthropic | Indexes content to improve Claude search results | No, keep allowed |
| Claude-User | Anthropic | Fetches pages when a Claude user asks | Optional |
| Google-Extended | Gemini training and grounding; not used for Search ranking | Yes | |
| Applebot-Extended | Apple | Use of content to train Apple's foundation models | Yes |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search; Perplexity says not for training | No, keep allowed |
| CCBot | Common Crawl | Open web archive widely used as AI training data | Yes |
Two details worth knowing. Google-Extended is a control token only: Google crawls with its normal user agents and checks the token to decide how the content may be used, and Google says it has no effect on inclusion or ranking in Google Search. Applebot-Extended likewise does not crawl; Apple says blocked pages can still appear in Siri, Spotlight and Safari results through the normal Applebot.
Copy-ready robots.txt: block training, allow AI search
robots.txt lives at the root of your domain (yoursite.com/robots.txt). Add these groups to your existing file rather than replacing it, so your current Googlebot rules and sitemap line stay in place:
- User-agent: GPTBot / Disallow: /
- User-agent: ClaudeBot / Disallow: /
- User-agent: Google-Extended / Disallow: /
- User-agent: Applebot-Extended / Disallow: /
- User-agent: CCBot / Disallow: /
Write each pair on two lines in the real file (User-agent on one line, Disallow on the next), with a blank line between groups. Do not add rules for OAI-SearchBot, Claude-SearchBot or PerplexityBot unless you want to leave those AI search tools. Anthropic notes that each subdomain needs its own robots.txt, so repeat this on shop.yoursite.com or blog.yoursite.com if they exist.
Open your current robots.txt
Visit yoursite.com/robots.txt in a browser and copy what is there. WordPress SEO plugins, Shopify and Wix each have their own place to edit it; check the platform's help page.
Add the training groups
Paste the five groups above below your existing rules. Keep any Sitemap line.
Test the file
Use Search Console's robots.txt report to confirm Google can still fetch it and that Googlebot is not blocked from your pages.
Allow time to take effect
OpenAI says robots.txt changes can take about 24 hours to be reflected in its search results. Other vendors re-read the file on their own schedule.
Recheck every quarter
Vendors add and rename bots. Re-read the vendor pages above and update the list.
Cloudflare AI bot settings: enforcement, not just a request
robots.txt is voluntary. Cloudflare's own robots.txt documentation puts it plainly: some crawler operators may ignore your directives. If your site runs through Cloudflare, you can block at the edge instead.
- AI bot policies (Security Settings > Configure AI bot policies) let you set Search, Agent and Training bots separately to Block, Block on pages with ads, or Allow.
- New defaults from 15 September 2026: for new domains, Cloudflare blocks Training and Agent bots on pages that display ads and keeps Search allowed. Mixed-purpose crawlers that do both search and training are blocked by any training-block setting.
- The older "Block AI bots" toggle is marked as deprecating on 15 September 2026 in Cloudflare's docs.
- AI Crawl Control (formerly AI Audit) is available on all plans and shows which AI services visit, and whether they follow robots.txt.
- Managed robots.txt can generate the rules from your policies and prepend them to your existing file.
Should a small business block AI crawlers at all?
There is no single right answer. It depends on whether your content is the product or the shop window.
Block training bots if
- You publish original research, courses, photography or writing that you sell or license
- Your pages carry content you only licensed for your own site, such as stock photos or partner material
- You have a policy or client contract that restricts reuse of published material
Leave them allowed if
- Your pages are mainly service descriptions, prices and contact details
- You want AI models to know about your business in general terms
- You have no time to maintain the list as vendors add bots
Either way, keep the search bots allowed unless you have a clear reason. Blocking training does not remove content already collected, and opting out is a request the vendor honours by its own policy. If image rights are part of the worry, our guide to website images and stock photo copyright covers what your licences allow.
What robots.txt cannot do
robots.txt controls crawling, not copying. It does not hide pages from people, remove content already collected, or stop a bot that ignores it. Anthropic says blocking its IP addresses instead of using robots.txt may not work reliably, because it stops its bot reading your opt-out. Some user-initiated fetchers, such as ChatGPT-User and Perplexity-User, may not follow robots.txt at all, according to their vendors.
And llms.txt is not a blocking tool either. It is an optional reading list for AI agents; see llms.txt explained. For the opposite goal, appearing in AI answers, read our guide to ChatGPT search and Google AI Overviews visibility.
Do it yourself or get help?
Editing robots.txt is a ten-minute job if your platform gives you access to it. The risk is a typo that blocks Googlebot, so always test in Search Console afterwards.
Before you finish
Our Speed & SEO Fix (S$390) includes a robots.txt and indexing review, and our Website Security Hardening (S$290) covers firewall and bot settings, including Cloudflare.
Frequently asked questions
No. OpenAI documents GPTBot and OAI-SearchBot as independent settings. Disallowing GPTBot signals that your content should not be used to train its models. ChatGPT search relies on OAI-SearchBot, and OpenAI says sites that block OAI-SearchBot will not be shown in ChatGPT search answers. Keep OAI-SearchBot allowed if you want to be cited.
Google says it does not. Google-Extended controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google's crawler documentation states it does not affect inclusion in Google Search and is not a ranking signal. AI Overviews are part of Search and follow normal Googlebot rules.
The major vendors say their documented crawlers do, and they publish IP ranges so you can verify them. But robots.txt is voluntary, unknown scrapers can ignore it, and some user-initiated fetchers may not apply it. If enforcement matters, use a CDN such as Cloudflare, which can block AI bot categories before they reach your site.
No. robots.txt only affects future crawling. Anthropic, for example, says a restriction signals that the site's future materials should be excluded from training datasets. Content already gathered or published elsewhere is not recalled. If you need something removed, contact the vendor through its published privacy or content removal process.
Possibly. Cloudflare set new defaults for new domains on 15 September 2026, blocking Training and Agent bots on pages with ads and allowing Search. Older zones keep whatever was configured. Open Security Settings and the AI bot policies section in the dashboard to see your current setting for each category.