AI crawlers come in two kinds, and the distinction decides whether blocking them costs you anything.
Training crawlers collect text to train models. Blocking them costs you no visibility whatsoever.
Answer crawlers fetch pages so an assistant can cite them in a reply. Blocking these removes you from AI answers — usually the opposite of what someone blocking "AI bots" intends.
The rules below block the first group and leave the second alone. Several other guides make this same distinction; what is added here is measured data on what sites actually do, and two caveats most lists omit — see below.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Diffbot
Disallow: /
# Deliberately NOT blocked — these fetch pages in order to cite you:
# OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User,
# PerplexityBot, Perplexity-UserPut it at /robots.txt, keep your existing rules, and check the
result with the checker or by fetching the file
yourself.
| Crawler | Operator | Purpose | Block? |
|---|---|---|---|
GPTBot | OpenAI | Collects content for model training | Safe to block |
ClaudeBot | Anthropic | Collects content for model training | Safe to block |
anthropic-ai | Anthropic | Older token, still seen in the wild | Safe to block |
Google-Extended | Controls Gemini training use. Does not affect Google Search ranking | Safe to block | |
Applebot-Extended | Apple | Training opt-out. Distinct from Applebot, which is search | Safe to block |
CCBot | Common Crawl | Public crawl corpus, widely used as training data | Safe to block |
Bytespider | ByteDance | Training data collection | Safe to block |
meta-externalagent | Meta | Training data collection | Safe to block |
Amazonbot | Amazon | Crawling for Amazon services incl. Alexa | Your call |
Diffbot | Diffbot | Commercial structured-data extraction | Your call |
OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results | Do not block |
ChatGPT-User | OpenAI | Fetches a page because a user asked about it | Do not block |
Claude-SearchBot | Anthropic | Indexes pages for use in Claude's answers | Do not block |
Claude-User | Anthropic | Fetches a page on a user's behalf | Do not block |
PerplexityBot | Perplexity | Indexes pages for citation in answers | Do not block |
Perplexity-User | Perplexity | Fetches a page on a user's behalf | Do not block |
Operator documentation: OpenAI · Anthropic · Google · Apple · Common Crawl. Check these before trusting any list, including this one — tokens change.
The mistake worth avoiding. Blocking Google-Extended
does not remove you from Google Search, and blocking
Googlebot does. They are separate tokens with separate effects, and
conflating them is the most expensive error available on this page.
Across 76 B2B SaaS domains, measured on 30 July 2026:
User-agent: * wildcard.Some AI browsing does not identify itself at all. OpenAI's ChatGPT Atlas browser reportedly sends a standard Chrome user-agent with no distinguishing token, which means robots.txt cannot address it — it is indistinguishable from an ordinary visitor.
Bing Copilot rides on Bingbot. You cannot block Copilot without also blocking Bing Search, because it is the same crawler. If someone tells you to "block all AI", ask them what that costs in Bing.
It is a request, not a control. Compliance is voluntary. A crawler that ignores it is not prevented by it, and nothing in a text file changes that.
Enforcement happens at the server or CDN. If you need a real
block rather than a stated preference, that is a firewall or bot-management rule
returning 403. In our edge probe, 3 of 74 domains enforced this way.
Note that CDN rules are invisible in your own robots.txt — so your stated policy
and your actual behaviour can silently diverge.
It does not remove anything already collected. A model trained before you added the rule is unaffected, and archives like Common Crawl retain prior snapshots.
That is a legitimate choice — just make it knowingly. Add the six answer crawlers to the list above. You will lose eligibility for citation in ChatGPT, Claude and Perplexity answers, and you will not get a warning when it happens, because there is no report anywhere that tells you that you were excluded.