7 of 76 B2B SaaS domains (9.2%) disallow at least one named AI crawler
in robots.txt. Only 1 of 76 (1.3%) disallows a crawler used
for answering rather than training.
None of the blocks were accidental. Every one came from a rule
naming a specific bot. Not a single domain caught an AI crawler through a
catch-all User-agent: * rule, which is the failure mode usually
assumed when this topic comes up.
Two limits on this result are serious enough to state before the detail rather than after it. First, this sample is large, well-resourced SaaS companies — the population most likely to have already thought about this. It says nothing about small business sites. Second, the edge-layer probe further down cannot verify itself as a real bot, so it measures something narrower than it appears to. Both are expanded in Limits.
Looking for what to actually do? The robots.txt guide gives copy-paste rules that block training crawlers while keeping you eligible for AI answers.
| Measure | Count | Rate |
|---|---|---|
| Domains blocking ≥1 AI crawler (robots.txt) | 7 / 76 | 9.2% |
| Domains blocking ≥1 search/answer crawler | 1 / 76 | 1.3% |
| Blocks arising from a wildcard rule | 0 / 7 | 0% |
| Domains that would not serve robots.txt at all | 4 / 80 | 5.0% |
| Domains treating AI-bot-shaped requests differently at the edge | 4 / 74 | 5.4% |
— of those, a hard 403 rather than rate limiting | 3 / 74 | 4.1% |
80 B2B SaaS domains were selected across seven product categories — project management, CRM and marketing automation, HR and payroll, design and collaboration, e-signature and documents, developer tooling, and billing. The list is published with the data. Selection was by category coverage, not by traffic rank, and it was fixed before any measurement was taken.
For each domain, robots.txt was fetched and parsed into
user-agent groups. Applicability was resolved the way the standard specifies:
longest matching user-agent group wins, a bot's own group takes precedence over
*, and an Allow beats a Disallow of equal
specificity. The question asked of each group was whether / is
disallowed for that crawler.
Sixteen named crawlers were tested, split into two classes, because the distinction is what makes the result interesting:
GPTBot, ClaudeBot, anthropic-ai,
CCBot, Google-Extended, Applebot-Extended,
Bytespider, meta-externalagent, Amazonbot,
Diffbot. Blocking these keeps your content out of model training. It
is a legitimate editorial choice and costs you nothing in visibility.
OAI-SearchBot, ChatGPT-User,
Claude-SearchBot, Claude-User,
PerplexityBot, Perplexity-User. These fetch pages in
order to cite them in answers. Blocking these removes you from AI answers
entirely, which is usually not what anyone intends.
Per-crawler block rates were low across the board, with a maximum of 3.9%.
| Crawler | Class | Blocked by | Rate |
|---|---|---|---|
GPTBot | training | 3 | 3.9% |
CCBot | training | 3 | 3.9% |
Amazonbot | training | 3 | 3.9% |
anthropic-ai | training | 2 | 2.6% |
ClaudeBot | training | 1 | 1.3% |
Google-Extended | training | 1 | 1.3% |
Applebot-Extended | training | 1 | 1.3% |
Bytespider | training | 1 | 1.3% |
meta-externalagent | training | 1 | 1.3% |
Diffbot | training | 1 | 1.3% |
OAI-SearchBot | search | 1 | 1.3% |
ChatGPT-User | search | 1 | 1.3% |
Claude-SearchBot | search | 1 | 1.3% |
PerplexityBot | search | 1 | 1.3% |
Claude-User | search | 0 | 0% |
Perplexity-User | search | 0 | 0% |
Listed because a rate without the underlying cases is not checkable. Each of
these is a public robots.txt you can read yourself.
| Domain | Bots blocked | Which |
|---|---|---|
| canva.com | 8 | GPTBot, ClaudeBot, anthropic-ai, CCBot, Applebot-Extended, Bytespider, meta-externalagent, Diffbot |
| figma.com | 8 | GPTBot, anthropic-ai, CCBot, Google-Extended, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot |
| calendly.com | 1 | CCBot |
| notion.com | 1 | Amazonbot |
| activecampaign.com | 1 | Amazonbot |
| pandadoc.com | 1 | Amazonbot |
| loom.com | 1 | GPTBot |
The two patterns worth noticing. Canva blocks training crawlers and leaves every search crawler free — content stays out of training sets while remaining quotable in AI answers. That is a coherent policy, and it is the one most sites appear to want.
Figma is the only domain in the sample that blocks search and answer crawlers, covering OpenAI, Anthropic and Perplexity's fetchers. Whether that is deliberate is not something a robots.txt file can tell you. It is the single case in 76 where the "you may be invisible to AI answers without knowing it" concern actually describes a real site.
gusto.com and justworks.com returned
403; brevo.com and drift.com returned
404. These are excluded from all robots.txt percentages above rather
than counted as non-blocking. A 403 on robots.txt is its
own kind of finding: a crawler that cannot read your rules does not get to follow
them.
A robots.txt file is a request, not an enforcement mechanism. CDN
bot-management rules can block a crawler at the edge without appearing in
robots.txt at all — so a site owner reading their own robots.txt can
be looking at rules that no longer describe what their server does.
To test that, a real page was requested from each domain four times, using user
agents carrying the GPTBot, OAI-SearchBot,
ClaudeBot and PerplexityBot tokens, and the status codes
compared against an ordinary browser baseline. 74 of 80 domains returned a clean
200 baseline and are included.
| Domain | GPTBot | OAI-SearchBot | ClaudeBot | PerplexityBot |
|---|---|---|---|---|
| rippling.com | 403 | 403 | 403 | 403 |
| bamboohr.com | 403 | 403 | 403 | 403 |
| lemonsqueezy.com | 403 | 403 | 403 | 403 |
| dropbox.com | 200 | 200 | 429 | 429 |
4 of 74 domains (5.4%) responded differently, and only 3 (4.1%) with a
hard block. Dropbox returned 429 — rate limiting, not a
block, and a different thing entirely.
The three 403 responses were uniform across all four bots, which
reads as a deliberate policy rather than an accident. Edge-layer blocking
was not more common than robots.txt blocking in this sample. It was less
common — which is the opposite of what the "invisible blocking" framing
predicts.
These are established SaaS companies with staff who own their robots.txt. That is the group most likely to have already made a deliberate choice, and the low rate here is consistent with competence rather than with the problem not existing. A sample of small business sites on hosted platforms could produce a completely different number in either direction, and nothing here should be read as describing them.
Cloudflare and comparable services verify crawlers by IP range and request signature, not by user-agent string alone. This probe sent bot-identifying user agents from an ordinary IP address, so it was an unverified bot claim throughout.
Real verified crawlers may be treated more permissively than this probe was, and unverified ones more harshly. So the correct reading of the edge result is "does this origin treat AI-bot-shaped requests differently" — not "would GPTBot be blocked." Those are different questions and only the first was measured.
Every figure describes 30 July 2026. robots.txt files and CDN rules both change without announcement, and a single snapshot cannot distinguish a standing policy from a temporary one.
Nothing here treats a block as a mistake. Keeping content out of training corpora is a reasonable position, and the data cannot tell you which of these blocks were intended. The only case where intent is genuinely in question is a site blocking search crawlers while permitting training crawlers, and that pattern did not occur even once.
Every percentage on this page is derived from those two files and can be recomputed from them. For example, the headline figure:
node -e "const a=require('./ai-crawler-robots-2026-07.json');
const ok=a.filter(d=>d.ok), bots=Object.keys(ok[0].bots);
const hit=ok.filter(d=>bots.some(b=>d.bots[b].blocked));
console.log(hit.length+'/'+ok.length, (100*hit.length/ok.length).toFixed(1)+'%');"
// 7/76 9.2%The data is published under CC BY 4.0. If you quote a number from it, a link back to this page is all that is asked.