Among technical sites, yes — 86 of 159 domains (54.1%) serve a real
llms.txt. Across the wider web it is far lower: 28% of 137K
domains (Ahrefs) and 8.7% of the top 1,000 (Rankability). And 97% of the
files that do exist are never fetched by anything — see below.
But the two groups with the most obvious reason to adopt it have the
least. Documentation platforms are the lowest category measured at 20%,
and AI companies sit below average at 48%. Mintlify,
GitBook, ReadMe, Read the Docs,
Docusaurus, Swagger and Stoplight all return
404 — several of them while generating llms.txt files for their own
customers.
The llms.txt proposal asks sites to publish a
markdown file at /llms.txt that indexes their documentation for
language models, on the reasoning that models have limited context and HTML pages
are mostly navigation.
Ahrefs studied 137,210 domains and found that 97% of llms.txt files received zero requests in May 2026 — nothing fetched them at all. Of the 3% that did get traffic, only 19.5% came from named AI tools; most of the rest was SEO tooling and general crawlers (Ahrefs, 15 June 2026).
That is the most decision-relevant fact on this page and it is not ours. If you are deciding whether to publish an llms.txt, the adoption rate below is the less important number.
This question has been measured before, at far larger scale:
| Study | Sample | Adoption |
|---|---|---|
| Ahrefs, Jun 2026 | 137,210 domains | 28% |
| Rankability, Jun 2026 | Top 1,000 (Tranco) | 8.7% |
| This study, Jul 2026 | 171 technical domains | 54.1% |
Our rate is high because our sample is deliberately skewed — 171 hand-picked technical domains, not a general or top-ranked population. Read together the three are consistent and tell a clearer story than any one alone: adoption rises steeply with how developer-facing a site is, from 8.7% of the general top 1,000 to 54.1% of API-first and dev-tool companies.
What is genuinely new here is the category breakdown, not the headline rate — specifically that documentation platforms are the lowest category at 20%, which is the opposite of what you would predict. Rankability already excludes soft-404s in their published methodology, so the soft-404 discussion below is a caution for readers rather than a novel contribution.
Checking your own site? The llms.txt checker runs the same classification used here against any domain, including the soft-404 test described below.
| Measure | Count | Rate |
|---|---|---|
Serves a real llms.txt | 86 / 159 | 54.1% |
| — and it conforms (H1 + markdown links) | 76 / 159 | 47.8% |
Serves llms-full.txt as well | 17 / 159 | 10.7% |
Returns HTML at /llms.txt (soft 404) | 10 / 159 | 6.3% |
Points to it from robots.txt | 12 / 159 | 7.5% |
Genuine 404 — no file | 62 / 159 | 39.0% |
Blocked the probe (403/429) — excluded | 12 / 171 | — |
10 of 159 domains return HTTP 200 and an HTML page at
/llms.txt. Their site catches the unknown path and renders a
normal page — often the homepage or a styled 404 — with a success status.
A probe that checks only the status code counts every one of these as adoption and reports 60.4% instead of 54.1%. Any published figure that does not say how it handled soft 404s should be treated as an upper bound — though the two larger studies cited above do handle it, Rankability explicitly.
The ten: ansible.com, apidog.com,
invisionapp.com, pandadoc.com, postmark.com,
remote.com, sendgrid.com, terraform.io,
trello.com, wandb.ai. One further domain,
deepgram.com, returns 200 with an entirely empty body.
| Category | Determinable | Serves llms.txt | Rate |
|---|---|---|---|
| API-first companies | 30 | 21 | 70% |
| Ecommerce & site builders | 12 | 7 | 58% |
| Dev tools & infrastructure | 32 | 18 | 56% |
| B2B SaaS | 47 | 26 | 55% |
| AI / ML companies | 23 | 11 | 48% |
| Documentation platforms | 15 | 3 | 20% |
API-first companies lead at 70%, which is the intuitive result: their product is their documentation, and developers increasingly read it through a model.
Only postman.com, redocly.com and
scalar.com serve one. Missing: mintlify.com,
gitbook.com, readme.com, readthedocs.org,
docusaurus.io, swagger.io, stoplight.io,
bump.sh, insomnia.rest, nextra.site.
The likely explanation is not hypocrisy, and it is more interesting than that. Several of these vendors ship llms.txt generation as a product feature — the file appears on their customers' documentation sites, not on their own marketing homepage. That would also help explain the high overall rate: adoption may be substantially a by-product of platform defaults rather than a series of deliberate decisions by individual site owners.
This measurement cannot distinguish those two causes, and the distinction matters, so it is stated as a hypothesis rather than a finding. Testing it would need per-site detection of the docs platform in use, which is the obvious next study.
Of 23 determinable AI/ML domains, 11 serve llms.txt. Absent with a
clean 404: anthropic.com, huggingface.co,
langchain.com, llamaindex.ai, groq.com,
fireworks.ai, chroma.com, scale.com,
stability.ai, labelbox.com.
openai.com and perplexity.ai blocked the probe entirely
and are excluded rather than counted as absent.
Present: assemblyai.com, cohere.com,
elevenlabs.io, mistral.ai, modal.com,
pinecone.io, qdrant.tech, replicate.com,
runpod.io, together.ai, weaviate.io.
Only 12 of 159 domains (7.5%) reference llms.txt
from robots.txt. There is no registered link relation for it and no
autodiscovery mechanism in the proposal, so in practice a consumer has to guess the
well-known path. A file that half of sites publish and 7.5% advertise is a
discovery problem more than an adoption problem.
Conformance here is deliberately loose: an H1 and at least one markdown link. That tests shape, not quality. A file that is current, accurate and actually indexes the documentation is a much higher bar than 47.8% of sites are being credited with, and nothing here measures whether the linked content is any good.
Eleven returned 403 and one 429 to a plainly
identified audit user agent. They are excluded from all denominators rather than
counted as non-adopters, because a blocked request is not evidence of absence.
Note this cuts the headline both ways: if all twelve had files, adoption would be
57.3%; if none did, 50.3%.
171 domains, selected for category coverage rather than traffic rank, fixed before measurement. Ahrefs' sample is 800× larger. Where the headline rate is concerned, prefer their number over ours; 54.1% describes technical sites specifically and is close to a best case. What this sample is good for is the relative comparison between categories, where a hand-built stratification beats a large undifferentiated crawl.
Everything describes 30 July 2026. Requests followed redirects, so a domain
serving its file from a www or docs subdomain is counted as present.
llms-full.txt and robots.txt findingsnode -e "const r=require('./llms-txt-2026-07.json');
const bad=x=>/^http_(403|429|0|timeout)/.test(x.llmsTxt.reason||'');
const det=r.filter(x=>!bad(x)), p=det.filter(x=>x.llmsTxt.present);
console.log(p.length+'/'+det.length, (100*p.length/det.length).toFixed(1)+'%');"
// 86/159 54.1%Published under CC BY 4.0. If you quote a number, a link back is all that is asked.