Does llms.txt Work? What a Year of Server Logs Says

Does llms.txt work? For getting found by AI, no. Ahrefs read the server logs of 137,210 domains for May 2026 and found that 97% of the llms.txt files on them were never requested, by anything. For helping a coding agent that is already reading your docs, yes. Mintlify tested 20 documentation sites and agents made 2.23 failed page requests per task on plain HTML, and 0.11 when the pages were Markdown with a link to llms.txt.

A month ago I wrote that DevRel now serves two audiences and that the team should own llms.txt and the MCP server because that is where discovery happens. Half of that holds up. The other half was wrong, and the logs show which half.

What llms.txt was supposed to do

The idea came from Jeremy Howard in September 2024. You put a Markdown file at /llms.txt with a short description of the site and a list of links to the pages that matter, and a model that lands on your domain reads that file first instead of fighting its way through your navigation, cookie banner and JavaScript. The pitch was robots.txt for the reading side rather than the blocking side.

That pitch needs one thing to be true: something has to fetch the file. Not a person, not an SEO plugin, but the bot or the agent that is about to answer a developer’s question using your docs.

Who fetches llms.txt?

Ahrefs looked at 137,210 domains for May 2026 using its web analytics and bot analytics products. 28% of those domains published a valid llms.txt. 97% of those files got zero requests for the whole month. Of the 3% that got any traffic at all, 96% of it was bots, and the biggest group of those bots was SEO audit tools at 21.7% of requests. The AI training crawlers were at 5.3%. GPTBot was the busiest single AI bot and it made 4.51% of the requests. The search-side crawlers from OpenAI, Perplexity and Anthropic combined added up to “only a couple of hundred fetches” across the whole dataset.

The detail I keep coming back to is the missing files. Ahrefs checked whether AI bots request /llms.txt on domains that do not have one. The answer was zero. “They never go looking.” A crawler that wanted the file would try the URL the way every crawler tries /robots.txt. None of them do.

A smaller study backs that up with the kind of numbers you can check against your own logs. Saaslinks watched 18 named AI crawler families for 14 days to 3 August 2026. Those crawlers fetched robots.txt 723 times, fetched 151 of the site’s own pages, and fetched llms.txt zero times. ChatGPT-User alone made 82 page requests. It did not want the index.

Google said the same thing twice. On 17 June 2025 John Mueller wrote on Bluesky, “FWIW no AI system currently uses llms.txt,” and told people to check their own server logs. A year later, in June 2026, he called the whole thing “purely speculative for now (the file has existed for years, yet none of the AI systems use it)” and pointed technical SEOs at WebMCP instead, the Google and Microsoft proposal that went into a Chrome origin trial at I/O 2026.

Adoption went up anyway

None of that slowed anyone down. Originality.ai counted 4,088 sites with an llms.txt in June 2025 across the 3 million it monitors. By May 2026 that was 36,120, an 8.8x rise in eleven months. The longer llms-full.txt variant went from 23 sites to 2,463.

The big sites were slower. Rankability checked the Tranco top 10,000 on 18 September 2026 and found 9.3% of the top 1,000 serving a confirmed file. Cloudflare has one, so do GitHub, Fastly, WordPress.org and Adobe. Originality found 7.4% of the Fortune 500 had one in March 2026, against 92.8% with a robots.txt.

So we have a file that tens of thousands of sites publish and that the crawlers it was written for do not read. On its own that would be the end of the post. It is not the end, because the crawlers are not the only reader any more.

Agents are reading your docs more than people are

Mintlify published 30 days of traffic across the documentation sites it hosts on 3 April 2026. AI agents made 357.6 million requests, 45.3% of the total. Browsers made 361.7 million, 45.8%. Claude Code on its own made 199.4 million requests, more than Chrome on Windows. Cursor made 142.3 million. ChatGPT made 1.8 million, NotebookLM 1.4 million. OpenAI’s Codex does not send an identifiable header, so the agent share is an undercount.

By August the line had been crossed. Mintlify’s 2026 State of Knowledge report, published 16 September, counts 257 million agent requests against 131 million human page loads for the month. MCP tool calls against its hosted servers tripled between February and August. And 83% of the agent traffic came in through what Mintlify calls machine-friendly routes: Markdown versions of pages, llms.txt, and agent skills.

Read those two sets of numbers side by side and they look like they disagree. Ahrefs says nothing reads llms.txt. Mintlify says agents arrive through it. Both are right, and the gap between them is the whole strategy question.

The crawler and the agent are different readers

Ahrefs measured the open web. Most of its 137,210 domains are shops, publishers and company sites, and the bots that visit those are training crawlers and search crawlers. GPTBot, ClaudeBot, PerplexityBot. Those bots fetch HTML in bulk to train or to ground a chat answer, and they have no reason to read a hand-written index when they can take every page.

Mintlify measured developer documentation, and the reader there is a coding agent sitting in a developer’s terminal with a task. Claude Code or Cursor is trying to call your API right now. It fetches a page because a CLAUDE.md, a skill, a search result or the developer pointed it there, and once it is on the page it follows links to find the next one. That agent does read llms.txt, when it is told where the file is. Ahrefs saw this too: in its data the Claude-Code user agent fetched llms.txt more than every AI retrieval bot, assistant and training crawler. That is the agent, not the crawler.

Nobody is probing for the file by default. A Claude Code issue from June 2025 asked for llms.txt to be picked up automatically the way Cursor lets you add the links in settings. The reply from the maintainer was “Claude should be able to fetch urls that you paste in,” and the issue closed without a built-in. You point the agent at the file, or you link to it from the page the agent is already reading. The file does nothing on its own.

That is where Mintlify’s test matters. The report describes 2,400 controlled tasks across 20 documentation sites. Agents answered correctly 94 to 99% of the time whatever format the docs were in, so the content was never the problem. The problem was requests for pages that do not exist. Agents guess URLs. On plain HTML they made 2.23 failed requests per task. On Markdown pages, 1.42. On Markdown pages that link to llms.txt, 0.11. Token use dropped by up to 60% on the same tasks.

Two caveats before you take that to a planning meeting. Mintlify sells the product that does this, so the test is a vendor benchmark, and the full method is in its report rather than a paper. And the 20x number is failed requests, not correct answers. The agent got the answer either way. It just burned fewer calls and fewer tokens getting there.

What developers say they trust

The human side moved as well, and in a direction that favours docs. Stack Overflow’s 2026 Developer Survey, published 6 October from over 30,000 developers in 169 countries, has 66% using a coding assistant or agent and 6.6% willing to trust AI with an important decision. Nearly half trust it when they can verify the output. 78.7% say source attribution is important or very important when they judge an AI answer.

Asked what context an AI needs to get things right, 73% said the underlying code and the technical documentation, second only to project goals at 85%. Asked where they go to check a specific answer, 72% go to a colleague, 63% to the code, 60% to the docs. Technical documentation is still the top learning source at 58.9%, ahead of AI code tools at 52.7%.

Put that next to the Mintlify traffic. The agent reads your docs to write the code, and the developer reads your docs to check what the agent wrote. Same page, two readers, and the second reader is the one deciding whether to trust the first.

What this means for a DevRel team

In September I said the new surfaces give DevRel its first measurement story an executive can read. I still think that. The thing I got wrong was which surface. Here is what the evidence supports now.

Stop treating llms.txt as discovery. Nobody arrives through it. If your plan for being recommended by ChatGPT is a file at the root, the Ahrefs logs say that plan is doing nothing, and Google has said so twice. Discovery still runs on whatever the search crawler and the training crawler can read in your HTML, and on what developers paste into the agent.

Publish it anyway, and link to it from every Markdown page. The cost is an afternoon. The payoff is an agent that stops guessing URLs on your site. The 20x in Mintlify’s test came from the link on the page, not from the file at the root. If your docs host cannot serve a Markdown version of each page, that is the thing to fix first, because the agent is reading the HTML and the HTML is where it got 2.23 failures per task.

Find your 404s. This one is free. Filter your docs logs by the Claude-Code and Cursor user agents and sort by status code. Every 404 is a URL an agent guessed from a pattern it saw elsewhere. The guessed URL tells you what the agent expected your docs to be called, and a redirect from the guess to the real page fixes it for every agent that follows.

Count the agent traffic, and count it separately. 24% of the companies Mintlify surveyed still lump agent traffic in with bots. If your analytics does that, your docs look like they lost half their readers over the past year when the readers moved from the browser to the terminal. The State of DevRel report has 61% of DevRel teams finding it hard to prove their impact and 18% able to link their work to revenue. A page that an agent fetched during a successful integration is a cleaner signal than a page view ever was, and right now most teams are throwing it away as bot noise.

Accept that the referral is gone. Cloudflare’s 1 July 2026 report has 52% of all crawler requests going to AI training, up from 22% in spring 2025, and more than half of all traffic on the internet now non-human. Some sectors have lost 40% of their human traffic in under a year. The agent that reads your docs does not send a human back to them. That was the measurement I used to lean on, the click from the chatbot to the signup page, and it is not coming. What you can measure is the API key that was created during a session where an agent was fetching your reference pages.

The MCP server is a different bet from the file. The llms.txt file is passive. An MCP server is something the agent calls, and Mintlify’s hosted servers saw those calls triple in six months. That is still a vendor’s number from a vendor’s platform, and the public registries are at a point where finding the right server is harder than building one, with roughly 9,400 published servers by April 2026 across the four big registries. Owning the server still looks right. Owning the file is cheap and should not be sold internally as more than it is.

The split in the brief was that DevRel serves the agent and the human. The logs say it is more specific than that. DevRel serves the agent that is already on the page, and the human who checks its work, and neither of them is the crawler the file was written for.

Frequently asked questions

Do ChatGPT or Claude read llms.txt?

Not when they crawl. Ahrefs’ May 2026 logs across 137,210 domains show GPTBot made 4.51% of the few llms.txt requests that happened and ClaudeBot under 1%. Neither probes for the file on sites that do not have it. A coding agent like Claude Code does fetch it when a page or a config file points at it.

Should I still publish an llms.txt file?

Yes if you run developer docs, no if you expect it to get you found. Put it at the root, keep it a short list of links to Markdown pages, and link to it from the Markdown version of every page. Do not spend a sprint on it. Mintlify’s tests put the win at fewer failed requests per task, not more traffic.

Why do agents fail on documentation?

They guess URLs. In Mintlify’s 2,400 controlled tests across 20 sites, agents got the answer right 94 to 99% of the time whatever the format. What went wrong was navigation: requests for pages that did not exist. Plain HTML averaged 2.23 failed requests per task, Markdown 1.42, Markdown with a link to llms.txt 0.11.

How much documentation traffic is now AI agents?

More than half on Mintlify-hosted docs. In August 2026 agents made 257 million requests against 131 million human page loads. In March 2026 it was 45.3% agents to 45.8% browsers, with Claude Code at 199.4 million requests and Cursor at 142.3 million. Other hosts have not published equivalent numbers.

The file was never the product; the page the agent lands on is, and a year of logs says to spend the budget there.