Does llms.txt work? We ship one, and the evidence says it barely matters

The verdict

Probably not — not for the thing most people ship it for, anyway. llms.txt gets pitched as a lever for AI visibility: add the file, get cited by ChatGPT, Perplexity, and Google’s AI answers. Three independent measurements, using three different methods, find no such effect. SE Ranking tested the file against citation frequency across roughly 300,000 domains and found no correlation. Ahrefs read the server logs of 137,000 sites and found that 97% of published llms.txt files were never fetched by anything at all. Otterly logged 90 days of AI bot traffic on a domain with a correctly implemented file and watched 0.1% of that traffic touch it.

We should be the last people telling you this. We sell white-hat GEO, and we ship an llms.txt on specmora.com right now. But the honest reading of the evidence is that the file has exactly one real, measurable audience — coding agents reading developer documentation — and no detectable effect on whether AI answers cite you. We keep ours because it costs nothing and that one audience is real. We don’t bill it as a growth lever, and you should be skeptical of anyone who does.

What llms.txt was designed to do

The proposal came from Jeremy Howard in September 2024. The idea, per the spec at llmstxt.org, is a markdown file at /llms.txt: an H1 with the site name, a short summary, then curated lists of links to the pages that matter, so a language model using a website at inference time can find the right content without burning its context window on navigation, ads, and markup.

Two things follow from that design that the marketing around the file tends to blur. First, llms.txt is not robots.txt for AI. It doesn’t control what crawlers may fetch; it’s a menu, not a gate. Second, it was never proposed as a ranking or citation signal. It helps a model that has already decided to use your site do so more efficiently. Whether anything actually reads the menu is an empirical question — and that’s the question the measurements answered.

No engine says it uses the file

Before the log data existed, there was already a telling silence: no major answer engine — not Google, not OpenAI, not Anthropic, not Perplexity — has stated that it uses llms.txt for search or answers. Google’s John Mueller put it bluntly in April 2025: “AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it).” He compared it to the keywords meta tag — a self-declared description of your site that search engines learned to ignore because it was free to fake, when they could just read the site instead.

That was one person’s observation about an absence. The studies that followed checked actual behavior at scale, and they backed him up.

What the measurements found

SE Ranking: no correlation across ~300,000 domains. SE Ranking analyzed nearly 300,000 domains — about 10% had the file — and tested whether having llms.txt related to how often a domain gets cited in LLM answers, using Spearman correlation, an XGBoost model, and SHAP factor analysis. No relationship. The detail that stings: when they removed the llms.txt variable from the model, its predictions got better. The file wasn’t even a weak signal. It was noise.

Ahrefs: 97% of the files are never read. In June 2026, Ahrefs analyzed server logs and bot analytics for 137,210 domains. About 28% published an llms.txt file — a high number, but Ahrefs’ customer base skews technical, so treat it as a ceiling. The headline: 97% of those files received zero requests in the measured month. Not few requests — none, from anything, AI or otherwise. And of the requests that did reach the remaining 3% of files, 96% came from bots, with the single biggest category being SEO audit tools at 21.7% — the industry inspecting its own artifact. AI retrieval bots, the ones that fetch pages to build answers for actual users, accounted for 1.1% of fetches.

Otterly: 0.1% of AI bot traffic. OtterlyAI instrumented a domain with a correctly implemented llms.txt and logged AI bot traffic for 90 days: more than 62,100 AI bot visits, of which 84 requested the file — about 0.1%, roughly a third of the traffic an average content page on the same site received. Their conclusion was direct enough that they removed the llms.txt check from their own GEO audit product, on the grounds that it was pulling attention away from factors that actually move citations.

Three unrelated methodologies — citation-level correlation, log analysis at scale, single-site logs in depth — arriving at the same place is about as strong as evidence gets in this space. The file is not being read by the systems people publish it for, and having it does not correlate with being cited.

Why it spread anyway

It’s worth being honest about why a file with no evidence behind it ended up on so many audit checklists, including — for a while — ones run by people who should know better. The analogy sold itself: robots.txt and sitemaps are files you put at your web root, and they matter, so a new file at the web root for the new kind of crawler feels like the same species of work. It’s cheap to recommend, easy to verify you did it, and it lets an agency show a deliverable in week one. None of that makes it do anything. The pattern is familiar from twenty years of SEO folklore — the tactics that spread fastest are the ones that are easy to execute, not the ones that are measured to work.

The one place it earns its keep

The Ahrefs bot breakdown contained one genuine surprise, and it points at the file’s actual constituency. Among the AI tools that did fetch llms.txt, coding agents and agentic infrastructure led at 10.5% of requests. GPTBot pulled the file most often overall (about 4.51% of fetches), but set it and a single research crawler aside and Claude Code, Anthropic’s coding agent, out-fetched every other AI retrieval bot, assistant, and training crawler in the dataset.

That fits the original design brief exactly. When a developer asks a coding agent a question about your API, the agent has to navigate your documentation inside a limited context window. A curated markdown map of the docs is genuinely useful to it — which is why the places llms.txt actually lives and works are developer-documentation platforms. Mintlify generates llms.txt (and a full-content llms-full.txt) automatically for every docs site it hosts, and Anthropic serves one for its own API documentation.

So the honest split is: if you publish developer docs, ship the file — it has a real reader today. If you publish a marketing site, the evidence says nothing is coming to read it.

Why we ship one anyway

Because the cost is as close to zero as anything in this business gets. Generating the file takes minutes, and increasingly your platform does it for you — Ahrefs’ own recommendation, after publishing the most damning data anyone has collected, was to let your CMS or framework generate it rather than hand-crafting one. Nothing penalizes its presence. And the coding-agent traffic is real, if small: as more research and buying gets delegated to agents, leaving a clean map out for them costs nothing and might occasionally help one.

Two cautions before you copy that reasoning. The real risk of llms.txt was never harm; it’s opportunity cost. If the file sits above server-side rendering, crawler access, or sourced content on your GEO checklist, the checklist is upside down — those are the levers with measured evidence behind them, and how AI engines choose sources walks through that evidence. Second, keep the file consistent with your live pages. A curated file that says something different from your site is the same shape as cloaking, which is exactly the manipulation risk Mueller flagged and the kind of mismatch engines have spent two decades learning to detect. If your CMS auto-generates it, read the output once — agents ingest the file trustingly, and Ahrefs even found a crawler studying llms.txt files as a prompt-injection surface.

Where this leaves you

llms.txt in 2026 is a cheap, harmless artifact with one real audience and no measured effect on citations. Ship it if it’s free, skip it without guilt if it isn’t, and spend the actual effort where the measurements point: being in the underlying index, being readable without JavaScript, and writing answer-first content a model can lift and attribute. That’s the work — we’ve laid out what it looks like in what white-hat GEO actually is — and no text file at your web root substitutes for it.

Sources

1. The /llms.txt file — Jeremy Howard’s original proposal (September 2024): a markdown file to help LLMs use a website at inference time (llmstxt.org).
2. LLMs.txt: Why Brands Rely On It and Why It Doesn’t Work — ~300,000-domain analysis; ~10% adoption; Spearman, XGBoost, and SHAP find no relationship with AI citations, and removing the variable improved the model (SE Ranking).
3. We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read — server-log analysis of 137,210 domains: 28% adoption, 97% of files got zero requests; SEO audit tools 21.7% of fetches, AI retrieval bots 1.1%; GPTBot the single most frequent AI fetcher of llms.txt (~4.51%), Claude Code the top coding-agent fetcher (Ahrefs, June 2026).
4. llms.txt and AI Visibility: Results from OtterlyAI’s GEO Study — 90-day log study: 84 of 62,100+ AI bot visits (~0.1%) requested the file; Otterly removed llms.txt from its GEO audit (OtterlyAI, February 2026).
5. Google Says LLMs.Txt Comparable To Keywords Meta Tag — John Mueller: no AI service has said it uses the file, and server logs show they don’t check for it (Search Engine Journal, April 2025).
6. llms.txt — Mintlify documentation — docs platform that auto-generates llms.txt and llms-full.txt for every hosted documentation site (Mintlify).
7. Anthropic API documentation llms.txt — a live example of the file’s real use case: a curated index of developer docs for agents (Anthropic).