Does llms.txt Actually Work? The Evidence in Mid-2026
No major engine has committed to llms.txt and most files never get read. What the evidence actually shows, and why I ship one on every site anyway.
July 6, 2026 · 6-min read
Mostly no, with one real exception. As of mid-2026 there is no credible evidence that llms.txt improves your visibility in ChatGPT, Perplexity, or Google's AI Overviews. No major engine has committed to reading it. Google has said outright that it does nothing for Search. The largest study to date found that 97% of published llms.txt files got zero requests from AI crawlers in the month measured. I still ship one on every site I build, and one of my ventures generates them. Both of those things are true, and the gap between them is what this post is about.
What llms.txt is, in thirty seconds#
llms.txt is a plain markdown file that sits at the root of your domain, proposed by Jeremy Howard in September 2024. Where robots.txt tells crawlers what they may not do, llms.txt tells language models what they should read. First a short description of the site, then a curated list of your most important pages, ideally with clean markdown versions behind them. The pitch is that an LLM with a limited context window shouldn't have to wade through your nav, your cookie banner, and your footer to figure out what you do. Hand it a map instead.
That's the whole spec. A file, some links, some descriptions. Which is part of why it spread: it costs almost nothing to adopt, so a lot of people adopted it without asking whether anything reads it.
What proponents claim#
The claims stack up roughly like this, from modest to ambitious:
- Token efficiency. A model retrieving your content burns fewer tokens on a curated markdown file than on rendered HTML.
- Narrative control. You decide which pages represent you, instead of letting a crawler guess.
- Citation lift. The big one: sites with llms.txt supposedly get cited more often by AI assistants, which is where the "AEO ranking factor" framing comes from.
The first claim is mechanically true whenever a tool actually fetches the file. The second is true in the same conditional way. The third is where the evidence gets thin, and it's the one most of the hype is built on.
What the evidence actually says#
Three buckets worth separating: what engines say, what crawler logs show, and what large-scale studies found.
The engines have not signed on#
Google is the clearest case because they've spoken on the record. John Mueller has said no AI system at Google uses llms.txt, and he compared the file to the keywords meta tag, a signal search engines abandoned because site owners control it and can say anything they like. His most generous framing was that it might serve as a temporary crutch for AI coding tools, saving them some tokens, and that it has nothing to do with Search. Google's own guidance on generative AI features says machine-readable files like llms.txt aren't needed to appear in them. OpenAI, Anthropic, and Perplexity have published nothing committing their consumer search products to reading it either. So the strong version of the claim, that llms.txt is a ranking or citation factor in AI search, has no engine behind it.
Some crawlers do fetch it#
This is the honest complication. If you watch server logs (crawler analytics is part of what we built into Cite-met, so I watch a lot of them), you do see AI user agents request llms.txt on some sites. Fetching isn't using, though. A crawler pulling the file tells you nothing about whether the content influenced an answer, and nobody outside the labs can see that far into the pipeline. The most you can say from logs is: the file is not universally ignored.
The adoption-versus-usage gap is stark#
Ahrefs ran the biggest study I've seen, across 137,000 domains. Adoption was surprisingly high, roughly 38,000 domains with a valid file. Usage was the opposite: 97% of those files got zero AI crawler requests in the month they measured. Tens of thousands of site owners did the work; almost nothing came to read it. That's about as clean a verdict as observational data gives you.
The one place it clearly earns its keep#
AI coding assistants. Cursor, Copilot, and Claude-based tools retrieve live documentation when a developer asks a product-specific question. A curated llms.txt reliably helps that retrieval land on the right page instead of a hallucinated endpoint. This is why Stripe, Vercel, Cloudflare, and Anthropic all ship one. Notice that this matches Mueller's dismissal exactly: his framing was a token-saving crutch for coding tools, and that's precisely the context where it works. If you're a developer-tool company with docs, shipping llms.txt is basic hygiene. For everyone else, it's a bet.
Why I ship it anyway#
Full disclosure first: Cite-met, one of my ventures, generates llms.txt as part of making AI-built sites legible to LLMs and search (static rendering, crawler analytics, the file itself). So I have every incentive to tell you it's a ranking silver bullet. It isn't, and I'd rather keep the credibility.
Here's the actual reasoning:
It's nearly free. Generated at build time, maintained automatically. The cost is minutes, not hours, and it never touches what human visitors see.
There's no downside case. Engines that ignore it just ignore it. Nobody has produced evidence of a penalty, and structurally there's nothing to penalize: it's a text file with links to pages you already publish.
It's the right shape for where agents are heading. Agents budget tokens. An agent visiting your site to answer a question wants exactly what llms.txt is: a short, curated, markdown map. Today's consumer crawlers mostly skip it. The agent traffic I expect over the next few years has every reason not to. Cheap insurance against a plausible future is a fine trade at this price.
Writing it is a useful forcing function. Deciding which ten pages represent your business is a positioning exercise disguised as a config file. Most teams have never made that call explicitly.
So the verdict I'd actually defend: llms.txt is a low-cost option on the future, a proven aid for developer docs, and not currently a visibility lever for anyone else. Anyone selling it as an AI-search ranking factor is ahead of the evidence.
What to rely on instead#
If your goal is getting recommended by AI assistants, the file is maybe 2% of the job. The other 98% looks like this:
Be trivially crawlable. Server-rendered HTML, content visible without JavaScript, fast responses, no bot-walls on your key pages. Most AI crawlers don't execute JS. If your product pages only exist client-side, no root file saves you.
Keep your entity consistent. Same name, same one-line description, same category everywhere: your site, LinkedIn, directories, schema markup. Models assemble an understanding of who you are from many sources, and contradictions blur it.
Publish things worth citing. Original data, honest comparisons, pages that answer a specific question completely in the first paragraph. Models cite sources the way careful writers do: they reach for the page that actually settles the question. I went deeper on this whole chain in why ChatGPT doesn't recommend your product, and the retrieval-and-citation mechanics live in the Agentic Marketing OS if you want the framework version.
Common questions#
Does Google use llms.txt?#
No. Google has said explicitly that llms.txt is not used for Search or AI Overviews, and John Mueller compared it to the long-dead keywords meta tag. Appearing in Google's AI features runs on the same signals as regular Search: crawlability, indexability, and content quality.
Should I still add one?#
If it takes under an hour, yes. The cost is trivial, there's no known downside, and it's genuinely useful if AI coding tools ever retrieve your docs. Just don't reallocate real SEO or content budget to it, and don't expect measurable citation lift.
Does llms.txt replace robots.txt or sitemap.xml?#
No. robots.txt controls crawler permissions and sitemap.xml enumerates URLs for indexing; both have decades of confirmed engine support. llms.txt is an advisory reading list with no confirmed support from major engines. Keep all three; trust the first two.
If you're deciding where llms.txt sits in a broader AI-visibility plan, the Marketing OS is the free place to start. The ChatGPT-recommendation piece above covers the citation side in more depth. If you'd rather have someone build the whole pipeline than read about it, that's roughly what my engagements look like. Either way: ship the file, spend your real effort on being worth citing.
Operator notes, monthly.
Working notes on agentic marketing, Claude Code skills, and the operating models behind four ventures. It ships when there is something worth reading.