I Shipped an llms.txt, Then Read the Data on Whether It Does Anything

This site serves an llms.txt. It also serves an llms-full.txt, a markdown version of every page, a public JSON API, and an OpenAPI spec. I built all of it in one sitting because I'd read enough posts telling me AI search was the next distribution channel and that this file was the price of entry.

Then I went looking for evidence that any of it gets read. The honest version of what I found: one of those things is doing real work, and the file everyone talks about is not it.

What is llms.txt, exactly?

llms.txt is a plain text file at the root of a domain, written in markdown, that describes what a site contains and links to the pages worth reading. Jeremy Howard of Answer.AI proposed it in September 2024. The pitch was simple: HTML pages are full of navigation, scripts, and cookie banners that waste an LLM's context window, so give the model a clean map instead.

It's a convention, not a standard. Nobody at OpenAI, Anthropic, Google, or Perplexity ever agreed to read it. That distinction matters more than it sounds, and it's most of the answer to the next question.

Does llms.txt work in 2026?

For getting cited in AI search: no. There's no measured effect, from two independent studies at real scale.

Ahrefs checked 137,210 domains in May 2026 using their bot analytics data. 28% of them, 38,360 sites, published a valid llms.txt. 97% of those files received zero requests that month. Not zero AI requests. Zero requests, from anything.

SE Ranking ran a different test across roughly 300,000 domains and looked for a relationship between having the file and being cited by LLMs. They found none, using Spearman correlation and then an XGBoost model. The detail that stuck with me: when they removed llms.txt as a variable, the model's predictions got better. The file wasn't a weak signal, it was noise.

Google's John Mueller has been blunter than either study: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)." He compared it to the keywords meta tag, which is about as harsh as that comparison gets in SEO. The keywords tag is the canonical example of metadata that site owners diligently fill in and no system reads.

So the situation is: no provider has committed to reading it, the crawlers mostly don't request it, and where the file exists at scale there's no citation lift to find. Three separate ways of asking the same question, three answers pointing the same direction.

Then who is requesting the 3%?

This is the part that actually changed how I think about the file, and it's buried in the Ahrefs numbers rather than in the headline.

Of the small slice of llms.txt files that did get fetched, AI bots accounted for 19.5% of requests. SEO audit tools accounted for 21.7%, general web crawlers 13.1%, and tech profiling tools 11.6%. More than half the traffic to llms.txt is tooling checking whether you have an llms.txt.

And inside that AI slice, the biggest single consumer wasn't a search product. Agentic infrastructure was 10.5%, training crawlers 5.3%, assistants 2.5%, and retrieval bots like PerplexityBot and OAI-SearchBot 1.1%. Claude Code alone out-fetched every retrieval and assistant bot combined.

Read that back and the file's real job becomes obvious. llms.txt isn't a search feature. It's a docs feature. It's the thing a coding agent grabs when it's trying to figure out your API without burning 40,000 tokens on rendered HTML. Mueller landed in the same place from the other direction, calling it a temporary crutch for saving tokens in AI coding tools.

If you sell a developer tool with real docs, that's a genuine use case and worth ten minutes. If you're publishing a blog and hoping ChatGPT starts recommending you, you've shipped a file for an audience that isn't showing up.

So why do I still have one?

Because mine costs nothing to keep, and I'd rather be honest about why than quietly delete it and pretend I was never fooled.

The version on this site isn't a hand-written file that rots the moment I publish a post. It's a route that generates itself from the same data the site renders: every tool in the directory grouped by category, every note with its excerpt, the API endpoints, the author bio. When I add a tool, the file updates. Maintenance cost after the initial build is zero, so the expected-value math is easy even at a 97% chance of nobody reading it.

What I won't do is tell you it's why anything happened. I don't have the traffic to produce a meaningful server log yet, and inventing a "since adding llms.txt, citations rose 30%" line would be the same move as quoting a Domain Rating I haven't earned. A real "no idea yet, probably nothing" is worth more than a fake number.

The part of that build I'd actually defend is the rest of it. Markdown content negotiation, so an agent asking for text/markdown gets clean markdown at the same URL a browser gets HTML. A typed JSON API for the directory. An OpenAPI spec describing all of it. Those aren't guesses about what a crawler might prefer. They're the same content, served in the format the client asked for, at a URL the client already found. That's the difference between building for a convention and building for a request that's actually arriving.

What actually gets you cited by AI, then?

Four things with real evidence behind them, roughly in order of how much they matter and how little they get talked about.

Server-render your content. This is the big one and it's mostly invisible because it fails silently. Vercel and MERJ tracked more than 500 million GPTBot fetches and found no evidence of JavaScript execution. GPTBot downloaded JS files about 11.5% of the time and never ran them. ClaudeBot downloaded them 23.84% of the time and never ran them either. If your copy only exists after a client-side render, the crawler is fetching an empty shell and there is nothing to cite. GPTBot alone made 569 million requests across Vercel's network in a single month, ClaudeBot another 370 million. That traffic is real, it's arriving now, and a client-rendered page throws all of it away.

Put statistics, sources, and quotes in the actual text. The GEO paper out of Princeton and Georgia Tech (Aggarwal et al., arXiv:2311.09735) tested this directly and found this class of tactic lifts citation likelihood by up to 40%. The gains skewed hardest toward smaller and lower-ranked sites, with visibility improving up to 115% for a page sitting around position five. That's the most encouraging finding in this whole space for anyone starting from nothing, and it costs nothing but doing the reading.

Answer the question near the top of the section. Retrieval pulls passages, not pages. A section that opens with two paragraphs of context before the answer gives the model nothing clean to lift. Look at the headings in this post: each one is a question someone actually types, and each answer starts in the first sentence under it.

Stop optimizing the things that don't move. Schema isn't the lever, however many agencies sell it as one. Ahrefs tracked 1,885 pages that added JSON-LD against matched controls and measured +2.2% on ChatGPT and +2.4% on Google AI Mode, both indistinguishable from noise, and -4.6% on AI Overviews. Length isn't the lever either. Their study of 174,000 pages cited in AI Overviews found an average cited length of 1,282 words but a Spearman correlation of 0.04 between length and being cited, with 53.4% of citations going to pages under 1,000 words. Keep schema if you already have it for rich results. Just don't expect it to buy citations, and don't pad a post to 3,000 words because a template told you to.

The 20-minute version, if you want a checklist

If you're a solo dev deciding what to actually do this afternoon:

• Check that AI crawlers aren't blocked in robots.txt. A blocked GPTBot or OAI-SearchBot is the one configuration mistake that makes every other thing on this list irrelevant. This takes two minutes and people get it wrong constantly.
• Confirm your content is in the HTML source. View source, search for a sentence from your page. If it isn't there, fix that before anything else.
• Generate llms.txt from your existing data if it's cheap. Ten minutes for a route that builds itself, not an hour for a hand-maintained file you'll forget to update. If it can't be automatic, skip it.
• Serve markdown to clients that ask for it, if you have an API or docs worth an agent reading.
• Then go write something specific enough to cite, with a number in it you actually measured.

That last one is doing most of the work, and it's the least fun to hear. The technical surface is a one-afternoon job with a hard ceiling. Everything above it is the same problem I've been circling since I started this site: distribution is still the hard part, and no file at the root of a domain fixes it.

The real cost of llms.txt isn't the file

It's what shipping it lets you believe.

There's a specific trap in this stuff where a technical task with a clear finish line stands in for the ambiguous work that actually matters. Ship llms.txt, tick a box, feel like you've addressed AI search. Same energy as buying a boilerplate because starting is uncomfortable. The file is fine. The 40 minutes you spend afterwards reading posts about how to optimize the file are not.

I'd have found this faster if I'd checked for evidence before building instead of after. That's the actual lesson, and it generalizes past this file: when a tactic is being recommended by roughly everyone and measured by roughly nobody, that gap is the whole story. Two studies covering more than 400,000 domains between them existed the entire time I was writing my route. I just hadn't looked yet.

The file's still up. It'll keep regenerating itself, and one of these years a provider might announce they read it, at which point I'll have had one since 2026 at a total cost of an afternoon and this post. That's a fine bet. It's just not a strategy, and I'd rather say so plainly than keep the confusion going for another 137,000 domains.