Does llms.txt work? What 4 studies found, and what to do instead

An SEO checklist tells you AI search needs an llms.txt file. A plugin generates one in a minute. It feels like progress, and then nothing changes, because the crawlers deciding what ChatGPT cites aren’t looking for it.

Four independent studies covering more than 437,000 domains point the same way: AI search crawlers rarely request the file, and sites that have one don’t get cited more. Here’s what the data shows, who should still bother, and what to do with that afternoon instead.

What is llms.txt?

llms.txt is a Markdown file at yoursite.com/llms.txt that summarises your site and links to the pages an AI model should read. Jeremy Howard proposed it in September 2024. A minimal one looks like this:

# Acme Analytics

> Product analytics for B2B SaaS teams. Self-serve, from $49/month.

## Docs
- [Quickstart](https://acme.com/docs/quickstart): install the SDK in 5 minutes
- [API reference](https://acme.com/docs/api): every endpoint and parameter

## Company
- [Pricing](https://acme.com/pricing)

It’s easy to confuse with robots.txt, but they do different jobs. robots.txt tells crawlers what they’re allowed to fetch, and AI search crawlers obey it. llms.txt can’t allow or block anything. It only suggests what to read, to whatever chooses to look.

llms.txt vs robots.txt vs sitemap.xml

Three files sit at the root of most sites and they’re easy to mix up. Only two of them affect whether AI search can use your pages:

If you only have time for one, fix robots.txt. A single wrong line there can hide your whole site from ChatGPT search. A missing llms.txt costs you nothing.

What the data shows

When something does fetch the file, it’s usually not an AI search crawler. Ahrefs found Slackbot fetched llms.txt files more often than PerplexityBot did, and 77% of the bots reading them weren’t AI tools at all: mostly SEO audit tools and general crawlers.

llms.txt is not requested by anything (other than humans checking).

rickette, who runs a blogging platform with 80,000 blogs, on Hacker News

What Google and OpenAI say

Google’s guidance on its AI features is direct: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” To show up in AI Overviews or AI Mode, a page needs to be indexed and eligible for a snippet. That’s it.

OpenAI’s crawler documentation explains why the file doesn’t help with ChatGPT either. ChatGPT search relies on OAI-SearchBot, which respects robots.txt and has nothing to do with llms.txt. The OpenAI bot that did show up most in Ahrefs’ llms.txt data was GPTBot, which collects training data. A GPTBot visit doesn’t put you in a ChatGPT answer.

How AI search actually finds your pages

If AI search crawlers aren’t reading llms.txt, how do they find your content? Through the same plumbing search has always used, with a few new names on the guest list.

Notice what every row has in common: robots.txt and a search index. That’s where an afternoon of technical work pays off. A page that’s blocked, missing from Bing, or invisible without JavaScript can’t be cited, no matter how good your llms.txt is.

Bing deserves special attention because it feeds two of the four. Verifying your site in Bing Webmaster Tools also gives you its AI Performance report, which shows how often Microsoft Copilot cites your pages. It’s free and takes about ten minutes.

Who actually publishes one: we checked 33 sites

On 25 October 2026 we requested /llms.txt from the AI companies, AI visibility tools and well-known SaaS sites:

The pattern is telling. The companies building AI assistants publish llms.txt for their developer documentation, so developers can feed the docs into a coding assistant. None of them publish it on their main sites. That’s the file’s real job, and it isn’t AI search.

Should you ship one? A quick decision guide

If you do ship one, generate it automatically from your docs or sitemap so it never goes stale. A hand-maintained llms.txt that drifts out of date is worse than none.

If you run docs: how to ship llms.txt properly

For documentation sites the file is worth having, so do it well. Developers paste it into tools like Cursor, Claude Code or ChatGPT to give the assistant your docs, and a clean file means the assistant writes better code against your API.

1. Generate it from your docs, don’t hand-write it. Most documentation platforms and static site generators can build it from your navigation or sitemap on every deploy.
2. Lead with one line on what your product does, then sections for getting started, core concepts, API reference and SDKs.
3. Link to clean Markdown versions of each page where you can. Assistants handle Markdown far better than HTML full of navigation.
4. Add an llms-full.txt with the full text of your docs in one file, for developers who want to paste everything at once.
5. Mention it in your docs (“Using an AI assistant? Point it at /llms.txt”). The people who benefit most won’t find it otherwise.
6. Check it after every major docs change. Broken links in a file meant for machines are easy to miss.

How to check whether anything reads your llms.txt

If you already have the file, look at who requests it. On most hosts you can filter request logs by path. With raw server logs:

# Count requests to llms.txt by user agent
grep "GET /llms.txt" access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

On Vercel, filter Logs by the path /llms.txt. On Cloudflare, use Security → Analytics and filter by path. An empty result after a month is normal.

While you’re in the logs, run the same check for the crawlers that do matter. Search for OAI-SearchBot, PerplexityBot and Bingbot. If they’re visiting your key pages, your access is fine. If they’re missing, or getting 403 errors, that’s a far bigger problem than any llms.txt, and it’s usually a CDN bot setting.

What actually decides whether AI cites you

If llms.txt isn’t the lever, what is? Three things, in this order:

1. Access. The AI search crawler can fetch your page and read the text. This is robots.txt, your CDN settings and server-rendered HTML, not llms.txt.
2. An answer page. You have a page that answers the exact question a buyer asked. In Something Inc’s study of 4,100 B2B queries, comparison and alternatives pages took 33% of AI citations, more than any other format.
3. Other people talking about you. Ahrefs’ study of 75,000 brands found branded web mentions correlated 0.664 with visibility in AI Overviews, three times more than backlinks. Mentions on review sites, comparison articles and Reddit threads move AI answers far more than any file on your own server.

That third point is why a competitor with a worse website can still be the one ChatGPT recommends. We break that down, with a 30-minute diagnosis, in why ChatGPT recommends your competitor.

If your agency or a tool says you need it

Ask them one question: which AI search crawler requests the file, and can they show you the log lines? If the answer is vague, the recommendation is a checklist item, not a strategy. A good AI visibility audit should talk about crawler access, the prompts your buyers use, the pages that get cited for them, and where your brand is missing. Adding a file takes a minute; that work takes weeks, and it’s the part that moves results.

What to do instead: a 30-minute checklist

1. Allow AI search crawlers in robots.txt (5 min). Add the rules below. OpenAI says changes take about 24 hours to apply.
2. Check your CDN’s bot settings (5 min). Cloudflare and other hosts can block AI crawlers even when robots.txt allows them.
3. Run the view-source test (5 min). Open a key page, view source, search for a body sentence. If it’s missing, your text depends on JavaScript.
4. Verify your site in Bing Webmaster Tools and turn on IndexNow (10 min). Microsoft is one of ChatGPT’s search partners.
5. Write down 5 questions your buyers ask AI (5 min) and check whether you have a page that answers each one directly. Our ChatGPT citation playbook covers what to publish next.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

Common mistakes

• Treating llms.txt as your AI strategy. It’s at most a small extra for documentation.
• Blocking OAI-SearchBot while adding llms.txt. The first undoes anything the second could do.
• Hand-writing the file once and forgetting it. Stale links in a file meant for AI are worse than no file.
• Measuring success by whether the file exists. Measure whether you’re named and linked in AI answers for your buyers’ questions.

So, does llms.txt work?

For getting cited in AI search, no. It’s rarely fetched, it makes no measurable difference to citations, and Google says outright you don’t need it. For documentation and developer tools, yes: generate it automatically and move on. Everyone else should spend that time on crawler access, Bing indexing and pages that answer real buyer questions. Those are what ChatGPT actually reads.