Does llms.txt Work? What the Evidence Says (and What We Do Instead)

llms.txt is the keywords meta tag of the AI era: cheap to add, easy to sell, and read by almost nothing that decides whether ChatGPT recommends you. That is our verdict after reading the studies and the crawler documentation. We still keep a file on our own site, because it takes ten minutes. We would not spend ten hours on it, and we would not pay anyone to “optimise” it. This page shows the evidence and the work that matters more.

Quick answer

llms.txt is a plain text file, proposed in 2024, that lists a site’s key pages in a format meant for AI tools. Evidence so far says it does not improve AI search visibility. Google says it does not use it. Ahrefs found 97% of files got no requests in a month. SE Ranking found no link to citations. It may help coding assistants read documentation. For a business trying to get recommended, crawler access, readable pages and outside proof matter far more.

What is llms.txt, exactly?

It is a Markdown file at the root of your site, at yoursite.com/llms.txt, that gives a short, tidy summary of who you are and which pages matter. It is meant for language models, the way robots.txt is meant for crawlers. The idea was that a model with a limited reading window could skip your menus and scripts and go straight to the useful parts.

Here is a complete, copy-paste template.

# Your Company Name

> One or two sentences: what you do, who for, where you are based.

Founded 20XX. Registered in the UK, company number XXXXXXXX.

## Key pages

- [Services](https://yoursite.com/services/): what we do and who it is for
- [Pricing](https://yoursite.com/pricing/): typical price ranges and what moves them
- [Case studies](https://yoursite.com/case-studies/): named results with dates
- [About](https://yoursite.com/about/): the people, credentials and registrations
- [Contact](https://yoursite.com/contact/): how to reach us

## Optional

- [Blog](https://yoursite.com/blog/): articles and guides

That is the whole thing. Nothing here changes how a page ranks or how a model ranks you. It is a signpost, and the question is whether anyone reads signposts.

What do the AI companies say about llms.txt?

The honest answer is that none of the big four has said they use it for search.

• Google. Its Search Central page on AI features says there are no additional requirements to appear in AI Overviews or AI Mode, and that you do not need to create new machine-readable files, AI text files or markup. In June 2025 a Google Search representative said on social media that no AI system currently uses llms.txt, and compared it to the old keywords meta tag. Treat the wording as reported, since we read it second-hand.
• OpenAI. Its crawler documentation describes OAI-SearchBot, GPTBot and ChatGPT-User and explains how robots.txt controls each. The page we read does not mention llms.txt.
• Anthropic. Its help page lists ClaudeBot, Claude-User and Claude-SearchBot and the robots.txt rules for them. It does not mention llms.txt either.
• Perplexity. Its documentation covers PerplexityBot and Perplexity-User and robots.txt. No mention of llms.txt there either.

The pattern is clear. Every company documents robots.txt as the control. In the pages we read, none documents llms.txt as an input. Absence of a mention is not proof that no one reads it, and that is why we also looked at logs.

Common mistake: treating “no one has said it doesn’t work” as evidence that it does. The burden of proof sits with the claim.

What do the studies show?

Three sources, from three angles. None is perfect, so we list the limits.

1. Ahrefs: who actually requests the file. Ahrefs published this in June 2026. It checked 137,210 domains that had traffic in May 2026 and tested each for a valid llms.txt file. It then used server logs from its own analytics products to see who asked for those files.

• 28% of the domains had a valid file. Ahrefs says its customers skew technical, so the real web figure is probably lower.
• 97% of those files received no requests at all in May.
• About 96% of the requests that did arrive came from bots. Named AI bots were 19.5% of requests, but most of that was AI agents (10.5%) and training crawlers (5.3%). Assistants were 2.5%. Retrieval bots, the ones that fetch pages to answer a user’s question, were 1.1%.
• The largest single group was SEO audit tools, at 21.7%.
• AI bots made zero requests for llms.txt on sites that did not have one. Nothing goes looking for it.

Ahrefs’ conclusion: the file has little effect on AI search visibility today, and its most plausible audience is coding and agent tools. The limits Ahrefs states are fair. A fetch does not prove a bot read or used the file, so these numbers are ceilings. The study also covers only the index file, and only the 3% of files that were requested drive most of the findings.

2. SE Ranking: does it correlate with being cited? In October 2026, SE Ranking analysed nearly 300,000 domains. 10.13% had an llms.txt file, and adoption was spread evenly across high- and low-traffic sites. It found no measurable relationship between having the file and how often a domain appeared as a source in sampled AI answers. When it removed the llms.txt variable from its prediction model, the model got more accurate. SE Ranking’s own phrase was that the file “might currently introduce more noise than value”. It did not say which AI models it sampled or exactly when it collected the data, and a correlation study cannot show cause. It is one more piece pointing the same way.

3. Evil Martians: one site’s own logs. An engineering studio logged AI agent and crawler requests to its own site from April 2026 and updated the numbers on 4 November 2026. Across about 593 direct fetches of llms.txt and llms-full.txt, AI companies’ crawlers accounted for 75. The rest came from SEO crawlers, readiness scanners and plain scripts. Named AI assistants fetched the file only 47 times, and Claude Code made 43 of them. This is one site and not a sample of the web. It matches the pattern: the readers are coding tools, not the assistants your buyers use.

What the data says so far. Sources: Ahrefs (June 2026), SE Ranking (October 2026).

Who does read llms.txt?

Two groups, and neither is your customer.

• Coding assistants. If you publish developer documentation, tools such as Claude Code do fetch these files while helping someone write code. That is a real use. If you run an API, a software library or a technical docs site, a well-kept llms.txt can be worth an afternoon.
• SEO and audit tools. They fetch it to report whether you have one. That tells you something about the tools and nothing about how you rank.

Ahrefs also raised a further risk: any file written for machines to trust is a possible route for prompt injection, where hostile text tells an AI tool what to do. Keep yours short, factual and under version control, as you would any code.

So should you add one? Our decision tree

Worth ten minutes in some cases. Never worth ten hours.

• You publish technical documentation or an API. Yes. Add it, link your key docs and keep it current.
• You run a normal business site and want recommendations in ChatGPT. Optional. Add the template above in ten minutes if you like. Do not expect movement and do not pay for it.
• Your site is small and rarely changes. Skip it. An outdated file is worse than none, because it points tools at pages that no longer exist.
• An agency has quoted you a line item for it. Ask what they expect to change and how they will measure it. We would be surprised by a good answer.

Our own position, in full. We keep an llms.txt on voltik.ai, because it costs almost nothing and may matter to agent tools later. It earns no points in our AI search optimisation checklist, and we never count it as a reason a client’s visibility moved.

What do we do instead?

We put the effort where the evidence points, in the order we do it. This is the same order as our 90-day plan, where the first 30 days are the high-return fixes.

The work that moves visibility. Each step is measurable.

1. Let the search crawlers in

OpenAI says sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers. Check robots.txt and any CDN or security setting. Cloudflare said in July 2025 that it was changing its default to block AI crawlers unless they pay, so a block can exist outside robots.txt.

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Common mistake: blocking GPTBot and assuming that blocks ChatGPT search. GPTBot is for training. OAI-SearchBot is the one that puts you in answers.

2. Make sure the page is readable without scripts

Vercel and MERJ reported in December 2024 that none of the major AI crawlers render JavaScript. Open your page, right-click, choose View page source, and search for a sentence. If it is missing, the bots may not see it. We find this most often on template builders and heavy page-builder setups.

3. Put one source of truth on a human page

This is what we do instead of relying on llms.txt. Publish a plain “facts” section on your About page that a visitor, a crawler and a model can all read: legal name, trading name, founded date, address, what you do, who you serve, price ranges, credentials and the names of your people. Use the same wording everywhere. Fragmented brand assets make AI see a split source of truth. Redirect old domains, and do not run two sites for one brand.

4. Name the people and show the proof

Put a real author, bio and credential line on every important article. Publish original numbers and dated case studies. Add Organization, Person and Article structured data that matches the visible text, as Google advises. Experts hidden behind a company byline give a model nothing to check.

5. Write pages that answer the buyer’s question

Question as the heading, answer in the first two sentences, evidence after. Include a price or range. In an August 2025 study of 15,000 prompts, Ahrefs found only about 8% of ChatGPT’s in-text citations pointed to a page in Google’s top 10 for the same query. Pages that answer one sub-question well get cited even when they do not rank for the headline phrase.

6. Earn mentions where AI already looks

In a study of 75,000 brands, Ahrefs found brand mentions across the web correlated with Google AI Overview visibility at 0.664, against 0.218 for backlinks. That is correlation, but it points off-site. Peec AI’s analysis of about 30 million AI citations put Reddit, YouTube, LinkedIn, Wikipedia, Forbes and G2 among the most cited sources, though Peec sells monitoring tools and every industry differs. Ask each AI tool which sources it cites for your category, then work out how to appear on the top five.

How can you check what AI bots do on your own site?

Do not trust anyone’s study over your own logs. If you can get your server’s access log, two commands tell you more than a month of guessing. Ask your host or developer for the file, then run these.

# Which AI crawlers visit, and how often
grep -Ei "OAI-SearchBot|GPTBot|ChatGPT-User|Claude-SearchBot|ClaudeBot|Claude-User|PerplexityBot" access.log \
  | grep -Eio "OAI-SearchBot|GPTBot|ChatGPT-User|Claude-SearchBot|ClaudeBot|Claude-User|PerplexityBot" \
  | sort | uniq -c | sort -rn

# Does anything fetch llms.txt?
grep "llms.txt" access.log | grep -Ei "bot|gpt|claude|perplexity"

If the first command returns nothing, your problem is access, not llms.txt. Look at robots.txt and your CDN settings. If the second returns nothing, you have just confirmed the Ahrefs finding on your own site.

Plan for honest timelines. Unblocking a crawler can show within days. Reviews, mentions and authority build over months. Nobody can promise a citation, and be wary of anyone who does. In our MVPs case study, the starting point was a 0% AI citation rate and a #23 AI search rank, and the programme reached #2 on three of four platforms through sustained content, mention and trust work, not a text file. See Why isn’t my business showing up in ChatGPT? for the diagnosis.

Questions people ask

Does llms.txt help SEO?

No evidence says it does. Google’s guidance says AI features need no special files, and Ahrefs and SE Ranking found no visible benefit. It is not a ranking factor in anything we have seen documented.

Do ChatGPT, Claude or Perplexity read llms.txt?

Their crawler documentation covers robots.txt and does not list llms.txt as an input. In Ahrefs’ logs, retrieval bots made 1.1% of requests to llms.txt files, and in one site’s logs the only regular AI reader was a coding tool.

Should I add llms.txt to my website?

If you publish developer docs, yes. Otherwise it is optional, takes about ten minutes, and should not be expected to change anything. Keep it accurate or remove it.

What goes in an llms.txt file?

A title, a one-line summary and a short list of your key pages with a few words on each, in Markdown. Use the template above.

Is llms.txt the same as robots.txt?

No. Robots.txt tells crawlers where they may go, and it is respected by the major AI search crawlers. Llms.txt is a suggested reading list with no enforcement and, so far, little uptake.

What is better than llms.txt for AI visibility?

Letting the search crawlers in, making pages readable without scripts, keeping one consistent set of facts, naming your experts and earning mentions on sites AI already cites. Our AI search optimisation checklist scores all of it out of 100.

Sources

• Ahrefs, “We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read”, 15 June 2026.
• SE Ranking, “Does LLMs.txt impact your AI visibility and citations? No, according to research”, 15 October 2026.
• Evil Martians, “Which AI actually reads your site? Two months of LLM traffic, measured”, 21 July 2026, updated 4 November 2026.
• Google Search Central, “AI features and your website”.
• Search Engine Roundtable, “Google: No AI System Currently Uses LLMs.txt”, June 2025.
• OpenAI, “Overview of OpenAI Crawlers”, developer documentation.
• Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?”, help centre.
• Perplexity, “Perplexity crawlers”, documentation.
• Vercel and MERJ, “The rise of the AI crawler”, 17 December 2024.
• Cloudflare, “Content Independence Day: no AI crawl without compensation”, 1 July 2025.
• Ahrefs, “Only 12% of AI Cited URLs Rank in Google’s Top 10 for the Original Prompt”, 11 August 2025.
• Ahrefs, “An Analysis of AI Overview Brand Visibility Factors (75K Brands Studied)”, 26 May 2025.
• Peec AI, “Top domains cited by AI search: Analysis based on 30M sources”, March 2026.

Want us to do this for you?

If you would rather have a team handle the access, proof and mentions work and report the numbers monthly, that is what we do. Book a free call and we will check what AI says about you today, or read about our AI search optimisation service.