llms.txt: what it is, who reads it, and whether it matters in 2026

What is llms.txt and who proposed it?

llms.txt is a Markdown file at the root of your website that gives AI systems a short, curated map of your most useful pages. Jeremy Howard proposed it on 3 September 2024, and the specification reached version 2 on 10 August 2026 (llmstxt.org, spec v2 updated 10 August 2026).

The proposal describes the file as holding "brief background information, guidance, and links to detailed markdown files" (llmstxt.org, spec v2 updated 10 August 2026). The design goal is a file "small enough to fit in context", so a model can read it in one pass.

That is the entire idea. An AI tool either fetches the file or ignores it, and the fetching part is where the evidence runs thin.

What goes inside an llms.txt file?

An llms.txt file holds a short description of your site, some guidance, and links to your most useful pages with a one-line note on each (llmstxt.org, spec v2 updated 10 August 2026). Ours lives at /llms.txt. For a site like ours, the file would cover three groups of pages.

• Services. SEO services, AI search optimization, the SNIFF audit at Rs 15,000 or USD 300, retainers from Rs 25,000 or USD 750 a month, and the free visibility check.
• Company. About, how we work, and client zero: our own numbers, published because we have no clients yet.
• Resources. The audit checklist, the glossary, and research posts such as what is Answer Engine Optimization.

Two rules keep it useful. Every link should be a page a person would actually want, not a dump of your sitemap. Every note should be a plain description, not a sales line, because the reader is a model summarising you for someone else.

Does Google use llms.txt?

No. Google says its AI features need no special files, and its AI optimisation guide names llms.txt among the tactics that do not affect Google Search visibility. Google's "AI features and your website" page says "You don't need to create new machine readable files, AI text files, or markup to appear in these features" (Google Search Central, last updated 10 December 2025). The same page states there is "no special schema.org structured data that you need to add".

Google's guide to optimising for AI search goes further. It lists tactics that "don't affect Google Search visibility", and LLMS.txt and other AI-specific files sit on that list alongside chunking content into tiny pieces, rewriting for AI, seeking inauthentic mentions and over-focusing on structured data (Google Search Central, May 2026, page updated 10 July 2026). The same guide says optimising for generative AI search "is optimizing for the search experience, and thus still SEO".

John Mueller of Google put it more bluntly in June 2026. He called llms.txt "purely speculative for now" and noted that "the file has existed for years, yet none of the AI systems use it" (Search Engine Journal, 2 June 2026). His practical test is worth keeping. In his words, "When an AI platform that brings you clients complains that it needs the file for your site, then I'd recommend taking the time to create one."

Do ChatGPT, Claude or Perplexity read llms.txt?

Nothing in their published crawler documentation says they do, and Google's John Mueller says "none of the AI systems use it" (Search Engine Journal, 2 June 2026). We read each vendor's own bot page on 5 September 2026. Here is what they do document.

• OpenAI documents three bots. OAI-SearchBot surfaces sites in ChatGPT search, and sites that block it will not appear in ChatGPT search answers. GPTBot is for training, and blocking it does not affect search visibility. ChatGPT-User fetches pages on a user's request, and robots.txt rules may not apply to it (OpenAI, checked 5 September 2026).
• Perplexity says PerplexityBot "is not used to crawl content for AI foundation models", and Perplexity-User serves user requests (Perplexity, checked 5 September 2026).
• Anthropic documents ClaudeBot for training, Claude-User for user-initiated fetches and Claude-SearchBot for its search index (Anthropic, checked 5 September 2026).

All three pages are about bots and robots.txt. None of them documents llms.txt.

There is one place the format clearly lives, and it is not search. Anthropic's engineering team wrote that "LLM-friendly documentation can commonly be found in flat llms.txt files on official documentation sites", and Anthropic publishes its own file for its developer docs (Anthropic, 11 September 2025). That is the use case: agents reading developer documentation. A restaurant, a law firm or a D2C brand has little of that to offer.

How is llms.txt different from robots.txt and sitemap.xml?

The three files do three different jobs: robots.txt controls access, sitemap.xml lists everything, and llms.txt curates a little. The proposal draws the lines itself (llmstxt.org, spec v2 updated 10 August 2026).

The access file is the one with documented effects. OpenAI states that sites which disallow OAI-SearchBot will not appear in ChatGPT search answers (OpenAI, checked 5 September 2026). No vendor documents any effect for llms.txt. Our own /robots.txt does the heavy lifting. Our /llms.txt is the polite extra. See llms.txt and AI crawlers in the glossary.

What controls AI visibility instead?

Three things: crawler access, indexing and pages that answer plainly. Google says its generative features use retrieval-augmented generation and "query fan-out" over indexed pages (Google Search Central, May 2026, page updated 10 July 2026). The system splits a question into several searches, pulls indexed pages, and writes from them. See RAG and query fan-out.

1. Crawler access. Check your robots.txt against each vendor's bot names. Blocking OAI-SearchBot removes you from ChatGPT search answers, while blocking GPTBot does not affect search visibility (OpenAI, checked 5 September 2026). Google-Extended is a separate robots.txt token that controls training of future Gemini models and grounding in Gemini apps and Vertex AI. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google Search Central, last updated 14 July 2026). See Google-Extended.
2. Indexing. Google's generative features draw on indexed pages (Google Search Central, May 2026, page updated 10 July 2026). A page that is not indexed cannot be retrieved. Fix indexing before you touch any AI file.
3. Clear answers with a visible author. State the answer in the first paragraph, use headings that match real questions, and show who wrote the page. Google's helpful content guidance asks "Is it self-evident to your visitors who authored your content?" and says that on E-E-A-T "trust is most important" (Google Search Central, last updated 10 December 2025). See E-E-A-T. That is ordinary SEO work, which is why we sell it as SEO services and AI search optimization rather than as file tricks.

One more number matters before you welcome every bot. Cloudflare measured that nearly 80% of AI-bot crawling is for training, with user-action and undeclared purposes under 5%. For the week of 1 to 7 August 2025, crawl-to-refer ratios were roughly 50,000:1 for Anthropic, 887:1 for OpenAI and 118:1 for Perplexity (Cloudflare, 28 August 2025). Those ratios move every month, so treat them as a snapshot. Most AI crawling sends nothing back, so decide bot by bot. See crawl-to-refer ratio and our post on Generative Engine Optimization.

Should you add one anyway?

Yes, if it takes little time and you expect nothing from it. That is what we did. Our file is at /llms.txt. It cost nothing, and Google says it is neither needed nor harmful to visibility. If a coding agent or a future search crawler wants a Markdown map of the site, one exists.

What we will not do is sell it as a tactic. If an agency puts llms.txt near the top of its AI visibility plan, ask what evidence it has that any engine reads the file. On 5 September 2026 we could find none in the documentation from Google, OpenAI, Anthropic or Perplexity.

Add one when you publish documentation that developers or coding agents read, when you already have a tidy list of key pages, or when a platform that sends you customers starts asking for it. Skip it when it would distract you from robots.txt, indexing or writing better pages. To see where you stand first, run our free visibility check.

How will we know if this changes?

We will re-check the evidence and add the date of each check to this post. Here is what we look at.

• Vendor crawler docs. The OpenAI, Anthropic and Perplexity bot pages linked above. If any of them starts naming llms.txt, that is the first signal.
• Google's guidance. The AI features page and the AI optimisation guide. Today both say the file is unnecessary.
• Server logs. Requests for /llms.txt, grouped by user agent, show who fetches the file. A declared search bot would be news. Scrapers and SEO tools are not.
• Search Console. The Generative AI performance report shows impressions, but no clicks, for AI Overviews and AI Mode, and reached all sites worldwide by 31 August 2026 (Google Search Console Help, 31 August 2026). See AI Overviews and AI Mode.
• Client zero. Our own numbers live at /client-zero/. If the file ever moves them, you will read it there.

What else do people ask about llms.txt?

Four questions come up most often, and the answers are short.

Does llms.txt help SEO?

No. Google lists AI-specific files among the tactics that do not affect Google Search visibility (Google Search Central, May 2026, page updated 10 July 2026). It does not hurt either. Treat it as a courtesy file, not a ranking input.

Is llms.txt the same as robots.txt?

No. robots.txt sets access permissions for crawlers. llms.txt is a curated Markdown summary that a tool may read on demand (llmstxt.org, spec v2 updated 10 August 2026), and no search crawler documents reading it. If you only have time for one, fix robots.txt.

Which AI tools read llms.txt?

No search engine or assistant documents reading it as of 5 September 2026. The format is used on documentation sites for coding agents, and Anthropic publishes one for its developer docs (Anthropic, 11 September 2025). That is the only confirmed use we have found.

Where do I put llms.txt?

At the root of your domain, as a Markdown file (llmstxt.org, spec v2 updated 10 August 2026). Keep it short and update it when key pages change.

Sources

• llmstxt.org, the llms.txt proposal and specification: https://llmstxt.org/ (spec v2 updated 10 August 2026)
• Google Search Central, AI features and your website: https://developers.google.com/search/docs/appearance/ai-features (last updated 10 December 2025)
• Google Search Central, AI optimization guide: https://developers.google.com/search/docs/fundamentals/ai-optimization-guide (May 2026, page updated 10 July 2026)
• Search Engine Journal, Google says llms.txt is purely speculative for now: https://www.searchenginejournal.com/google-says-llms-txt-is-purely-speculative-for-now/577576/ (2 June 2026)
• Anthropic engineering, Writing tools for agents: https://www.anthropic.com/engineering/writing-tools-for-agents (11 September 2025)
• OpenAI, crawler and bot documentation: https://developers.openai.com/api/docs/bots (checked 5 September 2026)
• Perplexity, crawler documentation: https://docs.perplexity.ai/guides/bots (checked 5 September 2026)
• Anthropic, Does Anthropic crawl data from the web: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler (checked 5 September 2026)
• Google Search Central, Google common crawlers (Google-Extended): https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (last updated 14 July 2026)
• Google Search Central, Creating helpful, reliable, people-first content: https://developers.google.com/search/docs/fundamentals/creating-helpful-content (last updated 10 December 2025)
• Cloudflare, AI crawler traffic by purpose and industry: https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/ (28 August 2025)
• Google Search Console Help, Generative AI performance report: https://support.google.com/webmasters/answer/16984139 (31 August 2026)

Sources and accuracy

Researched and written by the WoofRank founders. Every statistic and quote was checked against the primary source on 5 September 2026 and linked inline. WoofRank has no clients yet, so nothing here is a client result. Corrections to woofrank26@gmail.com.