llms.txt Explained: What It Is and Whether B2B SaaS Should Bother

llms.txt is a proposed standard: a single markdown file you place at yoursite.com/llms.txt that lists and summarises the pages you most want a large language model to read. Think of it as a hand-drawn map to your best content, written for machines instead of humans.

It sounds like exactly the kind of thing a B2B SaaS marketing team should jump on, and most AI-visibility checklists tell you to add one immediately. We run AI search and SEO campaigns for B2B SaaS companies every day, and llms.txt comes up in almost every kickoff. So we went and checked the one thing the checklists skip: does anything actually read the file? The honest answer is more useful than the hype, and it changes what you should do about it. This is the companion piece to our guide on AI crawlers, and the two share a lesson: a lot of AI-search effort is spent on signals that no crawler consumes yet.

What llms.txt actually is, and what it is not

llms.txt is a markdown file that gives a language model a short, curated index of your site so it doesn't have to guess which pages matter. It was proposed by Jeremy Howard of Answer.AI in September 2024, and the reasoning is sound. When a model needs context from your site, it has to decide what to pull, and raw web pages are full of navigation, scripts and markup that burn through the model's limited context window. A clean markdown summary is cheaper to read and less ambiguous.

Howard, who also co-founded fast.ai and co-created the ULMFiT approach that fed into modern language models, framed the file as a way to let authors do the selecting.

What an llms.txt file looks like

The spec is deliberately minimal. An llms.txt file is plain markdown with a defined shape: an H1 with your site or product name, a blockquote summary, then H2 sections that each hold a bullet list of links with short notes. Here is the pattern in practice.

Two files usually get mentioned together. The /llms.txt above is the short index that links out to detail. /llms-full.txt is a single expanded file that concatenates the actual content, meant for dropping an entire documentation set into a model in one shot.

The scale gap between them is large: Anthropic's own llms.txt index runs a few thousand tokens, while its llms-full.txt is close to half a million. A section literally titled "Optional" has special meaning in the spec too, marking links a model can skip when it needs a shorter context.

Note the .md links in the example. The proposal also suggests serving clean markdown versions of your pages so a model reads structured text instead of rendered HTML, and in our experience that idea is quietly more useful than the index file itself.

llms.txt vs robots.txt and sitemap.xml

Where teams go wrong is treating llms.txt as a third member of the robots.txt family. It sits at the site root like they do, but it does a completely different job, and conflating the three is the single most common mistake we see in briefs on this topic.

robots.txt blocks. sitemap.xml inventories everything. llms.txt selects and summarises. It is closest in spirit to a sitemap, but where a sitemap aims to be exhaustive, llms.txt is meant to be short and opinionated. The critical difference is enforcement. robots.txt and sitemaps are honoured by the crawlers they target because those crawlers agreed to. llms.txt is a suggestion that no major consumer has committed to reading, which is exactly where the evidence gets uncomfortable.

Does anything actually read llms.txt?

The data: almost nothing reads it

Overwhelmingly, no, and this is the fact that reframes the entire topic. In June 2026, Ahrefs analysed 137,000 domains and found that 97% of published llms.txt files received zero requests of any kind across a full month. Not from AI, not from humans, not from anything. It is the largest study on the question to date, and the number has held up as other trackers have run their own checks.

What makes that silence striking is that adoption is climbing fast. Originality.ai tracked an 8.8x year-on-year rise in sites publishing llms.txt, and roughly 28% of the technical-leaning sites in the Ahrefs sample now host one. Publishing is surging while consumption flatlines. That gap, between how many people are making the file and how rarely anything opens it, is the whole story of llms.txt in 2026.

When you look at the small share of files that did get traffic, most of it isn't AI at all. SEO audit tools, generic web crawlers and tech-profiling bots like BuiltWith made up the bulk. Named AI bots were under a fifth of requests, and the retrieval crawlers that actually decide whether you appear in an AI answer were a rounding error.

Read the bottom of that chart. The AI retrieval crawlers, the bots behind ChatGPT and Perplexity answers, accounted for 1.1% of llms.txt requests. Ahrefs noted that Slack's link-preview bot fetched llms.txt more often than PerplexityBot did.

Just as telling, the study found AI bots never went looking for the file on their own; they only fetched it when a link on the page or an explicit user instruction pointed them at it. That behaviour matters, because it means an llms.txt sitting quietly at your root is not being discovered and read by AI search the way a sitemap is discovered by Google.

What Google says about llms.txt

Google has been unusually direct about its own position. Its Search Advocate, John Mueller, compared llms.txt to the old keywords meta tag, the self-reported signal that search engines learned to ignore decades ago because anyone could claim anything.

That is not an offhand remark. Google's own AI-features optimisation guide reinforces it, stating in a mythbusting section that machine-readable files like llms.txt are not needed to appear in AI search. And at the July 2025 Search Central event, Gary Illyes reportedly told attendees the same thing: Google doesn't support llms.txt and isn't planning to.

There is one genuine wrinkle worth knowing, because it points to where the file does have value. Google's Chrome team shipped an experimental Lighthouse audit that checks for llms.txt, framed entirely around AI agents browsing your site rather than search ranking. So Google-the-search-company ignores it while Google-the-browser-team nudges you toward it. That split is the clue to the one audience that actually consumes the file.

The one audience that does use it: coding agents

That audience is AI coding agents. In the Ahrefs data, Anthropic's Claude Code was the second-biggest fetcher of llms.txt overall, pulling it more often than every AI search bot, assistant and training crawler combined. Tools like Cursor and Windsurf let developers pull documentation directly into their coding context, and a well-structured llms.txt, or better still clean markdown docs served through content negotiation, makes those tools noticeably more accurate.

This is real, verifiable demand, and it is growing. It is just a very different audience from the one most B2B SaaS teams picture when they add the file.

It is worth being precise about how those agents actually use it, because it shapes what a good file looks like. A coding agent working in an IDE does not stumble onto your llms.txt through the open web. A developer points it at your docs, or your documentation platform surfaces the file through a link or a connected tool, and only then does the agent pull the index and follow the links it needs.

The value is not visibility in the sense marketers mean it; it is accuracy. When the agent has a clean, current map of your API reference and guides, it writes code against your product correctly and stops inventing endpoints that do not exist.

For a developer-tools company, fewer hallucinated integrations is a genuine product outcome. For a company whose buyers never touch an agent, that outcome simply does not apply, which is why the audience question decides everything.

In the interest of showing the full picture, there is a counter-anecdote worth naming. One practitioner, Ray Martinez, posted server logs in mid-2025 claiming OpenAI pinged his llms.txt files every 15 minutes checking for freshness. It made the rounds, but it covers a handful of sites, shows a fetch rather than any use of the content, and runs directly against the 137,000-domain dataset. We flag it because good analysis shows the exceptions, but it does not change the conclusion.

Should your B2B SaaS publish one?

For most companies, the answer is: publish it only if it is effectively free, and do not expect it to move your AI search visibility. The cost-benefit is lopsided in a specific way. The benefit today is small and narrow, but so is the cost, because platforms increasingly generate the file for you without any effort on your part.

The mistake is not publishing an llms.txt; it is spending real engineering or content hours on one in the belief that it will earn you ChatGPT citations.

Start with the audience question

The question that settles it for any given company is about audience, not technology. When a client asks us whether they should invest in llms.txt, our first question back is always the same.

Here is the decision we actually use with clients, reduced to the three cases that cover almost everyone.

If you ship developer documentation, publish one and keep it current, because coding agents are a real and growing audience and the file demonstrably helps them. If your CMS or docs platform generates it automatically, leave it on; there is no reason to switch it off, and it costs you nothing. If neither is true and you are hoping for AI-search lift, treat it as a low priority and spend the hours on the crawlability and content work that AI answers actually reward.

This also explains the roster of big names that already publish one. Anthropic, Stripe, Cloudflare, Vercel, Supabase and Cursor all host an llms.txt, and the common thread is that they sell to developers and run their docs on Mintlify, which generates the file by default. Their presence on that list is a platform default meeting a genuine developer audience, not evidence that the file drives search visibility.

The case for publishing anyway

It would be unfair to present only the skeptical case, because there is a reasonable argument on the other side and we want you to weigh it yourself.

The pragmatic case for publishing anyway is that the file is cheap, it points to real and verifiable URLs rather than unverifiable keyword claims, and it may future-proof you if AI search does go agentic. Search Engine Land made a solid version of this in arguing that llms.txt is not the new meta keywords, and platforms like Yoast frame it explicitly as future-proofing. We think that case is legitimate. We just weigh it against current evidence and land on "do it if it is free, don't if it is not," which is a different conclusion from "everyone needs one now."

How to create one without wasting time or adding risk

Option 1: let a tool generate it

You have two routes: let a tool generate it, or write it by hand. For most teams the tool route is the right one, and there is a good chance the file already exists on your site without you realising. Mintlify has auto-generated llms.txt and llms-full.txt for every documentation site it hosts since late 2024, which is precisely why so many developer-tools companies appear on the adoption lists.

Yoast added a native llms.txt generator to its WordPress plugin in June 2025, billed as the first built into an SEO plugin, and Wix now produces one automatically. If you are on any of these, the file is already live or one toggle away, and that is genuinely all the effort it warrants for most sites.

Option 2: write it by hand

If you write it by hand, treat it as curation, not a data dump. Give it an H1 with your product name, a one or two line summary that a model could quote verbatim, and a short list of your genuinely important pages with a plain-language note on each. Link to clean markdown versions where you can. Resist the urge to list every URL on the site; the entire point is to be selective, and a bloated llms.txt is just a worse sitemap that no one reads. Keep it current, because a stale index that points a coding agent at a renamed or deleted page is worse than no index at all.

Risks to manage before you ship it

The risks are modest but real, and worth naming so nobody is caught out. The same Ahrefs study spotted a crawler openly identifying itself as a prompt-injection probe, systematically testing llms.txt files as an attack surface. Because agents are built to trust the file, a compromised or carelessly edited llms.txt is a place where malicious instructions could ride along into an agent's context.

The lower-stakes risks are cloaking and duplicate content: serving one llms.txt to bots and different content to users edges toward cloaking, and publishing large markdown copies of every page can create clutter if handled without care.

None of this is a reason to avoid llms.txt, but it is a reason to treat the file like code rather than marketing copy. Put it in version control, restrict who can edit it, and review changes before they ship. If a file is going to be trusted by autonomous agents, it deserves the same hygiene you would give any other trusted input.

Where llms.txt fits in a real AI-visibility strategy

The most useful way to hold llms.txt is as a small, cheap bet on an agentic future that may or may not arrive. It is not a lever you pull to appear in AI answers today, and treating it as one pulls attention away from the work that genuinely moves visibility right now. Across the B2B SaaS accounts we run, the brands that get cited and recommended in AI answers are winning on fundamentals that are already in effect, not on files that nothing reads.

Those fundamentals are concrete. Make sure the AI search and user-triggered crawlers can actually reach and parse your pages, which is the whole subject of our guide to AI crawlers and matters far more than any root-level file.

Structure your pages so answers are easy to lift, the focus of our work on content formats for AI search, because models reward content they can extract cleanly.

Build the third-party coverage, reviews and comparisons that teach models to recommend you, since that off-site footprint shapes AI answers more than anything on your own domain.

And track where you actually appear, using the approaches in our guide to measuring AI search visibility, so you are optimising against evidence rather than hope.

The full playbook lives in our generative engine optimisation guide, and if you would rather have a team run it for you, that is exactly what our AI search optimisation service is built to do.

So publish an llms.txt if your platform hands it to you or you sell to developers. Skip the hand-built project if you are hoping it will get you into ChatGPT, because the data does not support that yet. And spend the time you save on the crawlability, structure and authority work that AI answers actually reward today. That is the honest, current read, and it is the one we would give a client paying for our time.

Frequently asked questions about llms.txt

Is llms.txt the same as robots.txt?

No. robots.txt is an access-control file that tells crawlers what they may and may not crawl, and it is widely respected. llms.txt is a content guide that lists and summarises your best pages for language models. It blocks nothing and commands nothing, and unlike robots.txt no major AI platform has agreed to honour it.

Does Google use llms.txt for ranking or AI Overviews?

No. Google's John Mueller compared llms.txt to the old keywords meta tag and said none of the AI services check for it, and Google's own AI-features guide states machine-readable files like llms.txt are not needed to appear in AI search. Normal SEO and clean, crawlable content are what drive AI Overview visibility.

Does ChatGPT or Perplexity read llms.txt?

Not in any meaningful way today. In Ahrefs' June 2026 study of 137,000 domains, AI retrieval bots, the crawlers behind ChatGPT and Perplexity answers, made up just 1.1% of the requests that llms.txt files received, and 97% of files got no requests at all. The retrieval crawlers only fetch the file when a link or user instruction points them to it.

Who actually uses llms.txt then?

AI coding agents. Anthropic's Claude Code was the second-biggest fetcher of llms.txt in the Ahrefs data, ahead of every AI search bot. Tools like Cursor and Windsurf can pull your documentation into a developer's coding context, so an llms.txt has genuine value if you sell to developers who use those tools.

Should a B2B SaaS company create an llms.txt file?

Publish one if it is effectively free, meaning your CMS or docs platform generates it, or if you ship developer documentation that coding agents will use. Do not build a project around it expecting AI search citations, because the evidence does not support that yet. For most teams it is a low-effort, low-priority item rather than a growth lever.

How do I create an llms.txt file?

The easiest route is a tool. Mintlify auto-generates one for every docs site it hosts, Yoast added a native generator to its WordPress plugin in June 2025, and Wix produces one automatically. To write it by hand, add a markdown file at your domain root with an H1 for your product name, a short summary blockquote, and H2 sections listing your most important pages with a note on each. Keep it curated rather than exhaustive.