llms.txt and AI Crawlers: What Actually Matters

Someone forwards your agency a blog post about llms.txt, and the question lands on your desk: do we need this file, and what happens if we skip it?

Someone forwards your agency a blog post about llms.txt, and the question lands on your desk: do we need this file, and what happens if we skip it? It is a fair question and it deserves a straight answer rather than a hedge, because the honest version saves you a sprint of work that would not have moved anything.

What llms.txt actually is

llms.txt is a proposed convention: a plain-Markdown file at the root of your domain that lists your most important pages with a sentence about each, so a language model reading your site has a curated map instead of having to infer one. Think of it as a table of contents written for a machine that will only ever read a fraction of your site.

The critical thing to understand is what it is not. It is not a standard that any major AI company has committed to honouring. It is a proposal with real adoption among developer-tooling and documentation sites, and essentially no confirmed enforcement from the assistants most of your buyers use. Nobody at your company should present it as a requirement, and nobody should present skipping it as a risk.

So should you add one?

Yes, if it takes an afternoon. No, if it takes a sprint.

That is not a dodge — it is the actual cost-benefit. A small file listing twenty pages costs almost nothing to write and nothing to maintain if you keep it short. The upside is uncertain but non-zero, and the downside is an hour. That is a fine trade. What is not a fine trade is restructuring your content, hiring someone to generate a file for every subdirectory, or delaying work that demonstrably matters in order to ship it.

The failure mode to avoid: a business spends three weeks on llms.txt while its service pages still have no clear answer to what the company does and who it is for in the first hundred words. The second problem is the one costing them citations. The first is a rounding error.

The crawlers you can actually control

Separate two things that get conflated constantly: crawling for training, and fetching to answer a live question. They use different user agents, they are controlled separately, and blocking one does not block the other.

• Training crawlers collect pages to build future models. Blocking them keeps your content out of a training corpus and does nothing to your visibility today.
• Live retrieval fetchers pull a page at the moment a user asks a question, so the assistant can cite it. Blocking these removes you from answers right now.
• Search-index crawlers feed conventional search, which in turn feeds several AI answer surfaces. Blocking these is the most expensive mistake of the three.

The mistake to watch for is a blanket block. Somebody reads that AI is scraping our content, adds a broad disallow rule, and quietly removes the business from the exact surface it was trying to win. If you want to opt out of training but stay quotable in live answers, those are two different rules and you have to write them as two different rules.

Check what your site actually does today

Before changing anything, find out what is already in place. In most small businesses nobody remembers, and the rule that is hurting you was added by a developer who left.

• Open your robots.txt in a browser and read every line. Note any disallow rule that is not scoped to a specific path.
• Ask whoever runs your hosting or CDN whether bot filtering is switched on. Security products block AI fetchers by default more often than people expect.
• Check whether your key pages render their content in the HTML or assemble it in the browser with JavaScript. A fetcher that gets an empty shell has nothing to quote.
• Confirm your pages return a normal 200 and not a consent wall, an age gate, or a region redirect for a visitor with no cookies.

That last one catches more businesses than any file ever will. A cookie banner that blocks the page body until it is dismissed is invisible to your team, who clicked it once months ago, and total to a fetcher that has never been there.

What actually decides whether you get quoted

An assistant quoting your business is doing something close to what a rushed human researcher does: land on a page, look for a specific claim that answers the question, and check whether it can be attributed to somebody. Pages that survive that are pages that state things plainly.

• Answer the question in the first paragraph, then explain. Content that builds to its point over four sections gets skimmed and dropped.
• Make the specifics extractable: what you do, where you do it, who it is for, what is included, what is not. Vague scope is unquotable.
• Give claims an owner. A sentence like our clinic has offered same-day appointments since 2019 survives a quote. Industry-leading response times does not.
• Keep one canonical page per topic. Three near-identical service pages split the signal and none of them reads as the authoritative one.
• Put the facts in text, not only in an image, a PDF or a video.

None of that is new advice, which is the uncomfortable part. The businesses getting cited by assistants are mostly the ones that were already clear, already specific, and already structured. AEO rewarded the same work that good content always did; it just raised the penalty for padding.

A sane order of operations

If you are the person deciding where the next two weeks go, this is the sequence that produces results rather than activity.

• First, confirm nothing is blocking retrieval — robots rules, bot filtering, consent walls, JavaScript-only content.
• Second, fix the five pages that matter: services, pricing or how you charge, about, contact, and your single best explainer. Make each one answer its question in the opening lines.
• Third, make your basic facts consistent everywhere they appear — name, location, service area, hours, what you sell.
• Fourth, if you still have an afternoon spare, write the llms.txt file.
• Do not reorder this list because the fourth item is the one people are blogging about.

How to tell whether any of it worked

Be realistic about measurement, because this is where the topic usually collapses into wishful thinking. There is no clean dashboard for AI citations, and anyone selling you one is selling a sample, not a census.

What you can do: write down the ten questions a good-fit buyer would actually ask an assistant before hiring someone like you. Ask them yourself, in the assistants your buyers use, and record what comes back — who is named, what is said about you if anything, and which page gets cited. Repeat it on a fixed schedule rather than whenever someone is curious. It is a spot check, not analytics, and it is worth more than a tool that reports a number nobody can trace.

Then watch the thing that actually pays: whether people arriving at your site already know what you do. A buyer who opens with a question about whether you handle the permit side too has been briefed by something. That is the signal, and it shows up in sales conversations long before it shows up in a report.

Keep reading

• What is a fractional CMO? — the plain-English 2026 guide
• Fractional CMO cost in 2026 — real numbers, including ours
• AI CMO vs fractional CMO — how the models actually differ
• Compare Fracmo to agencies, in-house hires and DIY tools
• All Fracmo blog guides