llms.txt is a plain markdown file at the root of your domain that tells AI models what your site contains and where the good material is. Think of it as robots.txt inverted: instead of restricting crawlers, it curates for them.
You can write one by hand in ten minutes. No plugin, no library, no build step. The format is deliberately simple: a title, a short description, and a list of links with one-line summaries.
Here is the part most articles about this quietly skip. Nobody can currently prove that AI models read it. The specification is a proposal, not a standard. No major model provider has publicly committed to fetching it. Anyone telling you it definitely improves your ChatGPT visibility is guessing, and charging you for the guess.
I run one on this site anyway, and I think you should too. But for a specific reason that has nothing to do with hoping a standard gets adopted, and I will be precise about what that reason is.
llms.txtlives atyourdomain.com/llms.txtand is plain markdown. No special syntax, no validation step.- It curates rather than restricts. It says "here is what matters", not "you may not enter". That makes it the opposite of
robots.txt. - No model provider has publicly confirmed reading it. Treat guaranteed-AI-visibility claims as marketing.
- The real value is the markdown twins: clean
.mdversions of your pages, which remove the parsing problem entirely and pay off regardless of any standard. - The file that kills most implementations isn't the first one. It's the one that goes stale two months later.
What actually goes in an llms.txt file
One H1, an optional summary blockquote, then sections of links. That's the entire specification.
# Piotr Litwa
> Independent GTM and GA4 monitoring for EU businesses.
> One specialist, not an agency.
## Articles
- [Unassigned Traffic in GA4](https://piotrlitwa.com/articles/en/unassigned-traffic-ga4.html): The six causes of unassigned traffic in GA4 and how to fix each one. [Markdown](https://piotrlitwa.com/articles/md/en/unassigned-traffic-ga4.md)
## Services
- [GTM Monitoring](https://piotrlitwa.com/services.html#gtm): Weekly automated health checks of tags, triggers and consent. €150/month, no contract.Three rules separate a useful file from decoration.
Write descriptions for a machine that will never see your page. "Learn more about our services" is worthless. "Weekly automated GTM health checks, €150/month, no contract" is a fact a model can lift directly into an answer. Facts get quoted. Enthusiasm doesn't.
Link to markdown versions wherever you have them. This is the highest-value line in the file and the one people skip. A model reading your HTML has to fight through navigation, a cookie banner, and three CTAs. A model reading your .md file gets the content and nothing else.
Curate, don't dump. An llms.txt listing all 400 of your pages is a sitemap with extra steps. The point is to say which twenty pages actually represent what you know.
Building your llms.txt: the ten-minute version
1. Decide what belongs. Your best twenty to fifty pages. The ones you would send to someone who asked "what do you actually do, and what do you know about it?"
2. Write one factual line per page. Numbers, prices, versions, product names. Not teasers.
3. Generate the markdown twins. For each listed page, publish a .md version with the same content and headings, without the site chrome. If you have fifty pages, this is the part that takes real time. It's also the part that does the actual work.
4. Save it at /llms.txt in your domain root, served as plain text. Load yourdomain.com/llms.txt in a browser and confirm you get raw text, not a 404 and not a styled page.
5. Solve the sync problem. A stale llms.txt is worse than none, because it confidently describes pages that changed or vanished. If generating it isn't part of how you publish, it will be wrong within a quarter.
Step 5 is where most implementations die. The file gets written during one enthusiastic afternoon and nobody touches it again.
The part that actually matters: markdown twins
If you take one thing from this article, take this.
For every important page, publish a clean markdown version at a predictable URL, and link it from the HTML head:
<link rel="alternate" type="text/markdown" href="../md/en/unassigned-traffic-ga4.md">This site does it at /articles/md/en/<slug>.md for every article, in both languages.
Why this beats the index file: a model that finds your markdown gets your argument, your tables, and your headings with zero parsing risk. No navigation to skip. No consent banner to trip over. No JavaScript-rendered content it can't see at all.
And here is the difference that matters. Unlike llms.txt, this has value whether or not any AI specification is ever adopted, because a markdown twin is simply a cleaner representation of your content that happens to be trivially machine-readable. You aren't betting on a standard. You are removing an obstacle that exists today.
Does any of this actually work?
Let me be straight, because the marketing around this topic isn't.
What is true: the format is public, several hundred sites have adopted it, and it costs almost nothing to implement.
What isn't confirmed: that OpenAI, Anthropic, Google, or Perplexity read llms.txt when answering questions. None has publicly committed to it. Independent analyses have found that the large majority of published llms.txt files are never requested by any AI crawler at all.
So why run one?
Because the file is a byproduct of doing something unambiguously useful: maintaining clean, canonical, machine-readable versions of your content with honest summaries, kept in sync with what you publish. If the standard is adopted, you are already there. If it never is, you still have markdown twins that every crawler and model can read without a fight.
That's a reasonable bet on a cheap option. "This will get you cited in ChatGPT" isn't a reasonable claim, and you should discount anyone making it, including anyone selling you a tool to do it.
How to check whether anything actually fetches it
You don't have to take my word for the scepticism, or anyone else's enthusiasm. Your server logs settle it.
AI crawlers identify themselves. Grep your access logs for the request path and the user agent:
grep "llms.txt" access.logThe agents worth looking for:
| User agent | Who it is | What it does |
|---|---|---|
GPTBot | OpenAI | Crawls for training |
OAI-SearchBot | OpenAI | Crawls for ChatGPT search results |
ChatGPT-User | OpenAI | Fetches a page live when a user asks about it |
ClaudeBot | Anthropic | Crawls for training |
Claude-User | Anthropic | Live fetch during a conversation |
PerplexityBot | Perplexity | Crawls for its index |
Google-Extended | Controls Gemini training use |
Two things you will probably find.
Your llms.txt gets very few hits, if any. That's the honest state of the world right now, and it's exactly why I wouldn't build a strategy on it.
Your individual pages get plenty. GPTBot, ClaudeBot, and PerplexityBot are almost certainly crawling your content already, right now, without asking permission through any index file. They found you through links and sitemaps, like every crawler always has.
That second finding is the actionable one. The models are already reading your site. The question is not how to invite them. It is whether what they find is legible. A page buried under a cookie wall, rendered by JavaScript, wrapped in navigation, is hard for them to use. A markdown twin isn't.
That reframing is the whole point of this article, and it's why the file is the smaller half of the job.
WordPress, Webflow, and static sites
Static sites: trivial. Write the file, put it in the root, deploy. This site generates it as part of the publish step, which is precisely why it never goes stale.
WordPress: you don't need a plugin. Anything that serves a file at the root works, including uploading it directly. A plugin helps if you want auto-generation from your post list, but be careful: auto-generated files tend to dump every post, which defeats the entire purpose of curation.
Webflow, Shopify, and hosted platforms: check whether the platform lets you serve arbitrary root-level files. Several don't. If yours doesn't, skip the index file and put your effort into markdown twins, which matter more anyway.
Frequently asked questions
What is llms.txt? A plain markdown file at your domain root that tells AI models what your site contains and links to the pages that matter, ideally with markdown versions attached. It curates rather than restricts, which makes it the opposite of robots.txt.
Do AI models actually read llms.txt? No provider has publicly confirmed it, and independent analysis suggests most published files are never fetched. Treat it as a cheap bet, not a guaranteed visibility channel, and be sceptical of anyone selling it as the latter.
Do I need a plugin or a generator tool? No. It's plain markdown and you can write it by hand in ten minutes. A tool earns its place on the sync problem, not the writing problem: keeping the file accurate as you publish is the hard part.
What is the difference between llms.txt and robots.txt? robots.txt restricts crawlers. llms.txt curates content for models. One says where you may not go, the other says where the good material is. They serve opposite purposes and you can run both.
Is llms.txt just a sitemap? No. A sitemap lists every URL for completeness. llms.txt should list only your best pages, with descriptions, for comprehension. A file listing everything is a sitemap in markdown and adds nothing.
What should I do instead, if llms.txt is unproven? Publish clean markdown versions of your key pages and link them from the HTML head. That removes the parsing problem for every model regardless of which standard wins, and it's the part of this work with value either way.
Next steps
Write the file by hand first. It takes ten minutes and it will teach you more about what your site actually claims than any generator would.
Then do the part that matters: publish markdown twins of your important pages and link them from the HTML. That work pays off whether or not the specification ever becomes a standard, because it makes your content readable to machines without a fight, today.
Then solve the sync problem, because that is what kills these files. If generating llms.txt isn't part of how you publish, it will be quietly describing pages that no longer exist by the time anyone checks.
If you would rather have it audited and generated automatically, including the markdown twins and a coverage diff against your sitemap, that is what I built AI Visibility for. The free plan covers one site and 200 URLs, no credit card. And if you want a human to look at what AI models can actually see on your site, that is a scoping call.
Sources and further reading
- The llms.txt proposal (Jeremy Howard)
- Google crawler documentation (Google)
- Robots Exclusion Protocol (IETF RFC 9309)
Is your GA4 attribution telling the truth?
The Free GTM Audit inspects the tracking layer that feeds your channel data: broken tags, consent gaps, and firing order. Enter a URL, get a report in about 10 minutes. No signup.
Start Free GTM Audit