llms.txt vs robots.txt: what each file does and how they interact

robots.txt tells crawlers what they may not fetch, and it's an internet standard (RFC 9309) that every major crawler, AI ones included, says it respects. llms.txt suggests what's worth reading, and it's a proposal at llmstxt.org that no major AI provider has committed to reading. So the llms.txt vs robots.txt question isn't "which one do I need." It's a gate versus a reading list, and the gate always wins.

If you remember one thing from this article, make it that last sentence. Most of the confusion I see comes from people treating the two files as alternatives, or worse, editing one while the other silently cancels it.

Purpose: a gate and a reading list

robots.txt is about restriction. It's been around since 1994, it lives at /robots.txt on every host, and its job is to say "don't request these paths." It says nothing about what's good. A perfectly configured robots.txt can allow everything.

llms.txt is about curation. It's a markdown file at /llms.txt listing the pages you consider most useful, with short descriptions, so a language model can skip your navigation and cookie banners. It says nothing about permission. I covered the format in the llms.txt generator guide, so I'll keep the syntax section short.

Different jobs. They overlap in exactly one place: both are fetched by bots from the root of your domain. That's where the trouble starts.

Syntax side by side, with example files

Here's a realistic robots.txt for a small EU service business that wants AI search visibility but doesn't want its account area or internal search crawled:

# robots.txt for example.com

        User-agent: *
        Disallow: /account/
        Disallow: /search
        Allow: /

        User-agent: GPTBot
        Disallow: /account/
        Disallow: /search

        # Opts out of Gemini training and grounding. Search and AI Overviews are unaffected.
        User-agent: Google-Extended
        Disallow: /

        Sitemap: https://example.com/sitemap.xml
        

The rules are line-based: a User-agent line opens a group, Disallow and Allow take path prefixes, and when two rules match a URL, the longest (most specific) path wins. Sitemap isn't part of RFC 9309's core rules, but Google, Bing and the main AI crawlers read it.

Notice the GPTBot group repeats the disallows. That's deliberate, and I'll explain why in the Michał story below.

And here's a matching llms.txt:

# Example Analytics

        > Independent GTM and GA4 monitoring for EU businesses. Monthly retainer from 390 EUR, hourly consulting, Free GTM Audit.

        ## Services
        - [GTM monitoring](https://example.com/gtm.md): Weekly automated tag health checks, fixes within 48 hours
        - [GA4 audits](https://example.com/ga4-audit.md): Scope, deliverables and price

        ## Articles
        - [Consent Mode v2 checklist](https://example.com/consent-mode.md): How to verify consent signals in Tag Assistant

        ## Optional
        - [Case archive](https://example.com/cases.md): Older client write-ups
        

The only required element is the # title. The blockquote summary and ## sections of links are conventions from the proposal, and the Optional section marks material a tool can skip when context is tight. There's no Disallow equivalent, no user-agent targeting, no wildcard matching. It's a document, not a rule set.

Who reads each file

This is where the two files really part ways.

robots.txt is read by essentially every well-behaved crawler: Googlebot, Bingbot, and the AI crawlers GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot, Applebot. OpenAI, Anthropic, Google and Perplexity all document their user agents and state that their crawlers follow robots.txt. RFC 9309 also says crawlers shouldn't cache the file for more than 24 hours, so a change usually takes effect within a day.

There's a wrinkle with user-triggered fetchers. When a person asks a chatbot to open a specific URL, some vendors treat that request differently from crawling. Perplexity documents that Perplexity-User generally ignores robots.txt because a human initiated the fetch. Check each vendor's current bot page rather than assuming the crawler rules carry over.

It also helps to know that the big vendors split their bots by job, so one robots.txt group doesn't cover all of a vendor's traffic:

Vendor	Training crawler	Search / index crawler	User-triggered fetcher
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User
Perplexity	(none separate)	PerplexityBot	Perplexity-User
Google	Google-Extended token, no separate bot	Googlebot	n/a for Search

This is why "block AI" is three decisions, not one. Blocking GPTBot keeps you out of OpenAI's training data. It doesn't stop OAI-SearchBot from indexing you for ChatGPT search, and that's usually the one a business wants to keep.

llms.txt has no documented readers among the big four. My llms.txt evidence article goes through what Google said (it doesn't use it) and what server logs show (occasional fetches by training crawlers, almost nothing from live fetchers). The confirmed readers are developer tools, where a human pastes the URL in on purpose.

Enforcement: neither file stops anyone

People treat robots.txt as a lock. It's a sign on the door.

RFC 9309 is explicit that the rules "are not a form of access authorization." A crawler that ignores them faces no technical barrier, only reputational and, increasingly, legal risk. Cloudflare publicly accused Perplexity in 2025 of crawling sites that had disallowed it, using undeclared user agents. Whatever you make of that dispute, it shows the limit: robots.txt works for crawlers that choose to comply.

llms.txt enforces even less, because it doesn't try to. It can't stop or permit anything.

If you need actual blocking, do it where requests are answered:

• CDN or WAF rules that match user agent plus verified IP ranges (OpenAI publishes ranges for its bots).
• Cloudflare's AI crawler controls, which block at the edge with a 403 regardless of what robots.txt says.
• Authentication for anything that shouldn't be public. If it's sensitive, a Disallow line just advertises the path.

How they interact (and the mistakes I keep finding)

Two stories, both from the past year.

Ewa runs a skincare shop on Shopify with Cloudflare in front. She spent two days building markdown twins and a careful llms.txt, then asked me why nothing changed in her AI citations.

Her logs answered it in five minutes: 1,940 requests from GPTBot in 30 days, every one of them a 403. A developer had switched on Cloudflare's AI bot blocking a year earlier and forgotten about it. Her robots.txt said "welcome." Her edge said "go away." The llms.txt was never delivered once.

The fix took one toggle. The two days of llms.txt work weren't wasted, but they'd been sitting behind a locked door for a year.

Michał runs a B2B parts catalog with 6,200 product URLs. He added a User-agent: GPTBot group with Disallow: /koszyk/ (his cart) and assumed his existing * rules, which blocked /admin-preview/, still applied. They didn't.

Under RFC 9309, a crawler follows the single most specific group matching its name and ignores the rest. For GPTBot, his file now allowed everything except the cart, including 340 preview pages. Fix: copy every rule you still want into the named group.

The general interaction rules:

1. robots.txt is checked first. If /llms.txt is disallowed for a bot, a compliant bot never reads it.

2. Allowing only /llms.txt is pointless. Disallow: / plus Allow: /llms.txt lets a bot read your reading list and then forbids every page on it.

3. Listing a disallowed URL in llms.txt doesn't override anything. The block stands.

4. Blocking Google-Extended doesn't touch AI Overviews. Those come from Search, crawled by Googlebot. To stay out of them you'd need nosnippet-style controls, which also hurt normal results.

That last point is why I'd be careful with blanket "block all AI" templates copied from forums. Some of them block bots that send you search visibility and miss the ones that only take training data.

Not sure which AI crawlers can actually reach your pages? AI Visibility crawls your site the way an LLM does and shows what's reachable and readable, page by page.

llms.txt vs robots.txt at a glance

	robots.txt	llms.txt
Purpose	Restrict what crawlers request	Curate what's worth reading
Status	Internet standard, RFC 9309 (2022)	Proposal at llmstxt.org (2024)
Location	/robots.txt, per host and protocol	/llms.txt at the domain root
Format	User-agent, Disallow, Allow lines	Markdown: title, summary, link lists
Targets specific bots	Yes, by user-agent group	No
Read by Google, OpenAI, Anthropic, Perplexity	Yes, all document compliance	No public commitment from any
Enforcement	Voluntary compliance	None, it's advisory content
Size limit	Crawlers must parse at least 500 KiB	None defined
What happens if it's missing	Everything is allowed	Nothing changes
Can it hurt you	Yes, a wrong Disallow can deindex sections	Practically no

That bottom row is the one I'd stare at. robots.txt is the file that can cost you traffic overnight. llms.txt is the one people spend time on.

Which one to work on first

I'll be blunt: if you've got an hour, spend 55 minutes on robots.txt and 5 on llms.txt.

Here's the order I use when I check a client site:

• Fetch /robots.txt and read every group. Look for named AI groups that accidentally drop your * rules.
• Check the CDN or WAF for AI bot blocking that contradicts the file.
• Grep 30 days of logs for GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot and look at status codes. A wall of 403s or 429s means the file isn't the problem.
• Decide, per bot, whether you want training crawlers, search crawlers, or both. They're separate user agents for a reason.
• Only then publish or update llms.txt, and make sure every URL in it is allowed.

The fundamentals that actually earn citations, answering early and writing checkable facts, are in my LLM SEO article. Neither text file substitutes for them.

Frequently asked questions

What is the difference between llms.txt and robots.txt? robots.txt tells crawlers which paths they may not request, and it's a formal standard (RFC 9309) that major crawlers follow. llms.txt is a markdown reading list for language models, a proposal no major AI provider has committed to reading.

Does llms.txt replace robots.txt? No. They do different jobs, and robots.txt takes precedence for any compliant crawler. If a bot is blocked there, it won't read llms.txt or the pages it lists.

How do I block GPTBot in robots.txt? Add a group with User-agent: GPTBot and Disallow: /. To keep ChatGPT search visibility while opting out of training, block GPTBot and leave OAI-SearchBot allowed, since OpenAI documents them as separate crawlers.

Does blocking Google-Extended remove me from AI Overviews? No. Google-Extended controls whether your content is used for Gemini training and grounding, and Google says it doesn't affect Search. AI Overviews are part of Search and use pages crawled by Googlebot.

Is robots.txt legally binding? It's a technical convention, not a contract, and RFC 9309 says the rules aren't access authorization. Some jurisdictions treat machine-readable opt-outs as relevant for text and data mining (the EU's copyright rules include one), but don't rely on the file alone to protect anything sensitive.

Can I list pages in llms.txt that robots.txt blocks? You can, but it achieves nothing. Compliant crawlers follow robots.txt and won't fetch those URLs, so keep both files consistent.

Next steps

Open your own /robots.txt right now and read it as GPTBot would: find the most specific group that matches, ignore everything else, and see what's left. Then check your CDN for a block that overrules it. That's ten minutes, and it's the part of this topic that actually changes outcomes.

Then publish an llms.txt if you like. It's cheap. Just don't expect a reading list to open a door that your gate has shut. If you'd rather have someone go through the files, logs and CDN rules with you, that's a custom project.

---

Sources and further reading - RFC 9309: Robots Exclusion Protocol (IETF) - The llms.txt proposal (llmstxt.org) - Introduction to robots.txt (Google Search Central) - OpenAI crawlers and user agents (OpenAI)