# llms.txt vs robots.txt: what each file does and how they interact **Author:** Piotr Litwa - GTM & Analytics Specialist **Published:** 2026-08-07 **URL:** https://piotrlitwa.com/articles/en/llms-txt-vs-robots-txt.html **Language:** en **Keywords:** llms.txt vs robots.txt, robots.txt ai crawlers, block gptbot robots.txt, rfc 9309, llms.txt example --- `robots.txt` tells crawlers what they may not fetch, and it's an internet standard (RFC 9309) that every major crawler, AI ones included, says it respects. `llms.txt` suggests what's worth reading, and it's a proposal at llmstxt.org that no major AI provider has committed to reading. So the llms.txt vs robots.txt question isn't "which one do I need." It's a gate versus a reading list, and the gate always wins. If you remember one thing from this article, make it that last sentence. Most of the confusion I see comes from people treating the two files as alternatives, or worse, editing one while the other silently cancels it. > **Key Takeaways** > - `robots.txt` is the Robots Exclusion Protocol, standardized as RFC 9309 in September 2022. `llms.txt` is a community proposal from September 2024 with no formal standing. > - Neither file is enforcement. RFC 9309 itself says the rules aren't a form of access authorization. Real blocking happens at the server or CDN. > - If `robots.txt` disallows `GPTBot`, a compliant `GPTBot` won't fetch your `llms.txt` either. The reading list becomes moot. > - A crawler obeys only the most specific `User-agent` group that matches it. Add a `GPTBot` group and it stops reading your `*` rules entirely. > - `Google-Extended` controls Gemini training and grounding use, not Search. Blocking it doesn't remove you from AI Overviews. ## Purpose: a gate and a reading list `robots.txt` is about restriction. It's been around since 1994, it lives at `/robots.txt` on every host, and its job is to say "don't request these paths." It says nothing about what's good. A perfectly configured `robots.txt` can allow everything. `llms.txt` is about curation. It's a markdown file at `/llms.txt` listing the pages you consider most useful, with short descriptions, so a language model can skip your navigation and cookie banners. It says nothing about permission. I covered the format in the [llms.txt generator guide](https://piotrlitwa.com/articles/en/llms-txt-generator.html), so I'll keep the syntax section short. Different jobs. They overlap in exactly one place: both are fetched by bots from the root of your domain. That's where the trouble starts. ## Syntax side by side, with example files Here's a realistic `robots.txt` for a small EU service business that wants AI search visibility but doesn't want its account area or internal search crawled: ```text # robots.txt for example.com User-agent: * Disallow: /account/ Disallow: /search Allow: / User-agent: GPTBot Disallow: /account/ Disallow: /search # Opts out of Gemini training and grounding. Search and AI Overviews are unaffected. User-agent: Google-Extended Disallow: / Sitemap: https://example.com/sitemap.xml ``` The rules are line-based: a `User-agent` line opens a group, `Disallow` and `Allow` take path prefixes, and when two rules match a URL, the longest (most specific) path wins. `Sitemap` isn't part of RFC 9309's core rules, but Google, Bing and the main AI crawlers read it. Notice the `GPTBot` group repeats the disallows. That's deliberate, and I'll explain why in the Michał story below. And here's a matching `llms.txt`: ```markdown # Example Analytics > Independent GTM and GA4 monitoring for EU businesses. Monthly retainer from 390 EUR, hourly consulting, Free GTM Audit. ## Services - [GTM monitoring](https://example.com/gtm.md): Weekly automated tag health checks, fixes within 48 hours - [GA4 audits](https://example.com/ga4-audit.md): Scope, deliverables and price ## Articles - [Consent Mode v2 checklist](https://example.com/consent-mode.md): How to verify consent signals in Tag Assistant ## Optional - [Case archive](https://example.com/cases.md): Older client write-ups ``` The only required element is the `#` title. The blockquote summary and `##` sections of links are conventions from the proposal, and the `Optional` section marks material a tool can skip when context is tight. There's no `Disallow` equivalent, no user-agent targeting, no wildcard matching. It's a document, not a rule set. ## Who reads each file This is where the two files really part ways. `robots.txt` is read by essentially every well-behaved crawler: `Googlebot`, `Bingbot`, and the AI crawlers `GPTBot`, `OAI-SearchBot`, `ClaudeBot`, `PerplexityBot`, `CCBot`, `Applebot`. OpenAI, Anthropic, Google and Perplexity all document their user agents and state that their crawlers follow `robots.txt`. RFC 9309 also says crawlers shouldn't cache the file for more than 24 hours, so a change usually takes effect within a day. There's a wrinkle with user-triggered fetchers. When a person asks a chatbot to open a specific URL, some vendors treat that request differently from crawling. Perplexity documents that `Perplexity-User` generally ignores `robots.txt` because a human initiated the fetch. Check each vendor's current bot page rather than assuming the crawler rules carry over. It also helps to know that the big vendors split their bots by job, so one `robots.txt` group doesn't cover all of a vendor's traffic: | Vendor | Training crawler | Search / index crawler | User-triggered fetcher | |---|---|---|---| | OpenAI | `GPTBot` | `OAI-SearchBot` | `ChatGPT-User` | | Anthropic | `ClaudeBot` | `Claude-SearchBot` | `Claude-User` | | Perplexity | (none separate) | `PerplexityBot` | `Perplexity-User` | | Google | `Google-Extended` token, no separate bot | `Googlebot` | n/a for Search | This is why "block AI" is three decisions, not one. Blocking `GPTBot` keeps you out of OpenAI's training data. It doesn't stop `OAI-SearchBot` from indexing you for ChatGPT search, and that's usually the one a business wants to keep. `llms.txt` has no documented readers among the big four. My [llms.txt evidence article](https://piotrlitwa.com/articles/en/does-llms-txt-work.html) goes through what Google said (it doesn't use it) and what server logs show (occasional fetches by training crawlers, almost nothing from live fetchers). The confirmed readers are developer tools, where a human pastes the URL in on purpose. ## Enforcement: neither file stops anyone People treat `robots.txt` as a lock. It's a sign on the door. RFC 9309 is explicit that the rules "are not a form of access authorization." A crawler that ignores them faces no technical barrier, only reputational and, increasingly, legal risk. Cloudflare publicly accused Perplexity in 2025 of crawling sites that had disallowed it, using undeclared user agents. Whatever you make of that dispute, it shows the limit: `robots.txt` works for crawlers that choose to comply. `llms.txt` enforces even less, because it doesn't try to. It can't stop or permit anything. If you need actual blocking, do it where requests are answered: - **CDN or WAF rules** that match user agent plus verified IP ranges (OpenAI publishes ranges for its bots). - **Cloudflare's AI crawler controls**, which block at the edge with a `403` regardless of what `robots.txt` says. - **Authentication** for anything that shouldn't be public. If it's sensitive, a `Disallow` line just advertises the path. ## How they interact (and the mistakes I keep finding) Two stories, both from the past year. **Ewa** runs a skincare shop on Shopify with Cloudflare in front. She spent two days building markdown twins and a careful `llms.txt`, then asked me why nothing changed in her AI citations. Her logs answered it in five minutes: 1,940 requests from `GPTBot` in 30 days, every one of them a `403`. A developer had switched on Cloudflare's AI bot blocking a year earlier and forgotten about it. Her `robots.txt` said "welcome." Her edge said "go away." The `llms.txt` was never delivered once. The fix took one toggle. The two days of `llms.txt` work weren't wasted, but they'd been sitting behind a locked door for a year. **Michał** runs a B2B parts catalog with 6,200 product URLs. He added a `User-agent: GPTBot` group with `Disallow: /koszyk/` (his cart) and assumed his existing `*` rules, which blocked `/admin-preview/`, still applied. They didn't. Under RFC 9309, a crawler follows the single most specific group matching its name and ignores the rest. For `GPTBot`, his file now allowed everything except the cart, including 340 preview pages. Fix: copy every rule you still want into the named group. The general interaction rules: 1. **`robots.txt` is checked first.** If `/llms.txt` is disallowed for a bot, a compliant bot never reads it. 2. **Allowing only `/llms.txt` is pointless.** `Disallow: /` plus `Allow: /llms.txt` lets a bot read your reading list and then forbids every page on it. 3. **Listing a disallowed URL in `llms.txt` doesn't override anything.** The block stands. 4. **Blocking `Google-Extended` doesn't touch AI Overviews.** Those come from Search, crawled by `Googlebot`. To stay out of them you'd need `nosnippet`-style controls, which also hurt normal results. That last point is why I'd be careful with blanket "block all AI" templates copied from forums. Some of them block bots that send you search visibility and miss the ones that only take training data. **Not sure which AI crawlers can actually reach your pages?** [AI Visibility](https://aiv.piotrlitwa.com/?lang=en) crawls your site the way an LLM does and shows what's reachable and readable, page by page. ## llms.txt vs robots.txt at a glance | | `robots.txt` | `llms.txt` | |---|---|---| | Purpose | Restrict what crawlers request | Curate what's worth reading | | Status | Internet standard, RFC 9309 (2022) | Proposal at llmstxt.org (2024) | | Location | `/robots.txt`, per host and protocol | `/llms.txt` at the domain root | | Format | `User-agent`, `Disallow`, `Allow` lines | Markdown: title, summary, link lists | | Targets specific bots | Yes, by user-agent group | No | | Read by Google, OpenAI, Anthropic, Perplexity | Yes, all document compliance | No public commitment from any | | Enforcement | Voluntary compliance | None, it's advisory content | | Size limit | Crawlers must parse at least 500 KiB | None defined | | What happens if it's missing | Everything is allowed | Nothing changes | | Can it hurt you | Yes, a wrong `Disallow` can deindex sections | Practically no | That bottom row is the one I'd stare at. `robots.txt` is the file that can cost you traffic overnight. `llms.txt` is the one people spend time on. ## Which one to work on first I'll be blunt: if you've got an hour, spend 55 minutes on `robots.txt` and 5 on `llms.txt`. Here's the order I use when I check a client site: - Fetch `/robots.txt` and read every group. Look for named AI groups that accidentally drop your `*` rules. - Check the CDN or WAF for AI bot blocking that contradicts the file. - Grep 30 days of logs for `GPTBot`, `OAI-SearchBot`, `ClaudeBot` and `PerplexityBot` and look at status codes. A wall of `403`s or `429`s means the file isn't the problem. - Decide, per bot, whether you want training crawlers, search crawlers, or both. They're separate user agents for a reason. - Only then publish or update `llms.txt`, and make sure every URL in it is allowed. The fundamentals that actually earn citations, answering early and writing checkable facts, are in my [LLM SEO article](https://piotrlitwa.com/articles/en/llm-seo.html). Neither text file substitutes for them. ## Frequently asked questions **What is the difference between llms.txt and robots.txt?** `robots.txt` tells crawlers which paths they may not request, and it's a formal standard (RFC 9309) that major crawlers follow. `llms.txt` is a markdown reading list for language models, a proposal no major AI provider has committed to reading. **Does llms.txt replace robots.txt?** No. They do different jobs, and `robots.txt` takes precedence for any compliant crawler. If a bot is blocked there, it won't read `llms.txt` or the pages it lists. **How do I block GPTBot in robots.txt?** Add a group with `User-agent: GPTBot` and `Disallow: /`. To keep ChatGPT search visibility while opting out of training, block `GPTBot` and leave `OAI-SearchBot` allowed, since OpenAI documents them as separate crawlers. **Does blocking Google-Extended remove me from AI Overviews?** No. `Google-Extended` controls whether your content is used for Gemini training and grounding, and Google says it doesn't affect Search. AI Overviews are part of Search and use pages crawled by `Googlebot`. **Is robots.txt legally binding?** It's a technical convention, not a contract, and RFC 9309 says the rules aren't access authorization. Some jurisdictions treat machine-readable opt-outs as relevant for text and data mining (the EU's copyright rules include one), but don't rely on the file alone to protect anything sensitive. **Can I list pages in llms.txt that robots.txt blocks?** You can, but it achieves nothing. Compliant crawlers follow `robots.txt` and won't fetch those URLs, so keep both files consistent. ## Next steps Open your own `/robots.txt` right now and read it as `GPTBot` would: find the most specific group that matches, ignore everything else, and see what's left. Then check your CDN for a block that overrules it. That's ten minutes, and it's the part of this topic that actually changes outcomes. Then publish an `llms.txt` if you like. It's cheap. Just don't expect a reading list to open a door that your gate has shut. If you'd rather have someone go through the files, logs and CDN rules with you, that's a [custom project](https://piotrlitwa.com/services.html#custom). --- **Sources and further reading** - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (IETF) - [The llms.txt proposal](https://llmstxt.org/) (llmstxt.org) - [Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) (Google Search Central) - [OpenAI crawlers and user agents](https://platform.openai.com/docs/bots) (OpenAI) --- *Written by [Piotr Litwa](https://piotrlitwa.com/about.html) - independent GTM & Analytics specialist.*