robots.txt says what crawlers may access. sitemap.xml lists what exists. llms.txt explains what your site is. They're complementary, not competing — a well-configured site in 2026 has all three, each doing its own job.
The one-table answer
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Job | Access control | Page inventory | Curated summary |
| Audience | All crawlers | Search/AI crawlers | AI language models |
| Format | Plain-text directives | XML | Markdown |
| Consequence if missing | Everything default-allowed | Slower/incomplete discovery | AI infers your site from raw HTML |
| Age | 1994, universally honored | 2005, universally consumed | 2024, adoption still growing |
robots.txt: the permission layer
The oldest of the three, and the only one that restricts. Each User-agent block says what that crawler may fetch. Two modern realities changed how you should think about it:
- AI companies operate many distinct bots — training crawlers (GPTBot, ClaudeBot, Google-Extended) and search/browsing crawlers (OAI-SearchBot, PerplexityBot) — controllable independently. Silence means allowed: a bot you never mention crawls freely.
- It's voluntary. Reputable crawlers honor it; it is not an enforcement mechanism.
Build per-bot rules with the robots.txt Generator for AI Bots, and audit your live file with the AI Crawler Access Checker.
sitemap.xml: the inventory layer
A machine-readable list of every URL you want indexed, with optional freshness hints. It doesn't grant or deny anything — it accelerates discovery, which matters most for new sites, deep pages weakly linked from navigation, and young AI search indexes that haven't crawled the long tail of the web. Reference it from robots.txt (Sitemap: https://example.com/sitemap.xml) so the two files chain together. Small sites can generate one with the Sitemap Generator.
llms.txt: the comprehension layer
The newcomer, proposed in 2024. Where robots.txt and sitemap.xml serve crawler infrastructure, llms.txt serves model comprehension: a human-written Markdown summary — site name, one-line description, curated sections of links — that a language model can read in one pass instead of inferring your site's purpose from navigation menus and marketing copy.
Key differences from its siblings: it's curated (your 15 most important pages, not all 400), it's prose-friendly (descriptions, not just URLs), and its consumption is still uneven across AI platforms — publishing one is a cheap forward-looking bet, not a guaranteed signal. Full details in What is llms.txt?; build one with the llms.txt Generator.
How they work together
A concrete flow for an AI search crawler visiting your site:
- It reads robots.txt — may I crawl here, and what's off-limits?
- It finds your sitemap.xml referenced there — here's everything that exists.
- An AI assistant (or agent) reads llms.txt — here's what this site is and where the good stuff lives.
Misconfigurations usually come from confusing the layers: listing pages in your sitemap that robots.txt blocks (contradictory signals), treating llms.txt as a sitemap dump (defeats curation), or assuming llms.txt controls access (it doesn't — only robots.txt does).
Do you need all three?
- robots.txt: yes, always — if only to declare your AI-bot stance explicitly and point to your sitemap.
- sitemap.xml: yes for any site beyond a handful of pages.
- llms.txt: cheap and worthwhile for any site whose visibility to AI assistants matters — which, if you're reading this site, is yours.