Yes — OpenAI, Anthropic, and the other major AI companies state that their crawlers respect robots.txt. But "ChatGPT" itself isn't one single crawler, and the answer depends on which of its several distinct bots you're asking about.
Does ChatGPT read robots.txt?
OpenAI operates three separate crawlers under the ChatGPT/OpenAI umbrella, each respecting robots.txt independently:
GPTBot— used for training future models. If blocked, your content isn't used in OpenAI's training data going forward.OAI-SearchBot— powers search-style results shown inside ChatGPT. If blocked, your pages won't appear in ChatGPT's search-style citations.ChatGPT-User— fetches a specific page when a user asks ChatGPT to browse or read a URL live. If blocked, ChatGPT can't fetch that page even when a user explicitly requests it.
Blocking one doesn't block the others — check your current rules for all three with the AI Crawler Access Checker.
Does Claude read robots.txt?
Same pattern, different names: ClaudeBot (training), Claude-SearchBot (search indexing), and Claude-User (live browsing on a user's behalf). Anthropic documents all three as respecting robots.txt.
Does Perplexity read robots.txt?
Perplexity operates PerplexityBot (crawling/indexing for its answer engine) and Perplexity-User (live fetches triggered by a user's specific request). Both are documented as robots.txt-compliant.
If I block a bot, can it still browse my site when a user pastes my URL?
This depends on which bot is blocked. If you block ChatGPT-User (or the equivalent live-browsing bot for another platform) specifically, then no — the assistant won't be able to fetch that page even at a user's explicit request, and will typically tell the user it couldn't access the content. If you only block the training bot (GPTBot) and leave ChatGPT-User allowed, live fetches on request still work.
Does blocking these bots remove content already used in training?
No. robots.txt only controls future crawling. If a training bot already crawled and ingested your content before you added a block, that data's influence on already-released model versions isn't retroactively removed. Blocking going forward prevents further ingestion in future crawls and training runs, but isn't a takedown mechanism for the past.
Is there a way to fully guarantee AI companies never access my content?
Not through robots.txt alone. It's a voluntary standard: compliant crawlers honor it, but it isn't a technical barrier, and it can't stop a non-compliant or unlisted bot. If content truly must not be accessible to any automated client, the only reliable control is authentication (login-gating the content) — robots.txt communicates intent to well-behaved crawlers, it doesn't enforce access control.
Do these crawlers respect Disallow rules for specific paths, or only whole-site blocks?
Path-specific rules work the same way as they do for any other crawler — you can allow AI bots on most of your site while disallowing specific paths (e.g., /account/, /internal/) under each bot's User-agent block, or under a shared User-agent: * block if you want the same path rules to apply broadly.
How do I check what I've currently configured?
Enter your domain into the AI Crawler Access Checker — it parses your live robots.txt and reports allow/blocked/not-mentioned status for every bot covered in this article. If you need to change your rules, the robots.txt Generator for AI Bots builds a new file with presets or per-bot toggles.
Do these rules ever change?
Yes — AI companies periodically launch new crawlers or rename existing ones as products evolve. We refresh the bot list behind our tools monthly; if you're reading this guide some time after publication, re-run the access checker rather than relying solely on the list above.