AI crawler profile
What is ClaudeBot? How to allow or block Anthropic's crawler
Operator
Anthropic
Purpose
AI model training
robots.txt
Respects robots.txt
Official docs
ClaudeBot is Anthropic's primary web crawler, collecting publicly available content that may be used to train the Claude family of models. It is the Anthropic equivalent of OpenAI's GPTBot, and the user agent to target if you want to opt out of Claude training specifically.
Does ClaudeBot respect robots.txt?
Anthropic documents ClaudeBot and states it respects robots.txt directives and anti-circumvention signals.
Block ClaudeBot with robots.txt
User-agent: ClaudeBot Disallow: /
Explicitly allow ClaudeBot
User-agent: ClaudeBot Allow: /
Should you block ClaudeBot?
Blocking ClaudeBot opts your content out of Anthropic's model training — a legitimate choice for original content you don't want reproduced by AI. The trade-off: models trained without your content are less likely to know your brand or recommend it unprompted. There is no direct traffic loss today, since training crawlers don't send visitors. See our guide on whether to block AI bots for the full decision framework.
Check and monitor ClaudeBot on your site
- AI Crawler Access Checker — see whether your current robots.txt allows or blocks ClaudeBot.
- robots.txt Generator for AI Bots — build a robots.txt with per-bot rules for all 15 known AI crawlers.
- AI Bot Log Analyzer — paste server logs and count real ClaudeBot hits, entirely in your browser.
Frequently asked questions
What is ClaudeBot?
ClaudeBot is Anthropic's web crawler for collecting AI training data. Crawls content to train Anthropic's Claude models.
Does ClaudeBot respect robots.txt?
Anthropic documents ClaudeBot and states it respects robots.txt directives and anti-circumvention signals.
How do I block ClaudeBot?
Add "User-agent: ClaudeBot" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.
Does blocking ClaudeBot hurt my Google rankings?
No. ClaudeBot is separate from Googlebot, which handles Google Search indexing. Blocking ClaudeBot has no effect on your traditional search rankings.
How can I tell if ClaudeBot is crawling my site?
Search your server access logs for the string "ClaudeBot". Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including ClaudeBot, with per-path breakdowns.
More Anthropic agents
Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-07-15.