AI crawler profile
What is Claude-SearchBot? How to allow or block Anthropic's crawler
Operator
Anthropic
Purpose
AI search & live answers
robots.txt
Respects robots.txt
Official docs
Claude-SearchBot indexes pages to improve the quality of Claude's web-search answers and citations. Blocking it reduces your chances of being cited by Claude without affecting training data collection, which ClaudeBot handles separately.
Does Claude-SearchBot respect robots.txt?
Anthropic states Claude-SearchBot respects robots.txt directives.
Block Claude-SearchBot with robots.txt
User-agent: Claude-SearchBot Disallow: /
Explicitly allow Claude-SearchBot
User-agent: Claude-SearchBot Allow: /
Should you block Claude-SearchBot?
For most sites, no. Claude-SearchBot powers visibility: blocking it removes your pages from the cited answers Anthropic's assistant shows its users — the AI-era equivalent of de-indexing yourself from a search engine. Block it only if you deliberately don't want that audience. Our guide on blocking AI bots covers the trade-offs in detail.
Check and monitor Claude-SearchBot on your site
- AI Crawler Access Checker — see whether your current robots.txt allows or blocks Claude-SearchBot.
- robots.txt Generator for AI Bots — build a robots.txt with per-bot rules for all 15 known AI crawlers.
- AI Bot Log Analyzer — paste server logs and count real Claude-SearchBot hits, entirely in your browser.
Frequently asked questions
What is Claude-SearchBot?
Claude-SearchBot is Anthropic's agent for AI search and live answers. Indexes content to inform Claude's search-style answers.
Does Claude-SearchBot respect robots.txt?
Anthropic states Claude-SearchBot respects robots.txt directives.
How do I block Claude-SearchBot?
Add "User-agent: Claude-SearchBot" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.
Does blocking Claude-SearchBot hurt my Google rankings?
No. Claude-SearchBot is separate from Googlebot, which handles Google Search indexing. Blocking Claude-SearchBot has no effect on your traditional search rankings, but it does remove your pages from the AI answers Anthropic's assistant serves to its users.
How can I tell if Claude-SearchBot is crawling my site?
Search your server access logs for the string "Claude-SearchBot". Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including Claude-SearchBot, with per-path breakdowns.
More Anthropic agents
Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-07-15.