Blocking or allowing AI crawlers used to mean digging up each company's user-agent string and hand-writing User-agent / Disallow blocks from scratch. This tool does that for you: toggle each bot individually, or apply a preset, and get a ready-to-deploy robots.txt.
Why it matters for AI search: every major AI product — ChatGPT, Claude, Perplexity, Gemini — either trains on crawled web content, answers questions by browsing live, or both, and each behavior is controlled by a separate crawler with its own user-agent. Getting this wrong in either direction has a cost: leaving training bots unblocked when you didn't intend to allow them, or accidentally blocking a search/browsing bot and disappearing from AI-generated answers entirely.
How this tool works: start from a preset — Allow all, Block training bots only, or Block all — or fine-tune each bot with an individual toggle. The tool covers the major AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider, Applebot-Extended, Amazonbot, Meta-ExternalAgent, cohere-ai), labeled by operator and purpose (training vs. search) so you know exactly what each toggle controls. You can also add a standard User-agent: * block with your own disallowed paths, and include a Sitemap: line. The output updates live and can be copied or downloaded as robots.txt.
Limitations: this tool generates rules for AI bots plus one general-purpose block — it isn't a full robots.txt editor for complex multi-crawler setups with crawl-delay directives or dozens of path-specific rules for non-AI crawlers (like Googlebot-Image or Bingbot). If you already have a robots.txt with custom logic, treat this tool's output as a block to merge in rather than a full replacement. To check what your current, live robots.txt actually allows before making changes, use the companion AI Crawler Access Checker.