Check your server access logs (or CDN analytics) for AI crawler user agents like GPTBot, ClaudeBot, and PerplexityBot. Our free log analyzer does this in your browser: paste log lines, get hit counts and top paths per bot. If you see nothing, verify your robots.txt isn't blocking them first.
Method 1: Analyze your access logs (most detail)
Every request to your server — bot or human — lands in your access log with a user-agent string. AI crawlers identify themselves honestly in that string, so counting them is a text-matching exercise.
Getting your logs:
- nginx:
/var/log/nginx/access.log - Apache:
/var/log/apache2/access.log(or cPanel → Raw Access Logs on shared hosting) - Vercel / Netlify / Cloudflare: log export or drains on paid tiers
Analyzing them: paste lines into our AI Bot Log Analyzer — it matches against the full current AI-bot registry and reports hits per bot, each bot's purpose (training vs. search), and the top paths each bot fetched, entirely in your browser (logs contain visitor IPs; they never leave your machine).
For very large logs, pre-filter on the command line first:
grep -iE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|claude-searchbot|perplexity|google-extended|ccbot|bytespider|applebot-extended|amazonbot|meta-externalagent|cohere-ai" access.log > ai-hits.log
Method 2: CDN / hosting analytics (least effort)
If you're behind Cloudflare, its dashboard (Security → Bots, or Analytics with a user-agent filter) shows crawler traffic without touching raw logs — and Cloudflare Radar publishes industry-wide AI crawler trends for context. Most managed hosts expose some user-agent breakdown in their analytics; look for a "bots" or "crawlers" section.
Method 3: Verify the door is open
Zero AI-bot traffic sometimes just means the bots can't get in. Run your domain through the AI Crawler Access Checker to confirm robots.txt isn't blocking them, and check that your bot-protection layer (Cloudflare bot fight mode, etc.) isn't challenging legitimate AI crawlers with CAPTCHAs — an allowed bot that can't pass a challenge is blocked in practice.
Reading the results
What the traffic mix tells you:
- Search bots present (OAI-SearchBot, PerplexityBot, Claude-SearchBot): your content is being indexed for AI answers — the visibility signal you want. The paths they hit are the pages those platforms consider worth indexing.
- Only training bots (GPTBot, CCBot, Bytespider, Meta-ExternalAgent): your content feeds model training but isn't in AI search indexes yet. If visibility is the goal, work through how to get cited by ChatGPT.
- Live-browse bots (ChatGPT-User, Claude-User, Perplexity-User): actual users are asking assistants about your pages in real time — a small but high-intent signal worth noticing.
- Nothing at all: either access is blocked (check Method 3), your site is too new/low-authority to have been discovered, or your log sample is too short — AI crawler visits are bursty, so look at a week or more before concluding.
Spoofing caveat
User agents are self-declared, and some scrapers impersonate AI bots to borrow their reputation. For casual monitoring this rarely matters, but if you're making decisions based on the numbers (or being hammered by a claimed "GPTBot"), verify against the publishers' documented IP ranges — OpenAI, Anthropic, and Perplexity all publish the IP blocks their crawlers use.
Make it a habit
AI crawler activity is your GEO ground truth, and it changes as platforms launch new bots and re-crawl schedules shift. A monthly log check — same cadence as the bot-list refresh we apply to our own tools — is enough to catch trends: which bots are increasing, which content they're prioritizing, and whether your robots.txt changes actually took effect.