PerplexityBot: Perplexity's search crawler and the robots.txt dispute

PerplexityBot is the crawler Perplexity AI uses to build the index behind its answer engine. Perplexity's crawler documentation says it exists "to surface and link websites in search results on Perplexity" and is not used to collect content for training foundation models. If you want your pages to show up as cited sources in Perplexity answers, this is the agent you need to let in. If you want out, a robots.txt rule addressed to it is the documented way to do that, with caveats covered below.

The user-agent string

Perplexity documents one string for this crawler:

Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Perplexity notes that the version number may change. The robots.txt token is PerplexityBot. In log searches and firewall rules, match on the PerplexityBot/ substring rather than the full string. Perplexity's strings have not always appeared in exactly the documented form: Cloudflare's 2025 report quotes its sibling agent, Perplexity-User, with an extra AppleWebKit/537.36 (KHTML, like Gecko; ...) segment that the docs page does not show. An exact-match rule would miss variants like that.

Two Perplexity agents, two different rules

Perplexity runs two declared agents, and most confusion about "blocking Perplexity" comes from treating them as one.

	PerplexityBot	Perplexity-User
Job	Crawls ahead of time to build the search index	Fetches a page live when a user's question needs it
robots.txt	Documented as the way to opt in or out of search results	"generally ignores robots.txt rules" per Perplexity's docs
IP list	perplexitybot.json	perplexity-user.json

So a Disallow for PerplexityBot removes you from the pre-built index, but a page can still be read and cited when someone asks Perplexity about it directly. The PerplexityBot vs Perplexity-User comparison goes through that split in detail, and the Perplexity-User profile covers the live fetcher on its own.

Perplexity also notes that each robots.txt setting is evaluated independently and that changes "may take up to 24 hours" to be reflected. Give a new rule a day before judging it from your logs.

Verifying a request is really PerplexityBot

Perplexity's guidance is to match the user-agent and the source IP together, and to treat its JSON endpoints as the source of truth. The PerplexityBot file lives at perplexity.ai/perplexitybot.json. It has a creationTime timestamp and a prefixes array of ipv4Prefix entries. When we fetched it for this page, it held 13 single-address /32 prefixes, all IPv4, generated on 28 October 2026. That is a short list, so a strict allowlist is practical, but it also means a stale copy goes wrong quickly. Perplexity recommends automating periodic refreshes of WAF rules from the endpoint rather than pasting addresses in once.

Perplexity does not document a reverse-DNS hostname check for its crawlers, so the IP list is the only verification method it offers.

One Cloudflare-specific note from the same docs: Bot Fight Mode cannot be bypassed with custom WAF rules, so it may still challenge PerplexityBot even if you have written an allow rule for its IPs. If you want Perplexity traffic and you run Bot Fight Mode, check the firewall events after making the change.

Why the robots.txt record is disputed

Perplexity states that PerplexityBot respects robots.txt. Two independent investigations reported traffic linked to Perplexity reaching sites that had blocked its declared agents.

June 2024, Wired and Robb Knight. Developer Robb Knight, who had blocked PerplexityBot on his own site and on MacStories, found Perplexity still reaching MacStories with requests that did not carry its published user-agent. Wired then set up a new site, asked Perplexity to summarize it, and its server logged a visit from an IP address that was not on Perplexity's published list. As MacStories summarized, Wired reported that this address hit Condé Nast properties at least 822 times in three months.

August 2025, Cloudflare. Cloudflare published tests using newly registered domains that were not publicly linked and that disallowed all crawling. Perplexity still answered detailed questions about their content. Cloudflare said that when the declared agents were blocked, it saw a generic Chrome-on-macOS user-agent fetching from IPs outside Perplexity's published ranges, at 3 to 6 million requests a day, compared with 20 to 25 million a day from the declared agent. Cloudflare removed Perplexity from its verified bots list and added signatures for the undeclared traffic to its managed AI-crawler rules, including on free plans.

Perplexity's answer. In its response, Perplexity said Cloudflare had misattributed the 3 to 6 million daily requests, which it said came from BrowserBase, a third-party cloud browser service that Perplexity uses occasionally for fewer than 45,000 requests a day. It also argued that user-requested fetches are a different thing from crawling and should not be treated like bots.

Note what the dispute is about. Both reports concern traffic that did not identify itself as PerplexityBot. Neither showed the declared PerplexityBot agent ignoring a disallow. In practice that means a robots.txt rule reliably controls the declared crawler and nothing more.

Picking a setup

You want Perplexity citations. Allow the crawler explicitly. This matters when you also have a broad AI block, because a wildcard Disallow catches PerplexityBot too:

User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /private/

If you sit behind Cloudflare, check whether its AI-crawler blocking or Bot Fight Mode is turned on, since either can stop PerplexityBot regardless of your robots.txt. Our guide to optimizing for Perplexity covers what gets pages cited once the crawler can reach them.

You want out of Perplexity. Disallow both declared agents. The Perplexity-User line costs nothing even though it is documented as generally ignored:

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

Then enforce it at the edge using both JSON IP lists, because robots.txt is a request, not a block. Going by the reports above, undeclared traffic will not match either list, so the only defense against it is bot detection at your CDN. The AI crawler access checker shows how your current robots.txt treats PerplexityBot, and the AI bot log analyzer helps confirm what is actually reaching your server.

User-agent: PerplexityBot
Disallow: /

User-agent: PerplexityBot
Allow: /

if ($http_user_agent ~* "PerplexityBot") {
    return 403;
}

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} PerplexityBot [NC]
RewriteRule .* - [F,L]

(http.user_agent contains "PerplexityBot")

grep -i "PerplexityBot" /var/log/nginx/access.log | wc -l