What is PerplexityBot?
PerplexityBot is the web crawler operated by Perplexity AI, an AI-powered search engine that provides direct answers to questions by searching and synthesizing information from the web in real-time.
Unlike GPTBot or CCBot, which crawl for AI model training, PerplexityBot crawls to power user search queries — making it more similar to Googlebot than to a training data crawler. When a user asks Perplexity a question, PerplexityBot fetches and reads web pages to construct an answer.
How Perplexity AI Works
Perplexity AI is a conversational search engine:
- User asks a question
- Perplexity identifies relevant web pages
- PerplexityBot fetches and reads those pages
- Perplexity synthesizes an answer with citations
- User gets a summarized answer with source links
This means allowing PerplexityBot can drive referral traffic to your site — users who see your site cited in Perplexity answers may click through.
PerplexityBot vs Traditional Search Crawlers
| Aspect | PerplexityBot | Googlebot |
|---|---|---|
| Purpose | AI search answers | Search index |
| Traffic value | Medium (citations) | High (organic search) |
| Crawl trigger | User queries + indexing | Continuous discovery |
| Content usage | Summarized in answers | Snippet + ranking |
| Attribution | Yes, with links | Yes, via rankings |
User Agent
Primary:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://docs.perplexity.ai/docs/perplexitybot)
Alternative form seen in logs:
PerplexityBot
Perplexity’s Crawler Ecosystem
Perplexity operates multiple bots:
| Bot | Purpose |
|---|---|
| PerplexityBot | Web indexing and crawling |
| Perplexity-User | Real-time fetching for user queries |
This guide focuses on PerplexityBot — the indexing crawler.
Should You Allow or Block PerplexityBot?
PerplexityBot is an AI search bot, not a training data collector. This distinction matters significantly:
Allow PerplexityBot if:
- You want your content to appear in Perplexity AI answers
- You want citation-based referral traffic from Perplexity
- You’re a content creator, publisher, or blogger seeking visibility
- You want to reach Perplexity’s growing user base (tens of millions)
- You run a business that benefits from AI-powered search discovery
Block PerplexityBot if:
- You object to your content being summarized without full page visits
- You run paywalled content and Perplexity is bypassing your paywall
- You’re concerned about content reproduction without adequate attribution
- You prefer users to visit your full page rather than read a summary
Unlike GPTBot or CCBot, blocking PerplexityBot means losing visibility in Perplexity’s search results — similar to blocking Googlebot would hurt Google rankings. Think carefully before blocking.
The Summarization Controversy
Perplexity has faced criticism from publishers:
- Content is summarized and users may not need to visit the original
- This can reduce page views for publishers
- Some media companies (Forbes, News Corp) have requested blocking or licensing deals
- Perplexity has announced a Publisher Program offering revenue sharing
This creates a nuanced decision: you lose AI search visibility by blocking, but you may lose page views by allowing.
How to Block PerplexityBot
If you choose to block:
User-agent: PerplexityBot
Disallow: /
Block specific content types:
User-agent: PerplexityBot
Disallow: /premium/
Disallow: /members/
Disallow: /paywalled/
Allow: /
Server-Level Blocking
Nginx:
if ($http_user_agent ~* "PerplexityBot") {
return 403;
}
Apache:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} PerplexityBot [NC]
RewriteRule .* - [F,L]
Does PerplexityBot Respect robots.txt?
Generally yes. Perplexity has publicly committed to honoring robots.txt directives. However, Perplexity faced controversy in 2024 when reports surfaced that it was sometimes ignoring robots.txt for real-time user queries.
For the indexing crawler (PerplexityBot), compliance is generally reported as good.
Verifying PerplexityBot
To confirm a request is from Perplexity:
host [IP address]
# Should resolve to Perplexity AI infrastructure
Perplexity documents their bot and IP ranges at their official documentation.
PerplexityBot Traffic Volume
Perplexity AI has grown rapidly since 2023:
- Handles hundreds of millions of queries per month
- Growing user base driven by ChatGPT competition
- Significant crawl volume as it expands its index
Sites in tech, science, finance, and knowledge domains tend to see more PerplexityBot traffic.
Impact on Referral Traffic
Data from early adopters shows:
- Sites cited in Perplexity answers can receive meaningful referral traffic
- Traffic quality is high (users are actively researching topics)
- Click-through rates vary by content type and answer format
Publishers who have embraced Perplexity report it as an emerging traffic source, though smaller than Google at this stage.
Comparison with Other AI Search Bots
| Bot | Company | Search Product | Traffic Potential |
|---|---|---|---|
| PerplexityBot | Perplexity AI | Perplexity.ai | Growing (tens of millions) |
| OAI-SearchBot | OpenAI | SearchGPT / ChatGPT Search | Very high potential |
| Googlebot | Google Search | Dominant | |
| Bingbot | Microsoft | Bing + Copilot | Significant |
Test PerplexityBot Access to Your Site
Use our AI Bot Checker to verify if PerplexityBot can access your website and whether your content could appear in Perplexity AI answers.
Related AI Training Bots (different purpose — collects data, not search):
- GPTBot - OpenAI’s AI training crawler
- ClaudeBot - Anthropic’s AI training crawler
- CCBot - Common Crawl data collector
- Bytespider - ByteDance aggressive AI crawler
Search Engine Bots:
- Googlebot - Google’s primary search crawler
- Bingbot - Microsoft Bing crawler
- Applebot - Apple’s crawler for Siri and Spotlight
For comprehensive bot testing, explore our free bot detection tools.