AI Bot Blocker & Checker
Check whether AI crawlers like GPTBot, ClaudeBot, PerplexityBot, and Common Crawl can access your website — and control how your content is used.
Explore our complete suite of Bot Detection Tools to test search engines, social media crawlers, and SEO analytics bots.
Why Block or Monitor AI Bots?
🛡 Protect Your Content
AI companies scrape public websites to train models. Blocking prevents unconsented reuse of original text, research, or media.
🧪 Verify Your robots.txt Configuration
Most reputable AI bots follow restrictions — but many do not, so testing access is essential.
⚖️ Control Instead of Fully Blocking
Some crawlers (e.g., Perplexity or AI-search agents) may provide value, while pure dataset scrapers may not. Choose which ones to allow.
For comprehensive guidance, read our robots.txt configuration guide and Understanding Bot Traffic.
AI Bots This Tool Detects (28 Crawlers)
AI Training Bots (typically blocked):
- GPTBot — OpenAI model training crawler
- ClaudeBot — Anthropic dataset and research crawler
- anthropic-ai — Additional Anthropic training crawler
- Google-Extended — Google AI/ML training data
- Google-CloudVertexBot — Google Cloud AI training
- Applebot-Extended — Apple AI data crawler
- CCBot — Common Crawl archive bot (used by many AI companies)
- Meta-ExternalAgent — Meta / Facebook AI ingest bot
- cohere-ai — Cohere LLM data indexer
- Bytespider — TikTok/ByteDance AI training
- Omgili/Webzio — Data collection for AI datasets
- Diffbot — Knowledge graph and AI training
- AI2Bot — Allen Institute for AI research
- PanguBot — Huawei AI training crawler
AI Search Bots (typically allowed):
- ChatGPT-User — ChatGPT Search
- OAI-SearchBot — OpenAI search features
- Claude-Web — Claude AI search
- Claude-SearchBot — Claude search integration
- PerplexityBot — Perplexity AI search
- Gemini-Deep-Research — Google Gemini research
- DuckAssistBot — DuckDuckGo AI Chat
- Amazonbot — Amazon Alexa AI features
- YouBot — You.com AI search
- MistralAI-User — Mistral AI search
Note: AI search bots provide value by surfacing your content in AI-powered search results. Training bots scrape data to build commercial AI models.
How It Works
- Enter your domain
- We fetch and analyse your
robots.txt - Tool simulates real AI bot requests
- Results compare access vs allowed rules
- You get recommended allow/deny rules
Results Include:
- ✓ Which bots can and cannot access your website
- ✓ Whether robots.txt rules are respected
- ✓ Risk level for model training exposure
- ✓ Copy-paste blocking instructions
Who Uses This
- Publishers & Researchers protecting original work
- E-commerce blocking product and pricing scrapers
- Communities & Forums keeping discussions private
- Legal & Compliance enforcing policy boundaries
How to Block AI Bots (Example)
User-agent: GPTBot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
⚠️ Some AI crawlers may ignore robots.txt. Most major AI companies comply, but results may vary.
Start Protecting Your Content Now
Enter your domain above to instantly check which AI bots can access your website. Test 28 AI crawlers including GPTBot, ClaudeBot, and Perplexity — get blocking recommendations in seconds. Free tool, no signup required.