When webmasters started logging aggressive, unfamiliar bot traffic in 2023, many traced it back to a single user agent: Bytespider. The crawler belongs to ByteDance — the Chinese company behind TikTok, Douyin, and CapCut — and it quickly earned a reputation as the most aggressive and controversial AI crawler on the web.

Here’s why Bytespider stands apart from other AI bots, and what you can do about it.

What Is Bytespider?

Bytespider is ByteDance’s official web crawler, used to collect training data for the company’s AI models. ByteDance has been investing heavily in AI to compete with OpenAI, Google, and Anthropic — and like those companies, it needs massive amounts of web data to train its models.

Unlike most AI crawlers, Bytespider has attracted controversy not just for what it collects, but for how aggressively it collects it.

The Three Problems with Bytespider

1. Unusually High Crawl Volume

Site owners and server admins have consistently reported Bytespider crawling at volumes that exceed other AI bots:

  • Multiple requests per second from different IP addresses
  • Crawling the same pages repeatedly in short intervals
  • Continuing to crawl during high server load without backing off
  • Bandwidth consumption reported in the hundreds of gigabytes monthly on large sites

For comparison, most AI training crawlers (GPTBot, ClaudeBot) are relatively polite — they crawl at moderate rates and back off when servers respond slowly. Bytespider does not reliably do this.

2. Inconsistent robots.txt Compliance

This is the most serious technical concern. Standard robots.txt practice is the internet’s social contract for bots — crawlers are expected to read and honor the rules in your robots.txt file.

Multiple webmasters have reported that Bytespider continued crawling their sites after adding:

User-agent: Bytespider
Disallow: /

ByteDance officially states that Bytespider respects robots.txt, but the real-world evidence is mixed. This inconsistency may be due to:

  • Multiple Bytespider variants that don’t all honor the same rules
  • Delays between adding robots.txt rules and Bytespider updating its crawl schedule
  • Different behavior across Bytespider versions

The practical implication: Don’t rely solely on robots.txt to block Bytespider. Use server-level or CDN-level blocking.

3. Geopolitical and Data Jurisdiction Concerns

ByteDance is a Chinese company, which means:

  • Data collected by Bytespider is processed by a company subject to Chinese law
  • Under China’s National Intelligence Law, Chinese companies can be required to cooperate with intelligence agencies
  • ByteDance’s relationship with TikTok data has already been the subject of US congressional hearings
  • Multiple countries (including the US, UK, and EU members) have restricted ByteDance products on government devices

For most websites, this is a background concern. For government sites, healthcare, legal, or any organization with data sensitivity policies, it may be a blocking requirement.

How Bytespider Compares to Other AI Crawlers

Feature Bytespider GPTBot ClaudeBot CCBot
Crawl volume Very High Moderate Moderate Moderate
Bandwidth impact Very High Low-Medium Low-Medium Low-Medium
robots.txt compliance Inconsistent Good Good Good
Backs off under load Inconsistent Yes Yes Yes
Data destination ByteDance (China) OpenAI (US) Anthropic (US) Public dataset
Spoofing risk Higher Lower Lower Lower

Bytespider’s User Agent

The primary user agent:

Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)

Variants seen in logs:

Bytespider
bytespider

Note: Bytespider spoofing has been reported — bots impersonating ByteDance’s user agent. Always verify with reverse DNS when investigating suspicious traffic.

How to Block Bytespider Effectively

Given the robots.txt compliance issues, a layered blocking approach is recommended.

Layer 1: robots.txt (necessary but not sufficient)

User-agent: Bytespider
User-agent: bytespider
Disallow: /

Always add this — it handles cases where Bytespider does respect it, and signals your intent clearly.

Layer 2: Nginx server-level block

if ($http_user_agent ~* "Bytespider") {
    return 403;
}

Layer 3: Apache .htaccess

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
RewriteRule .* - [F,L]

Layer 4: Cloudflare WAF (most effective)

For high-traffic sites, Cloudflare stops the requests before they reach your server:

  1. Go to Security → WAF → Custom Rules
  2. Create rule: User Agent contains bytespider (case-insensitive)
  3. Action: Block

Cloudflare is especially valuable because it handles the blocking at the network edge, eliminating any server load from Bytespider entirely.

Verify the block is working

# Check if Bytespider is still in your logs after blocking
grep -i "bytespider" /var/log/nginx/access.log | tail -20

# Compare volume before and after
grep -i "bytespider" /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

Diagnosing Bytespider Traffic

If you’re seeing unusual bandwidth usage or server load, check whether Bytespider is involved:

# Total Bytespider requests in log
grep -i "bytespider" access.log | wc -l

# Requests per hour
grep -i "bytespider" access.log | awk '{print $4}' | cut -d: -f1-2 | sort | uniq -c | sort -rn | head -24

# Most crawled URLs
grep -i "bytespider" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

# Bandwidth consumed (approximate)
grep -i "bytespider" access.log | awk '{sum += $10} END {print sum/1024/1024 " MB"}'

Should Every Site Block Bytespider?

Not necessarily — but the bar for blocking Bytespider is lower than for other AI crawlers.

Block Bytespider if:

  • You’ve noticed unusual server load or bandwidth spikes
  • You have geopolitical or data policy concerns about ByteDance
  • You block other AI training crawlers (for consistency)
  • You run a media, news, or content site protecting IP
  • You’re a government, healthcare, or sensitive-sector organization

Consider allowing if:

  • You specifically want visibility in ByteDance’s AI products
  • You’re targeting Asian markets (ByteDance has strong regional presence)
  • Server capacity is not a concern

For most site owners, Bytespider is the one AI training crawler worth blocking at the server level rather than relying solely on robots.txt.

Check Your Bytespider Exposure

Use our AI Bot Checker to verify whether Bytespider can access your website and test your blocking configuration.

Related guides: