AI crawler profile
What is Bytespider? How to allow or block ByteDance's crawler
Operator
ByteDance
Purpose
AI model training
robots.txt
Ignores robots.txt
Official docs
None published
Bytespider is ByteDance's aggressive web crawler, believed to gather training data for its AI products including the Doubao models. It has drawn criticism for crawling at very high volume and for continuing to fetch pages on sites that have disallowed it.
Does Bytespider respect robots.txt?
ByteDance publishes no crawler documentation, and multiple independent reports have found Bytespider ignoring robots.txt; blocking it reliably requires firewall or CDN rules.
Block Bytespider with robots.txt
User-agent: Bytespider Disallow: /
⚠ Because Bytespider's robots.txt compliance is unreliable, pair this with a firewall or CDN rule matching the user-agent string if blocking actually matters to you.
Explicitly allow Bytespider
User-agent: Bytespider Allow: /
Should you block Bytespider?
Blocking Bytespider opts your content out of ByteDance's model training — a legitimate choice for original content you don't want reproduced by AI. The trade-off: models trained without your content are less likely to know your brand or recommend it unprompted. There is no direct traffic loss today, since training crawlers don't send visitors. See our guide on whether to block AI bots for the full decision framework.
Check and monitor Bytespider on your site
- AI Crawler Access Checker — see whether your current robots.txt allows or blocks Bytespider.
- robots.txt Generator for AI Bots — build a robots.txt with per-bot rules for all 15 known AI crawlers.
- AI Bot Log Analyzer — paste server logs and count real Bytespider hits, entirely in your browser.
Frequently asked questions
What is Bytespider?
Bytespider is ByteDance's web crawler for collecting AI training data. ByteDance's crawler, used for training its AI models.
Does Bytespider respect robots.txt?
ByteDance publishes no crawler documentation, and multiple independent reports have found Bytespider ignoring robots.txt; blocking it reliably requires firewall or CDN rules.
How do I block Bytespider?
Add "User-agent: Bytespider" followed by "Disallow: /" to your robots.txt file. Because this bot's robots.txt compliance is not reliable, enforce the block with firewall or CDN rules (for example a Cloudflare WAF rule matching the user agent) if it matters to you.
Does blocking Bytespider hurt my Google rankings?
No. Bytespider is separate from Googlebot, which handles Google Search indexing. Blocking Bytespider has no effect on your traditional search rankings.
How can I tell if Bytespider is crawling my site?
Search your server access logs for the string "Bytespider". Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including Bytespider, with per-path breakdowns.
Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-07-15.