ClaudeBot: what this bot is and how to allow or block it
ClaudeBot is a bot from Anthropic that collects pages to train AI models. It does not affect Claude answers with search: you can block it without losing visibility in AI search.
ClaudeBot User-Agent
This is how ClaudeBot identifies itself in requests to your site. In robots.txt you only write the token — ClaudeBot.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])
How to allow ClaudeBot
Add this group to robots.txt at the site root. Close service sections in the same group: a bot with its own group does not read the «User-agent: *» rules.
User-agent: ClaudeBot Allow: /
robots.txt is not everything. If your site uses Cloudflare or other bot protection, it may serve ClaudeBot a 403 error or a challenge page even when robots.txt allows everything.
How to block ClaudeBot
To block the whole site, add the group below to robots.txt. The rule works for bots that respect robots.txt.
User-agent: ClaudeBot Disallow: /
What happens if you block ClaudeBot
New model versions will not learn from your pages. Visibility in answers with search does not change — search bots are responsible for that.
How to check whether ClaudeBot visits your site
Search the web server logs for the token. The User-Agent string is easy to fake, so to be sure, check the IP address against the ranges if Anthropic publishes them.
grep "ClaudeBot" /var/log/nginx/access.log
Does your site let ClaudeBot in
Free 15-second check: we read your robots.txt and request your site as AI bots — you see which ones the server and bot protection let in and which they block.
Other Claude bots
Claude-SearchBotsearchClaude-Useron requestanthropic-aitraining
Questions and answers
Should I block ClaudeBot?
It is your call. Blocking ClaudeBot does not remove your site from AI answers with search, but models will not learn about you in training either. If content is your main asset, block it; if awareness matters, leave it open.
How do I know whether ClaudeBot visits my site?
Search the web server logs for the token. The User-Agent string is easy to fake, so to be sure, check the IP address against the ranges if Anthropic publishes them.
Is it enough to allow ClaudeBot in robots.txt?
robots.txt is not everything. If your site uses Cloudflare or other bot protection, it may serve ClaudeBot a 403 error or a challenge page even when robots.txt allows everything.