Claude-SearchBot: what this bot is and how to allow or block it
Claude-SearchBot is a search bot from Anthropic: it finds and reads pages so that Claude can find your site and cite it in answers. Block it and your site drops out of those answers.
Claude-SearchBot User-Agent
Anthropic publishes the full User-Agent string in the bot's documentation. In robots.txt the token Claude-SearchBot is enough.
How to allow Claude-SearchBot
Add this group to robots.txt at the site root. Close service sections in the same group: a bot with its own group does not read the «User-agent: *» rules.
User-agent: Claude-SearchBot Allow: /
robots.txt is not everything. If your site uses Cloudflare or other bot protection, it may serve Claude-SearchBot a 403 error or a challenge page even when robots.txt allows everything.
How to block Claude-SearchBot
To block the whole site, add the group below to robots.txt. The rule works for bots that respect robots.txt.
User-agent: Claude-SearchBot Disallow: /
What happens if you block Claude-SearchBot
Claude will stop finding and reading your pages: your site will not appear in answers with search, and the assistant will not link to it.
How to check whether Claude-SearchBot visits your site
Search the web server logs for the token. The User-Agent string is easy to fake, so to be sure, check the IP address against the ranges if Anthropic publishes them.
grep "Claude-SearchBot" /var/log/nginx/access.log
Does your site let Claude-SearchBot in
Free 15-second check: we read your robots.txt and request your site as AI bots — you see which ones the server and bot protection let in and which they block.
Other Claude bots
Claude-Useron requestClaudeBottraininganthropic-aitraining
Questions and answers
Should I block Claude-SearchBot?
No, if you want Claude to find and recommend your site. Blocking search bots only makes sense for pages that should not appear in answers.
How do I know whether Claude-SearchBot visits my site?
Search the web server logs for the token. The User-Agent string is easy to fake, so to be sure, check the IP address against the ranges if Anthropic publishes them.
Is it enough to allow Claude-SearchBot in robots.txt?
robots.txt is not everything. If your site uses Cloudflare or other bot protection, it may serve Claude-SearchBot a 403 error or a challenge page even when robots.txt allows everything.