AI bot reference

anthropic-ai: what this bot is and how to allow or block it

anthropic-ai is a bot from Anthropic that collects pages to train AI models. It does not affect Claude answers with search: you can block it without losing visibility in AI search.

training Claude · Anthropic

anthropic-ai User-Agent

Anthropic publishes the full User-Agent string in the bot's documentation. In robots.txt the token anthropic-ai is enough.

How to allow anthropic-ai

Add this group to robots.txt at the site root. Close service sections in the same group: a bot with its own group does not read the «User-agent: *» rules.

User-agent: anthropic-ai
Allow: /

robots.txt is not everything. If your site uses Cloudflare or other bot protection, it may serve anthropic-ai a 403 error or a challenge page even when robots.txt allows everything.

How to block anthropic-ai

To block the whole site, add the group below to robots.txt. The rule works for bots that respect robots.txt.

User-agent: anthropic-ai
Disallow: /

What happens if you block anthropic-ai

New model versions will not learn from your pages. Visibility in answers with search does not change — search bots are responsible for that.

How to check whether anthropic-ai visits your site

Search the web server logs for the token. The User-Agent string is easy to fake, so to be sure, check the IP address against the ranges if Anthropic publishes them.

grep "anthropic-ai" /var/log/nginx/access.log

Does your site let anthropic-ai in

Free 15-second check: we read your robots.txt and request your site as AI bots — you see which ones the server and bot protection let in and which they block.

Check a site

Other Claude bots

All AI bots and robots.txt rules →

Questions and answers

Should I block anthropic-ai?

It is your call. Blocking anthropic-ai does not remove your site from AI answers with search, but models will not learn about you in training either. If content is your main asset, block it; if awareness matters, leave it open.

How do I know whether anthropic-ai visits my site?

Search the web server logs for the token. The User-Agent string is easy to fake, so to be sure, check the IP address against the ranges if Anthropic publishes them.

Is it enough to allow anthropic-ai in robots.txt?

robots.txt is not everything. If your site uses Cloudflare or other bot protection, it may serve anthropic-ai a 403 error or a challenge page even when robots.txt allows everything.