Applebot-Extended: what this bot is and how to allow or block it
Applebot-Extended is not a separate crawler but a robots.txt token: with it a site allows or forbids Apple to use pages it has already crawled for training AI models. It does not affect search rankings.
Applebot-Extended User-Agent
Apple publishes the full User-Agent string in the bot's documentation. In robots.txt the token Applebot-Extended is enough.
How to allow Applebot-Extended
Add this group to robots.txt at the site root. Close service sections in the same group: a bot with its own group does not read the «User-agent: *» rules.
User-agent: Applebot-Extended Allow: /
robots.txt is not everything. If your site uses Cloudflare or other bot protection, it may serve Applebot-Extended a 403 error or a challenge page even when robots.txt allows everything.
How to block Applebot-Extended
To block the whole site, add the group below to robots.txt. The rule works for bots that respect robots.txt.
User-agent: Applebot-Extended Disallow: /
What happens if you block Applebot-Extended
New model versions will not learn from your pages. Visibility in answers with search does not change — search bots are responsible for that.
How to check whether Applebot-Extended visits your site
Search the web server logs for the token. The User-Agent string is easy to fake, so to be sure, check the IP address against the ranges if Apple publishes them.
grep "Applebot-Extended" /var/log/nginx/access.log
Does your site let Applebot-Extended in
Free 15-second check: we read your robots.txt and request your site as AI bots — you see which ones the server and bot protection let in and which they block.
Questions and answers
Should I block Applebot-Extended?
It is your call. Blocking Applebot-Extended does not remove your site from AI answers with search, but models will not learn about you in training either. If content is your main asset, block it; if awareness matters, leave it open.
How do I know whether Applebot-Extended visits my site?
Search the web server logs for the token. The User-Agent string is easy to fake, so to be sure, check the IP address against the ranges if Apple publishes them.
Is it enough to allow Applebot-Extended in robots.txt?
robots.txt is not everything. If your site uses Cloudflare or other bot protection, it may serve Applebot-Extended a 403 error or a challenge page even when robots.txt allows everything.