meta-externalfetcher: what this bot is and how to allow or block it
meta-externalfetcher is a bot from Meta that opens a page when a person asks: when someone using Meta AI asks it to read a link or the assistant looks up an answer right in the chat. It does not crawl the whole site — it reads one page at a time.
meta-externalfetcher User-Agent
This is how meta-externalfetcher identifies itself in requests to your site. In robots.txt you only write the token — meta-externalfetcher.
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
How to allow meta-externalfetcher
Add this group to robots.txt at the site root. Close service sections in the same group: a bot with its own group does not read the «User-agent: *» rules.
User-agent: meta-externalfetcher Allow: /
robots.txt is not everything. If your site uses Cloudflare or other bot protection, it may serve meta-externalfetcher a 403 error or a challenge page even when robots.txt allows everything.
How to block meta-externalfetcher
To block the whole site, add the group below to robots.txt. The rule works for bots that respect robots.txt.
User-agent: meta-externalfetcher Disallow: /
What happens if you block meta-externalfetcher
When someone using Meta AI asks it to open your page, the assistant will not be able to read it and will answer without your site.
How to check whether meta-externalfetcher visits your site
Search the web server logs for the token. The User-Agent string is easy to fake, so to be sure, check the IP address against the ranges if Meta publishes them.
grep "meta-externalfetcher" /var/log/nginx/access.log
Does your site let meta-externalfetcher in
Free 15-second check: we read your robots.txt and request your site as AI bots — you see which ones the server and bot protection let in and which they block.
Questions and answers
Should I block meta-externalfetcher?
Usually not: meta-externalfetcher opens a page only when a person asks. Blocking it stops people using Meta AI from reading your site through it.
How do I know whether meta-externalfetcher visits my site?
Search the web server logs for the token. The User-Agent string is easy to fake, so to be sure, check the IP address against the ranges if Meta publishes them.
Is it enough to allow meta-externalfetcher in robots.txt?
robots.txt is not everything. If your site uses Cloudflare or other bot protection, it may serve meta-externalfetcher a 403 error or a challenge page even when robots.txt allows everything.