meta-webindexer: what this bot is and how to allow or block it
meta-webindexer is a search bot from Meta: it finds and reads pages so that Meta AI can find your site and cite it in answers. Block it and your site drops out of those answers.
meta-webindexer User-Agent
This is how meta-webindexer identifies itself in requests to your site. In robots.txt you only write the token — meta-webindexer.
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
How to allow meta-webindexer
Add this group to robots.txt at the site root. Close service sections in the same group: a bot with its own group does not read the «User-agent: *» rules.
User-agent: meta-webindexer Allow: /
robots.txt is not everything. If your site uses Cloudflare or other bot protection, it may serve meta-webindexer a 403 error or a challenge page even when robots.txt allows everything.
How to block meta-webindexer
To block the whole site, add the group below to robots.txt. The rule works for bots that respect robots.txt.
User-agent: meta-webindexer Disallow: /
What happens if you block meta-webindexer
Meta AI will stop finding and reading your pages: your site will not appear in answers with search, and the assistant will not link to it.
How to check whether meta-webindexer visits your site
Search the web server logs for the token. The User-Agent string is easy to fake, so to be sure, check the IP address against the ranges if Meta publishes them.
grep "meta-webindexer" /var/log/nginx/access.log
Does your site let meta-webindexer in
Free 15-second check: we read your robots.txt and request your site as AI bots — you see which ones the server and bot protection let in and which they block.
Other Meta AI bots
meta-externalagenttrainingmeta-externalfetcheron request
Questions and answers
Should I block meta-webindexer?
No, if you want Meta AI to find and recommend your site. Blocking search bots only makes sense for pages that should not appear in answers.
How do I know whether meta-webindexer visits my site?
Search the web server logs for the token. The User-Agent string is easy to fake, so to be sure, check the IP address against the ranges if Meta publishes them.
Is it enough to allow meta-webindexer in robots.txt?
robots.txt is not everything. If your site uses Cloudflare or other bot protection, it may serve meta-webindexer a 403 error or a challenge page even when robots.txt allows everything.