Published 12.11.2026
theStacc tested 88,640 local business websites. At 3% of them, robots.txt allows AI search bots, but the server refuses them. The owners most likely don't know.
How they tested
The sites came from Overture Maps, an open places dataset, across 30 industries. Each homepage was requested as a browser, as Googlebot and as eight bots from OpenAI, Anthropic and Perplexity. Bots disallowed in robots.txt were not requested. Every refusal was rechecked at least 24 hours later, and only confirmed ones counted. The crawl ran from October 31 to November 2, 2026.
What they found
- 2,523 of 83,504 sites (3%, about 1 in 33) refused an AI search bot that their robots.txt allows. A browser and Googlebot still got the page.
- The most refused were Claude-SearchBot (1.8% of sites), OAI-SearchBot (1.4%) and PerplexityBot (1.1%).
- 83% of these refusals look identical across many sites. The authors read this as shared hosting, platform or CDN rules rather than owners' choices.
- On one platform for health practices, 949 sites refused GPTBot and Claude's bots although their robots.txt allows them. The refusals came from the CDN in front of the sites.
- On Wix, 79% of sites served a challenge page to AI search bots. But 68.5% of those did the same to Googlebot: that's general bot protection, not an AI rule.
Caveats
- Requests came from the authors' servers, not from OpenAI or Anthropic addresses. Protection that verifies IPs may let the real bots in.
- Only homepages were tested, in a single crawl.
Why this happens
robots.txt is a set of rules for the bot: it reads the file and decides whether to follow it. But the response comes from the server, the CDN or the host's protection. If your host blocks AI bots for all customers, an open robots.txt won't change that. We saw the same in our check of 100 well-known services.
What to do
- Request your homepage as a bot. For example,
curl -I -A "OAI-SearchBot" https://your-site. You want a 200, not a 403 or a challenge page. - Check your protection settings. In Cloudflare, look at AI bot blocking, Bot Fight Mode and WAF rules. On a site builder or host, ask support whether they have a shared rule for AI bots.
- Ask for an exception. If the platform is refusing the bots, ask it to exempt AI search bots.
A free aiseen.cloud check does this for you: it reads robots.txt and requests your homepage as AI bots. 15 seconds, no sign-up.
Source: theStacc study.