Published 06.11.2026
The agency Blend tested 200 stores, and 29 refused its AI agent. All 24 with a readable robots.txt allow AI search crawlers. The authors blame bot protection, but the agent was unknown, not ChatGPT's or Claude's bot.
What they found
The agency Blend Commerce tested 200 stores in the US and UK across 20 sectors, from September 8 to 15, 2026. The agent used a plain Chromium browser and, according to Blend, did not pose as a bot from any AI company.
- 29 of 200 stores (14.5%) refused the agent on both attempts.
- Large stores refused most often: 16 of 80. Mid-sized stores: 12 of 80. Emerging stores: 1 of 40.
- robots.txt was readable at 24 of the 29. All 24 allow the search crawlers from ChatGPT, Perplexity and Claude.
Blend's conclusion: the refusals come from bot protection, not from robots.txt rules.
What this doesn't prove
- The agent was unknown. A store can admit named agents and crawlers on its allow list and block everything else. The test doesn't show how these stores respond to the real ChatGPT or Claude bots.
- The study has a commercial interest. Blend sells services to online stores.
- The cause is the authors' conclusion. None of the stores that refused the agent responded. For 5 of the 29, the robots.txt rules are unknown.
Why an open robots.txt guarantees nothing
robots.txt only tells a bot the rules: the bot reads the file and decides whether to follow them. Bot protection answers the request at the server or CDN and can refuse it whatever the file says. So the file can't tell you whether a bot really gets in.
We saw the same thing in our check of 100 well-known services: 14 gave us no pages at all, although 12 of them allow AI bots in robots.txt. The same caveat applies: our requests don't come from OpenAI or Anthropic addresses.
What to check on your site
- Protection or CDN logs. Look for OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. See whether they get a 200 or a 403.
- Protection settings. In Cloudflare, that means AI bot blocking, Bot Fight Mode and WAF rules. Other services have similar settings.
- Search and training bots separately. Search bots matter for AI answers with links. You can block training bots on their own.
The free aiseen.cloud check tests your robots.txt and how your server responds to AI bots: 15 seconds, no sign-up.
Sources: Blend Commerce report, PPC Land.