Published 03.11.2026 · Data (CSV, CC BY 4.0)
We checked 100 well-known online services the way AI crawlers see them. 14 of them didn't serve a single page to our checker — openai.com among them. Yet 12 of those 14 allow AI bots in robots.txt: the real gatekeeper today is the firewall, not robots.txt.
What we checked
On 27 October 2026 we ran the free aiseen.cloud check on 100 well-known online services in 10 categories: CRM, website builders, email marketing, online learning, booking, hosting, project tools, payments, SEO and AI tools. Each check requests the site as a regular browser and as AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and others), reads robots.txt, llms.txt, structured data and up to 10 pages with and without JavaScript. The score out of 100 is a sum of published rules, not an AI's opinion: methodology.
The main finding: robots.txt is no longer the gate
14 of 100 sites didn't serve a single page to our checker — not to the AI user agents, and not to a regular browser user agent either. Ten answered with a Cloudflare bot challenge or captcha (HTTP 403), two returned a page with no readable content, two timed out.
Then we read their robots.txt. 12 of the 14 don't disallow any major AI bot. Only Midjourney blocks training bots (GPTBot, CCBot, Google-Extended), and Udemy's robots.txt was not readable for us either. openai.com's robots.txt has a single group: User-agent: *, Allow: /.
| Site | What our checker got | robots.txt for AI bots |
|---|---|---|
| openai.com | bot challenge, HTTP 403 (Cloudflare) | allows |
| perplexity.ai | page without readable content (Cloudflare) | allows |
| midjourney.com | bot challenge, HTTP 403 | blocks training bots only |
| pipedrive.com, freshworks.com, kit.com, activecampaign.com, skillshare.com, codecademy.com | bot challenge, HTTP 403 | allow |
| bluehost.com | captcha page, HTTP 403 | allows |
| udemy.com | bot challenge, HTTP 403 | not readable |
| zoho.com | page without readable content | allows |
| revolut.com, similarweb.com | connection timeout | allow |
An honest caveat. Our requests come from a server, not from OpenAI's or Anthropic's official IP ranges. Bot protection may let verified crawlers through — GPTBot from OpenAI's own addresses, for example. But the long tail of AI fetchers isn't on those allowlists: newer assistants, agents, browsing tools and checkers like ours. For them, a site that says "welcome" in robots.txt is still a closed door.
Five more numbers from the 86 sites that opened
- Median AI readiness: 76 out of 100. Top: Netlify 93, SimplyBook.me 92, GetResponse 91, ElevenLabs 91. Bottom: REG.RU 22, Duolingo 32, Stepik 41, Trello 45.
- 30% block at least one AI search or user-triggered bot (OAI-SearchBot, PerplexityBot, Claude-User…) in robots.txt or at the server. Blocking training bots is a fair choice; blocking search bots removes a site from AI answers that cite sources. One booking service blocks 18 AI bots.
- Only half (50%) keep their main text in HTML without JavaScript. The rest lose part or all of the text for crawlers that don't run scripts.
- 56% already have an llms.txt.
- Only 13% have complete Organization markup — name, logo, contacts and profiles that help AI tell who the company is.
What to check on your own site
- Open your firewall or CDN logs and look for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. Are they getting 200 or 403?
- In Cloudflare, review "Block AI bots", Bot Fight Mode and custom WAF rules: they override robots.txt.
- Decide separately for search bots (they bring you into AI answers) and training bots (they don't).
- Make sure the main text is in the HTML, not only rendered by JavaScript.
- Run the same free check on your site — 15 seconds, no sign-up.
Data
The full table for all 100 sites is on the AI readiness rating page, rechecked monthly. Download the data: CSV. License CC BY 4.0 — cite with a link to aiseen.cloud.