AiseenBot — the aiseen.cloud crawler
aiseen.cloudAiseenBot visits a site when someone runs a check of it on aiseen.cloud. It reads robots.txt, the sitemap, llms.txt and a few pages: up to 10 in a free check, as many as ordered in an audit.
User-Agent
Mozilla/5.0 (compatible; AiseenBot/1.0; +https://aiseen.cloud/bots/)
How it behaves
- Follows robots.txt: the rules for AiseenBot, or for all robots (*) if there are none.
- No more than 2 requests to a site at once, with a pause between them.
- Reads public pages only: it doesn't fill in forms or enter protected sections.
Requests on behalf of AI bots
To check whether the server lets AI bots in, we request the home page once with the User-Agent of each of these bots: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-User, ClaudeBot, PerplexityBot, Googlebot, bingbot, YandexBot, MistralAI-Index, meta-webindexer, Kimi-SearchBot, Claude-Agent, Google-Agent, Google-GeminiNotebook, Google, Google-Lens. These requests come from our server, not from OpenAI, Anthropic or Google — the IP address tells them apart in your logs.
An audit adds a few more requests: to a non-existent address (how the site answers a 404) and to the www and http variants of the address (whether mirrors are merged).
How to block AiseenBot
Add this group to robots.txt — blocked pages won't be included in checks. Questions about the crawler: [email protected].
User-agent: AiseenBot Disallow: /