Reference

AI bots and AiseenBot

AiseenBot reads up to 10 pages of a site in the free check and from 30 in an audit, respects robots.txt and makes no more than 2 requests at a time. Below are the AI bots: which bring visitors from ChatGPT, Claude, Perplexity and Alice, which collect texts for training, and how to allow or block each one.

AS

AiseenBot — the aiseen.cloud crawler

aiseen.cloud

AiseenBot visits a site when someone runs a check of it on aiseen.cloud. It reads robots.txt, the sitemap, llms.txt and a few pages: up to 10 in a free check, as many as ordered in an audit.

User-Agent

Mozilla/5.0 (compatible; AiseenBot/1.0; +https://aiseen.cloud/bots/)

How it behaves

  • Follows robots.txt: the rules for AiseenBot, or for all robots (*) if there are none.
  • No more than 2 requests to a site at once, with a pause between them.
  • Reads public pages only: it doesn't fill in forms or enter protected sections.

Requests on behalf of AI bots

To check whether the server lets AI bots in, we request the home page once with the User-Agent of each of these bots: OAI-SearchBot, ChatGPT-User, GPTBot, Claude-User, ClaudeBot, PerplexityBot, Googlebot, bingbot, YandexBot, MistralAI-Index, meta-webindexer, Kimi-SearchBot, Claude-Agent, Google-Agent, Google-GeminiNotebook, Google, Google-Lens. These requests come from our server, not from OpenAI, Anthropic or Google — the IP address tells them apart in your logs.

An audit adds a few more requests: to a non-existent address (how the site answers a 404) and to the www and http variants of the address (whether mirrors are merged).

How to block AiseenBot

Add this group to robots.txt — blocked pages won't be included in checks. Questions about the crawler: [email protected].

User-agent: AiseenBot
Disallow: /

AI system bots

The robots.txt tokens that aiseen.cloud checks. Search and on-request bots bring your site into AI answers with a link; training bots collect texts for future model versions.

GP

ChatGPT

OpenAI

robots.txt tokens

More bots — not part of the score

  • OAI-AdsBotadsdocschecks ad landing pages in ChatGPT: visits only pages submitted as ads, not used for training

robots.txt rule

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Allow: /
CL

Claude

Anthropic

robots.txt tokens

More bots — not part of the score

  • Claude-AgentagentdocsClaude's agentic browser: opens pages and acts on a user's task (including Claude Duet); verify by IP — claude.com/.well-known/sail

robots.txt rule

User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
GM

Gemini & AI Overviews

Google

robots.txt tokens

More bots — not part of the score

  • Google-AgentagentdocsGoogle's agents: browse sites and act on a user's request; generally ignore robots.txt
  • Google-GeminiNotebookon requestdocsfetches pages a user added as sources in Gemini Notebook (formerly Google-NotebookLM); generally ignores robots.txt
  • Googleon requestsourcepage fetch for Gemini — per an independent test, not confirmed by Google
  • Google-Lenson requestsourcepage fetch for AI Mode, other uses possible — per an independent test, not confirmed by Google
CP

Microsoft Copilot

Microsoft

robots.txt tokens

DA

DuckAssist

DuckDuckGo

robots.txt tokens

robots.txt rule

User-agent: DuckAssistBot
Allow: /
CC

Common Crawl

Common Crawl

robots.txt tokens

How to allow or block

Ready-made groups for robots.txt at the site root.

Allow AI search

Your site appears in ChatGPT, Claude and Perplexity answers with a link. Block service sections in every group.

# Search and user requests: AI answers with a link to your site come from here
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: YandexAdditional
User-agent: YandexAdditionalBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Kimi-SearchBot
User-agent: Kimi-User
Allow: /
Disallow: /admin/   # your service sections

# Model training: your call; Disallow: / closes the site to training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: MistralAI-Training
User-agent: KimiBot
Allow: /
Disallow: /admin/   # your service sections

Sitemap: https://example.com/sitemap.xml

Googlebot, bingbot, YandexBot aren't in the sample: they are regular search crawlers, and your site most likely already has rules for them.

Block model training

Your texts won't be used to train future models. This doesn't block AI search — see the group on the left.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: MistralAI-Training
User-agent: KimiBot
Disallow: /

Questions and answers

Why do my logs show GPTBot or ClaudeBot from your IP address?

That's an aiseen.cloud check: we request the home page with the User-Agent of AI bots to see what your server answers them. The real bots come from their companies' addresses.

Should I block training bots?

It's your choice. Blocking them doesn't remove the site from AI search, but a model without search will know less about you. Better not to block search and on-request bots: they bring the site into answers with a link.

If I block Google-Extended, will my site disappear from Google?

No. Google-Extended only controls whether your texts can be used to train Gemini. Google Search and AI Overviews read the site via Googlebot.

robots.txt allows the bot, but AI still can't see the site. Why?

Bot protection (Cloudflare, the hosting WAF) often gets in the way: it answers bots with a 403 or a captcha. The free check on aiseen.cloud shows what the server answered each bot.

What are llms.txt and llms-full.txt?

llms.txt is a short site map for AI agents: what the site is and links to its key pages. llms-full.txt is the same pages in full, as one text file. Both live at the site root.