Free toolsAI crawler checker

Can AI crawlers read your site?

Paste your robots.txt to see which AI crawlers and answer bots are allowed, blocked or partly blocked. Pasted text is parsed locally. Fetching a domain sends that public domain to our server. This checks directives, not firewall or HTTP access.

Your robots.txt
Load sample·Clear

Open yoursite.com/robots.txt, copy everything, and paste it here.

AI crawler access
2 blocked · 8 partly blocked
GPTBot
OpenAI · Training · Disallow: /
Blocked
OAI-SearchBot
OpenAI · ChatGPT search · /admin/, /cart
Partly blocked
ChatGPT-User
OpenAI · Live browsing · /admin/, /cart
Partly blocked
ClaudeBot
Anthropic · Training · /admin/, /cart
Partly blocked
Claude-User
Anthropic · Live browsing · /admin/, /cart
Partly blocked
PerplexityBot
Perplexity · Search index · /private/
Partly blocked
Google-Extended
Google · Gemini training · /admin/, /cart
Partly blocked
Applebot-Extended
Apple · AI training · /admin/, /cart
Partly blocked
CCBot
Common Crawl · Open dataset · Disallow: /
Blocked
Bytespider
ByteDance · Training · /admin/, /cart
Partly blocked

Checked against the most specific matching User-agent group, falling back to *. Blocking training crawlers may not affect live answer bots.

Ten AI user agents

Training crawlers and live answer bots from OpenAI, Anthropic, Google, Perplexity and more.

Group matching

Uses the most specific User-agent group, then falls back to *.

Private

Pasted contents stay in your browser. Fetching a domain uses our server.

A little more clarity.

Are these tools really free?

Yes, and there is no signup. The response explorer requires your own OpenRouter API key; provider charges may apply. Other browser checks do not require a key.

How accurate is a one-off check?
Does the crawler checker change my site?
Can I use the output with clients?

Be part of the conversation.

Explore a model response with your own provider key.