Can AI crawlers read your site?
Paste your robots.txt to see which AI crawlers and answer bots are allowed, blocked or partly blocked. Pasted text is parsed locally. Fetching a domain sends that public domain to our server. This checks directives, not firewall or HTTP access.
Open yoursite.com/robots.txt, copy everything, and paste it here.
Checked against the most specific matching User-agent group, falling back to *. Blocking training crawlers may not affect live answer bots.
Training crawlers and live answer bots from OpenAI, Anthropic, Google, Perplexity and more.
Uses the most specific User-agent group, then falls back to *.
Pasted contents stay in your browser. Fetching a domain uses our server.