Technical SEO

Robots.txt Checker

Retrieve the site robots file and review its crawler groups and discovery hints.

InfiniteRanks editorial teamUpdated 23 October 2026No account required

Run the robots.txt checker

Public website check

This tool makes bounded requests to the public URL or domain you enter. It does not sign in or change the target website.

What robots.txt checker helps you do

Retrieve the site robots file and review its crawler groups and discovery hints. Disallow: /admin/ blocks cooperative crawlers from that path. It does not protect private records; authentication must still enforce access to sensitive pages.

Prepare a useful input

Example input: https://example.com/ Save the original input alongside the result so another person can reproduce the check and judge whether a proposed change addresses the same page or file.

Read the result before making changes

A robots file is a voluntary crawl policy, not a security layer or an index-removal command. Record the checked URL and observation time. Compare the evidence with the intended page behavior, then verify the affected page again after a change instead of assuming the issue is resolved.

Read policy scope before editing a rule

Robots policies apply to the origin where they are served. A rule on a marketing subdomain does not automatically govern a help subdomain. Check the precise hostname, protocol and requested path before reasoning about an exclusion. A wildcard group and a named-agent group can lead to different applicable rules, so keep the literal group boundaries in your review. If a private address appears in robots.txt, remember that the file itself is public. Remove secrets from public locations and enforce authentication instead of expecting a crawler instruction to conceal them. Test a narrow set of representative paths after publishing.

Verify the result and continue

Check the public root file with the exact hostname visitors use. A correct repository file does not help if the host serves a generated replacement or an HTML error page at that location. Review the sitemap declaration and named-agent groups whenever hosting or crawler policy changes. Continue with the robots.txt generator or explore the technical seo tool cluster.

How this tool works

Retrieve the site robots file and review its crawler groups and discovery hints. The server fetches the submitted public URL with bounded redirects and response sizes. The result reports observed HTML, headers or network evidence for this specific check.

What the result does and does not establish

A robots file is a voluntary crawl policy, not a security layer or an index-removal command.

Use this check as part of a repeatable website review. Record what you found, make a specific change and run the same check again.

Browse all toolsContent and technical review workflow

Reference standards

Be part of the conversation.

Explore a model response with your own provider key.