Technical SEO

X-Robots-Tag Checker

Read HTTP robot directives that may apply even when a document has no HTML head.

InfiniteRanks editorial teamUpdated 23 October 2026No account required

Run the x-robots-tag checker

Public website check

This tool makes bounded requests to the public URL or domain you enter. It does not sign in or change the target website.

What x-robots-tag checker helps you do

Read HTTP robot directives that may apply even when a document has no HTML head. A PDF can send X-Robots-Tag: noindex in its response headers. Review server and CDN rules if every file in a directory inherits the same restriction.

Prepare a useful input

Example input: https://example.com/report.pdf Save the original input alongside the result so another person can reproduce the check and judge whether a proposed change addresses the same page or file.

Read the result before making changes

Headers describe the current response only; cached responses and different user agents may receive different directives. Record the checked URL and observation time. Compare the evidence with the intended page behavior, then verify the affected page again after a change instead of assuming the issue is resolved.

Check non-HTML files and inherited headers

A downloads directory can send an X-Robots-Tag policy for every PDF through a server rule. That may be desirable for receipts but wrong for a public research report. Inspect the requested file and its redirect destinations because the policy can differ between them. If the HTML head looks unrestricted while the header says noindex, find the CDN, web-server or application middleware rule. Make a narrow change for the intended resource set rather than removing privacy-conscious defaults everywhere. Re-request the exact file after cache invalidation and keep the response headers in the release record.

Verify the result and continue

Fetch an affected file after changing the server rule and compare its headers with an unaffected control file. A cached header can survive an application release. Document whether the policy belongs to the CDN, server or route handler so a later infrastructure change does not silently restore the restriction. Continue with the url status code checker or explore the technical seo tool cluster.

How this tool works

Read HTTP robot directives that may apply even when a document has no HTML head. The server fetches the submitted public URL with bounded redirects and response sizes. The result reports observed HTML, headers or network evidence for this specific check.

What the result does and does not establish

Headers describe the current response only; cached responses and different user agents may receive different directives.

Use this check as part of a repeatable website review. Record what you found, make a specific change and run the same check again.

Browse all toolsContent and technical review workflow

Reference standards

Be part of the conversation.

Explore a model response with your own provider key.