What the check does
The combined URL workspace fetches the public robots.txt file and checks supported rules against the submitted path. Complex user-agent groups, wildcards, server variations, and rendered behavior may still require manual review.
How to use the result
A blocked result means the crawler may not fetch the page content. Fix the rule before relying on a page-level noindex or another submission. An allowed result does not prove that the page will be crawled or indexed.
Related guidance
Read Robots.txt vs Noindex and keep blocked or noindex URLs out of the XML sitemap.
Run the URL preflight
Get the technical result before routing another discovery request.
Check URLs Free →