← Free tools
robots.txt Tester
Fetch and parse any site's robots.txt file. Optionally test whether a specific URL is blocked.
Fetch and parse any site's robots.txt file. Optionally test whether a specific URL is blocked.
Revuin's robots.txt tester fetches and parses any site's robots.txt file, checks whether a specific URL is blocked for a given user agent, and lists every sitemap it references — a fast way to confirm your crawl directives actually do what you think they do, free and with no login required.
Enter a domain
The site whose robots.txt you want to fetch and parse.
Enter a test URL and user agent
Check whether that specific path is allowed or disallowed.
Review the result
See the matching rule and every sitemap the file references.
A plain-text file at a site's root that tells search engine crawlers which parts of the site they are and aren't allowed to crawl.
Yes — a single misplaced rule can accidentally block Google from crawling your entire site or key sections of it, which is exactly what this tool is built to catch before it causes damage.
Yes — you can test against a specific user agent, since some sites apply different crawl rules to different bots.
Yes — completely free, no login required.
Robots.txt blocks crawling (the bot never fetches the page); a noindex meta tag allows crawling but tells search engines not to include the page in results — blocking a page in robots.txt actually prevents Google from ever seeing a noindex tag on it.
At the root of a domain, e.g. example.com/robots.txt — it won't be recognized anywhere else.
Major search engines (Google, Bing) respect it as a standard, though it's technically a voluntary directive, not an enforced access restriction — sensitive content should never rely on robots.txt alone for protection.