Free robots.txt Tester
Check whether search engines and AI crawlers may read a page, and see the exact line that decides
How the tester reads a robots.txt file
+
Google retired its own robots.txt tester in 2023. This one follows the same published rules (the robots.txt standard, RFC 9309, and Google's documentation), and adds the AI crawlers that did not exist when the old tool was built.
A crawler follows one group only
If the file has a section written for a crawler by name, that crawler follows it and ignores the general section. A common mistake is to add a section for Googlebot and forget that it no longer inherits the rules under “User-agent: *”.
The longest matching rule wins
When an Allow and a Disallow both match an address, the longer, more specific one decides. If they are the same length, Allow wins. The order of lines in the file does not matter.
Wildcards
A star matches any run of characters, and a dollar sign at the end means “the address ends here”. So “Disallow: /*.pdf$” blocks addresses ending in .pdf and nothing else.
Test a draft before you publish it
Paste a file you are about to upload and test real addresses against it. A wrong line in robots.txt can remove a whole site from search, so it is worth a minute.
Blocked is not the same as hidden
robots.txt stops a crawler reading a page. It does not remove the page from search results if other sites link to it. To keep a page out of results, let it be read and add a noindex tag.
What this cannot tell you
Whether a crawler obeys the file. Reputable ones do. The tester also reads the file as it is served to us; a few sites serve different files to different visitors.
The tester checks one address at a time. For a whole-site check, including pages hidden from search by mistake, run the full audit.
Check which AI crawlers your whole site blocksGuide: sitemaps, robots.txt and indexing
This page in other languages: Español · Português · Français · Deutsch