gizmobench

Robots.txt Tester

Paste your robots.txt, name the crawler and give it a URL, and the page says allowed or disallowed and then shows its working: the rule that won, the rule it beat, the length of each pattern and the user-agent group both came from. What is different here is what does not happen. This page never fetches robots.txt from a host you type, so it works on a file that is not published yet, on a staging site behind a login, and on a diff in a pull request. It also names its rule set instead of leaving you to guess: matching follows the rules Google documents, longest pattern first with Allow taking a tie, and the page lists what it does not model, from Crawl-delay to the first-match reading that other crawlers still use. Comments, the spellings Google forgives, a missing colon, a byte order mark and percent-encoded paths are all read the way Google reads them, and each one that could surprise you is reported under the verdict.

verdictagent: Googlebot

ALLOWED

winning

Allow: /private/public (15 chars, line 4)

beaten

Disallow: /private (8 chars, line 3)

group

User-agent: * (line 2, 3 rules)

Verdict
Allowed
Winning rule
Allow: /private/public
Basis
Longest match

Allowed for /private/public/a.html as Googlebot: Longest match. Winning rule Allow /private/public.

Nothing in this file needs reporting: every line is a field Google reads, and every rule can match a path. to see how those are matched.

  • /private/public/a.htmlboth rules match
    allowed, the longer path wins
  • /private/notes.htmlonly the Disallow matches
    disallowed, the only rule that matches
  • Allow and Disallow of equal lengthsame path, both rules written
    allowed, Allow takes the tie
Which rules these are. The matching here is the set Google documents and ships in its open source parser: the group whose user-agent token names your crawler, falling back to the group named with a star; the longest matching pattern, counting the characters of the pattern itself; and Allow ahead of Disallow when the two are the same length. Crawlers disagree about all three. A first-match crawler obeys whichever rule it reads first, and older crawlers support neither the star wildcard nor the end-of-URL anchor, so the same file can be read a second way. Not modelled here: Crawl-delay, Host and Clean-param, which Google ignores and some other crawlers obey; what your server returns for /robots.txt, where a 404 means no rules at all and a server error is handled differently again; and any copy of the file a crawler is still holding from an earlier visit.

Google reads the first 512,000 bytes of a robots.txt file, which is 500 KiB, and ignores the rest. This page stops at the same limit, and says so with both numbers when your paste is longer. URLs up to 2,048 characters are tested. The pasted file, the crawler and the URL are kept in this browser alone, files up to 200,000 characters, and the Start over button above the tool forgets them.

Accuracy. Matching follows Google's documented robots.txt rules: the most specific user-agent group, then the longest matching path, with Allow winning a tie of equal length. It reads only the text you paste and fetches nothing, from your site or anyone else's. robots.txt asks a crawler not to fetch a URL: it is not access control and it hides nothing from someone holding the link.

Common questions

Why does an Allow beat a Disallow here?
Because Google compares the length of the two patterns and takes the longer one. In the file on the page, Disallow: /private and Allow: /private/public both match /private/public/a.html. The Disallow pattern is 8 characters and the Allow pattern is 15 characters, so the Allow wins and the URL is allowed. When the two patterns are exactly the same length, such as Disallow: /admin/ against Allow: /admin/, the Allow takes the tie. The length counted is the pattern's own characters, wildcards and the end-of-URL anchor included, not the length of the part of the URL that matched.
Does this fetch robots.txt from my site?
No. It reads the text you paste into the box and nothing else, and it makes no network request of any kind. That is a deliberate limit, not a missing feature: a tester that fetches a URL you type is making a request from its own server to a host you named. Reading pasted text instead means you can test a file that is not published yet, one on a staging host behind a login, or one sitting in a pull request. Copy the contents of the file, paste them in, and the whole check happens in your browser. Nothing is uploaded and there is no account.
Whose rules are these, and will Bing read my file the same way?
The rules implemented here are the ones Google documents and ships in its open source robots.txt parser: the group whose user-agent token names your crawler, the star group as the fallback, the longest matching pattern, and Allow ahead of Disallow at equal length. Crawlers genuinely disagree about this. The older convention, and several crawlers still following it, take the first rule that matches rather than the longest, which can flip a verdict on a file like the sample. Wildcards and the end-of-URL anchor are not part of the original convention at all. You can type any crawler name into the Agent field and the same Google rules are applied, and for a name outside the Google crawlers this page lists it warns that Google's matching rules are being applied to that name and that another crawler may read the same file differently.
What do the star and the dollar sign do in a rule?
A star stands for any run of characters, including none, and a dollar sign as the last character of a pattern anchors the match to the end of the URL. So Disallow: /*.pdf$ matches /reports/2026.pdf but not /reports/2026.pdf?print=1, because the query string comes after what the anchor pinned. A dollar sign anywhere other than the end of the pattern is an ordinary dollar sign. Both characters count toward the pattern length that decides which rule wins, which is why Allow: /reports/*.pdf$ at 15 characters beats Disallow: /*.pdf$ at 7.
Which group applies when a file has several?
The most specific one. A group whose User-agent line names your crawler is used, and the group named with a star is the fallback for every crawler no group names. Groups that name the same crawler are merged and read as one, and the page says when that happened. Two details catch people out and both are handled here: Googlebot-Image, Googlebot-News and Googlebot-Video follow the Googlebot group when no group names them, because Google publishes two tokens for each of those crawlers; and AdsBot crawlers ignore the star group entirely, so a rule that is meant to reach them has to name them.
Does a Disallow keep a page out of search results?
No, and this is the mistake that costs the most. A Disallow asks a crawler not to fetch the URL. It does not remove the URL from an index, and a page that is linked to from elsewhere can still appear in results without its content being read. To keep a page out of results, let the crawler fetch it and return a noindex meta tag or an X-Robots-Tag header. A URL blocked in robots.txt is never fetched, so a noindex sitting on it is never read. Noindex written inside robots.txt has not been supported since September 2019, and this page reports it as a line that does nothing.
Is robots.txt a way to hide a private URL?
It is not. The file is public, it sits at a fixed address on your own site, and anyone can read every path listed in it, which means a Disallow line is a signpost to the thing you did not want fetched. It is not access control: it asks a crawler to stay away and a crawler is free to ignore it. Anything that must not be reached needs a login, a token or a firewall in front of it. This page exists to tell you which rule a well behaved crawler will follow, not to tell you a URL is protected.
How large can the file be, and what happens past that?
Google reads the first 512,000 bytes of a robots.txt file, which is 500 KiB, and ignores everything after it. This page stops at the same limit, and when your paste is longer it says so, quoting the size of your file against the 512,000-byte limit. The rules past the cut are left out of the verdict rather than quietly counted. The URL you test can be up to 2,048 characters, and a longer one is refused with its own length quoted rather than trimmed to fit.
Why is my rule being read as something shorter?
Almost always a comment. A hash starts a comment wherever it appears on the line, so Disallow: /page#section is read as Disallow: /page, and the page reports it with the line number. Percent-encode it as %23 to keep it in the path. Two more line-level habits are reported the same way: a missing colon, which Google's parser forgives by reading the first space as the separator while other crawlers skip the line, and the misspellings its parser accepts, such as Dissallow, which other crawlers skip.

Matching follows Google's documented robots.txt rules: the most specific user-agent group, then the longest matching path, with Allow winning a tie of equal length. It reads only the text you paste and fetches nothing, from your site or anyone else's. robots.txt asks a crawler not to fetch a URL: it is not access control and it hides nothing from someone holding the link.