Why does an Allow beat a Disallow here?

Because Google compares the length of the two patterns and takes the longer one. In the file on the page, Disallow: /private and Allow: /private/public both match /private/public/a.html. The Disallow pattern is 8 characters and the Allow pattern is 15 characters, so the Allow wins and the URL is allowed. When the two patterns are exactly the same length, such as Disallow: /admin/ against Allow: /admin/, the Allow takes the tie. The length counted is the pattern's own characters, wildcards and the end-of-URL anchor included, not the length of the part of the URL that matched.

Does this fetch robots.txt from my site?

No. The check reads pasted robots.txt text and makes no network request. The URL field supplies a path to test against that text; it does not retrieve the file at the URL or inspect a live site. You can test an unpublished file, and the report describes only the pasted version.

Whose rules are these, and will Bing read my file the same way?

The rules implemented here are the ones Google documents and ships in its open source robots.txt parser: the group whose user-agent token names your crawler, the star group as the fallback, the longest matching pattern, and Allow ahead of Disallow at equal length. Crawlers genuinely disagree about this. The older convention, and several crawlers still following it, take the first rule that matches rather than the longest, which can flip a verdict on a file like the sample. Wildcards and the end-of-URL anchor are not part of the original convention at all. You can type any crawler name into the Agent field and the same Google rules are applied, and for a name outside the Google crawlers this page lists it warns that Google's matching rules are being applied to that name and that another crawler may read the same file differently.

What do the star and the dollar sign do in a rule?

A star stands for any run of characters, including none, and a dollar sign as the last character of a pattern anchors the match to the end of the URL. So Disallow: /*.pdf$ matches /reports/2026.pdf but not /reports/2026.pdf?print=1, because the query string comes after what the anchor pinned. A dollar sign anywhere other than the end of the pattern is an ordinary dollar sign. Both characters count toward the pattern length that decides which rule wins, which is why Allow: /reports/*.pdf$ at 15 characters beats Disallow: /*.pdf$ at 7.

Which group applies when a file has several?

The most specific one. A group whose User-agent line names your crawler is used, and the group named with a star is the fallback for every crawler no group names. Groups that name the same crawler are merged and read as one, and the page says when that happened. Two details catch people out and both are handled here: Googlebot-Image, Googlebot-News and Googlebot-Video follow the Googlebot group when no group names them, because Google publishes two tokens for each of those crawlers; and AdsBot crawlers ignore the star group entirely, so a rule that is meant to reach them has to name them.

Does a Disallow keep a page out of search results?

No, and this is the mistake that costs the most. A Disallow asks a crawler not to fetch the URL. It does not remove the URL from an index, and a page that is linked to from elsewhere can still appear in results without its content being read. To keep a page out of results, let the crawler fetch it and return a noindex meta tag or an X-Robots-Tag header. A URL blocked in robots.txt is never fetched, so a noindex sitting on it is never read. Noindex written inside robots.txt has not been supported since September 2019, and this page reports it as a line that does nothing.

Is robots.txt a way to hide a private URL?

It is not. The file is public, it sits at a fixed address on your own site, and anyone can read every path listed in it, which means a Disallow line is a signpost to the thing you did not want fetched. It is not access control: it asks a crawler to stay away and a crawler is free to ignore it. Anything that must not be reached needs a login, a token or a firewall in front of it. This page exists to tell you which rule a well behaved crawler will follow, not to tell you a URL is protected.

How large can the file be, and what happens past that?

Google reads the first 512,000 bytes of a robots.txt file, which is 500 KiB, and ignores everything after it. This page stops at the same limit, and when your paste is longer it says so, quoting the size of your file against the 512,000-byte limit. The rules past the cut are left out of the verdict rather than quietly counted. The URL you test can be up to 2,048 characters, and a longer one is refused with its own length quoted rather than trimmed to fit.

Why is my rule being read as something shorter?

Almost always a comment. A hash starts a comment wherever it appears on the line, so Disallow: /page#section is read as Disallow: /page, and the page reports it with the line number. Percent-encode it as %23 to keep it in the path. Two more line-level habits are reported the same way: a missing colon, which Google's parser forgives by reading the first space as the separator while other crawlers skip the line, and the misspellings its parser accepts, such as Dissallow, which other crawlers skip.

Is this Google Search Console's robots.txt tester?

No. This page checks your pasted file using Google's documented matching rules. To examine the Googlebot group, enter Googlebot in Agent and supply the URL or path to test. The verdict explains the rule match for that input. It does not connect to Search Console, inspect Google's cached robots.txt or report live crawl activity.

Robots.txt Tester

Paste robots.txt, enter a crawler name and test a URL or path. The verdict shows the winning rule, a competing rule, pattern lengths and the matching user-agent group. Matching uses Google's documented rules: the longest matching pattern wins, with Allow taking equal-length ties. The check reads your pasted file in the browser and does not fetch a deployed copy.

robots.txtpasted, not fetched

verdictagent: Googlebot

URL

ALLOWED

winning

Allow: /private/public (15 chars, line 4)

beaten

Disallow: /private (8 chars, line 3)

group

User-agent: * (line 2, 3 rules)

Verdict: Allowed

Agent

Winning rule: Allow: /private/public

Basis: Longest match

Allowed for /private/public/a.html as Googlebot: Longest match. Winning rule Allow /private/public.

Nothing in this file needs reporting: every line is a field Google reads, and every rule can match a path. to see how those are matched.

/private/public/a.htmlboth rules match
allowed, the longer path wins
/private/notes.htmlonly the Disallow matches
disallowed, the only rule that matches
Allow and Disallow of equal lengthsame path, both rules written
allowed, Allow takes the tie

Which rules these are. The matching here is the set Google documents and ships in its open source parser: the group whose user-agent token names your crawler, falling back to the group named with a star; the longest matching pattern, counting the characters of the pattern itself; and Allow ahead of Disallow when the two are the same length. Crawlers disagree about all three. A first-match crawler obeys whichever rule it reads first, and older crawlers support neither the star wildcard nor the end-of-URL anchor, so the same file can be read a second way. Not modelled here: Crawl-delay, Host and Clean-param, which Google ignores and some other crawlers obey; what your server returns for /robots.txt, where a 404 means no rules at all and a server error is handled differently again; and any copy of the file a crawler is still holding from an earlier visit.

Google reads the first 512,000 bytes of a robots.txt file, which is 500 KiB, and ignores the rest. This page stops at the same limit, and says so with both numbers when your paste is longer. URLs up to 2,048 characters are tested. The pasted file, the crawler and the URL are kept in this browser alone, files up to 200,000 characters. Clear this site's saved data in your browser settings to forget them.

Accuracy. Matching follows Google's documented robots.txt rules: the most specific user-agent group, then the longest matching path, with Allow winning a tie of equal length. It reads only the text you paste and fetches nothing, from your site or anyone else's. robots.txt asks a crawler not to fetch a URL: it is not access control and it hides nothing from someone holding the link.

Common questions

Why does an Allow beat a Disallow here?: Because Google compares the length of the two patterns and takes the longer one. In the file on the page, Disallow: /private and Allow: /private/public both match /private/public/a.html. The Disallow pattern is 8 characters and the Allow pattern is 15 characters, so the Allow wins and the URL is allowed. When the two patterns are exactly the same length, such as Disallow: /admin/ against Allow: /admin/, the Allow takes the tie. The length counted is the pattern's own characters, wildcards and the end-of-URL anchor included, not the length of the part of the URL that matched.
Does this fetch robots.txt from my site?: No. The check reads pasted robots.txt text and makes no network request. The URL field supplies a path to test against that text; it does not retrieve the file at the URL or inspect a live site. You can test an unpublished file, and the report describes only the pasted version.
Whose rules are these, and will Bing read my file the same way?: The rules implemented here are the ones Google documents and ships in its open source robots.txt parser: the group whose user-agent token names your crawler, the star group as the fallback, the longest matching pattern, and Allow ahead of Disallow at equal length. Crawlers genuinely disagree about this. The older convention, and several crawlers still following it, take the first rule that matches rather than the longest, which can flip a verdict on a file like the sample. Wildcards and the end-of-URL anchor are not part of the original convention at all. You can type any crawler name into the Agent field and the same Google rules are applied, and for a name outside the Google crawlers this page lists it warns that Google's matching rules are being applied to that name and that another crawler may read the same file differently.
What do the star and the dollar sign do in a rule?: A star stands for any run of characters, including none, and a dollar sign as the last character of a pattern anchors the match to the end of the URL. So Disallow: /*.pdf$ matches /reports/2026.pdf but not /reports/2026.pdf?print=1, because the query string comes after what the anchor pinned. A dollar sign anywhere other than the end of the pattern is an ordinary dollar sign. Both characters count toward the pattern length that decides which rule wins, which is why Allow: /reports/*.pdf$ at 15 characters beats Disallow: /*.pdf$ at 7.
Which group applies when a file has several?: The most specific one. A group whose User-agent line names your crawler is used, and the group named with a star is the fallback for every crawler no group names. Groups that name the same crawler are merged and read as one, and the page says when that happened. Two details catch people out and both are handled here: Googlebot-Image, Googlebot-News and Googlebot-Video follow the Googlebot group when no group names them, because Google publishes two tokens for each of those crawlers; and AdsBot crawlers ignore the star group entirely, so a rule that is meant to reach them has to name them.
Does a Disallow keep a page out of search results?: No, and this is the mistake that costs the most. A Disallow asks a crawler not to fetch the URL. It does not remove the URL from an index, and a page that is linked to from elsewhere can still appear in results without its content being read. To keep a page out of results, let the crawler fetch it and return a noindex meta tag or an X-Robots-Tag header. A URL blocked in robots.txt is never fetched, so a noindex sitting on it is never read. Noindex written inside robots.txt has not been supported since September 2019, and this page reports it as a line that does nothing.
Is robots.txt a way to hide a private URL?: It is not. The file is public, it sits at a fixed address on your own site, and anyone can read every path listed in it, which means a Disallow line is a signpost to the thing you did not want fetched. It is not access control: it asks a crawler to stay away and a crawler is free to ignore it. Anything that must not be reached needs a login, a token or a firewall in front of it. This page exists to tell you which rule a well behaved crawler will follow, not to tell you a URL is protected.
How large can the file be, and what happens past that?: Google reads the first 512,000 bytes of a robots.txt file, which is 500 KiB, and ignores everything after it. This page stops at the same limit, and when your paste is longer it says so, quoting the size of your file against the 512,000-byte limit. The rules past the cut are left out of the verdict rather than quietly counted. The URL you test can be up to 2,048 characters, and a longer one is refused with its own length quoted rather than trimmed to fit.
Why is my rule being read as something shorter?: Almost always a comment. A hash starts a comment wherever it appears on the line, so Disallow: /page#section is read as Disallow: /page, and the page reports it with the line number. Percent-encode it as %23 to keep it in the path. Two more line-level habits are reported the same way: a missing colon, which Google's parser forgives by reading the first space as the separator while other crawlers skip the line, and the misspellings its parser accepts, such as Dissallow, which other crawlers skip.
Is this Google Search Console's robots.txt tester?: No. This page checks your pasted file using Google's documented matching rules. To examine the Googlebot group, enter Googlebot in Agent and supply the URL or path to test. The verdict explains the rule match for that input. It does not connect to Search Console, inspect Google's cached robots.txt or report live crawl activity.

Matching follows Google's documented robots.txt rules: the most specific user-agent group, then the longest matching path, with Allow winning a tie of equal length. It reads only the text you paste and fetches nothing, from your site or anyone else's. robots.txt asks a crawler not to fetch a URL: it is not access control and it hides nothing from someone holding the link.