robots.txt Tester
and which rule actually wins

What a robots.txt Tester Actually Tests

A robots.txt file is a plain text file at the root of a host telling automated clients which paths they may fetch. Unlike most things in this corner of the web it is a real standard: RFC 9309, published in 2022 on the IETF Standards Track after twenty-five years as a convention. This robots.txt tester implements it rule by rule.

One word matters more than any other here: robots.txt asks. It does not prevent, stop or block anything. A compliant crawler reads your file and chooses to obey it; one that ignores it fetches the page anyway. That is the design, not a flaw — a notice on a door, not a lock. If you need a lock, you have to refuse the request at the server or the edge, which is what the guide to blocking an IP address at every layer covers.

The file is made of groups. A group starts with one or more User-agent: lines, followed by the Allow: and Disallow: rules for them. A crawler obeys the group naming its own product token and ignores every other group — including the * group. That catches people out constantly: give Googlebot a group of its own and it stops reading your * group entirely.

So a tester has two jobs: does the file say what you think, and will the crawler you care about ever see it? The second is about HTTP, and it does the real damage.

How to Read Your robots.txt Test Result

When you test robots.txt here you get four answers rather than one, because "robots.txt is not working" is four problems with four different fixes.

Every finding is labelled with its authority: RFC 9309 for the standard, vendor docs for behaviour a named crawler documents and the standard does not, HTTP layer for how your server answered, and TrustMyIP view for our own opinion with its threshold beside it. Only findings against RFC 9309 make a file invalid; ours never do. That is the main reason one robots.txt checker calls a file broken and another calls it fine — most present their preferences as errors, so a robots.txt validator that never separates the two leaves you guessing which findings matter.

The per-path answer prints three things other testers leave out: the line that decided it, that rule's length in octets, and the rule it beat. If you have ever wondered why a rule you wrote is ignored, that is the answer.

Search Console will not give you this. Its robots.txt report shows the file Google last fetched and offers "Request a recrawl", but does not test a URL against the rules; its help page sends you to URL Inspection for that, which answers for Google's crawler on a property you have verified. Nothing there answers for GPTBot.

Which robots.txt Rule Wins: Allow vs Disallow

This is the question the whole tool exists to answer, and the rule is short. RFC 9309 section 2.2.2:

The most specific match found MUST be used. The most specific match is the match that has the most octets.

Octets, not lines. Not first-wins, not last-wins, not allow-beats-disallow: count the bytes in each matching value and the longest decides. Only on an exact tie does the standard say the allow "SHOULD be used" — a SHOULD, not a MUST.

Take the pattern nearly every WordPress site serves:

For /wp-admin/admin-ajax.php both rules match. The allow is 24 octets, the disallow 10, so the allow wins — in either order. A great deal of advice online says rule order matters. It does not.

How a crawler picks its group

Group selection is case-insensitive on the crawler name. With no group naming the crawler, RFC 9309 says it "MUST obey the group with a user-agent line with the * value, if present"; with no * group either, nothing applies.

Two refinements are not in the RFC's text. Google states that "if there are multiple groups in a robots.txt file that are relevant to a specific user agent, Google's crawlers internally merge the groups", so rules split across two User-agent: Googlebot blocks all apply. And a value is read up to the first character that is not a letter, hyphen or underscore, so User-agent: Googlebot/2.1 names Googlebot and User-agent: Foo Bar is the group for Foo. A strict grammar parser rejects both. This tester follows Google, whose parser the standard was written from, and says when your file depends on the difference.

The two metacharacters, and the one that is not

Only two characters are special inside a path. The * wildcard matches "zero or more instances of any character", slashes included, and $ at the very end anchors the match to the end of the URL. A $ anywhere else is an ordinary dollar sign. There is no regular-expression support at all — matching already starts at the first octet of the path.

Sitemap: belongs to no group: it is an extension, takes an absolute URL, and applies wherever it appears.

# starts a comment anywhere, including mid-path: Disallow: /a#b is the rule /a, which blocks far more than you meant.

Two rules that look like they do something and do not

A bare Disallow: blocks nothing: the grammar allows an empty value, and an empty pattern matches nothing. It is the conventional way to say "no restrictions". A value not starting with / or * also matches nothing, because comparison begins at the first octet — Disallow: admin is silently dead.

Paths are compared after percent-encoding

A rule and a URL are escaped before comparison, and Google escapes them differently: in a rule it encodes only the non-ASCII octets, while a real request has everything encoded. So Disallow: /my page matches nothing, while Disallow: /café/ matches the real request. Nothing is decoded, so /a%2Fb is not /a/b. The tester prints the compared form when it differs from what you typed, and flags a rule that can never match.

A file has a size ceiling

Section 2.5 requires a crawler to parse "at least 500 kibibytes", and Google enforces exactly that. On a site that generates a rule per URL, the rules at the bottom may never be read, so the important ones belong at the top.

robots.txt and AI Crawlers: What a Disallow Can and Cannot Reach

Most questions about robots.txt now are really about AI, and the honest answer has three parts. A green tick without them is not the truth.

Some agents document that they ignore robots.txt. Perplexity states that Perplexity-User generally ignores it, because those fetches are triggered by a person asking a question rather than by a crawl. A perfect Disallow for that token changes nothing, and the reality column says so.

Some tokens are not crawlers. Google-Extended and Applebot-Extended control whether already-collected content may train models. Nothing fetches under either name, so a Disallow there is a use rule, not a crawl rule, and has no effect on search. Our AI crawler blocker builds the rules and explains which can be enforced by IP.

Some crawlers follow another crawler's group. Apple documents that if your instructions do not mention Applebot but do mention Googlebot, "the Apple robot will follow Googlebot instructions". So a Googlebot group written for search reasons quietly governs Apple too. The tester marks that answer as coming from the Googlebot group rather than from *, because you would never guess it from the file.

The vendors that document compliance are consistent:

So the same line can be honoured, ignored or meaningless depending on who reads it, which is why this page answers per crawler. Where a vendor publishes nothing, it says "unverified" rather than guessing. For the practical version, the guide to blocking AI crawlers such as GPTBot and ClaudeBot walks through which methods hold.

robots.txt has a counterpart. Where it refuses, llms.txt invites: a curated list of the pages worth reading. Unrelated files, opposite purposes, and our llms.txt checker tests that one against its own format.

The 500 Error That Closes Your Whole Site

Rarely explained, and the most damaging thing in the file. RFC 9309 section 2.3.1.4:

If the robots.txt file is unreachable ... the crawler MUST assume complete disallow.

Unreachable means a 5xx, a 429, a timeout, a refused connection or a failed TLS handshake, and "complete disallow" means every URL on the host. A crawler that has seen this for more than thirty days may fall back to cached rules or treat the site as allowed — but a month of not being crawled is not a recovery plan.

A 4xx is safer. Section 2.3.1.3 treats an unavailable file as no rules, and a crawler "MAY access any resources on the server". A missing robots.txt is valid; a broken one is not.

Three more delivery failures the tester looks for:

• The soft 404. Your server returns its own 404 page, or an application shell, with status 200. To a crawler that is a robots.txt full of HTML, which parses to no rules at all — while your monitoring sees a 200 and reports nothing wrong. It is the most common way a site loses every rule it thinks it has.
• A 403 or a 401. Technically "no rules", but usually an authentication layer or firewall sitting in front of the file.
• Long redirect chains. Section 2.3.1.2 asks crawlers to follow "at least five consecutive redirects, even across authorities". Five is a floor, not a promise. Serve the file where crawlers ask for it.

Content type matters too: the standard calls robots.txt a text file "served as text/plain", and text/html is usually the fingerprint of the soft 404 above. Our headers analyzer shows what your server sends. On caching, section 2.4 asks crawlers not to reuse a cached copy for more than twenty-four hours, and permits longer when the file cannot be fetched — so a change takes about a day, and only affects future crawls.

Two things follow from the answers. A rule that needs enforcing rather than requesting belongs in the server configuration — the walkthrough for refusing an address in .htaccess is the shortest route on Apache. And robots.txt cannot tell you whether a visit claiming to be a named AI crawler was genuine; a reverse-DNS and IP check can, which is what the ClaudeBot verifier does for one of the busiest.

Five Mistakes That Quietly Cost You Traffic

1. The staging file that reached production

User-agent: * plus Disallow: / is occasionally intended. Far more often it is a staging configuration that got deployed. Reported as a warning rather than an error, because it is legal — but on a live site it is almost certainly wrong.

It is also not as total as it looks. Of AdsBot-Google, AdsBot-Google-Mobile, Mediapartners-Google and APIs-Google, Google says "the global user agent (*) is ignored", and of Google-Safety that it "ignores robots.txt rules". A site-wide block stops Search and leaves those four crawling; only a rule naming them reaches them.

2. Rules written for the wrong group

A crawler that finds its own group stops reading *:

Googlebot is not kept out of /wp-admin/ here. It found its own group, and that group says nothing about /wp-admin/. An empty group is sharper still: name a crawler, give it no rules, and it is completely unrestricted.

The reverse trap is as common. Google lists a second token for several crawlers, and says "you need to match only one crawler token for a rule to apply". So a Googlebot rule also governs Googlebot-Image, Googlebot-Video, Googlebot-News, Google-InspectionTool and Google-CloudVertexBot — the last crawls for Vertex AI, which no Search rule hints at. Writing a separate Googlebot-Image group does not release it from the Googlebot one: the two are read together.

3. Expecting Noindex: to work

Noindex: is not in RFC 9309 and Google does not support it here. The line does nothing at all. Deindexing belongs in a page-level tag or header instead, and the FAQ below covers why blocking a page makes that impossible.

4. Blocking the CSS and JavaScript the page needs

A Disallow on /assets/ or /static/ stops a crawler rendering the page as a browser would, and Google renders before deciding what a page says.

5. Misspellings and missing colons that work for some crawlers only

Google's parser corrects a short list of frequent misspellings on purpose — Dissalow and Disalow among them — matches field names by prefix, and accepts a space for the colon when the line has exactly two words. A stricter parser drops those lines. So a typo can leave a rule working for Google and missing everywhere else, which is worse than one that fails everywhere. The tester reports each leniency it applied.

curl -sI https://example.com/robots.txt shows the status and content type by hand. It will not tell you which rule wins, nor that Disallow: /Private leaves /private wide open — paths are compared byte for byte.

What This Tester Cannot Tell You

Four honest limits:

It cannot make a crawler obey you. Everything here is what a compliant crawler concludes; a scraper that ignores robots.txt fetches the page regardless.

It cannot tell you what actually visited. The rules say what should happen; only your access logs say what did. A user-agent string is trivially forged, so a log line claiming to be Googlebot proves nothing on its own — that takes our bot IP verifier.

It cannot know undocumented behaviour. Where a vendor publishes how its crawler treats robots.txt, this page uses that and names the source; otherwise the answer says "unverified". One token is marked as in common use rather than vendor-verified, because Bing's crawler page is rendered by JavaScript and could not be read on the review date.

It cannot remove anything from search. Deindexing is a different mechanism, which blocking actively prevents.

And not to overclaim: this matcher has been compared against Google's own parser over more than two million questions with no unexplained disagreement, and against Scrapy's parser over the same set. It has not yet been run against a live website from the server it is published on.

robots.txt Questions People Actually Ask