Our position: we are Geonode and we sell proxies to people who collect data, so robots.txt sits directly in the middle of our customers' work. The honest framing is that honouring it is in your interest, not just the site's. A crawler that respects the file, identifies itself and paces itself is one that site operators can choose to allow. A crawler that ignores it is a nuisance to be blocked, and blocking is cheap for them and expensive for you. We also want to be clear about what the standard itself says on the subject: the RFC states plainly that these rules "are not a form of access authorization" — so honouring robots.txt is necessary and not sufficient, and terms of service are a separate question.
What robots.txt Is and Is Not
The RFC's own framing is the clearest available:
It may be inconvenient for service owners if crawlers visit the entirety of their URI space. This document specifies the rules originally defined by the "Robots Exclusion Protocol" that crawlers are requested to honor when accessing URIs.
These rules are not a form of access authorization.
That last sentence does double duty. It means a disallowed path is not protected — nothing enforces the rule — and it means an allowed path is not thereby authorised, because permission comes from terms and law rather than from a text file.
The security section makes the first half explicit and is worth quoting because so many people get it backwards:
The Robots Exclusion Protocol is not a substitute for valid content security measures. Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable.
So Disallow: /admin/secret-reports/ tells the world that path exists. If you are writing a robots.txt, that is an argument for not listing sensitive paths in it at all — use authentication, which the RFC explicitly recommends.
The Format
A file is a sequence of groups. Each group begins with one or more user-agent lines and is followed by rules.
User-agent: *
Disallow: /admin/
Disallow: /search?
Allow: /search/help
User-agent: BadBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
Three special characters that crawlers MUST support:
| Character | Meaning | Example |
|---|---|---|
# | Line comment | allow: / # comment in line |
$ | End of the match pattern | allow: /this/path/exactly$ |
* | Zero or more of any character | allow: /this/*/exactly |
An empty group at the end is meaningful: the RFC notes that "the last group may have no rules, which means it implicitly allows everything". So a trailing User-agent: quxbot with nothing under it grants that crawler unrestricted access.
An empty Disallow: with no path also means everything is allowed — it is the conventional way to say "no restrictions".
Sitemap: is not part of the core grammar. The ABNF in the RFC includes a note to implementers to "define additional lines you need (for example, Sitemaps)", so it is a widely supported extension rather than a required feature. Crawl-delay is in the same category: common, honoured by many crawlers, and not in the standard.
How User-Agent Matching Works
More specific than most people assume, and worth getting right.
The token is a substring of your User-Agent header. The RFC's example: a header of Mozilla/5.0 (compatible; ExampleBot/0.1; https://www.example.com/bot.html) corresponds to a robots.txt line of user-agent: ExampleBot, and it notes that "the product token (ExampleBot) is a substring of the User-Agent HTTP header."
Matching is case-insensitive. "Crawlers MUST use case-insensitive matching to find the group that matches the product token and then obey the rules of the group."
Multiple matching groups are merged. "If there is more than one group matching the user-agent, the matching groups' rules MUST be combined into one group." So two separate User-agent: ExampleBot blocks in the same file produce one combined rule set rather than the second overriding the first.
You get one group, not several. If a group names your token, you use that group and ignore the * group entirely — the wildcard is a fallback, not a baseline that specific rules add to. This surprises people: a crawler with its own group is not also subject to the general rules.
And if nothing matches at all: "If no group matches the product token and there is no group with a user-agent line with the * value, or no groups are present at all, no rules apply."
Matching Is by Specificity, Not Order
The most commonly misstated rule in this whole area.
To evaluate if access to a URI is allowed, a crawler MUST match the paths in "allow" and "disallow" rules against the URI. The matching SHOULD be case sensitive. The matching MUST start with the first octet of the path. The most specific match found MUST be used. The most specific match is the match that has the most octets.
Not first-match-wins. Not last-match-wins. Longest match wins.
The RFC's own example:
User-Agent: foobot
Allow: /example/page/
Disallow: /example/page/disallowed.gif
For example.com/example/page/disallowed.gif, the Disallow line is longer, so it applies — despite appearing second and despite an Allow covering the parent path.
Reverse the order in the file and nothing changes. Order is irrelevant.
Two further rules complete the picture. Ties go to Allow: "If an 'allow' rule and a 'disallow' rule are equivalent, then the 'allow' rule SHOULD be used." And no match means allowed: "If no match is found amongst the rules in a group for a matching user-agent or there are no rules in the group, the URI is allowed."
One more detail worth knowing: "The /robots.txt URI is implicitly allowed", so a file that disallows everything does not disallow itself.
Path matching also involves percent-encoding normalisation. Octets outside ASCII and those in the reserved range "MUST be percent-encoded" before comparison, and a percent-encoded ASCII octet in the URI "MUST be unencoded prior to comparison" unless it is reserved. In practice: use a maintained library rather than writing this yourself.
Status Codes Change Everything
The rules here are precise, frequently ignored, and the difference between two of them is large.
Success. "If the crawler successfully downloads the robots.txt file, the crawler MUST follow the parseable rules."
Redirects. "The crawlers SHOULD follow at least five consecutive redirects, even across authorities." A file reached within five redirects "MUST be fetched, parsed, and its rules followed in the context of the initial authority". Beyond five, a crawler "MAY assume that the robots.txt file is unavailable."
Unavailable — 4xx. "If a server status code indicates that the robots.txt file is unavailable to the crawler, then the crawler MAY access any resources on the server." A 404 means no restrictions.
Unreachable — 5xx. This is the one people get wrong: "If the robots.txt file is unreachable due to server or network errors, this means the robots.txt file is undefined and the crawler MUST assume complete disallow."
A 5xx means stop entirely. Not "carry on as before", not "use the cached copy indefinitely" — complete disallow. The RFC does allow a long-term escape: if the file is undefined "for a reasonably long period of time (for example, 30 days), crawlers MAY assume that the robots.txt file is unavailable... or continue to use a cached copy."
Parsing errors. "Crawlers MUST try to parse each line of the robots.txt file. Crawlers MUST use the parseable rules." A malformed line does not invalidate the file; you use what you can read.
The practical implication for anyone writing a crawler: a temporary outage at the target should pause your crawl, not accelerate it. Getting this backwards means hammering a site that is already struggling.
Caching and Limits
Two operational requirements that catch out home-grown implementations.
Refresh at least daily. "Crawlers MAY cache the fetched robots.txt file's contents... Crawlers SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable."
Fetching the file once when your crawler starts and running for a week is not compliant. Sites change their rules, and a long-running job needs to notice.
Parse at least 500 KiB. "The parsing limit MUST be at least 500 kibibytes." Large sites have large files, and a parser that truncates at an arbitrary smaller size will silently miss rules — which is the worst possible failure mode here, because it produces a crawler that believes it is compliant and is not.
Reading a Real File
Working through a realistic example.
User-agent: *
Disallow: /search
Allow: /search/about
Disallow: /*?sessionid=
Disallow: /*.pdf$
Crawl-delay: 2
User-agent: GPTBot
Disallow: /
User-agent: PartnerBot
Disallow:
Sitemap: https://example.com/sitemap_index.xml
Line by line:
Disallow: /search blocks /search, /search/, /search/results — anything beginning with that string, since matching starts at the first octet and there is no $.
Allow: /search/about is longer, so it wins for that specific path. Longest match, not order.
Disallow: /*?sessionid= uses the wildcard to block any path containing a session parameter, regardless of what precedes it.
Disallow: /*.pdf$ blocks URLs ending in .pdf. Without the $, it would also block /report.pdf.html.
Crawl-delay: 2 is an extension rather than standard, and honouring it is good practice.
User-agent: GPTBot with Disallow: / excludes that crawler entirely. Note that GPTBot gets only this rule — it is not also subject to the * group, so the Crawl-delay above does not apply to it.
User-agent: PartnerBot with an empty Disallow: grants unrestricted access.
Sitemap: points at the URL inventory, which is the most immediately useful line in most files. We covered what to do with it in how to find all pages on a website.
Honouring It in Code
Use a maintained parser. The specificity-matching rule alone is a common source of bugs, and percent-encoding normalisation is worse.
Python: urllib.robotparser is in the standard library and adequate for simple cases; Google's open-source robotstxt parser and its Python bindings implement RFC 9309 precisely. Scrapy has RobotsTxtMiddleware built in and enabled by default in new projects — check that nobody disabled it.
Node: several maintained packages implement the standard.
Go: libraries exist that follow the RFC's matching rules.
Whatever you use, get these four things right:
Refresh every 24 hours, not once at start-up. Treat 5xx as complete disallow, and 404 as no restrictions. Match on your actual product token, and make sure your User-Agent header contains it. Follow up to five redirects, applying the rules in the context of the original host.
And a fifth that is not in the RFC but matters as much: log what you skipped. A crawler that silently excludes half a site because of a rule you did not expect is one where the missing data is discovered weeks later by someone asking why a report looks wrong.
What robots.txt Does Not Cover
Worth being explicit, because people over-read it in both directions.
It says nothing about what you may do with data you fetch. Copyright, database rights and terms of service all apply independently.
It is not permission. The RFC says so directly. An allowed path is a path the site has not asked crawlers to avoid, which is not the same as consent to bulk collection.
It has no rate limit in the standard. Crawl-delay is an extension. Being polite is on you.
It does not distinguish purposes. A file cannot say "indexing yes, AI training no" in standard syntax, though many sites now approximate this by naming specific AI crawler tokens. The EU's text and data mining framework contemplates machine-readable rights reservations, and the mechanisms for expressing them are still settling.
It cannot stop anyone. It is a request. Enforcement is rate limiting, blocking and legal process — which is the practical reason to be the crawler an operator chooses to allow rather than the one they have to stop.
People Also Ask
Is robots.txt legally binding?
Not in itself. RFC 9309 states that its rules "are not a form of access authorization" — it is a request that crawlers are asked to honour. Legal obligations come from terms of service, copyright, database rights and jurisdiction-specific law, all of which apply regardless of what the file says.
Are robots.txt rules matched in order?
No, and this is the most commonly repeated error about it. The specification requires that "the most specific match found MUST be used", where most specific means the most octets. Longest match wins, order is irrelevant, and ties go to Allow.
What happens if robots.txt returns a 404?
The file is "unavailable", and the RFC says the crawler "MAY access any resources on the server". No file means no restrictions. This is different from a server error.
What if robots.txt returns a 500?
The file is "unreachable" and undefined, and the crawler "MUST assume complete disallow". A server error means stop entirely, not carry on. After a long period — the RFC suggests 30 days — a crawler may treat it as unavailable or keep using a cached copy.
How often should I fetch robots.txt?
At least every 24 hours. The RFC says crawlers "SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable". Fetching once at start-up and running for a week is not compliant.
Does a specific user-agent group override the wildcard group?
It replaces it. If a group names your product token, you follow that group and ignore the * group entirely — specific rules do not add to general ones. Two groups naming the same token are merged into one.
What do * and $ mean in robots.txt?
* matches zero or more of any character, and $ marks the end of the match pattern. Both are characters crawlers MUST support. Disallow: /*.pdf$ blocks URLs ending in .pdf; without the $ it would also block /file.pdf.html.
Can I use robots.txt to hide sensitive pages?
No, and doing so makes things worse. The RFC's security section notes that "listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable". Anyone can read the file. Use authentication, which the specification explicitly recommends instead.
Wrapping Up
robots.txt is short, standardised and routinely implemented wrongly. Three rules account for most of the errors.
Longest match wins, not first or last. A Disallow further down the file can override an Allow above it and vice versa, purely on length. A 5xx means complete disallow, which is the opposite of what an intuitive implementation does — a struggling site should get less traffic from you, not the same amount. And the file must be refreshed at least daily, because sites change their minds and a week-old cached copy is not compliance.
Use a maintained parser rather than writing your own, make sure your User-Agent header actually contains the token you expect to be matched on, and log what you skipped so that missing data is a decision rather than a surprise.
And keep the standard's own caveat in view. These rules "are not a form of access authorization" — honouring them makes you a crawler an operator can live with, and it does not settle the separate questions of what the terms allow and what you may do with what you collect.