
How to Read a robots.txt File: A Complete Guide
`robots.txt` stopped being a convention in 2022. It is now a standard, [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.txt), with defined matching semantics and required behaviour. Most explanations of it are older than that and get two things wrong: they say rules are matched in order (they are not) and they say a missing file means the same as an unreachable one (it does not). Here is what the specification actually says, and how to read a real file correctly.

Geonode Team
Published: 2026-09-02
