robots.txt

Definition

robots.txt is a plain-text file at the root of a website in which the owner states which paths automated clients are asked not to request. It is a published convention, not an access control: it expresses the owner’s wishes and does not enforce them.

In plain terms

The format was standardised as RFC 9309. Rules are grouped by the name of the client they address (User-agent) and list paths with Allow and Disallow. A well-behaved crawler fetches the file first and follows it.

Two limits are worth remembering. It speaks about paths, not about how fast you may go, so request rates are a separate matter (see Rate Limiting). And it says nothing about what you may do with the data: the site’s terms and any licence (Data License) are separate documents. It is also not security: never rely on it to hide anything.

Example

A small, invented file:

User-agent: *
Disallow: /private/
Disallow: /search

User-agent: ExampleBot
Disallow: /

Common misunderstandings

• “robots.txt protects private pages.” It is only a request; anyone can ignore it, and listing a path in it can even advertise the path.
• “If robots.txt allows it, I may use the data however I like.” Permission to fetch a page is not permission to store, derive from or publish its content.
• “It limits how often I may request.” It covers paths; a delay directive exists in some implementations but is not part of the standard.