Robots Exclusion Protocol

Document Type Replaced Internet-Draft (individual)
Authors Martijn Koster  , Gary Illyes  , Henner Zeller  , Lizzi Harvey 
Last updated 2019-07-01
Replaced by draft-koster-rep
Stream (None)
Intended RFC status (None)
Expired & archived
plain text htmlized pdfized bibtex
Stream Stream state (No stream defined)
Consensus Boilerplate Unknown
RFC Editor Note (None)
IESG IESG state Replaced by draft-koster-rep
Telechat date
Responsible AD (None)
Send notices to (None)

This Internet-Draft is no longer active. A copy of the expired Internet-Draft can be found at


This document standardizes and extends the "Robots Exclusion Protocol" <> method originally defined by Martijn Koster in 1996 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers.


Martijn Koster (
Gary Illyes (
Henner Zeller (
Lizzi Harvey (

(Note: The e-mail addresses provided for the authors of this Internet-Draft may no longer be valid.)