Robots Exclusion Protocol
draft-rep-wg-topic-00

Document Type Replaced Internet-Draft (individual)
Last updated 2019-07-01
Replaced by draft-koster-rep
Stream (None)
Intended RFC status (None)
Formats
Expired & archived
plain text pdf html bibtex
Stream Stream state (No stream defined)
Consensus Boilerplate Unknown
RFC Editor Note (None)
IESG IESG state Replaced by draft-koster-rep
Telechat date
Responsible AD (None)
Send notices to (None)

This Internet-Draft is no longer active. A copy of the expired Internet-Draft can be found at
https://www.ietf.org/archive/id/draft-rep-wg-topic-00.txt

Abstract

This document standardizes and extends the "Robots Exclusion Protocol" <http://www.robotstxt.org/> method originally defined by Martijn Koster in 1996 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers.

Authors

Martijn Koster (m.koster@greenhills.co.uk)
Gary Illyes (garyillyes@google.com)
Henner Zeller (henner@google.com)
Lizzi Harvey (lizzi@google.com)

(Note: The e-mail addresses provided for the authors of this Internet-Draft may no longer be valid.)