Robots Exclusion Protocol
draft-rep-wg-topic-00
| Document | Type |
Replaced Internet-Draft
(individual)
Expired & archived
|
|
|---|---|---|---|
| Authors | Martijn Koster , Gary Illyes , Henner Zeller , Lizzi Sassman | ||
| Last updated | 2019-07-01 | ||
| Replaced by | draft-koster-rep | ||
| RFC stream | (None) | ||
| Intended RFC status | (None) | ||
| Formats | |||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | Replaced by draft-koster-rep | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
This Internet-Draft is no longer active. A copy of the expired Internet-Draft is available in these formats:
Abstract
This document standardizes and extends the "Robots Exclusion Protocol" <http://www.robotstxt.org/> method originally defined by Martijn Koster in 1996 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers.
Authors
Martijn Koster
Gary Illyes
Henner Zeller
Lizzi Sassman
(Note: The e-mail addresses provided for the authors of this Internet-Draft may no longer be valid.)