Robots Exclusion Protocol
draft-rep-wg-topic-00
| Document | Type | Replaced Internet-Draft (individual) | |
|---|---|---|---|
| Last updated | 2019-07-01 | ||
| Replaced by | draft-koster-rep | ||
| Stream | (None) | ||
| Intended RFC status | (None) | ||
| Formats |
Expired & archived
pdf
htmlized
bibtex
|
||
| Stream | Stream state | (No stream defined) | |
| Consensus Boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | Replaced by draft-koster-rep | |
| Telechat date | |||
| Responsible AD | (None) | ||
| Send notices to | (None) | ||
https://www.ietf.org/archive/id/draft-rep-wg-topic-00.txt
Abstract
This document standardizes and extends the "Robots Exclusion Protocol" <http://www.robotstxt.org/> method originally defined by Martijn Koster in 1996 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers.
Authors
Martijn Koster
(m.koster@greenhills.co.uk)
Gary Illyes
(garyillyes@google.com)
Henner Zeller
(henner@google.com)
Lizzi Harvey
(lizzi@google.com)
(Note: The e-mail addresses provided for the authors of this Internet-Draft may no longer be valid.)