Robots Exclusion Protocol
RFC 9309
Revision differences
Document history
| Date | By | Action |
|---|---|---|
|
2026-05-20
|
(System) | Changed metadata: changed keywords to '['robot', 'crawler', 'robots.txt']' from '[]' |
|
2026-05-20
|
(System) | Metadata update from RFC Editor |
|
2026-05-20
|
(System) | Changed author "L. Sassman": changed name from "Lizzi Sassman" to "L. Sassman", cleared country (was "Switzerland") |
|
2026-05-20
|
(System) | Changed author "H. Zeller": changed name from "Henner Zeller" to "H. Zeller", cleared country (was "USA") |
|
2026-05-20
|
(System) | Changed author "G. Illyes": changed name from "Gary Illyes" to "G. Illyes", cleared country (was "Switzerland") |
|
2026-05-20
|
(System) | Changed author "M. Koster": changed name from "Martijn Koster" to "M. Koster", changed affiliation from "Stalworthy Computing, Ltd." to "None (in header)", cleared country (was … Changed author "M. Koster": changed name from "Martijn Koster" to "M. Koster", changed affiliation from "Stalworthy Computing, Ltd." to "None (in header)", cleared country (was "United Kingdom") |
|
2026-05-20
|
(System) | Metadata update from RFC Editor |
|
2022-09-12
|
(System) | Received changes through RFC Editor sync (added Errata tag) |
|
2022-09-09
|
(System) | Received changes through RFC Editor sync (created alias RFC 9309, changed abstract to 'This document specifies and extends the "Robots Exclusion Protocol" method originally … Received changes through RFC Editor sync (created alias RFC 9309, changed abstract to 'This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.', changed pages to 12, changed standardization level to Proposed Standard, changed state to RFC, added RFC published event at 2022-09-09, changed IESG state to RFC Published) |
|
2022-09-09
|
(System) | RFC published |