Paper | Illyes: The case for the Robots Exclusion Protocol
slides-aicontrolws-the-case-for-the-robots-exclusion-protocol-01
| Slides | IAB Workshop on AI-CONTROL (aicontrolws) Team | |
|---|---|---|
| Title | Paper | Illyes: The case for the Robots Exclusion Protocol | |
| Abstract | In 1994, Martijn Koster (a webmaster himself) came up with the idea of robots.txt after automatic clients (crawlers) were overwhelming his site. With more input … In 1994, Martijn Koster (a webmaster himself) came up with the idea of robots.txt after automatic clients (crawlers) were overwhelming his site. With more input from other webmasters, the Robots Exclusion Protocol was born, and it was adopted by search engines and other crawler operators to help website owners manage their server resources easier. It functioned as a de facto standard for over 20 years. |
|
| State | Active | |
| Other versions | ||
| Last updated | 2024-09-09 |
slides-aicontrolws-the-case-for-the-robots-exclusion-protocol-01
Not available as plain text.
Download as PDF.