Last Call Review of draft-koster-rep-08
review-koster-rep-08-intdir-lc-weber-2022-05-29-00

Moin!

I am an assigned INT directorate reviewer for draft-koster-rep.
These comments were written primarily for the benefit of the Internet Area
Directors. Document editors and shepherd(s) should treat these comments just
like they would treat comments from any other IETF contributors and resolve
them along with any other Last Call comments that have been received. For more
details on the INT Directorate, see
https://datatracker.ietf.org/group/intdir/about/

While the document technically defines the content of the robots.txt files it
could do a better job in describing with examples the semantic of how robots
interpret them. Especially in 2.2.1 the notation of "Crawlers MUST find the
group that matches the product token exactly" should be better explained. I
assume it does not mean being fully equal but instead a substring match in the
User-Agent Header, so in the example would also match a http user agent of
ExampleBotnet/1.2. Is that understanding correct at least?

Also the examples in 5 seem a lot more arbirtary than what the ROBOTSTXT
website has and it should explain all the outcomes, e.g in 5.1  it would allow
access to all crawlers and all paths, but the foobot, barbot and bazbot
/example/disallowed.gif. An example with a * group would be better and more
realistic.

So long
-Ralf