Robots Exclusion Protocol
draft-koster-rep-12

# Éric Vyncke, INT AD, review of # Éric Vyncke, INT AD, review of draft-koster-rep-09
CC @evyncke

Thank you for the work put into this document. As a webmaster for some minor websites, this robots.txt is simple but efficient.

Please find below some non-blocking COMMENT points (but replies would be appreciated even if only for my own education).

Special thanks to Ted Hardie for the shepherd's detailed write-up even if I had liked to have the justification of the intended status. 

I hope that this helps to improve the document,

Regards,

-éric

## COMMENTS

### Section 1.1

As always, I personally find weird when an informational document uses BCP14 normative language; but the IESG has agreed on this use of BCP14. So, let it be ;-)

### Section 2.3

Suggest to remove the "http" example.

### Section 2.3.1.2

In `The server may respond to a robots.txt fetch request` should this be a `MAY`(to contradict my previous comment on the use of BCP14)

### Section 2.5

I must admit that this is the first time that I read "kibibyte"... Please use normal units or add a reference, e.g., https://simple.wikipedia.org/wiki/Kibibyte (BTW, thank you, I have learned a new word!)

The sentence is rather hard to parse, could it be replaced by "Crawlers MUST be able to parse at least 500 KiB)" ?

## Notes

This review is in the ["IETF Comments" Markdown format][ICMF], You can use the
[`ietf-comments` tool][ICT] to automatically convert this review into
individual GitHub issues. 

[ICMF]: https://github.com/mnot/ietf-comments/blob/main/format.md
[ICT]: https://github.com/mnot/ietf-comments

Thank you for this work which formally standardizes long standing Internet practice.

** I recommend an editorial pass to consistently use client, crawlers or robots.

** Is robots.txt practically used beyond the HTTPS, HTTP and FTP schemes?  If not, it might be worth writing an applicability statement around these limited protocols, suggesting but not committing to broader use, and then be much more specific in Section 2.3 about error handling.

** Section 1.
   This document specifies the rules
   originally defined by the "Robots Exclusion Protocol" [ROBOTSTXT]
   that crawlers are expected to obey when accessing URIs.

“… expected to obey when accessing URIs” seems too strong.  Maybe “… are requested to honor.”  There is no enforcement mechanism or even requirement that a crawler even check for robots.txt.

** Section 2.2.2.
Crawlers SHOULD allow the following special characters.

-- The subject of the normative language seems reversed.  Isn’t this really (roughly) “the robots.txt file supports the following additional grammar”?  It is in the ABNF.

-- In a sense, the crawler is free to ignore anything the robots.txt says.  Why is providing flexibility with a SHOULD any different than the rest of the content?

** Section 2.2.4.  I’m not familiar with the operational practices that add additional “protocols” into robots.txt.  How would a ‘sitemap’ file be added to robots.txt?  Sitemaps is an XML blob.  It would make for a non-conforming robot.txt and an invalid XML.  

** Section 2.3.1.3 and 2.3.1.4.  These sections map neatly to HTTP error codes.  This protocol is scoped as being generic.  What is the expected behavior if a protocol doesn’t distinguish between unavailable and unreachable?

** Section 2.3.1.3 and 2.3.1.4.  Could equivalent error semantics be described for FTP.

** Section 2.3.1.4. 

   For
   other undefined status codes, the crawler MUST assume the robots.txt
   is unreachable.

Are those other HTTP code, or is this sentence a reference to other protocols?

** Section 2.3.1.4

   If the robots.txt is undefined for a reasonably long period of time
   (for example, 30 days), clients MAY assume the robots.txt is
   Unavailable

What does this persistent “unavailability” suggest about the crawler’s behavior.  Since this is a “MAY”, does it continue to assume “complete disallow” or follow Section 2.3.1.3 behavior of “complete allow”?

** Section 2.3.1.5.

(a)   Crawlers SHOULD try to parse each line of the robots.txt file.

(b)   Crawlers MUST use the parseable rules.

I appreciate that there is context from the Section header, but as written these appear conditional.  Since (a) is a SHOULD, then (b) won’t always be true since I can choose not to parse certain lines.  Is there a reason why (a) can’t be a MUST?

** Section 3.  Recommend acknowledging the obvious that this protocol is for crawlers willing to following directives in the file.  Also, paths, not URIs, are in the file.

OLD
   The Robots Exclusion Protocol is not a substitute for more valid
   content security measures.  Listing URIs in the robots.txt file
   exposes the URI publicly and thus makes the URIs discoverable.

NEW
The Robots Exclusion Protocol provides no access control and is advisory in nature to conforming crawlers. Non-compliant crawlers can avoid accessing the robots.txt file or ignore directives in it.  Listing paths in the robots.txt file exposes them publicly and thus makes the discoverable.

Thanks for the draft.

In writing this comment I'm going to state up front that what's in here is normally pretty far from my field of expertise - so please excuse any confusion or inaccuracies - happy to be corrected.

In section 2.1:

 Crawlers MUST find the group that matches the product token exactly,
   and then obey the rules of the group. 

To me this seems to conflict with:

The matching of the product token MUST be case-insensitive. 

I was also a little confused here - since I noticed in 2.2 it states: 

product-token = identifier / "*"

This would seem to imply a wildcard - if I'm reading this correctly, which also seems to not correlate with an exact match.

Thank you for the work on this document.

Many thanks to Todd Herr for his ART ART review: https://mailarchive.ietf.org/arch/msg/art/oaAMYwH7lWH4Xty01nSh1qSu7XU/.

I wonder if the reference to RFC 2616 should not be replaced with RFC 9111, or is there a reason to keep the old obsoleted reference?

Francesca

# GEN AD review of draft-koster-rep-09

CC @larseggert

## Nits

All comments below are about very minor potential issues that you may choose to
address in some way - or ignore - as you see fit. Some were flagged by
automated tools (via https://github.com/larseggert/ietf-reviewtool), so there
will likely be some false positives. There is no need to let me know what you
did with these suggestions.

### Outdated references

Reference `[RFC2616]` to `RFC2616`, which was obsoleted by `RFC7235`,
`RFC7234`, `RFC7233`, `RFC7232`, `RFC7231`, and `RFC7230` (this may be on
purpose).

### Grammar/style

#### Section 2.2.1, paragraph 5
```
he URI. The matching SHOULD be case sensitive. The most specific match found
                               ^^^^^^^^^^^^^^
```
This word is normally spelled with a hyphen.

#### Section 2.2.1, paragraph 7
```
 and disallow rule is equivalent, the allow SHOULD be used. If no match is f
                                  ^^^^^^^^^
```
After "the", the verb "allow" doesn't fit. Is "allow" spelled correctly? If
"allow" is the first word in a compound adjective, use a hyphen between the two
words. Using the verb "allow" as a noun may be non-standard.

## Notes

This review is in the ["IETF Comments" Markdown format][ICMF], You can use the
[`ietf-comments` tool][ICT] to automatically convert this review into
individual GitHub issues. Review generated by the [`ietf-reviewtool`][IRT].

[ICMF]: https://github.com/mnot/ietf-comments/blob/main/format.md
[ICT]: https://github.com/mnot/ietf-comments
[IRT]: https://github.com/larseggert/ietf-reviewtool

Hi,

Thanks for this document, it looks like a useful update and it was a pretty easy read.  I just have a few comments/clarifications for the authors to consider that may improve this document:

1.
   Crawlers MUST find the group that matches the product token exactly,
   and then obey the rules of the group.

   The matching of the product token MUST be case-insensitive

For me, matching exactly and also being case-insensitive seem to slightly contradict each other.  Would the text be more clear if these were combined into a single sentence?

2.
   The most specific match is the match that has the most octets.

Should the document specify that the match is from the root of the path?  Otherwise it implies that any substring match of the path may be acceptable.

3. For the examples:

Would it be helpful to include a line that has no space between the allow: and the match (e.g., allow:/foobar)?

4. 
  Disallow: /*.gif$

Would this also be allowed without the initial '/' character?  Is this worth clarifying/specifying?

Thanks,
Rob

Thanks for working on this specification. I have skimmed through the document due to shortage of my time and haven't notices any transport protocol related issues.