Guidance on Interoperation and Implementation Reports for Advancement to Draft Standard
RFC 5657

[Ballot comment]
1. Section 3 - Format

> The format of implementation and interoperability reports MUST be
  ASCII text with line-breaks for readability.

This is slightly inconsistent with the recommendation for formating text in the Internet-Drafts at http://www.ietf.org/ietf/1id-guidelines.html

> Internet-Drafts must be in ASCII. No 8-bit characters are currently allowed. If you need to include code points, a suggestion might be to use the unicode convention: U+XXXX, where X is a hexadecimal digit

2. s/Author  Identify the author of the report/Author: Identify the author of the report/

[Ballot discuss]
I find this document extremely useful and well written, and I plan to vote Yes after the DISCUSS issues are addressed:
The second and more difficult issue in my DISCUSS was resolved by the second IETF Last Call. There is still one issue left, which can probably be addressed with an RFC Editor note (I believe it was agreed):

Section 5.5 should mention MIB browsers used to test MIB modules. I believe that this is important because testing by means of generic SNMP clients (MIB browsers) that different agents implementations return the same set of values for objects when running in similar conditions is the common practice for implementation reports of documents that define MIB modules. Examples may be the agents implementations of RMON (RFC 2819) and RMON-2 (RFC 4502)

[Ballot comment]
1. Section 3 - Format

> The format of implementation and interoperability reports MUST be
  ASCII text with line-breaks for readability.

This is slightly inconsistent with the recommendation for formating text in the Internet-Drafts at http://www.ietf.org/ietf/1id-guidelines.html

> Internet-Drafts must be in ASCII. No 8-bit characters are currently allowed. If you need to include code points, a suggestion might be to use the unicode convention: U+XXXX, where X is a hexadecimal digit

2. s/Author  Identify the author of the report/Author: Identify the author of the report/

[Ballot discuss]
I find this document extremely useful and well written, and I plan to vote Yes after the DISCUSS issues are addressed:

1. Section 5.5 should mention MIB browsers used to test MIB modules. I believe that this is important because testing by means of generic SNMP clients (MIB browsers) that different agents implementations return the same set of values for objects when running in similar conditions is the common practice for implementation reports of documents that define MIB modules. Examples may be the agents implementations of RMON (RFC 2819) and RMON-2 (RFC 4502)

2. Section 5.6:

>  Optional features need not be shown to be implemented everywhere.
  However, they do need to be implemented somewhere, and more than one
  independent implementation is required.  If an optional feature does
  not meet this requirement, the implementation report must say so and
  explain why the feature must be kept anyway versus being evidence of
  a poor-quality standard.

This seems to be in conflict with RFC 2026:

>  The requirement for at least two independent and interoperable
  implementations applies to all of the options and features of the
  specification.  In cases in which one or more options or features
  have not been demonstrated in at least two interoperable
  implementations, the specification may advance to the Draft Standard
  level only if those options or features are removed.

Is this relaxation intentional? If it is than I believe that this should be called explicitly in this document and also documented by an IESG note in the respective approved Draft Standard RFC.

[Ballot comment]
Section 5.3., paragraph 0:
> 5.3.  Schemas, languages and formats

  As another example for this section, you may also want to add an
  informative reference to draft-bradner-metricstest, which is the basis
  of a (to-be-submitted) work item in IPPM that will define how
  performance metrics advance along the standards track.

[Ballot discuss]
INTRODUCTION, paragraph 14:
>    Advancing a protocol to Draft Standard requires documentation of the
>    interoperation and implementation of the protocol.

  DISCUSS: It's not only when going PS->DS, it's also from DS->IS where
  we need an interop report (see the text you quoted from RFC2026 in
  Section 1). This entire document, however, is written with a focus on
  PS->DS. I'd prefer if this document were rephrased to also explicitly
  cover the DS->IS case. (Alternatively, it needs to be made clear that
  it is *not* covering that, but then what is?)

[Ballot comment]
1. Section 3 - Format

> The format of implementation and interoperability reports MUST be
  ASCII text with line-breaks for readability.

This is slightly inconsistent with the recommendation for formating text in the Internet-Drafts at http://www.ietf.org/ietf/1id-guidelines.html

> Internet-Drafts must be in ASCII. No 8-bit characters are currently allowed. If you need to include code points, a suggestion might be to use the unicode convention: U+XXXX, where X is a hexadecimal digit

2. s/Author  Identify the author of the report/Author: Identify the author of the report/

[Ballot discuss]
I find this document extremely useful and well written, and I plan to vote Yes after the DISCUSS issues are addressed:

1. Neither this document nor RFC 2026 make clear whether an interoperation report is a mandatory requirement for advancing an RFC from Proposed to Draft. This document just says:

>  This documentation should be submitted to the Area Director with the protocol action request.(see Section 6)

Are there cases when the IESG can decide that such a documentation is not needed? I think that we should be clear on this respect.

2. Section 5.5 should mention MIB browsers used to test MIB modules. I believe that this is important because testing by means of generic SNMP clients (MIB browsers) that different agents implementations return the same set of values for objects when running in similar conditions is the common practice for implementation reports of documents that define MIB modules. Examples may be the agents implementations of RMON (RFC 2819) and RMON-2 (RFC 4502)

3. Section 5.6:

>  Optional features need not be shown to be implemented everywhere.
  However, they do need to be implemented somewhere, and more than one
  independent implementation is required.  If an optional feature does
  not meet this requirement, the implementation report must say so and
  explain why the feature must be kept anyway versus being evidence of
  a poor-quality standard.

This seems to be in conflict with RFC 2026:

>  The requirement for at least two independent and interoperable
  implementations applies to all of the options and features of the
  specification.  In cases in which one or more options or features
  have not been demonstrated in at least two interoperable
  implementations, the specification may advance to the Draft Standard
  level only if those options or features are removed.

Is this relaxation intentional? If it is than I believe that this should be called explicitly in this document and also documented by an IESG note in the respective approved Draft Standard RFC.

[Ballot comment]
It would be nice if the document talks about early implementations (as discussed during the IETF LC) and how they my not match the final document being progressed. But I don't think this is a blocker for the document.

Proto writeup: "shepherd" is Tim Polk.

  (1.a)  Who is the Document Shepherd for this document?  Has the
      Document Shepherd personally reviewed this version of the document
      and, in particular, does he or she believe this version is ready
      for forwarding to the IESG for publication?

The sponsoring AD, Tim Polk, has decided to to fulfill the shepherd's role
for this document.  The sponsoring AD has personally reviewed this version
of the document and is confident it is ready for forwarding to the IESG
for publication.

  (1.b)  Has the document had adequate review both from key members of
      the interested community and others?  Does the Document Shepherd
      have any concerns about the depth or breadth of the reviews that
      have been performed?

The document has received extensive review from members of the IETF
community who are interested in progressing documents up the maturity
ladder, including several IESG members.  The breadth and depth of review
justifies IETF Last Call.

  (1.c)  Does the Document Shepherd have concerns that the document
      needs more review from a particular or broader perspective, e.g.,
      security, operational complexity, someone familiar with AAA,
      internationalization or XML?

No.

  (1.d)  Does the Document Shepherd have any specific concerns or
      issues with this document that the Responsible Area Director
      and/or the IESG should be aware of?  For example, perhaps he or
      she is uncomfortable with certain parts of the document, or has
      concerns whether there really is a need for it.  In any event, if
      the interested community has discussed those issues and has
      indicated that it still wishes to advance the document, detail
      those concerns here.

No, I am comfortable with all parts of this document.

  (1.e)  How solid is the consensus of the interested community behind
      this document?  Does it represent the strong concurrence of a few
      individuals, with others being silent, or does the interested
      community as a whole understand and agree with it?

The interested community is strongly supportive of this document, but it
may represent strong concurrence of a few individuals.  While the document
was called out in the IETF Last Call for promotion of RFC 3852 to Draft
Standard, it is unclear how many people have read it.  IETF Last Call is
expected to help publicize the document and identify any differences of
opinion within the larger IETF community.

  (1.f)  Has anyone threatened an appeal or otherwise indicated extreme
      discontent?  If so, please summarise the areas of conflict in
      separate email messages to the Responsible Area Director.  (It
      should be in a separate email because this questionnaire is
      entered into the ID Tracker.)

No.

  (1.g)  Has the Document Shepherd personally verified that the
      document satisfies all ID nits?  (See
      http://www.ietf.org/ID-Checklist.html and
      http://tools.ietf.org/tools/idnits/).  Boilerplate checks are not
      enough; this check needs to be thorough.  Has the document met all
      formal review criteria it needs to, such as the MIB Doctor, media
      type and URI type reviews?

ID-nits identified four possible issues, three require attention.  There were
no other issues in the opinion of the shepherd.

(a) Need either 5378 boilerplate or disclaimer
(b) No reference to 2119 but uses conformance language.
(c) No IANA Considerations section; should be as follows:

X. IANA Considerations

  This document has no actions for IANA.

[Note: The fourth issue was a reference to an obsolete RFCC.  I reviewed the
reference to obsolete RFC 4234 and confirmed it is correct and appropriate.]

  (1.h)  Has the document split its references into normative and
      informative?  Are there normative references to documents that are
      not ready for advancement or are otherwise in an unclear state?
      If such normative references exist, what is the strategy for their
      completion?  Are there normative references that are downward
      references, as described in [RFC3967]?  If so, list these downward
      references to support the Area Director in the Last Call procedure
      for them [RFC3967].

All references are informative.

  (1.i)  Has the Document Shepherd verified that the document IANA
      consideration section exists and is consistent with the body of
      the document?  If the document specifies protocol extensions, are
      reservations requested in appropriate IANA registries?  Are the
      IANA registries clearly identified?  If the document creates a new
      registry, does it define the proposed initial contents of the
      registry and an allocation procedure for future registrations?
      Does it suggested a reasonable name for the new registry?  See
      [I-D.narten-iana-considerations-rfc2434bis].  If the document
      describes an Expert Review process has Shepherd conferred with the
      Responsible Area Director so that the IESG can appoint the needed
      Expert during the IESG Evaluation?

Does not exist, but no registries or other IANA actions will be needed...

  (1.j)  Has the Document Shepherd verified that sections of the
      document that are written in a formal language, such as XML code,
      BNF rules, MIB definitions, etc., validate correctly in an
      automated checker?

No formal language.

  (1.k)  The IESG approval announcement includes a Document
      Announcement Write-Up.  Please provide such a Document
      Announcement Writeup?  Recent examples can be found in the
      "Action" announcements for approved documents.  The approval
      announcement contains the following sections:

      Technical Summary

        Advancing a protocol to Draft Standard requires documentation of the
        interoperation and implementation of the protocol.  Historic reports
        have varied widely in form and level of content and there is little
        guidance available to new report preparers.  This document updates
        the existing processes and provides more detail on what is
        appropriate in an interoperability and implementation report.

      Working Group Summary

        This is not the product of an IETF working group.  This document
        has received significant review from members of the community, and
        those reviewers have been supportive.  Additional review is needed
        from the broader community to ensure that opposing views have been
        considered.  That review is expected from IETF Last Call.

      Document Quality

        The process described in this document has been executed to create
        an implementation report for RFC 3852.  The person who created that
        report indicated the process was appropriate and more effectively
        achieved the goals of implementation than a report referencing each
        occurrence of RFC 2119 conformance language.  This report did not
        conform precisely to the suggested report format, but in retrospect
        would have been more readable if it had conformed precisely.