IP Performance Metrics (IPPM) Standard Advancement Testing
draft-ietf-ippm-metrictest-05

[Ballot comment]
Overall this is a fine document.

I concur with those who have commented regarding RFC 2119 language -- in many places I think you could just as easily change the text to use "needs to", "ought to", and "might" with no harm to the meaning. (For example: "The methods proposed here *might* be generally applicable....")

[Ballot comment]
I think that the use of 2119 language in here is superfluous, if not just wrong. I don't see what difference it makes to a reader of this document to say, e.g., "The conditions for which the ADK test has been passed with the specified confidence level must be documented" instead of "MUST be documented."

State changed to In Last Call from Last Call Requested.

The following Last Call Announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Subject: Last Call:  (IPPM standard advancement testing) to BCP


The IESG has received a request from the IP Performance Metrics WG (ippm)
to consider the following document:
- 'IPPM standard advancement testing'
  as a BCP

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2011-12-15. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

This document includes a normative reference to RFC 2330, which is an
Informational RFC, however, RFC 2330 has been used as a normative
reference in several other IPPM working group documents, though some
predate the requirement to split normative and informative references.  One
example of an existing normative reference to RFC 2330 is found in RFC
6049.  The IESG is particularly interested in determining whether the
community considers RFC 2330 sufficiently mature to serve as a normative
reference for standards track and BCP publications.



The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-ippm-metrictest/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-ippm-metrictest/


No IPR declarations have been submitted directly on this I-D.

[Ballot discuss]
RFC 6410 reduced the number of levels to two: "Proposed Standard" and "Internet Standard".  Has this draft been reviewed keeping RFC 6410 in mind?  I caught a couple of required changes in s1 and s3.5.  Maybe in s1:

OLD:

beyond the Proposed Standard level,

New:

to the Internet Standard level,

Maybe in s3.5:

OLD:

advanced to Draft Standard or Internet Standard

NEW:

advanced to Internet Standard

[Ballot discuss]


cleared




RFC 6410 reduced the number of levels to two: "Proposed Standard" and "Internet Standard".  Has this draft been reviewed keeping RFC 6410 in mind?  I caught a couple of required changes in s1 and s3.5.  Maybe in s1:

OLD:

beyond the Proposed Standard level,

New:

to the Internet Standard level,

Maybe in s3.5:

OLD:

advanced to Draft Standard or Internet Standard

NEW:

advanced to Internet Standard




According to the IETF's TLP you need to put:



/*

  Copyright (c) 2011 IETF Trust and the persons identified
  as authors of the code. All rights reserved.

  Redistribution and use in source and binary forms, with
  or without modification, is permitted pursuant to, and subject
  to the license terms contained in, the Simplified BSD License
  set forth in Section 4.c of the IETF Trust’s Legal Provisions
  Relating to IETF Documents (http://trustee.ietf.org/license-info).

*/

code goes here



in Annex B.  Great choice for the code as there's no copyright issue with NIST.



[Ballot comment]
I support each of Dan's discuss points, and with Sean's observation about code components.

There are several places where 2119 keywords are used in non-meaningful ways. It would strengthen
the document to review and remove the instances that are not truly needed. (The use of RECOMMENDED
at the top of page 10 is a particularly strong example.)

When you update the document to take RFC 6410 into account, please note
that while interop reports are no longer required in general, if you have consensus
to do so, you can still require them for advancing these metric definitions. If you
chose not to require them, please consider text strongly encouraging them.

There are several places where you REQUIRE something "whenever possible". That
leaves a lot of vagueness in the specification (is "it costs too much" a reasonable excuse for
"not possible"?). Please consider removing the exception clause, or at least explicitly discussing
what to do or what it means when meeting that requirement is not possible.

The first paragraph of 3.1 (stating an implementation MUST implement all the MUSTs in a specification)
is vacuous - please remove it.

Or perhaps the intent of the whole section was to clarify what's required to be reported in an
implementation report and you were asking for an explicit statement that the implementation
implemented all the MUSTs - if so, please clarify. That would be consistent with the second paragraph
of 3.1 which appears to be asking that the report document which options were supported in "sufficient
detail". But what defines sufficient, and for what purpose? Please consider continuing that sentence
("in sufficient detail to ").

Please consider addressing the following nits.

The string "state of the art" isn't adding anything useful to the text - please remove it.

The third paragraph of section 3.5 does not parse - should "by" be deleted?

Why is there a link to xml.resource.org in the first paragraph of Appendix A.

[Ballot comment]
I agree with the DISCUSS comments of others:

1) This should be a BCP, not Standards Track.
2) This should be updated to take into account RFC 6410.

I also think that the use of 2119 language in here is superfluous, if not just wrong. I don't see what difference it makes to a reader of this document to say, e.g., "The conditions for which the ADK test has been passed with the specified confidence level must be documented" instead of "MUST be documented."

[Ballot comment]
I have the same question Dan has about BCP vs Standards Track.

There's a couple of extra references that are unused:

== Unused Reference: 'RFC2680' is defined on line 1070, but no explicit
      reference was found in the text

== Unused Reference: 'RFC2681' is defined on line 1073, but no explicit
      reference was found in the text

== Unused Reference: 'RFC4719' is defined on line 1094, but no explicit
      reference was found in the text

Dan already pointed out the downref issue.

[Ballot comment]
I have the same question Dan has about BCP vs Standards Track.

Are you planning on removing the summary of changes to the draft?

There's a couple of extra references that are unused:

== Unused Reference: 'RFC2680' is defined on line 1070, but no explicit
      reference was found in the text

== Unused Reference: 'RFC2681' is defined on line 1073, but no explicit
      reference was found in the text

== Unused Reference: 'RFC4719' is defined on line 1094, but no explicit
      reference was found in the text

Dan already pointed out the downref issue.

[Ballot discuss]


This is a discuss-discuss that I think we should talk about on the call:

So isn't this draft updating RFC 2026?  It says 2026 defines some rules, but they're kind of hard to apply to our RFCs, so here's our process to determine if they should progress to Internet Standard.




RFC 6410 reduced the number of levels to two: "Proposed Standard" and "Internet Standard".  Has this draft been reviewed keeping RFC 6410 in mind?  I caught a couple of required changes in s1 and s3.5.  Maybe in s1:

OLD:

beyond the Proposed Standard level,

New:

to the Internet Standard level,

Maybe in s3.5:

OLD:

advanced to Draft Standard or Internet Standard

NEW:

advanced to Internet Standard




According to the IETF's TLP you need to put:



/*

  Copyright (c) 2011 IETF Trust and the persons identified
  as authors of the code. All rights reserved.

  Redistribution and use in source and binary forms, with
  or without modification, is permitted pursuant to, and subject
  to the license terms contained in, the Simplified BSD License
  set forth in Section 4.c of the IETF Trust’s Legal Provisions
  Relating to IETF Documents (http://trustee.ietf.org/license-info).

*/

code goes here



in Annex B.  Great choice for the code as there's no copyright issue with NIST.



[Ballot comment]
1. The text about changes between different I-D versions needs to be taken out in the final version of the document.

2. Are [morton-advance-metrics] and [morton-advance-metrics-01] previous versions of the same document? In any case it does not make sense to include two versions of the same document as references.

3. page 14:

> For example, if 802.1Q
  Ethernet Virtual LANs (VLAN) are applied

Drop 'Ethernet' - the Virtual LANs are not Ethernet-specific

4. runing idnits points to a number of unused references

[Ballot discuss]
This is a very useful document, but some clarifications are needed before I can approve it

1. In section 2 the following statement is being made:

> Metric
  specification options chosen by implementors have to be documented.
  It is REQUIRED to use identical implementation options wherever
  possible for any test proposed here. 

I am questioning the reason for requiring the use of the same implementation options in testing. In real life different implementations may pick different implenmentation options. It looks to me that a stable and interoperable metrics definition would yield consistent results even if different implementation options are being chosen.

2. In Section 3.3

> In some cases, a goodness of fit test may not be possible or show
  disappointing results.  To clarify the difficulties arising from
  different implementation options, the individual options picked for
  every compared implementation SHOULD be documented in sufficient
  detail.  Based on this documentation, the underlying metric
  specification should be improved before it is promoted to a standard.

I believe that test needs to be added here to clarify what 'improve' means. If we deal only with clarification of the differences between implementation options, or even exclusion of one or more of the options from the specifications the advancement on the standards track may still be possible. However if the 'improvement' changes the definition or adds new options recycling at Proposed Standard level is necessary.

3. Why is this document Standards Track and not BCP?

4. idnits indicates two downrefs to RFC 2330 and RFC 4459 (both Informational) - where these Last Called?

[Ballot comment]
I have just one small editorial comment.  I can't parse this test from
Section 2; it might need some clarification:

  o  The error induced by the sample size must be small enough to
      minimize its influence on the test result.  This may have to be
      respected, especially if two implementations measure with
      different average probing rates.

State changed to In Last Call from Last Call Requested.

The following Last Call Announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Subject: Last Call:  (IPPM standard advancement testing) to Proposed Standard


The IESG has received a request from the IP Performance Metrics WG (ippm)
to consider the following document:
- 'IPPM standard advancement testing'
  as a Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2011-10-24. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This document specifies tests to determine if multiple independent
  instantiations of a performance metric RFC have implemented the
  specifications in the same way.  This is the performance metric
  equivalent of interoperability, required to advance RFCs along the
  standards track.  Results from different implementations of metric
  RFCs will be collected under the same underlying network conditions
  and compared using state of the art statistical methods.  The goal is
  an evaluation of the metric RFC itself, whether its definitions are
  clear and unambiguous to implementors and therefore a candidate for
  advancement on the IETF standards track.




The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-ippm-metrictest/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-ippm-metrictest/


No IPR declarations have been submitted directly on this I-D.

(1.a) Who is the Document Shepherd for this document? Has the
Document Shepherd personally reviewed this version of the
document and, in particular, does he or she believe this
version is ready for forwarding to the IESG for publication?

Document sheperd is Henk Uijterwaal.

(1.b) Has the document had adequate review both from key WG members
and from key non-WG members?

Yes, the document was reviewed by about 10 people in the group, including
the AD at the time. Many of them provided comments, the people with the most
extensive comments are listed in the acknowledgement section.

Part of the work is based on a thesis by Matthias Wieser. His thesis has
been reviewed by his supervisor.


(1.c) Does the Document Shepherd have concerns that the document
needs more review from a particular or broader perspective,

No.

(1.d) Does the Document Shepherd have any specific concerns or
issues with this document that the Responsible Area Director
and/or the IESG should be aware of?

No, there are no such concerns.

(1.e) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with
others being silent, or does the WG as a whole understand and
agree with it?

Good, the topic has been under discussion for a long time and there is
general consensus on the document. Besides the people contributing and
reviewing, the topic is well understood in the WG.


(1.f) Has anyone threatened an appeal or otherwise indicated extreme
discontent?

No.

(1.g) Has the Document Shepherd personally verified that the
document satisfies all ID nits?

There are 2 references to framework documents (RFC2330 and 4459). The
framework documents are published as informational, which is a bit strange
as these are basic building blocks for this document.

There are a few unused references, these can be removed by the editor.


(1.h) Has the document split its references into normative and
informative?

Yes
Are there normative references to documents that
are not ready for advancement or are otherwise in an unclear
state?

No

(1.i) Has the Document Shepherd verified that the document IANA
consideration section exists and is consistent with the body
of the document?

Yes, it is empty, so please remove it upon publication.

(1.j) Has the Document Shepherd verified that sections of the
document that are written in a formal language, such as XML
code, BNF rules, MIB definitions, etc., validate correctly in
an automated checker?

I checked that the code is syntaxtically correct on a unix box.

(1.k) The IESG approval announcement includes a Document
Announcement Write-Up. Please provide such a Document
Announcement Write-Up? Recent examples can be found in the
"Action" announcements for approved documents. The approval
announcement contains the following sections:

Technical Summary
This document specifies tests to determine if multiple independent
instantiations of a performance metric RFC have implemented the
specifications in the same way. This is the performance metric
equivalent of interoperability, required to advance RFCs along the
standards track. Results from different implementations of metric
RFCs will be collected under the same underlying network conditions
and compared using state of the art statistical methods. The goal is
an evaluation of the metric RFC itself, whether its definitions are
clear and unambiguous to implementors and therefore a candidate for
advancement on the IETF standards track.


Working Group Summary
The normal WG process was followed and the document has been discussed for
several years. The document as it is now, reflects WG consensus, with nothing
special worth noticing.

Document Quality
Good