Guidelines for Considering New Performance Metric Development
draft-ietf-pmol-metrics-framework-12

[Ballot discuss]
(Updating to reflect -10)

1) cleared.

2) cleared.

3) cleared.

5) Section 6.2 is the section of the document that is going to guide reviewers. It's not as clear as it needs to be. For instance, reviewers are being asked to assess a metric according the the qualification "use cases" and that is the only place the string appears in the document. What are you requiring with respect to use cases?

6) cleared.

[Ballot discuss]
(Updating to reflect -10)

1) Section 6.3 is not adding any new process (it is recommending using the directorate framework we already have, asking an AD to set up and use a new team). It could stop with it's first sentence, and that sentence does not need to be restated in this document.  I think you have the same document if you delete this section.  If you agree, please delete it. If you don't agree, the document needs to be more explicit on what the section is trying to achieve.

2) cleared.

3) cleared.

5) Section 6.2 is the section of the document that is going to guide reviewers. It's not as clear as it needs to be. For instance, reviewers are being asked to assess a metric according the the qualification "use cases" and that is the only place the string appears in the document. What are you requiring with respect to use cases?

6) cleared

[Ballot discuss]
I'm picking up Tim's discuss position.

In my opinion, the sections that establish a new directorate and make it part of
the official process require an
"Updates 2026".  In particular, section 6.3
integrates the directorate into the approval process for new work items.
I would
also note that this is a much stronger role than any other directorate has been
given.  Secdir, mib doctors, etc.
have a review role but they are not asked to
approve new work items.

It would be more appropriate to excise all text about the directorate; forming
the directorate and reviewing metrics
related efforts does not require an RFC.
I also suggest that the directorate make itself available for early review
upon
request, but strongly oppose any attempt to establish a gating function.

If the authors do not wish to remove this text or modify these functions, I
believe another Last Call is in order after
adding "updates 2026" to the header,
since I do not believe these changes have community consensus.

[Ballot comment]
In general I think this document is adding some rather weak requirements on process (lots of "SHOULD"s).

I have passed my DISCUSS to Robert, so I am agreeing with his DISCUSS point # 2:

5.4.2.  Normative parts of Performance Metric definition

  The normative part of a Performance Metric definition MUST define at
  least the following:

  (i) Metric Name

  Performance Metric names MUST be unique within the set of metrics
  being defined and MAY be descriptive.

Is there a formal syntax for performance metric names? How are these names to be used (e.g. are they going to be used by IANA to guaranty uniqueness)?


6.4.  Performance Metrics Entity Interaction with other WGs

  The Performance Metrics Entity SHALL work in partnership with the
  related protocol development WG when considering an Internet Draft
  that specifies Performance Metrics for a protocol.  A sufficient
  number of individuals with expertise must be willing to consult on
  the draft.  If the related WG has concluded, comments on the proposal
  should still be sought from key RFC authors and former chairs, or

I think "or" --> "and/or". I.e., there is not reason to choose the former over the latter if both are available.

  from the WG mailing list if it was not closed.

[Ballot comment]
1) "defined in a similar way" is very vague.

2) The language in 5.3.1 stating that measurements need to be non-overlapping and contiguous to be meaningfully composed caused me to pause. It's certainly easier to build meaningful compositions from measurements with those properties, but it's possible to build meaningful compositions when the properties, particularly contiguity, aren't met. 

3) Consider adding a discussion to section 5.4.2 (v) noting that the metric definition should discuss how the metric might vary depending which measurement point is chosen. The time between a SIP request and final response can be significantly different at the UAC and the UAS for example. Also consider noting that these kinds of differences should be taken into account when composing metrics. 

4) In 5.4.3 it would help readers to note that 3550 calls this interarrival jitter. 

5) It's not really clear what you want readers to do with the information in section 5.5. Please consider a short introduction to the section discussing whether these are observations that a metric designer needs to keep in mind, or if you are asking reviewers to ensure a specification says something about these things, or if you had a completely different audience in mind.

[Ballot discuss]
(Editing point 2 to adopt Alexey's discuss)

1) Section 6.3 is not adding any new process (it is recommending using the directorate framework we already have, asking an AD to set up and use a new team). It could stop with it's first sentence, and that sentence does not need to be restated in this document.  I think you have the same document if you delete this section.  If you agree, please delete it. If you don't agree, the document needs to be more explicit on what the section is trying to achieve.

2) I'm not sure Section 5.4.2 (i) is clear about the domain over which it is requiring names to be unique. It could be read to mean "unique within a given document defining metrics". Did you intend it to mean "Unique within the set of standardized metrics for a given protocol"? What mechanisms are you advising be used to ensure this uniqueness?  Should this document advise metric designers to help establish and manage registries per thing being measured, or are you anticipating something more lightweight? (Adopting Alexey's discussion point:) Does there need to be a formal syntax defined for these names?

3) Why did you make a new metric for the example in 5.4.5 instead of exemplifying one that is already well specified?  (Or have I missed the specification? It's related to discussion in 3550 and 3661, but I don't see this defined as a metric there). If it's new, could this section be misconstrued as actually defining the metric? I'd hate to see some future document trying to normatively refer an implementer to this section...

5) Section 6.2 is the section of the document that is going to guide reviewers. It's not as clear as it needs to be. For instance, reviewers are being asked to assess a metric according the the qualification "use cases" and that is the only place the string appears in the document. What are you requiring with respect to use cases?

6) Why didn't Section 6.6 recommend a particular Area?

[Ballot comment]
Agree with everyone else that the process stuff needs to be changed.

> Intended status: BCP

  Agree with Ron - this should not be a BCP.


Section 2.1., paragraph 1:
>    The Performance Metrics Entity is a directorate that coordinates the
>    Performance Metric development in the IETF.

  "Entity" is a weird name for a directorate. Why don't you just call it
  a directorate?


Section 3., paragraph 2:
>    intended to supercede existing working methods within WGs that have

  Nit: s/supercede/supersede/


Section 5.3.1., paragraph 6:
>    In the context of flow records in IP Flow Informatin eXport (IPFIX),

  Nit: s/Informatin/Information/


Section 5.4.3., paragraph 3:
>    definition is to assist implementers to achieve consistents results.

  Nit: s/consistents/consistent/

[Ballot comment]
draft-ietf-pmol-metrics-framework-08

The reference to RFC 2119 appears right after the Abstract. I would place it inside the Terminology section (Section 2 instead).

Section 1.1 uses the present tense to describe events that happened in the past (e.g., "the IETF has two active WGs" or "there is a gap"). This also related to Adrian's comment on the same section. This section could be more explicit about what is describing, which seems to be the creation of the PMOL WG.

When talking about SBCs in Section 5.5.4, you may want to reference RFC 5853

[Ballot discuss]
Section 6 describes a policy to accept new metrics and establishes a directorate. As you know, RFC 5226 defines a set of policies to be applied when something needs to be registered with the IANA. However, in this case new metrics do not seem to be registered with the IANA. So, what is the process trying to prevent? Someone writing a new metrics proposal and referencing this document? (this relates to Ron's discuss). I think the information about what to look for when reviewing proposals for new metrics is useful. Couldn't the document just provide it as information for those who will be reviewing proposals submitted to the IETF and not as a process definition?

[Ballot comment]
6.4.  Performance Metrics Entity Interaction with other WGs

  The Performance Metrics Entity SHALL work in partnership with the
  related protocol development WG when considering an Internet Draft
  that specifies Performance Metrics for a protocol.  A sufficient
  number of individuals with expertise must be willing to consult on
  the draft.  If the related WG has concluded, comments on the proposal
  should still be sought from key RFC authors and former chairs, or

I think "or" --> "and/or". I.e., there is not reason to choose the former over the latter if both are available.

  from the WG mailing list if it was not closed.

[Ballot discuss]
In general I think this document is adding some rather weak requirements on process (lots of "SHOULD"s).

I also have the following question which is another way of formulating one of Robert's DISCUSS points:

5.4.2.  Normative parts of Performance Metric definition

  The normative part of a Performance Metric definition MUST define at
  least the following:

  (i) Metric Name

  Performance Metric names MUST be unique within the set of metrics
  being defined and MAY be descriptive.

Is there a formal syntax for performance metric names? How are these names to be used (e.g. are they going to be used by IANA to guaranty uniqueness)?

[Ballot comment]
It is a very good idea to publish an RFC for defining the task of a directorate. However, I do want to make a few suggestions:

1. The name "Entity" is a bit odd. The established IETF terminology is "directorate"

2. I would not make the directorate a formal requirement or mandatory part of the process. I think it is better cast as a review organization that can help the working group, the IETF, and the IESG in making the right decisions.

[Ballot discuss]
In my opinion, the sections that establish a new directorate and make it part of the official process require an
"Updates 2026".  In particular, section 6.3 integrates the directorate into the approval process for new work items.
I would also note that this is a much stronger role than any other directorate has been given.  Secdir, mib doctors, etc.
have a review role but they are not asked to approve new work items.

It would be more appropriate to excise all text about the directorate; forming the directorate and reviewing metrics
related efforts does not require an RFC.  I also suggest that the directorate make itself available for early review
upon request, but strongly oppose any attempt to establish a gating function.

If the authors do not wish to remove this text or modify these functions, I believe another Last Call is in order after
adding "updates 2026" to the header, since I do not believe these changes have community consensus.

[Ballot comment]
Section 1.1

  Although the IETF has two active Working Groups (WGs) dedicated to
  the development of Performance Metrics, they each have strict
  limitations in their charters:

You seem to fail to list the PMOL WG. Is Section 1.1 simply trying to
justify the creation of PMOL? If so it can be removed from the
document because the WG seems to exist :-)

[Ballot comment]
Section 1.1

  Although the IETF has two active Working Groups (WGs) dedicated to
  the development of Performance Metrics, they each have strict
  limitations in their charters:
                                                                                   
You seem to fail to list the PMOL WG. Is Section 1.1 simply trying to
justify the creation of PMOL? If so it can be removed from the
document because the WG seems to exist :-)

[Ballot comment]
Section 1.1

  Although the IETF has two active Working Groups (WGs) dedicated to
  the development of Performance Metrics, they each have strict
  limitations in their charters:
                                                                                   
You seem to fail to list the PMOL WG. Is Section 1.1 simply trying to
justify the creation of PMOL? If so it can be removed from the
document because the WG seems to exist :-)

[Ballot discuss]
I agree with Ron that if what is wanted is a Directorate, it should
simply be created by the AD. Furthermore, I am worried that codifying
the directorate in this document is implying changes to the IETF
process that are not acceptable.

I strongly support the guidelines part of this document (which feels
Informational), but recommend that all reference to the directorate
be removed.

[Ballot discuss]
Point 1:

This document appears to have two purposes:

"The purpose of this document is to define a framework and a process for developing Performance Metrics for protocols above and below the IP-layer (such as IP-based applications that operate over reliable or datagram transport protocols), that can be used to characterize traffic on live networks and services.  As such, this document does not define any Performance Metrics."

-and-

"This document refers to the implementation of a Performance Metrics Entity, whose goal is to advice and support the Performance Metric development at the IETF.  A recommendation about the Performance Metrics Entity is made in Section 6.6."

You don't need an RFC to form a directorate. Just do it.

Point 2:

Does this document need to be a BCP? Or, asked another way, do you need a club to prevent someone outside of BMWG/PMOL/IPPM from developing a benchmarking metric using some other method? (And who would those people be?)

[Ballot comment]
1) "defined in a similar way" is very vague.

2) The language in 5.3.1 stating that measurements need to be non-overlapping and contiguous to be meaningfully composed caused me to pause. It's certainly easier to build meaningful compositions from measurements with those properties, but it's possible to build meaningful compositions when the properties, particularly contiguity, aren't met. 

3) Consider adding a discussion to section 5.4.2 (v) noting that the metric definition should discuss how the metric might vary depending which measurement point is chosen. The time between a SIP request and final response can be significantly different at the UAC and the UAS for example. Also consider noting that these kinds of differences should be taken into account when composing metrics. 

4) In 5.4.3 it would help readers to note that 3550 calls this interarrival jitter. 

5) It's not really clear what you want readers to do with the information in section 5.5. Please consider a short introduction to the section discussing whether these are observations that a metric designer needs to keep in mind, or if you are asking reviewers to ensure a specification says something about these things, or if you had a completely different audience in mind.

[Ballot discuss]
1) Section 6.3 is not adding any new process (it is recommending using the directorate framework we already have, asking an AD to set up and use a new team). It could stop with it's first sentence, and that sentence does not need to be restated in this document.  I think you have the same document if you delete this section.  If you agree, please delete it. If you don't agree, the document needs to be more explicit on what the section is trying to achieve.

2) I'm not sure Section 5.4.2 (i) is clear about the domain over which it is requiring names to be unique. It could be read to mean "unique within a given document defining metrics". Did you intend it to mean "Unique within the set of standardized metrics for a given protocol"? What mechanisms are you advising be used to ensure this uniqueness?  Should this document advise metric designers to help establish and manage registries per thing being measured, or are you anticipating something more lightweight?

3) Why did you make a new metric for the example in 5.4.5 instead of exemplifying one that is already well specified?  (Or have I missed the specification? It's related to discussion in 3550 and 3661, but I don't see this defined as a metric there). If it's new, could this section be misconstrued as actually defining the metric? I'd hate to see some future document trying to normatively refer an implementer to this section...

5) Section 6.2 is the section of the document that is going to guide reviewers. It's not as clear as it needs to be. For instance, reviewers are being asked to assess a metric according the the qualification "use cases" and that is the only place the string appears in the document. What are you requiring with respect to use cases?

6) Why didn't Section 6.6 recommend a particular Area?

State changed to In Last Call from Last Call Requested.

The following Last Call Announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Subject: Last Call:  (Guidelines for Considering New Performance Metric Development) to BCP


The IESG has received a request from the Performance Metrics for Other
Layers WG (pmol) to consider the following document:
- 'Guidelines for Considering New Performance Metric Development'
  as a BCP

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2011-01-27. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-pmol-metrics-framework/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-pmol-metrics-framework/

(1.a) Who is the Document Shepherd for this document? Has the
Document Shepherd personally reviewed this version of the
document and, in particular, does he or she believe this
version is ready for forwarding to the IESG for publication?
Al Morton is the shepherd, has read every version since the
draft was created and subsequently adopted on the charter,
and believes it is now ready for publication.

(1.b) Has the document had adequate review both from key WG members
and from key non-WG members? Does the Document Shepherd have
any concerns about the depth or breadth of the reviews that
have been performed?
The draft has been reviewed by many active working group members and
benefits from their comments. It has also been circulated to other
performance-oriented working groups.

(1.c) Does the Document Shepherd have concerns that the document
needs more review from a particular or broader perspective,
e.g., security, operational complexity, someone familiar with
AAA, internationalization or XML?
No.

(1.d) Does the Document Shepherd have any specific concerns or
issues with this document that the Responsible Area Director
and/or the IESG should be aware of? For example, perhaps he
or she is uncomfortable with certain parts of the document, or
has concerns whether there really is a need for it. In any
event, if the WG has discussed those issues and has indicated
that it still wishes to advance the document, detail those
concerns here. Has an IPR disclosure related to this document
been filed? If so, please include a reference to the
disclosure and summarize the WG discussion and conclusion on
this issue.
There are no concerns and no IPR disclosures.

(1.e) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with
others being silent, or does the WG as a whole understand and
agree with it?
There was active discussion of this draft in the WG and many points addressed,
all amicably.
After several WG Last Calls, the latest closing with comments that were
efficiently resolved, both the active and the recently passive participants
now appear to be satisfied.

(1.f) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarize the areas of conflict in
separate email messages to the Responsible Area Director. (It
should be in a separate email because this questionnaire is
entered into the ID Tracker.)
No.
(1.g) Has the Document Shepherd personally verified that the
document satisfies all ID nits? (See
http://www.ietf.org/ID-Checklist.html and
http://tools.ietf.org/tools/idnits/). Boilerplate checks are
not enough; this check needs to be thorough. Has the document
met all formal review criteria it needs to, such as the MIB
Doctor, media type and URI type reviews?
Yes.

There is one warning:
== Outdated reference: draft-ietf-sipping-rtcp-summary has been published
as RFC 6035



(1.h) Has the document split its references into normative and
informative? Are there normative references to documents that
are not ready for advancement or are otherwise in an unclear
state? If such normative references exist, what is the
strategy for their completion? Are there normative references
that are downward references, as described in [RFC3967]? If
so, list these downward references to support the Area
Director in the Last Call procedure for them [RFC3967].
The references are split.
There is one informative reference to a draft that was since updated
(see above).
Also, this ref is likely to be published soon:
[I-D.ietf-ippm-spatial-composition]
Morton, A. and E. Stephan, "Spatial Composition of
Metrics", draft-ietf-ippm-spatial-composition-16 (work in
progress), August 2010.

No down-refs.

I would also suggest to add a reference to
http://tools.ietf.org/html/draft-ietf-ippm-metrictest
which will be the published draft that uses material from
existing ref [I-D.bradner-metricstest].

(1.i) Has the Document Shepherd verified that the document IANA
consideration section exists and is consistent with the body
of the document? If the document specifies protocol
extensions, are reservations requested in appropriate IANA
registries? Are the IANA registries clearly identified? If
the document creates a new registry, does it define the
proposed initial contents of the registry and an allocation
procedure for future registrations? Does it suggest a
reasonable name for the new registry? See [RFC5226]. If the
document describes an Expert Review process has Shepherd
conferred with the Responsible Area Director so that the IESG
can appoint the needed Expert during the IESG Evaluation?
N/A

(1.j) Has the Document Shepherd verified that sections of the
document that are written in a formal language, such as XML
code, BNF rules, MIB definitions, etc., validate correctly in
an automated checker?
N/A

(1.k) The IESG approval announcement includes a Document
Announcement Write-Up. Please provide such a Document
Announcement Write-Up? Recent examples can be found in the
"Action" announcements for approved documents. The approval
announcement contains the following sections:

Technical Summary
Relevant content can frequently be found in the abstract
and/or introduction of the document. If not, this may be
an indication that there are deficiencies in the abstract
or introduction.
This document describes a framework and a process for developing
performance metrics of protocols and applications transported over
over IETF-specified protocols, and that can be used to characterize
traffic on live networks and services.

There's a steady (but small) stream of performance-oriented proposals
that appear in many areas of the IETF. The authors of such proposals
can usually benefit from the expertise of those who have tackled
similar tasks, and much of that expertise is embedded in this memo.
When others review such proposals, the framework forms an outline of
points to consider, and the memo defines a possible process for review.


Working Group Summary
Was there anything in WG process that is worth noting? For
example, was there controversy about particular points or
were there decisions where the consensus was particularly
rough?
There were no areas of rough consensus, as the elements of reasonable
performance metrics appear to be agreed in the practicing community.

Document Quality
Are there existing implementations of the protocol? Have a
significant number of vendors indicated their plan to
implement the specification? Are there any reviewers that
merit special mention as having done a thorough review,
e.g., one that resulted in important changes or a
conclusion that the document had no substantive issues? If
there was a MIB Doctor, Media Type or other expert review,
what was its course (briefly)? In the case of a Media Type
review, on what date was the request posted?
This framework has already been used in the review of several approved
drafts, such as:
http://tools.ietf.org/html/draft-ietf-pmol-sip-perf-metrics
and
http://tools.ietf.org/html/rfc5814