Reporting IP Network Performance Metrics: Different Points of View
draft-ietf-ippm-reporting-metrics-09

I have no objection to the publication of this document, but I think
it would be helpful if the document title reflected the fact that the 
metrics being reported are IP network performance metrics.Perhaps...

  Reporting IP Network Performance Metrics: Different Points of View


I also have some small Comments as follows...

---

I think the document would benefit from a further read-through to fix
some of the English and readability issues.  Leaving these to the RFC
Editor risks errors of meaning being introduced during the edit 
process.

---

Section 3.

   This section gives an overview of recommendations

And...

Section 3.1.

   This section gives an overview of reporting recommendations for the
   loss, delay, and delay variation metrics.

But...

Section 3.1

   The minimal report on measurements MUST include both Loss and Delay
   Metrics.

This "MUST" is not a recommendation. You need to decide whether you are
writing recommendations (which seems wholy appropriate since there are
no operational or interop implications of missing out some measurements)
or writing requirements.

Notwithstanding the resolution of the above point, I am not convinced 
that you really need to use RFC 2119 language in this document.

---

Section 3.1

"We have calculated a waiting time" needs a forward reference to the 
place in the document where this calculation is performed.

---

Section 3.1

"99.9%-ile" is really ugly!

---

A bit puzzled by Section 4.1.1 where you have

            n
           ---
           \
 D = t  +   >  (t  +  q )
      0    /     i     i
           ---
          i = 1

Presume you decided to not consider queue at the source node because you
consider it as the generator of the packets and not subject to queuing.
This is slightly suspect in my opinion and depends on the nature of
- the source node
- the definition of the path.

Given this I wonder whether it is right to exclude q at the source or
to include q at the destination. In any case, it would be helpful to 
explain your choices. But (of course) given the numbers being used to
arrive at D using this formula including or excluding one queue time is
not really significant.

It would also be nice to note that there are n+1 nodes on your path and
to clarify that q(i) is the delay due to queuing at node at the far end
of the ith link.

---

Not sure why section 4.3 is present in this document. It doesn't seem to
leverage or be leveraged by anything else in the document. What is more,
the concluding sentence ("After waiting sufficient time, packet loss can
probably be attributed to one of these causes.") is rather vague and out
of scope for the practice of measurement. Recall, the objective of ippm
isto takemeasurementsandprovide reports, not to make qualatative 
assessments of the results.

---

Section 10

Are there no security considerations associated with running the 
tests over longer periods of time? What if keys roll during the 
measurement period? Don'tlong periods offer more chance of seeing an
attack?

Substantive suggestions; please respond to these:

-- Section 5.2 --
   When both the sample mean and median are available, a comparison will
   sometimes be informative, because these two statistics are equal only
   when the delay distribution is perfectly symmetrical.

I'm not a statistician, but I don't think that's true.  For example, this has a symmetrical distribution with 5 as the mean and median:

1 1 4 4 5 6 6 9 9

But this also has mean and median of 5, and its distribution is not symmetrical:

1 2 3 4 5 6 6 9 9

Am I missing something?

========
Editorial suggestions.  No need to respond to these; take them or modify them as you please:

Throughout: there's no reason to hyphenate "point of view".

-- Introduction --
       in a way that facilitates determining affects on user
       applications,

"effects", not "affects".

-- Section 2 --
      [RFC5136] using a definition of raw capacity without the
      restrictions of data uniqueness or congestion-awareness.

Don't hyphenate "congestion awareness".

-- Section 3 --
Don't hyphenate "long term" here.  (The rule is that a compound modifier is hyphenated, but if it's not being used as a modifier (an adjective or adverb), it shouldn't be hyphenated.)

-- Section 3.1 --
   We have calculated a waiting time above that
   should be sufficient to differentiate between packets that are truly
   lost or have long finite delays under general measurement
   circumstances, 51 seconds.  Knowledge of specific conditions can help
   to reduce this threshold, but 51 seconds is considered to be
   manageable in practice.

"above"?  Does this need to be re-worded?  Maybe "above which it", or some such?  And 51 seconds seems oddly precise: does 50 seconds really not work, and is it really not appropriate to call it 55 or 60 ?  (Just asking; I have no idea of the answer here.)

   For example, the 99.9%-ile minus the minimum

I suggest just spelling out "percentile" here (and in 5.2); you're not tight on column-inches.  If you're worried, you can compensate by removing the extraneous "identified as" in the net paragraph.

-- Section 3.2 --

   The result would ideally appear in the same form as though a
   continuous measurement was conducted.

Needs subjunctive mood: "had been conducted."

   intervals it is possible to present the results as "metric A was less
   than or equal to objective X during Y% of time.

Missing closing quote.

   NOTE that numerical thresholds of acceptability are not set in IETF
   performance work and are explicitly excluded from the IPPM charter.

Once the RFC is published, its connection with the IPPM working group is not obvious.  I suggest just saying, "and are out of scope for this document," or some such.

-- Figure 2 --
I suggest moving "where j is the hop number where the loop begins" out of the figure, since you already have two other "wheres" out there.  You also don't say what "n" is, and should.  I see from below that it's the number of hops.  So make it, "where n is the total number of hops, j is the hop number where the loop begins, C is the number of times a packet circles the loop, and TTL is the packet's initial Time-to-Live value".

-- Section 4.3 --
In bullet 5, I would add a comma after "checking", to break up the length and to avoid confusion about what "and" conjoins.

-- Section 5.1.2 --

   As further evidence of overlap, consider the Cumulative Distribution
   Function (CDF) of Delay when the value positive infinity is assigned
   to all lost packets.

I suggest quoting "positive infinity" to set it off clearly.

   Although infinity is a familiar mathematical concept, it is somewhat
   disconcerting to see any time-related metric reported as infinity, in
   the opinion of the authors.

This is consensus of the WG, not opinion of authors, right?  I suggest just ending the sentence at the comma.  If you need to waffle, make it "it can be somewhat disconcerting".

-- Section 5.3 --
   the most efficient practice is to distinguish between truly lost and
   delayed packets with a sufficiently long waiting time, and to
   designate the delay of non-arriving packets as undefined.

Again, it's easy to misread what the "and" conjoins.  How about this way?:
NEW
   the most efficient practice is to distinguish between packets
   that are truly lost and those that are delayed with a sufficiently
   long waiting time, and to designate the delay of non-arriving
   packets as undefined.

-- Section 7.5 --
Last paragraph begins with a lower-case "w".

- Please address the OPS-DIR review by Bert Wijnen

    The one thing that concerns me a little bit is the fact that this
    document uses RFC2119 language. I think that is in-appropriate.
    Using lower case for the MUST, SHOULD and RECOMMEND in the document
    is perfectly fine I think. 

- Support Adrian's comment regarding the title "Reporting IP Network Performance Metrics: Different Points of View"


-  Next to the question "How will the results be used?", it would have been nice to ask the question "Which audience will read the results"
Network Characterization = network operator
Application Performance Estimation = application designer, service developer, etc..
Actually, this is what you did, without clearly mentioning it, asking the question about "how", and answering with "two main audience categories"

-

   2.  Application Performance Estimation - describes the network
       conditions in a way that facilitates determining affects on user
       applications, and ultimately the users themselves.  This point-
       of-view looks outward, toward the user(s), accepting the network
       as-is.

What do you mean "accepting the network as-is."?
It's not because the results will be used for application performance estimation that you can't optimize your network. 

- "The scope of this memo primarily covers the design and reporting of
   the loss and delay metrics [RFC2680] [RFC2679]."

What do you mean by design of metric? Do you mean choosing the measurement characteristics of a metric?
Note: multiple occurrences of "metric design" in the draft. 

-  Section 2
   "These memos effectively describe two
   different categories of metrics,

   o  [RFC3148] includes restrictions of congestion control and the
      notion of unique data bits delivered, and

   o  [RFC5136] using a definition of raw capacity without the
      restrictions of data uniqueness or congestion-awareness.

   It might seem at first glance that each of these metrics has an
   obvious audience (Raw = Network Characterization, Restricted =
   Application Performance), but reality is more complex and consistent
   with the overall topic of capacity measurement and reporting.  For
   example, TCP is usually used in Restricted capacity measurement
   methods, while UDP appears in Raw capacity measurement. "

I was not sure what you meant by Raw and Restricted.

However, I saw a definition way down in the document, in section 6 and 7
   Raw capacity refers to the metrics defined in [RFC5136] which do not
   include restrictions such as data uniqueness or flow-control response
   to congestion...

   Restricted capacity refers to the metrics defined in [RFC3148] which
   include criteria of data uniqueness or flow-control response to
   congestion...

Please add those "definitions" in section 2.
It's specifically important since RFC5136 and RFC3148 don't mention Raw/Restricted

- I learned to avoid "we", "our", "us" in RFCs.
I double-checked if it's still the case with the RFC-editor. I will let you know the answer.

- I would add an extra point to "For these and other reasons, such as"
Something such as: 

	o the ability to drill down to a specific measurement interval for deeper analysis

Justification: most of the time, when checking SLA, we check with large measurement interval, but want to ability to do a postmortem analysis

- I don't understand
  "Fortunately, application performance estimation activities are not
   adversely affected by the estimated worst-case transfer time.  
   Although the designer's tendency might be to set the Loss Threshold
   at a value equivalent to a particular application's threshold, this
   specific threshold can be applied when post-processing the
   measurements. "

- "We can say that the Delay and Loss metrics are Orthogonal"
Orthogonal -> orthogonal?

- section 7.4.  Bulk Transfer Capacity Reporting

   When BTC of a link or path is estimated through some measurement
   technique, the following parameters SHOULD be reported:


Also transport type, link layer type, tunneling yes/no, etc...?

- Personal preference, no need to modify the document unless you feel like it.
All my customers are interested in delay, loss, and delay variation (jitter).
It would have been nice to have a clear pointer in the table of content, with a clear entry "Effect of POV on the Delay Variation Metric . . . . . . . . . . . . . ." instead of addressing delay variation in "delay metric" section 5.1.3

-

Section 4.1 of [RFC3393] describes this specification and its
   rationale (ipdv = inter-packet delay variation in the quote below).
Use IPDV (Remember you used Packet Delay Variation (PDV)) in the document, and refer to RFC5481
Several ipdv instances in the draft.

-

  "Network Characterization has traditionally used Poisson-distributed
   inter-packet spacing, as this provides an unbiased sample."

Is this correct? or Poisson-distributed start, with fixed inter-packet spacing, to match, for example, a voice/video application

As per your reply to Eliot Lear's Apps Directorate review, please un-2119 the document. I don't think it's appropriate for this document.

You say "packets of type-P". Shouldn't that be "packets of type P" without the hyphen? Also, "type C"? With the hyphens, I'm not quite sure what you're talking about. Perhaps this is just unclear to someone outside the area.

  Thank you for considering the minor comments and editorial comments
  raised by Vijay Gurbani in the Gen-ART Review posted on 8-May-2012.

- 3.1: what does it mean to say the 51 seconds value
was "calculated above" when its (now, presumably)
done in 4.1.1. (Couldn't you have arranged that 42
seconds was the answer?)

- 8.2: might have been a nice thing to include some
reasonable representative sample sizes for some
statistics for some measurements. Definitely too
much to try add something with broad coverage, but
one good, and one bad, set of example numbers 
would be a fine addition if someone had time.