Reporting IP Network Performance Metrics: Different Points of View
RFC 6703
Yes
No Objection
Note: This ballot was opened for revision 08 and is now closed.
(Martin Stiemerling; former steering group member) Yes
(Wesley Eddy; former steering group member) Yes
(Adrian Farrel; former steering group member) No Objection
I have no objection to the publication of this document, but I think it would be helpful if the document title reflected the fact that the metrics being reported are IP network performance metrics.Perhaps... Reporting IP Network Performance Metrics: Different Points of View I also have some small Comments as follows... --- I think the document would benefit from a further read-through to fix some of the English and readability issues. Leaving these to the RFC Editor risks errors of meaning being introduced during the edit process. --- Section 3. This section gives an overview of recommendations And... Section 3.1. This section gives an overview of reporting recommendations for the loss, delay, and delay variation metrics. But... Section 3.1 The minimal report on measurements MUST include both Loss and Delay Metrics. This "MUST" is not a recommendation. You need to decide whether you are writing recommendations (which seems wholy appropriate since there are no operational or interop implications of missing out some measurements) or writing requirements. Notwithstanding the resolution of the above point, I am not convinced that you really need to use RFC 2119 language in this document. --- Section 3.1 "We have calculated a waiting time" needs a forward reference to the place in the document where this calculation is performed. --- Section 3.1 "99.9%-ile" is really ugly! --- A bit puzzled by Section 4.1.1 where you have n --- \ D = t + > (t + q ) 0 / i i --- i = 1 Presume you decided to not consider queue at the source node because you consider it as the generator of the packets and not subject to queuing. This is slightly suspect in my opinion and depends on the nature of - the source node - the definition of the path. Given this I wonder whether it is right to exclude q at the source or to include q at the destination. In any case, it would be helpful to explain your choices. But (of course) given the numbers being used to arrive at D using this formula including or excluding one queue time is not really significant. It would also be nice to note that there are n+1 nodes on your path and to clarify that q(i) is the delay due to queuing at node at the far end of the ith link. --- Not sure why section 4.3 is present in this document. It doesn't seem to leverage or be leveraged by anything else in the document. What is more, the concluding sentence ("After waiting sufficient time, packet loss can probably be attributed to one of these causes.") is rather vague and out of scope for the practice of measurement. Recall, the objective of ippm isto takemeasurementsandprovide reports, not to make qualatative assessments of the results. --- Section 10 Are there no security considerations associated with running the tests over longer periods of time? What if keys roll during the measurement period? Don'tlong periods offer more chance of seeing an attack?
(Barry Leiba; former steering group member) No Objection
Substantive suggestions; please respond to these:
-- Section 5.2 --
When both the sample mean and median are available, a comparison will
sometimes be informative, because these two statistics are equal only
when the delay distribution is perfectly symmetrical.
I'm not a statistician, but I don't think that's true. For example, this has a symmetrical distribution with 5 as the mean and median:
1 1 4 4 5 6 6 9 9
But this also has mean and median of 5, and its distribution is not symmetrical:
1 2 3 4 5 6 6 9 9
Am I missing something?
========
Editorial suggestions. No need to respond to these; take them or modify them as you please:
Throughout: there's no reason to hyphenate "point of view".
-- Introduction --
in a way that facilitates determining affects on user
applications,
"effects", not "affects".
-- Section 2 --
[RFC5136] using a definition of raw capacity without the
restrictions of data uniqueness or congestion-awareness.
Don't hyphenate "congestion awareness".
-- Section 3 --
Don't hyphenate "long term" here. (The rule is that a compound modifier is hyphenated, but if it's not being used as a modifier (an adjective or adverb), it shouldn't be hyphenated.)
-- Section 3.1 --
We have calculated a waiting time above that
should be sufficient to differentiate between packets that are truly
lost or have long finite delays under general measurement
circumstances, 51 seconds. Knowledge of specific conditions can help
to reduce this threshold, but 51 seconds is considered to be
manageable in practice.
"above"? Does this need to be re-worded? Maybe "above which it", or some such? And 51 seconds seems oddly precise: does 50 seconds really not work, and is it really not appropriate to call it 55 or 60 ? (Just asking; I have no idea of the answer here.)
For example, the 99.9%-ile minus the minimum
I suggest just spelling out "percentile" here (and in 5.2); you're not tight on column-inches. If you're worried, you can compensate by removing the extraneous "identified as" in the net paragraph.
-- Section 3.2 --
The result would ideally appear in the same form as though a
continuous measurement was conducted.
Needs subjunctive mood: "had been conducted."
intervals it is possible to present the results as "metric A was less
than or equal to objective X during Y% of time.
Missing closing quote.
NOTE that numerical thresholds of acceptability are not set in IETF
performance work and are explicitly excluded from the IPPM charter.
Once the RFC is published, its connection with the IPPM working group is not obvious. I suggest just saying, "and are out of scope for this document," or some such.
-- Figure 2 --
I suggest moving "where j is the hop number where the loop begins" out of the figure, since you already have two other "wheres" out there. You also don't say what "n" is, and should. I see from below that it's the number of hops. So make it, "where n is the total number of hops, j is the hop number where the loop begins, C is the number of times a packet circles the loop, and TTL is the packet's initial Time-to-Live value".
-- Section 4.3 --
In bullet 5, I would add a comma after "checking", to break up the length and to avoid confusion about what "and" conjoins.
-- Section 5.1.2 --
As further evidence of overlap, consider the Cumulative Distribution
Function (CDF) of Delay when the value positive infinity is assigned
to all lost packets.
I suggest quoting "positive infinity" to set it off clearly.
Although infinity is a familiar mathematical concept, it is somewhat
disconcerting to see any time-related metric reported as infinity, in
the opinion of the authors.
This is consensus of the WG, not opinion of authors, right? I suggest just ending the sentence at the comma. If you need to waffle, make it "it can be somewhat disconcerting".
-- Section 5.3 --
the most efficient practice is to distinguish between truly lost and
delayed packets with a sufficiently long waiting time, and to
designate the delay of non-arriving packets as undefined.
Again, it's easy to misread what the "and" conjoins. How about this way?:
NEW
the most efficient practice is to distinguish between packets
that are truly lost and those that are delayed with a sufficiently
long waiting time, and to designate the delay of non-arriving
packets as undefined.
-- Section 7.5 --
Last paragraph begins with a lower-case "w".
(Benoît Claise; former steering group member) No Objection
- Please address the OPS-DIR review by Bert Wijnen
The one thing that concerns me a little bit is the fact that this
document uses RFC2119 language. I think that is in-appropriate.
Using lower case for the MUST, SHOULD and RECOMMEND in the document
is perfectly fine I think.
- Support Adrian's comment regarding the title "Reporting IP Network Performance Metrics: Different Points of View"
- Next to the question "How will the results be used?", it would have been nice to ask the question "Which audience will read the results"
Network Characterization = network operator
Application Performance Estimation = application designer, service developer, etc..
Actually, this is what you did, without clearly mentioning it, asking the question about "how", and answering with "two main audience categories"
-
2. Application Performance Estimation - describes the network
conditions in a way that facilitates determining affects on user
applications, and ultimately the users themselves. This point-
of-view looks outward, toward the user(s), accepting the network
as-is.
What do you mean "accepting the network as-is."?
It's not because the results will be used for application performance estimation that you can't optimize your network.
- "The scope of this memo primarily covers the design and reporting of
the loss and delay metrics [RFC2680] [RFC2679]."
What do you mean by design of metric? Do you mean choosing the measurement characteristics of a metric?
Note: multiple occurrences of "metric design" in the draft.
- Section 2
"These memos effectively describe two
different categories of metrics,
o [RFC3148] includes restrictions of congestion control and the
notion of unique data bits delivered, and
o [RFC5136] using a definition of raw capacity without the
restrictions of data uniqueness or congestion-awareness.
It might seem at first glance that each of these metrics has an
obvious audience (Raw = Network Characterization, Restricted =
Application Performance), but reality is more complex and consistent
with the overall topic of capacity measurement and reporting. For
example, TCP is usually used in Restricted capacity measurement
methods, while UDP appears in Raw capacity measurement. "
I was not sure what you meant by Raw and Restricted.
However, I saw a definition way down in the document, in section 6 and 7
Raw capacity refers to the metrics defined in [RFC5136] which do not
include restrictions such as data uniqueness or flow-control response
to congestion...
Restricted capacity refers to the metrics defined in [RFC3148] which
include criteria of data uniqueness or flow-control response to
congestion...
Please add those "definitions" in section 2.
It's specifically important since RFC5136 and RFC3148 don't mention Raw/Restricted
- I learned to avoid "we", "our", "us" in RFCs.
I double-checked if it's still the case with the RFC-editor. I will let you know the answer.
- I would add an extra point to "For these and other reasons, such as"
Something such as:
o the ability to drill down to a specific measurement interval for deeper analysis
Justification: most of the time, when checking SLA, we check with large measurement interval, but want to ability to do a postmortem analysis
- I don't understand
"Fortunately, application performance estimation activities are not
adversely affected by the estimated worst-case transfer time.
Although the designer's tendency might be to set the Loss Threshold
at a value equivalent to a particular application's threshold, this
specific threshold can be applied when post-processing the
measurements. "
- "We can say that the Delay and Loss metrics are Orthogonal"
Orthogonal -> orthogonal?
- section 7.4. Bulk Transfer Capacity Reporting
When BTC of a link or path is estimated through some measurement
technique, the following parameters SHOULD be reported:
Also transport type, link layer type, tunneling yes/no, etc...?
- Personal preference, no need to modify the document unless you feel like it.
All my customers are interested in delay, loss, and delay variation (jitter).
It would have been nice to have a clear pointer in the table of content, with a clear entry "Effect of POV on the Delay Variation Metric . . . . . . . . . . . . . ." instead of addressing delay variation in "delay metric" section 5.1.3
-
Section 4.1 of [RFC3393] describes this specification and its
rationale (ipdv = inter-packet delay variation in the quote below).
Use IPDV (Remember you used Packet Delay Variation (PDV)) in the document, and refer to RFC5481
Several ipdv instances in the draft.
-
"Network Characterization has traditionally used Poisson-distributed
inter-packet spacing, as this provides an unbiased sample."
Is this correct? or Poisson-distributed start, with fixed inter-packet spacing, to match, for example, a voice/video application
(Brian Haberman; former steering group member) No Objection
(Gonzalo Camarillo; former steering group member) No Objection
(Pete Resnick; former steering group member) No Objection
As per your reply to Eliot Lear's Apps Directorate review, please un-2119 the document. I don't think it's appropriate for this document. You say "packets of type-P". Shouldn't that be "packets of type P" without the hyphen? Also, "type C"? With the hyphens, I'm not quite sure what you're talking about. Perhaps this is just unclear to someone outside the area.
(Ralph Droms; former steering group member) No Objection
(Robert Sparks; former steering group member) No Objection
(Ron Bonica; former steering group member) No Objection
(Russ Housley; former steering group member) No Objection
Thank you for considering the minor comments and editorial comments raised by Vijay Gurbani in the Gen-ART Review posted on 8-May-2012.
(Sean Turner; former steering group member) No Objection
(Stephen Farrell; former steering group member) No Objection
- 3.1: what does it mean to say the 51 seconds value was "calculated above" when its (now, presumably) done in 4.1.1. (Couldn't you have arranged that 42 seconds was the answer?) - 8.2: might have been a nice thing to include some reasonable representative sample sizes for some statistics for some measurements. Definitely too much to try add something with broad coverage, but one good, and one bad, set of example numbers would be a fine addition if someone had time.
(Stewart Bryant; former steering group member) No Objection