Packet Reordering Metrics
draft-ietf-ippm-reordering-13

[Ballot comment]
I might be a good idea to have a recommended value for dT.

The C code would be more useful if it had a license under which it could be used.

[Ballot comment]
> Metrics defined in this memo SHOULD be registered in the IANA IPPM
> METRICS REGISTRY as described in initial version of the registry
> [RFC4148].

I think it would be better to say "IANA is asked to register..."
instead of using SHOULD.

[Ballot comment]
I don't think I understand what you mean by these metrics being computable "on the fly". Several of the metrics require the device computing the metrics to save state for *every* packet ever received after the first packet loss - am I misunderstanding this? I don't see this as desirable (or for that matter even implementable :-) I imagine that you intend the metrics to have either a limited window of some sort for the s[i] vector or for the space over which the measurement is made. I can't seem to find a description of this in the document. Clearly some upper bound on the amount of reorder that was measurable would bound the amount of storage needed for the state and I would be happy that two device might implement it in ways that produced the same result. Anyway, perhaps I just missed some part of the document - let me know.

The C code would be more useful if it had a license under which it could be used.

IANA Comments:

Upon approval of this document, the IANA will register 8 new parameters in the IANA IPPM METRICS REGISTRY found at the following location:
http://www.iana.org/assignments/ianaippmmetricsregistry-mib

[Ballot discuss]
A number of comments were submited on the mreview list. These lead I believe to the need for changes in Sections 4.5 and 9.

The following includes inputs from Bert Wijnen and Randy Presuhn, and responses from Al Morton, who agreed I believe that the document needs to be revised following the comments.

1. Initial comment from Bert:

The confusion for me is in section 4.5.
Specifically the end of sect 4.5.4 which says:

  Gap(s[j'])    =  (j')  -  (j)

  Otherwise:

  The Type-P-packet-Reordering-Gap-Stream for the packet is 0.

  Gaps may also be expressed in time:

  GapTime(s[j']) = DstTime(j') - DstTime(j)

So explain to me what it means that if I express it in msg sequences the calculated value for Type-P-packet-Reordering-Gap-Stream could be 2 while if we were counting in milliseconds, then the value could be 150 for the same small set of messages.
So what should I (as a management application) do with those 2 values?
And how do I know if it is one or the other?

----

response from Al Morton:

Thanks for your re-statement of the issue, I think it is much more clear to me now.  This is a metric that can have multiple output parameters, so we have to be specific when talking about the "value" of Type-P-Packet-Reordering-Gap-Stream.
Saying:

    The Type-P-packet-Reordering-Gap-Stream for the packet is 0.

is not precise, and we should specifically say:

    The Gap(s[j']) parameter for the packet s[j'] is 0.

There seem to be a few places where the optional status of GapTime could be clarified, as well.  So, here are some suggested resolutions for this issue, below.

Hope this helps, it certainly helped me to set the time stamp issue aside and focus on the metric and its parameters.
Let me know if there more I can explain or do.

Al

4.5.2 Parameters

We use the same parameters defined earlier, but add the convention that index i' is greater than i, likewise j' > j, and define:

+  Gap(s[j']), the Reordering Gap of packet s[j'] in units of integer
+ messages

and the OPTIONAL parameter:

+  GapTime(s[j']), the Reordering Gap of packet s[j'] in units of time

...

4.5.4 Definition of Reordering Gap

A reordering gap is the distance between successive reordering discontinuities. The Type-P-Packet-Reordering-Gap-Stream metric assigns a value for Gap(s[j']) to (all) packets in a stream (and a value for GapTime(s[j']), when reported).

If:

        The packet s[j'] is found to be a reordering discontinuity, based on the arrival of reordered packet s[i'] with extent e', and

        an earlier reordering discontinuity s[j], based on the arrival of reordered packet s[i] with extent e was already detected, and

        i' > i, and

        there are no reordering discontinuities between j and j',

then the Reordering Gap for packet s[j'] is the difference between the arrival positions the reordering discontinuities, as shown below:

Gap(s[j'])    =  (j')  -  (j)

Gaps MAY also be expressed in time:

GapTime(s[j']) = DstTime(j') - DstTime(j)

Otherwise:

Gap(s[j']) (and GapTime(s[j']) ) for packet s[j'] is 0.

---------

Bert responds:

l, I think your seuggested changes are an improvement.

But I am still somewhat confused.
I still wonder what an NM application needs to do if it sees a value of 2 for one Type-P-packet-Reordering-Gap-Stream measurement, while is see for example 150 for another (so the first one was probably measure in msgs seq numbers and the second in milliseconds or some such).

Do we need two metrics for that?
Maybe a mandatory one that shows the metric in seq numbers and another (possibly OPTIONAL) one that shows the metric in time inetrval (I would then also ask to choose the granularity).

--------

Al Morton responds:

I think I better understand the NM reporting aspect of the issue now.
The problem seems to be that if a metric has multiple output parameters, we need to be able to refer to each one separately.

If defined two metrics in Sec 4.5, they could be called:
      Type-P-Packet-Reordering-Gap-Stream
      Type-P-Packet-Reordering-GapTime-Stream
and then we could easily refer to them in the IANA section, and get a metric number assigned to each one in the registry.

However, I see that we would have to do the same thing in Sec 4.6.
There are four output parameters that we defined, so we would need a new metric name for each one:
      Type-P-Packet-Reordering-Free-Run-x-numruns-Stream
      Type-P-Packet-Reordering-Free-Run-q-squruns-Stream
      Type-P-Packet-Reordering-Free-Run-p-numpkts-Stream
      Type-P-Packet-Reordering-Free-Run-a-accpkts-Stream

I suppose an alternative to adding all these new metric names would be to simply revise the IANA section to reflect the existence of output parameters, for example:

ietfReorderingFreeRunx OBJECT-IDENTITY
    STATUS      current
    DESCRIPTION
        "Type-P-Packet-Reordering-Free-Run-Stream, param x, number of runs"
    REFERENCE
        "Reference RFC xxxx, section 4.6"
    ::= { ianaIppmMetrics nn }    -- IANA assigns nn

So, please let me know your preference: new metric names, or revise the IANA section to reflect the output parameters where needed.

As far as the time stamp granularity goes, I think we can safely say that we have not specified the granularity for any of the time metrics in the current registry. This is a bigger problem than just the reordering draft, and some sort of global solution may be warranted.

-----

Bert's final response, to which I subscribe:

I am not sure it would be correct for me to tell you what solution I prefer and that that solution then gets adopted.
I think it would be much better if the problem is brought back to the WG and then discussed and a decision taken there.

Don't you think so?

My personal preference would be to have a separate metric name and OID assignment for each metric (at least so is my current thinking).

W.r.t. the time granularity... if that is a wider problem, then I suggest that that also gets discussed in the WG and that solutions get developed.

--------------------


2. A second issue raised by Randy Presihn and myself. From the dialog that we entertained with the editor:

> Finally, (in response to your earlier message) time stamp resolution
> is an implementation detail here in IPPM, unless you claim compliance
> with the active measurement protocol. Using different Timestamp
> resolutions certainly affects the degree to which the results can be
> compared, but that's just one of the possible errors and
> uncertainties, and it doesn't make comparison impossible.

Even if we accept this point of view, without getting into the question of accuracy or precision of the measurements, the document does not specify the UNITS in which they are reported.

I believe that this problem is general for all metrics defined in this document.

[Ballot discuss]
A number of comments were submited on the mreview list. The DISCUSS includes inputs from Bert Wijnen and Randy Presuhn, and responses from Al Morton, who agreed I believe that the document needs to be revised following the comments.

1. Initial comment from Bert:

The confusion for me is in section 4.5.
Specifically the end of sect 4.5.4 which says:

  Gap(s[j'])    =  (j')  -  (j)

  Otherwise:

  The Type-P-packet-Reordering-Gap-Stream for the packet is 0.

  Gaps may also be expressed in time:

  GapTime(s[j']) = DstTime(j') - DstTime(j)

So explain to me what it means that if I express it in msg sequences the calculated value for Type-P-packet-Reordering-Gap-Stream could be 2 while if we were counting in milliseconds, then the value could be 150 for the same small set of messages.
So what should I (as a management application) do with those 2 values?
And how do I know if it is one or the other?

----

response from Al Morton:

Thanks for your re-statement of the issue, I think it is much more clear to me now.  This is a metric that can have multiple output parameters, so we have to be specific when talking about the "value" of Type-P-Packet-Reordering-Gap-Stream.
Saying:

    The Type-P-packet-Reordering-Gap-Stream for the packet is 0.

is not precise, and we should specifically say:

    The Gap(s[j']) parameter for the packet s[j'] is 0.

There seem to be a few places where the optional status of GapTime could be clarified, as well.  So, here are some suggested resolutions for this issue, below.

Hope this helps, it certainly helped me to set the time stamp issue aside and focus on the metric and its parameters.
Let me know if there more I can explain or do.

Al

4.5.2 Parameters

We use the same parameters defined earlier, but add the convention that index i' is greater than i, likewise j' > j, and define:

+  Gap(s[j']), the Reordering Gap of packet s[j'] in units of integer
+ messages

and the OPTIONAL parameter:

+  GapTime(s[j']), the Reordering Gap of packet s[j'] in units of time

...

4.5.4 Definition of Reordering Gap

A reordering gap is the distance between successive reordering discontinuities. The Type-P-Packet-Reordering-Gap-Stream metric assigns a value for Gap(s[j']) to (all) packets in a stream (and a value for GapTime(s[j']), when reported).

If:

        The packet s[j'] is found to be a reordering discontinuity, based on the arrival of reordered packet s[i'] with extent e', and

        an earlier reordering discontinuity s[j], based on the arrival of reordered packet s[i] with extent e was already detected, and

        i' > i, and

        there are no reordering discontinuities between j and j',

then the Reordering Gap for packet s[j'] is the difference between the arrival positions the reordering discontinuities, as shown below:

Gap(s[j'])    =  (j')  -  (j)

Gaps MAY also be expressed in time:

GapTime(s[j']) = DstTime(j') - DstTime(j)

Otherwise:

Gap(s[j']) (and GapTime(s[j']) ) for packet s[j'] is 0.

---------

Bert responds:

l, I think your seuggested changes are an improvement.

But I am still somewhat confused.
I still wonder what an NM application needs to do if it sees a value of 2 for one Type-P-packet-Reordering-Gap-Stream measurement, while is see for example 150 for another (so the first one was probably measure in msgs seq numbers and the second in milliseconds or some such).

Do we need two metrics for that?
Maybe a mandatory one that shows the metric in seq numbers and another (possibly OPTIONAL) one that shows the metric in time inetrval (I would then also ask to choose the granularity).

--------

Al Morton responds:

I think I better understand the NM reporting aspect of the issue now.
The problem seems to be that if a metric has multiple output parameters, we need to be able to refer to each one separately.

If defined two metrics in Sec 4.5, they could be called:
      Type-P-Packet-Reordering-Gap-Stream
      Type-P-Packet-Reordering-GapTime-Stream
and then we could easily refer to them in the IANA section, and get a metric number assigned to each one in the registry.

However, I see that we would have to do the same thing in Sec 4.6.
There are four output parameters that we defined, so we would need a new metric name for each one:
      Type-P-Packet-Reordering-Free-Run-x-numruns-Stream
      Type-P-Packet-Reordering-Free-Run-q-squruns-Stream
      Type-P-Packet-Reordering-Free-Run-p-numpkts-Stream
      Type-P-Packet-Reordering-Free-Run-a-accpkts-Stream

I suppose an alternative to adding all these new metric names would be to simply revise the IANA section to reflect the existence of output parameters, for example:

ietfReorderingFreeRunx OBJECT-IDENTITY
    STATUS      current
    DESCRIPTION
        "Type-P-Packet-Reordering-Free-Run-Stream, param x, number of runs"
    REFERENCE
        "Reference RFC xxxx, section 4.6"
    ::= { ianaIppmMetrics nn }    -- IANA assigns nn

So, please let me know your preference: new metric names, or revise the IANA section to reflect the output parameters where needed.

As far as the time stamp granularity goes, I think we can safely say that we have not specified the granularity for any of the time metrics in the current registry. This is a bigger problem than just the reordering draft, and some sort of global solution may be warranted.

-----

Bert's final response, to which I subscribe:

I am not sure it would be correct for me to tell you what solution I prefer and that that solution then gets adopted.
I think it would be much better if the problem is brought back to the WG and then discussed and a decision taken there.

Don't you think so?

My personal preference would be to have a separate metric name and OID assignment for each metric (at least so is my current thinking).

W.r.t. the time granularity... if that is a wider problem, then I suggest that that also gets discussed in the WG and that solutions get developed.

--------------------


2. A second issue raised by Randy Presihn and myself is related to the resolution of the timestamps. From the dialog that we entertained with the editor:

> Finally, (in response to your earlier message) time stamp resolution
> is an implementation detail here in IPPM, unless you claim compliance
> with the active measurement protocol. Using different Timestamp
> resolutions certainly affects the degree to which the results can be
> compared, but that's just one of the possible errors and
> uncertainties, and it doesn't make comparison impossible.

Even if we accept this point of view, without getting into the question of accuracy or precision of the measurements, the document does not specify the UNITS in which they are reported

[Ballot comment]
Please clearly distinguish active measurement and passive measurement.
  At times, it is unclear which setting was being discussed.

  In Section 3, given the recent discussion about "monotonically
  increasing" sequence numbers on the IETF list, it would be good to be
  clear about what is required here.

  In Section 3.5 says:
  >
  > It is possible to detect Sequence Discontinuities with payload byte
  > numbering, but only when the complete pattern of payload sizes is
  > stored at the Destination, or when payload size is constant and then
  > the byte numbering adds needless complexity over message numbering.
  >
  Does the complete pattern really need to be stored at both sides?  If
  there is an active stream, one can pseudorandomly generate the sizes
  with a shared seed.  Please explain further?

  Section 4.2.4 says:
  >
  > If a receiver intends to restore order, then its buffer capacity
  > determines its ability to handle packets that are reordered. For
  > cases with single reordered packets, the extent e gives the number
  > of packets that must be held in the receiver's buffer while waiting
  > for the reordered packet to complete the sequence. For more complex
  > scenarios, the extent may be an overestimate of required storage
  > (see section 4.4 on Reordering Byte Offset and the Examples
  > section).
  >
  It seems like this does not give a correct buffer size estimate for
  protocols like TCP which adaptively retransmit and do not need to
  keep all reordered packets.  Since this is supposed to be a generic
  algorithm, it is probably sufficient to describe the situation.

[Ballot discuss]
Section 8.3 says:
  >
  > It may be possible to identify that a certain packet or stream of
  > packets is part of a sample. With that knowledge at the destination
  > and/or the intervening networks, it is possible to change the
  > processing of the packets (e.g. increasing or decreasing delay) that
  > may distort the measured performance.  It may also be possible to
  > generate additional packets that appear to be part of the sample
  > metric. These additional packets are likely to perturb the results
  > of the sample measurement. 
  >
  > To discourage the kind of interference mentioned above, packet
  > interference checks, such as cryptographic hash, may be used.
  >
  There are two issues in this section.

  First, to thwart an attacker that gives special treatment to
  measurement traffic in an attempt to generate false metrics, one needs
  to make the measurement traffic indistinguishable from real traffic.
  This document doesn't tell how to achieve this goal.  Encrypting both
  the measurement and the real traffic would be one answer, but of course,
  most real traffic is not encrypted today.  Perhaps the best thing to
  do is describe the potential consequences if the attacker is able to
  distinguish measurement and the real traffic.

  Second, an attacker might attempt to forge measurement traffic, which
  is most easily accomplished by replaying legitimate measurement
  traffic.  Use of "interference checks, such as cryptographic hash"
  will not detect replayed traffic.  Also, cryptographic keys are
  needed.  How are they managed?  Can one use TLS or DTLS?

[Ballot comment]
I wasn't completely clear that this definition is IP-version
independent (for example, there is a generic reference
to RFC 791 but not RFC 2460).

Gen-ART review comments from Sharon Chisholm:

1. In section 2, first paragraph, the term 'backwards step' is used but
not defined, either explicitly or with reference to another document.
Similarly, the terms 'non-reversing reference value' is also used
without definition of reference.

2. In section 3.5, I had difficulty parsing the second paragraph my
first pass or two. Does it mean:
'It is possible to detect Sequence Discontinuities with payload byte
  numbering, but only when the complete pattern of payload sizes is
  stored at the Destination, or when payload size is constant and then,

  IN THIS CASE, the byte numbering adds needless complexity over
message numbering'

PROTO Write-Up

Sheperd note for draft-ietf-ippm-reordering-11.txt
--------------------------------------------------

Questions
---------
1a. Yes, the draft has been reviewed by both chairs and we both believe
    that the draft is ready for publication.
1b. The concepts behind the draft have been discussed since 2001,
    resulting in a number of individual submission drafts which were
    merged into this draft.  This draft has been discussed for several
    years, it has been stable for about a year now.  Reviews were done
    by a number of key people in the group.
1c. No.
1d. No.
1e. The WG believes that this work is needed.  All but one group
    support this document.  The objections against the draft by the
    group from CSU are known to the AD.  However, we believe that there
    is rough consensus to support the draft in the WG.
1f. No.
1g. Yes.
1h. References have been split, there are no references to other IDs,
    only to RFC's and other documents that have already been published.

Protocol writeup
----------------

  This memo defines metrics to evaluate if a network has maintained
  packet order on a packet-by-packet basis. It provides motivations
  for the new metrics and discusses the measurement issues, including
  the context information required for all metrics. The memo first
  defines a reordered singleton, and then uses it as the basis for
  sample metrics to quantify the extent of reordering in several
  useful dimensions for network characterization or receiver design.
  Additional metrics quantify the frequency of reordering and the
  distance between separate occurrences. We then define a metric
  oriented toward assessing reordering affects on TCP. Several
  examples of evaluation using the various sample metrics are
  included. An Appendix gives extended definitions for evaluating
  order with packet fragmentation.