Framework for TCP Throughput Testing
draft-ietf-ippm-tcp-throughput-tm-13

[Ballot discuss]
(1) I don't quite understand how the IPR declaration and the spec match up.
The IPR declaration says you don't need any license for this since the unpublished
patent application only covers doing this spec in conjunction with other
connectionless traffic such as UDP. But the draft says that "In fact, Layer 2/3
tests are required to verify the integrity of the network before conducting TCP
tests.  Examples include iperf (UDP mode) and manual packet layer test
techniques where packet throughput, loss, and delay measurements are
conducted." That seems to be very close to saying that you practically would
(maybe) need a license. I think more clarity here is needed between the draft
and the IPR declaration before I could understand if a license is actually
needed to implement the putative-RFC or not. As it is, I don't understand the
combination.

(2) DISCUSS DISCUSS - is an IPR declaration like that ok? It doesn't say
anything about license terms but does indicate that there's a related thing
that you might do (UDP) and that doing both may require a license. If a similar
spec for UDP were to be submitted with the same kind of IPR declaration then
presumably that too would say you don't need a license. However, were the text
for both in one I-D then IETF LC would perhaps generate comments that no
license terms are disclosed so the IETF consensus might change. This could be
seen as a way of gaming the IPR declaration rules (splitting your drafts so
that none by itself needs a license, whereas the obvious/useful combination would)
but I'm not sure that our process provides a way to catch that other than
someone remembering that the various things might be combined. The possible
action here would be to require that IPR declarations not say (like this one)
that you don't need any license at all because this draft doesn't do <>
where <> is something not described in the draft in question.  [Note: I'm
not implying anyone's trying to game anything here and I'm not at all
confident in the "possible action" above but I'd like to ask the question in the
hope that someone else has a good answer.]

[Ballot comment]
1. I support the issues raised by Stewart and Ralph in their DISCUSS and COMMENT entries.

2. This metrics definitions in this document would have benefitted from following the recommendations in Section 5.4 of http://www.ietf.org/id/draft-ietf-pmol-metrics-framework-08.txt.

[Ballot comment]
While I share concerns about RFC 2119 language and IPR, I will let other ADs hold those DISCUSSes. Otherwise, this is a very well written and informative paper.

[Ballot comment]
I really can't figure out why this document uses 2119 language at all, and without explanation I would really prefer to see it removed. However, even if it is to remain, I don't understand when it is and isn't being used. For instance, part of Ralph's comments contain this:

  At the end, a TCP Throughput Test Device (TCP TTD) should generate a
  report with the calculated BDP and a set of Window Size experiments.
  Window Size refers to the minimum of the Send Socket Buffer and TCP
  RWND.  The report should include TCP Throughput results for each TCP
  Window Size tested.

Why are those shoulds and not SHOULDs? (Again, I think lowercase is fine, but I don't see why other things are not all lowercase). The one instance I could find that sounds like it might be 2119-worthy is:

  A compliant TCP TTD SHOULD
  provide a warning message when the expected test throughput will
  exceed 10% of the network bandwidth capacity.

But that worries me a little. Is this supposed to be a compliance document for test devices? If so, why is it Informational?

I'm not holding up the document on this basis, but I think the WG and the AD need to answer these questions.

[Ballot discuss]
(1) I don't quite understand how the IPR declaration and the spec match up.
The IPR declaration says you don't need any license for this since the unpublished
patent application only covers doing this spec in conjunction with other
connectionless traffic such as UDP. But the draft says that "In fact, Layer 2/3
tests are required to verify the integrity of the network before conducting TCP
tests.  Examples include iperf (UDP mode) and manual packet layer test
techniques where packet throughput, loss, and delay measurements are
conducted." That seems to be very close to saying that you practically would
(maybe) need a license. I think more clarity here is needed between the draft
and the IPR declaration before I could understand if a license is actually
needed to implement the putative-RFC or not. As it is, I don't understand the
combination.

(2) DISCUSS DISCUSS - is an IPR declaration like that ok? It doesn't say
anything about license terms but does indicate that there's a related thing
that you might do (UDP) and that doing both may require a license. If a similar
spec for UDP were to be submitted with the same kind of IPR declaration then
presumably that too would say you don't need a license. However, were the text
for both in one I-D then IETF LC would perhaps generate comments that no
license terms are disclosed so the IETF consensus might change. This could be
seen as a way of gaming the IPR declaration rules (splitting your drafts so
that none by itself needs a license, whereas the obvious/useful combination would)
but I'm not sure that our process provides a way to catch that other than
someone remembering that the various things might be combined. The possible
action here would be to require that IPR declarations not say (like this one)
that you don't need any license at all because this draft doesn't do <>
where <> is something not described in the draft in question.  [Note: I'm
not implying anyone's trying to game anything here and I'm not at all
confident in the "possible action" above but I'd like to ask the question in the
hope that someone else has a good answer.]

[Ballot comment]
1. Terms given definitions and acronyms in the Terminology section are
sometimes unnecessarily expanded in the body of the document.  Why not
just use the acronyms throughout the document?

2. In section 3:

  - More important, the TCP test host MUST be capable to generate
  and receive stateful TCP test traffic at the full link speed of the
  network under test.

Which link?  There are several links in the NUT as depicted in Figure
1.1.

3. Add "Bandwidth Delay Product" to the Terminology section.

4. Is "Send Buffer" equivalent to "Send Socket Buffer"?

5. In section 5.2:

  At the end, a TCP Throughput Test Device (TCP TTD) should generate a
  report with the calculated BDP and a set of Window Size experiments.
  Window Size refers to the minimum of the Send Socket Buffer and TCP
  RWND.  The report should include TCP Throughput results for each TCP
  Window Size tested.

Is there any advice for the range of window sizes that should be
tested?  Section 3.3.1 requires:

  The TCP TTD MUST allow the Send Buffer and Receive Window sizes to
  be set higher than the BDP, other wise TCP performance will be
  limited.

Should the window sizes in the test always be greater than BDP?

[Ballot discuss]
I have found a few places in the document where some clarification
seems to be required.  They should be easy to fix up, but I've put
them in a Discuss because I think they are central to the correct
utilization of the specified measurement techniques.

1. In section 3, step 2:

  These
  measurements refers to [RFC2681] and [RFC4898] in order to measure
  RTD and associated RTT.

Shouldn't this sentence refer to "RTT and BB", based on the citations
of RFC 2681 and RFC 4898?  Or is there another source for techniques
related to measurement of BB?

2. In section 3.2.1:

  Complementing the definition from Section 1.1, Round Trip Time(RTT)
  is the elapsed time between the clocking in of the first bit of a
  TCP segment sent and the receipt of the last bit of the corresponding
  TCP Acknowledgment.  Round Trip Delay (RTD) is used synonymously to
  twice the Link Latency.  RTT measurements SHOULD use techniques
  defined in [RFC2681] or statistics available from MIBs defined in
  [RFC4898].

Why not give this more precise definition for RTT in section 1.1?
RTD is defined in terms of the undefined "Link Latency".  As far as I
can tell, the only reference to RTD is in section 3.  In my opinion,
the definition of RTD in section 3.2.1 and the reference to RTD in
section 3 could both be omitted.

3. RFC 2861, cited as the source of techniques to measure RTT, uses the
term "Round-trip delay", which is given essentially the same
definition as RTT in this document.  Is the intention that RTT in this
document is equivalent to Round-trip delay in RFC 2861?

4. A little later in section 3.2.1, there is a list of techniques for
measuring RTT, none of which mention RFC 2861 or RFC 4898.  For
clarity, it would be good to consolodiate the suggestions to use RFC
2861 and RFC 4898 with the later list of techniques.

5. Is the bandwidth measured using the techniques in section 3.2.2 the
BB?

6. In section 4.1.1:

  Then, to obtain the Maximum Achievable TCP Throughput (Layer 4), we
  simply use: (MTU - 40) in Bytes X 8bits X max FPS.

Does the constant 40 in this computation imply IPv4?

[Ballot comment]
"Note that the primary focus of this methodology is managed business
class IP networks; i.e. those Ethernet terminated services for which
organizations are provided an SLA from the network provider. "

I suspect that you mean e.g. ("for example") and not "that is", after
all I would expect high quality service from all of the various links
you specify on page 12, most of which are not Ethernet.

=======

The following paragraphs seem contradictory since it would seem
reasonable to specify TCP throughput as part of an SLA.

"  - This methodology is not intended to measure TCP Throughput as part
  of an SLA, or to compare the TCP performance between service
  providers or to compare between implementations of this methodology
  in dedicated communications test instruments.

"  In contrast to the above exclusions, the primary goal is to define a
  method to conduct a practical end-to-end assessment of sustained
  TCP performance within a managed business class IP network. "

Please can you clarify the difference between these two cases.
Is the distinction one of in service vs out of service?

=========

Please clarify the following:

"TCP Throughput testing may require cooperation between the end-user
  customer and the network provider.  As an example, in an MPLS (Multi-
  Protocol Label Switching) network architecture, the testing should be
  conducted either on the CPE or on the CE device and not on the PE
  (Provider Edge) router."

The customer owns the CPE and would normally own the CE, so by definition
there is no provider involvement. Are you perhaps considering the case
of provider managed CE?

==========

Aside WRT Round Trip Time baselining and the various methods of measuring
this, running NTP between the two systems will tell you a lot about the RTT
characteristics.

==========

In the business customer environment, these settings are not
generally adjustable by the user.

I think that you mean "... by a user on an 'ordinary' host"

=========

Section 4 needs some rearrangement, for example "Ideal TCP Transfer
Time" is pulled out of a hat at the top of section 4.1, but
there is no further information until section 4.1.2, and even then
there is no definition. Please provide a formal definition (or
reference to one) before the term is used.

==========

Given that the environment is stated as managed service with
operating under SLA, the need for the following co-ordination
is not clear:

"If the throughput test
is expected to exceed 10% of the provider bandwidth, then the test
should be coordinated with the network provider.  This does not
include the customer premise bandwidth, the 10% refers directly to
the provider's bandwidth (Provider Edge to Provider router)."

[Ballot comment]
The Abstract says:

  In this framework,
  TCP and IP parameters are specified and should be configured as
  recommended.

I think this could be refined a little to give the scope of the recommendation. For example:
  ... in order to perform the specific tests described.

> Framework for TCP Throughput Testing
> draft-ietf-ippm-tcp-throughput-tm-12.txt
>
> (1.a) Who is the Document Shepherd for this document? Has the
> Document Shepherd personally reviewed this version of the
> document and, in particular, does he or she believe this
> version is ready for forwarding to the IESG for publication?
>
> Document sheperd is Henk Uijterwaal.
>
> (1.b) Has the document had adequate review both from key WG members
> and from key non-WG members?
>
> Yes. The document was reviewed by about a dozen people in the group.
> Many of them provided comments, the people with the most extensive
> comments are listed in the acknowledgement section.
The -07 version was initially submitted for publication and a fair number
of comments were raised by the AD. In consultation with the AD, the
draft was sent to the TCPM group for review. Comments by this group
were included and for the -12 all reviewers confirmed that they were
happy with the end result.

The WGLC was also sent to the TCPM group and attracted no comments.



>
> (1.c) Does the Document Shepherd have concerns that the document
> needs more review from a particular or broader perspective,
>
> No.
>
> (1.d) Does the Document Shepherd have any specific concerns or
> issues with this document that the Responsible Area Director
> and/or the IESG should be aware of?
>
> No, there are no such concerns.
>
> (1.e) How solid is the WG consensus behind this document? Does it
> represent the strong concurrence of a few individuals, with
> others being silent, or does the WG as a whole understand and
> agree with it?
>
Good, with some 6 members of the group reviewing it and the external
review by the TCPM group.

>
> (1.f) Has anyone threatened an appeal or otherwise indicated extreme
> discontent?
>
> No.
>
> (1.g) Has the Document Shepherd personally verified that the
> document satisfies all ID nits?

Yes, there were none.

> (1.h) Has the document split its references into normative and
> informative?
>
> Yes
> Are there normative references to documents that
> are not ready for advancement or are otherwise in an unclear
> state?
>
> No
>
> (1.i) Has the Document Shepherd verified that the document IANA
> consideration section exists and is consistent with the body
> of the document?
>
> Yes, it is empty, so please remove it upon publication.
>
> (1.j) Has the Document Shepherd verified that sections of the
> document that are written in a formal language, such as XML
> code, BNF rules, MIB definitions, etc., validate correctly in
> an automated checker?
>
> N/A.
>
> (1.k) The IESG approval announcement includes a Document
> Announcement Write-Up. Please provide such a Document
> Announcement Write-Up? Recent examples can be found in the
> "Action" announcements for approved documents. The approval
> announcement contains the following sections:
>
> Technical Summary
> This document describes a framework for measuring sustained TCP
> throughput performance in an end-to-end managed network environment.
> This document is intended to provide a practical methodology to help
> users validate the TCP layer performance of a managed network, which
> should provide a better indication of end-user experience. In the
> framework, various TCP and network parameters are identified that
> should be tested as part of the network verification at the TCP layer.
>
>
> Working Group Summary
> The normal WG process was followed and the document has been discussed for
> several years. The document as it is now, reflects WG consensus, with nothing
> special worth noticing.
>
> Document Quality
> Good

State changed to In Last Call from Last Call Requested.

The following Last Call Announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Subject: Last Call:  (Framework for TCP Throughput Testing) to Informational RFC


The IESG has received a request from the IP Performance Metrics WG (ippm)
to consider the following document:
- 'Framework for TCP Throughput Testing'
  as an Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2011-04-20. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-ippm-tcp-throughput-tm/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-ippm-tcp-throughput-tm/

Framework for TCP Throughput Testing
draft-ietf-ippm-tcp-throughput-tm-07.txt

(1.a) Who is the Document Shepherd for this document? Has the
Document Shepherd personally reviewed this version of the
document and, in particular, does he or she believe this
version is ready for forwarding to the IESG for publication?

Document sheperd is Henk Uijterwaal.

(1.b) Has the document had adequate review both from key WG members
and from key non-WG members?

Yes. The document was reviewed by about a dozen people in the group.
Many of them provided comments, the people with the most extensive
comments are listed in the acknowledgement section.

(1.c) Does the Document Shepherd have concerns that the document
needs more review from a particular or broader perspective,

No.

(1.d) Does the Document Shepherd have any specific concerns or
issues with this document that the Responsible Area Director
and/or the IESG should be aware of?

No, there are no such concerns.

(1.e) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with
others being silent, or does the WG as a whole understand and
agree with it?

Decent, with some 6 members of the group reviewing it.

(1.f) Has anyone threatened an appeal or otherwise indicated extreme
discontent?

No.

(1.g) Has the Document Shepherd personally verified that the
document satisfies all ID nits?

* In line 825, there is a strange spacing "... Window to ..."
* There is a spelling error synonomously instead of synonymously.
* Section 5 should be removed upon publication.

These are minor and can be fixed by the editor.

(1.h) Has the document split its references into normative and
informative?

Yes
Are there normative references to documents that
are not ready for advancement or are otherwise in an unclear
state?

No

(1.i) Has the Document Shepherd verified that the document IANA
consideration section exists and is consistent with the body
of the document?

Yes, it is empty, so please remove it upon publication.

(1.j) Has the Document Shepherd verified that sections of the
document that are written in a formal language, such as XML
code, BNF rules, MIB definitions, etc., validate correctly in
an automated checker?

N/A.

(1.k) The IESG approval announcement includes a Document
Announcement Write-Up. Please provide such a Document
Announcement Write-Up? Recent examples can be found in the
"Action" announcements for approved documents. The approval
announcement contains the following sections:

Technical Summary
This document describes a framework for measuring sustained TCP
throughput performance in an end-to-end managed network environment.
This document is intended to provide a practical methodology to help
users validate the TCP layer performance of a managed network, which
should provide a better indication of end-user experience. In the
framework, various TCP and network parameters are identified that
should be tested as part of the network verification at the TCP layer.

Working Group Summary
The normal WG process was followed and the document has been discussed for
several years. The document as it is now, reflects WG consensus, with nothing
special worth noticing.

Document Quality
Good