Applicability Statement for RFC 2544: Use on Production Networks Considered Harmful
draft-ietf-bmwg-2544-as-08

[Ballot comment]
I agree with the intend of the draft. Hence no objection.

Text from Al Morton
We're saying that almost any test using RFC2544 methods
on a production network will produce a form of harm from wrong
and misleading results with inherent wasted time and resources,
and that's just one of the ways we defined harm in version 05
clarifications.  In the test and measurement community
(the main audience of this draft), frequent flawed results are
the end of the world with too few transport ships ready to go.

I agree with Al: "a form of harm from wrong and misleading results"

While I understand Stewart's point (a customer could use RFC2544 to test his CIR), the Device Under Test (DUT) is a black box, and BMWG does black box testing. Per the "black box" definition, we can't control or even monitor what's in it.
By testing live network, the DUT becomes the network (or a circuit). And we have no way to know what we influence, if anything This is the way I understand harmful.

That being said. Harmful might be understood in a different way.
The title "Use on Production Networks Considered Harmful" might be a little bit alarming...
While the text in the draft is more realistic
- abstract  "Overload of shared resources would likely be harmful to user traffic performance on a production network"
- section 5 "Overload of shared resources would likely be harmful to user traffic performance on a production network"

Text from Scott Bradner
We feel that use of the 2544 in production networks is harmful in the sense that IETF has been using that term for a very long time. Some of the tests would harm other users of the network and some of the tests would get bad (misleading) results.

Any wording that would soften the word "harmful" would be a step in the right direction IMHO.
Alternatively, your own definition of harmful would also help. For example, "By harmful, we mean that the consequences are unknown/unpredictable/etc ..." or "By harmful, we mean: a form of harm from wrong and misleading results"

[Ballot comment]
I am retaining my Yes ballot, and I thank the authors for addressing my
Comments entered against revision 05.

It continues to be my opinion that, in stating an important point about
the use of these tests on live, and dynamically routed, shared-resource
IP/MPLS networks, the document overstates its case an implies that the
use on network paths in dedicated-resource networks (such as MPLS-TE or
MPLS-TP) is harmful when it is neither obvious nor proven that this is
the case. In this, I agree with Stewart's Discuss and wish that the
authors would make a clearer and more upfront statement about the scope
of the network in which this may be harmful.

The solution to this may be as simple as a change to the title to
something like:

        Use of RFC 2544 Benchmarking Test Methodologies on
    Production Networks with Shared Resources Considered Harmful

In making this change you would not be commenting about TE or transport
networks, but would be making the clear statement about where you know
the harm to be.

---

New typo introduced...
Section 4.2
  In other words, devices operating on the Internet may be configured
  to discard any traffic they observe in this address range, as it is
  intended for laboratory ITE use only.  Thus, testers using the
  assigned testing address ranges are connected to the Internet and
  test packets are forwarded across the Internet, it is likely that the
  packets will be discarded and the test will not work.

s/Thus, testers/Thus, if testers/

[Ballot discuss]
The steps that I agreed with the sponsoring Area Director to resolve my Discuss were that the comments posted by Adrian would be included and I would then re-review the draft to see it it addressed my concerns. I note that a number of Adrian's comments have not been included in version 6 of the draft.

I would additionally like to propose two changes to the draft that I think would address the fundamental basis of the my concerns, and if these are acceptable to the authors I will review the updated text with a view to clearing my discuss.

I think one of the following document titles would be more appropriate:

a) RFC 2544 Applicability Statement: Use on Production Networks Employing Contended Resources Considered Harmful

or

b) RFC 2544 Applicability Statement: Describing When Used on Production Networks Can be Considered Harmful

My preference is for (a), but I can accept (b).

Then in the Abstract and Introduction there needs to be some scoping text that describes the network situation considered by the authors.

This document is scoped to the case where the production network provides a service using contended resources. Production networks using un-contended resources are out of scope.

[Ballot comment]
  To directly address this situation, the past and present Chairs of
  the IETF Benchmarking Methodology Working Group (BMWG) have prepared
  this Applicability Statement for RFC 2544.

I'm going to push this also, where Pete gave it up... though it's still non-blocking.  I very strongly ask you to change this.

(1) Working group documents are the products of working groups, not of individuals.

(2) Statements like this imply that people in positions of leadership rammed something through, forcing "consensus" with a bulldozer.

(3) At best, statements like this appear to be appeal-to-authority arguments.  We should avoid that when it's not necessary, and I don't see that it's necessary here.

If BMWG has strong consensus on what this document says, then say *that*.  That statement means more to me than one that says that four chairs thought it was important.  (I'll also note that when you couple that statement with the Acknowledgments section, which recognizes five people and doesn't mention the working group, one *could* take away from this that there were nine people who read and agreed with this.  Saying instead -- explicitly, rather than the usual implicit message we give -- that this represents *strong* consensus of the BMWG makes it very clear.)

Apart from that, I'll let Stewart and Adrian sort out whether or not you're overstating anything.

[Ballot comment]
I have reviewed this document on a number of occasions before it came
for IESG review. The text is much improved and I thank the authors for
their work.

Isuport the publication of this document, but I agree with Stewart that
the case appears to be overstated. I believe that much of the problem is
stylistic, but it leads to some statements that are stronger than they
need to be. It is simply not the case that all use of 2544 mechanisms on
production networks are perforce harmful.

What we need to do is strike a balance between pointing out the risks
and indicating that use of 2544 is inadvisable because of those risks on
the one hand, and trying to use an Informational RFC to ban people from
doing stupid things.

Here are some suggested edits to improve matters...

Abstract
OLD
  Recent application of the methods
  beyond their intended scope is cause for concern.
NEW
NOTE
I don't think this statement is expanded upon in the document (and if it
were, I would be suggesting that pointing fingers is not helpful) and I
don't think it matters. Your advice presumably holds true whether or not
people have started to do silly things.
END

Abstract
OLD
  The methods
  described in RFC 2544, where overload is a possible outcome, would no
  doubt be harmful to user traffic performance on a production network.
NEW
  The methods described in RFC 2544 generate traffic that load the
  network. In heavily utilized networks, the combination of user
  traffic and benchmarking traffic could cause network overload that
  would potentially harm user traffic performance.
  an
NOTE
The phrase "would no doubt be harmful" is not correct when applied to
"a possible outcome". In fact, it is the overload itself that is the
harmful effect.
END

Abstract
OLD
  This memo clarifies the scope of RFC 2544 and other benchmarking work
  for the IETF community.
NEW
  This memo clarifies the scope of RFC 2544 and other benchmarking work.
NOTE
I think that, although you are working within the IETF community, your
actual intention is wider visiblity.
END

Section 1
OLD
  This memo clarifies the scope of RFC 2544 [RFC2544], which discusses
  and defines several tests that may be used to characterize the
  performance of a network interconnecting device, and other
  benchmarking work for the IETF community.
NEW
  This memo clarifies the scope and use of benchmarking tests including
  RFC 2544 [RFC2544], which discusses and defines several tests that
  may be used to characterize the performance of a network
  interconnecting device.
END

Section 1
OLD
  Benchmarking Methodologies (beginning with [RFC2544]) have always
  relied on test conditions that can only be produced and replicated
  reliably in the laboratory.  Thus it was unfortunate to find that
  this foundation methodology was being cited in several unintended
  specifications and products performing applications such as:

  1.  Validation of telecommunication service configuration, such as
      the Committed Information Rate (CIR).

  2.  Validation of performance metrics in a telecommunication Service
      Level Agreement (SLA), such as frame loss and latency.

  3.  Telecommunication service activation testing, where traffic that
      shares network resources with the test might be adversely
      affected.
NEW
  Benchmarking Methodologies (beginning with [RFC2544]) rely on test
  conditions that can only be produced and replicatedreliably in the
  laboratory.  These methodologies are not appropriate for inclusion in
  wider specifications such as:
 
  1.  Validation of telecommunication service configuration, such as
      the Committed Information Rate (CIR).

  2.  Validation of performance metrics in a telecommunication Service
      Level Agreement (SLA), such as frame loss and latency.

  3.  Telecommunication service activation testing, where traffic that
      shares network resources with the test might be adversely
      affected.
NOTE
I think that the commentary is itself "unfortunate." It is cleaner to
state what is not approriate, and move on.
END

Section 1
OLD
  Although RFC 2544 is held up as the standard reference for such
  testing, we believe that the actual methods used vary from RFC 2544
  in significant ways.  Since the only citation is to RFC 2544, the
  modifications are opaque to the standards community and to users in
  general (an undesirable situation).  There is risk of harm to user
  traffic from applying the test traffic and methods described in
  [RFC2544] on a production network, because overload in shared
  resources is a possible outcome.

  To directly address this situation, the past and present Chairs of
  the IETF Benchmarking Methodology Working Group (BMWG) have prepared
  this Applicability Statement for RFC 2544.
NEW
  RFC 2544 is used as a key reference for such testing implying that
  the mechanisms are directly derived from RFC 2544.  However, the
  actual methods often vary from the mechanisms described in RFC 2544,
  and the modifications are opaque to the standards community and to
  users in general.

  Since applying the test traffic and methods described in RFC 2544 on
  a production network risks causing overload in shared resources,
  there is a direct risk ofharming user traffic if these mechanisms are
  misued.

  To directly address this situation, the past and present Chairs of
  the IETF Benchmarking Methodology Working Group (BMWG) have prepared
  this Applicability Statement for RFC 2544.
NOTE
Again, the commentary is unnecessarily perjorative when direct report
would be cleaner.
END

Section 4
OLD
4.  Why RFC 2544 Methods are intended for ITE
NEW
4.  Why RFC 2544 Methods are intended only for ITE
END

Section 4
OLD
  The following sections discuss some of the reasons why RFC 2544
  [RFC2544] methods were intended only for isolated laboratory use, and
  the difficulties of applying these methods outside the lab
  environment.
NEW
  The following sections discuss some of the reasons why RFC 2544
  [RFC2544] methods are applicable only for isolated laboratory use,
  and the consequences of applying these methods outside the lab
  environment.
NOTE
I don't think that the intention when 2544 was written has as much
bearing as you imply. What is important is the applicability now. That
is, you could have intended use only in ITE but discovered wide
applicability later.

I also think that "difficulties" implies "it is hard, but you can do
it." That is not the message you intend to send.
END

Section 4.1
OLD
  All of the tests described in RFC 2544 require that the tester and
  device under test are the only devices on the networks that are
  transmitting data.  The presence of other unwanted traffic on the
  network would mean that the specified test conditions have not been
  achieved.

  If any unwanted traffic appears and the amount varies over time, the
  repeatability of any test result will likely depend to some degree on
  the unwanted traffic.

  The presence of unwanted or unknown traffic makes accurate,
  repeatable, and consistent measurements of the performance of the
  device under test very unlikely, since the complete details of test
  conditions will not be reported.
NEW
  All of the tests described in RFC 2544 require that the tester and
  device under test are the only devices on the networks that are
  transmitting data.  The presence of other traffic on the network
  would mean that the specified test conditions have not been achieved
  and the results could not be guaranteed to be valid.

  If any other traffic appears and the amount varies over time, the
  repeatability of any test result will likely depend to some degree on
  the amount and variation of the other traffic.

  The presence of other traffic makes accurate, repeatable, and
  consistent measurements of the performance of the device under test
  very unlikely, since the complete details of test conditions will not
  be reported.
NOTE
Two things...
The traffic *is* wanted. It is just not wanted by you.
Need to say why it matters if the test conditions are not achieved.
END

Section 4.2
OLD
  In other words, devices operating on the Internet may be configured
  to discard any traffic they observe in this address range, as it is
  intended for laboratory ITE use only.  Thus, testers using the
  assigned testing address ranges MUST NOT be connected to the
  Internet.
NEW
  In other words, devices operating on the Internet may be configured
  to discard any traffic they observe in this address range, as it is
  intended for laboratory ITE use only.  Thus, if testers using the
  assigned testing address ranges are connected to the Internet, and
  test packets are forwarded across the Internet it is likely that the
  packets will be discarded and the testing will not work correctly.
NOTE
I think that there is no logical progression here. It would appear to be
perfectly safe to attach the DUT to the Intrnet precisely because the
test packets will be discarded and not propagated further. OTOH, the
test itself will not work if any packets are discarded and this might
happen if those packets are routed over the Internet.
END
 
Section 4.2
OLD
  We note that a range of IPv6 addresses has been assigned to BMWG for
  laboratory test purposes, in [RFC5180] (as amended by errata).  Also,
  the strong statements in the Security Considerations Section of this
  memo make the scope even more clear; this is now a standard fixture
  of all BMWG memos.
NOTE
I couldn't decide whether it was the Security Considerations Section of
this memo that you intended or of 5180. Seems like, given the
juxtoposition, that you meant 5180.
But surely this is not the right section of this document to mention
that material.
I suggest moving "Also..."  to Section 7 of this document, and rewording
it a little.
END

Section 5
OLD
  There will be
  unanticipated difficulties when applying these methods outside the
  lab environment.
NOTE
I think you are anticipating the diffuclties in this document.
Try s/unanticipated/undesriable/
END

Section 5
NOTE
s/resource exhaust/resource exhaustion/
END

Section 5
OLD
  Operating test equipment on production networks according to the
  methods described in [RFC2544], where overload is a possible outcome,
  would no doubt be harmful to user traffic performance.  These tests
  MUST NOT be used on production networks and as discussed above, the
  tests will never produce a reliable or accurate benchmarking result
  on a production network.
NOTE
As per this paragraph in the Abstract.
END

Section 6
OLD
6.  What to do without RFC 2544?
NEW
6.  Considering Performance Testing in Production Networks
NOTE
The scope of "life without 2544" may be a little larger than the purpose
of this Section.
END

Section 6
OLD
  The world will not spin off axis while waiting for appropriate and
  standardized methods to emerge from the consensus process.
NOTE
I sense a certain frustration amongst the authors. I don't think this
statement is helpful or respectful to the readers for whom production
testing *is* the whole world.

Would it not be more constructive to suggest that individuals who find
the need for methods and mechanisms to carry out performance testing on
produciton networks to be extremely pressing should participate actively
in the work of the IPPM working group with a view to helping develop the
necessary specifications?
END

[Ballot discuss]
I have read various version of this text, and I think that the authors are still overstating their case.

There are some types of production network that use IETF technology and are designed to provide ingress rate policing, guaranteed bandwidth and path isolation. It is not clear why it should be harmful to the network to run these tests over such a network.

I would support an RFC that advised operators of the issues, but I am not convinced that the use of the technology is necessarily always harmful to their networks.

I think it is important to distinguish between the case where the network is broken for other users, and the case where a test protocol does not work. In the former case the term harmful is applicable, but using the term "harmful" in the essentially "private" case of a test protocol that does not work as expected dilutes the meaning of the IETF keyword "harmful".

[Ballot comment]
If anything needs an "Updates:" header, it seems to me this document does. Is there a reason you have not put in an "Updates: 2544"?

(I wish 2544 were standards track. I wish this one were as well. Sailed ships I expect. No action needed; I'll just go weep in my beer.)

Section 1:

  To directly address this situation, the past and present Chairs of
  the IETF Benchmarking Methodology Working Group (BMWG) have prepared
  this Applicability Statement for RFC 2544.

The mention of the past and present Chairs seems a bit self-aggrandizing. Could you simply say: "This Applicability Statement for RFC 2544 (or even just "This document") directly addresses this situation."

IANA has reviewed draft-ietf-bmwg-2544-as-04, which is currently in
Last Call, and has the following comments:

IANA notes that the authors suggest that there are no IANA actions required by
this document. However, IANA notes the following text in the document in
Section 4.2"

"We note that a range of IPv6 addresses has been assigned to BMWG for
laboratory test purposes, in [RFC5180]. Also, the strong statements
in the Security Considerations Section of this memo make the scope
even more clear; this is now a standard fixture of all BMWG memos."

and strongly suggests that it should be updated to refer to the erratum noting
the actual IPv6 assignment:

http://www.rfc-editor.org/errata_search.php?rfc=5180

Other than that request, IANA understands that, upon approval of this document,
there are no IANA Actions that need completion.

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Subject: Last Call:  (RFC 2544 Applicability Statement: Use on Production Networks Considered Harmful) to Informational RFC


The IESG has received a request from the Benchmarking Methodology WG
(bmwg) to consider the following document:
- 'RFC 2544 Applicability Statement: Use on Production Networks
  Considered Harmful'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2012-06-26. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  Benchmarking Methodology Working Group (BMWG) has been developing key
  performance metrics and laboratory test methods since 1990, and
  continues this work at present.  Recent application of the methods
  beyond their intended scope is cause for concern.  The methods
  described in RFC 2544, where overload is a possible outcome, would no
  doubt be harmful to user traffic performance on a production network.
  This memo clarifies the scope of RFC 2544 and other benchmarking work
  for the IETF community.




The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-bmwg-2544-as/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-bmwg-2544-as/ballot/


No IPR declarations have been submitted directly on this I-D.

(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)?  Why
is this the proper type of RFC?  Is this type of RFC indicated in the
title page header?

  Informational, as indicated on the title page.
  All BMWG RFCs are traditionally Informational,
  in part because they do not define protocols and
  the traditional conditions for standards track advancement
  did not apply.  However, they are specifications and
  the RFC 2119 terms are applicable to identify the
  level of requirements.

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

  Benchmarking Methodology Working Group (BMWG) has been developing key
  performance metrics and laboratory test methods since 1990, and
  continues this work at present.  Recent application of the methods
  beyond their intended scope is cause for concern.  This memo
  clarifies the scope of RFC 2544 and other benchmarking work for the
  IETF community.

Working Group Summary

  Working group consensus was achieved in a single WGLC, after allowing
  a year for review and many useful comments and exchanges.

Document Quality

  The single-sentence message of this memo was clear from first publication.
  Most comments helped to make the wording more concise.

Personnel

  Document Shepherd: Bill Cerveny

  Area Director: Ronald Bonica

(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

  The document shepherd has reviewed this document three times.  There are no
  outstanding issues, as far as the document shepherd is concerned.

(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed? 

  No

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

  No

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

  There are no specific concerns or issues that the document shepherd
  has with this document.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

  Each author has confirmed that there are no known IPR concerns, nor
  are any IPR disclosures required.

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

  No

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

  Many WG members have reviewed this draft, including long-time participants,
  new participants, and (most importantly) participants who work at companies
  most affected by the statements in this memo. WG consensus was called
  by Area Director Ron Bonica, since the current chair and *all past chairs*
  are co-authors.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

  No appeals have been threatened.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

  ID nits have been checked.  There was only one nit identified and this
  appears to be an error. This one id nit complained about addresses
  falling within "non-RFC5735-compliant IPv4 addresses" As far as I can
  tell, the only addresses are in section 4.2 and these are exactly within
  the range described in RFC5735.

  The tools team has been notified as this appears to be a bug or error in
  the nits checking program.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

  Not applicable

(13) Have all references within this document been identified as
either normative or informative?

  Yes

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

  No

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in the
Last Call procedure.

  No

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

  No

(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

  There are no IANA considerations.

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

  None

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

  Not applicable.