Applicability Statement for RFC 2544: Use on Production Networks Considered Harmful
draft-ietf-bmwg-2544-as-08

Note: This ballot was opened for revision 05 and is now closed.

(Ron Bonica) Yes

(Adrian Farrel) Yes

Comment (2012-09-11 for -06)
No email
send info
I am retaining my Yes ballot, and I thank the authors for addressing my
Comments entered against revision 05.

It continues to be my opinion that, in stating an important point about
the use of these tests on live, and dynamically routed, shared-resource 
IP/MPLS networks, the document overstates its case an implies that the
use on network paths in dedicated-resource networks (such as MPLS-TE or
MPLS-TP) is harmful when it is neither obvious nor proven that this is
the case. In this, I agree with Stewart's Discuss and wish that the
authors would make a clearer and more upfront statement about the scope
of the network in which this may be harmful.

The solution to this may be as simple as a change to the title to 
something like:

        Use of RFC 2544 Benchmarking Test Methodologies on
    Production Networks with Shared Resources Considered Harmful

In making this change you would not be commenting about TE or transport
networks, but would be making the clear statement about where you know
the harm to be.

---

New typo introduced...
Section 4.2
   In other words, devices operating on the Internet may be configured
   to discard any traffic they observe in this address range, as it is
   intended for laboratory ITE use only.  Thus, testers using the
   assigned testing address ranges are connected to the Internet and
   test packets are forwarded across the Internet, it is likely that the
   packets will be discarded and the test will not work.

s/Thus, testers/Thus, if testers/

(Stewart Bryant) (was Discuss) No Objection

Comment (2012-10-22)
No email
send info
Thank you for your work you have done to address my concerns.

(Gonzalo Camarillo) No Objection

(Benoît Claise) No Objection

Comment (2012-09-13 for -06)
No email
send info
I agree with the intend of the draft. Hence no objection.

Text from Al Morton
We're saying that almost any test using RFC2544 methods
on a production network will produce a form of harm from wrong
and misleading results with inherent wasted time and resources,
and that's just one of the ways we defined harm in version 05
clarifications.  In the test and measurement community
(the main audience of this draft), frequent flawed results are
the end of the world with too few transport ships ready to go.

I agree with Al: "a form of harm from wrong and misleading results"

While I understand Stewart's point (a customer could use RFC2544 to test his CIR), the Device Under Test (DUT) is a black box, and BMWG does black box testing. Per the "black box" definition, we can't control or even monitor what's in it.
By testing live network, the DUT becomes the network (or a circuit). And we have no way to know what we influence, if anything This is the way I understand harmful.

That being said. Harmful might be understood in a different way.
The title "Use on Production Networks Considered Harmful" might be a little bit alarming...
While the text in the draft is more realistic
- abstract  "Overload of shared resources would likely be harmful to user traffic performance on a production network"
- section 5 "Overload of shared resources would likely be harmful to user traffic performance on a production network"

Text from Scott Bradner
We feel that use of the 2544 in production networks is harmful in the sense that IETF has been using that term for a very long time. Some of the tests would harm other users of the network and some of the tests would get bad (misleading) results.

Any wording that would soften the word "harmful" would be a step in the right direction IMHO.
Alternatively, your own definition of harmful would also help. For example, "By harmful, we mean that the consequences are unknown/unpredictable/etc ..." or "By harmful, we mean: a form of harm from wrong and misleading results"

(Ralph Droms) No Objection

(Wesley Eddy) No Objection

(Stephen Farrell) No Objection

Comment (2012-08-27 for -05)
No email
send info
- section 1: "unintended specifications" sound quite odd, I think you
could usefully re-phrase that.

(Brian Haberman) No Objection

(Russ Housley) No Objection

Barry Leiba No Objection

Comment (2012-08-29 for -05)
No email
send info
   To directly address this situation, the past and present Chairs of
   the IETF Benchmarking Methodology Working Group (BMWG) have prepared
   this Applicability Statement for RFC 2544.

I'm going to push this also, where Pete gave it up... though it's still non-blocking.  I very strongly ask you to change this. 

(1) Working group documents are the products of working groups, not of individuals.

(2) Statements like this imply that people in positions of leadership rammed something through, forcing "consensus" with a bulldozer.

(3) At best, statements like this appear to be appeal-to-authority arguments.  We should avoid that when it's not necessary, and I don't see that it's necessary here.

If BMWG has strong consensus on what this document says, then say *that*.  That statement means more to me than one that says that four chairs thought it was important.  (I'll also note that when you couple that statement with the Acknowledgments section, which recognizes five people and doesn't mention the working group, one *could* take away from this that there were nine people who read and agreed with this.  Saying instead -- explicitly, rather than the usual implicit message we give -- that this represents *strong* consensus of the BMWG makes it very clear.)

Apart from that, I'll let Stewart and Adrian sort out whether or not you're overstating anything.

(Pete Resnick) No Objection

Comment (2012-08-28 for -05)
No email
send info
If anything needs an "Updates:" header, it seems to me this document does. Is there a reason you have not put in an "Updates: 2544"?

(I wish 2544 were standards track. I wish this one were as well. Sailed ships I expect. No action needed; I'll just go weep in my beer.)

Section 1:

   To directly address this situation, the past and present Chairs of
   the IETF Benchmarking Methodology Working Group (BMWG) have prepared
   this Applicability Statement for RFC 2544.

The mention of the past and present Chairs seems a bit self-aggrandizing. Could you simply say: "This Applicability Statement for RFC 2544 (or even just "This document") directly addresses this situation."

(Robert Sparks) No Objection

(Martin Stiemerling) No Objection

(Sean Turner) No Objection