Applicability Statement for RFC 2544: Use on Production Networks Considered Harmful
draft-ietf-bmwg-2544-as-08

   To directly address this situation, the past and present Chairs of
   the IETF Benchmarking Methodology Working Group (BMWG) have prepared
   this Applicability Statement for RFC 2544.

I'm going to push this also, where Pete gave it up... though it's still non-blocking.  I very strongly ask you to change this. 

(1) Working group documents are the products of working groups, not of individuals.

(2) Statements like this imply that people in positions of leadership rammed something through, forcing "consensus" with a bulldozer.

(3) At best, statements like this appear to be appeal-to-authority arguments.  We should avoid that when it's not necessary, and I don't see that it's necessary here.

If BMWG has strong consensus on what this document says, then say *that*.  That statement means more to me than one that says that four chairs thought it was important.  (I'll also note that when you couple that statement with the Acknowledgments section, which recognizes five people and doesn't mention the working group, one *could* take away from this that there were nine people who read and agreed with this.  Saying instead -- explicitly, rather than the usual implicit message we give -- that this represents *strong* consensus of the BMWG makes it very clear.)

Apart from that, I'll let Stewart and Adrian sort out whether or not you're overstating anything.

I agree with the intend of the draft. Hence no objection.

Text from Al Morton
We're saying that almost any test using RFC2544 methods
on a production network will produce a form of harm from wrong
and misleading results with inherent wasted time and resources,
and that's just one of the ways we defined harm in version 05
clarifications.  In the test and measurement community
(the main audience of this draft), frequent flawed results are
the end of the world with too few transport ships ready to go.

I agree with Al: "a form of harm from wrong and misleading results"

While I understand Stewart's point (a customer could use RFC2544 to test his CIR), the Device Under Test (DUT) is a black box, and BMWG does black box testing. Per the "black box" definition, we can't control or even monitor what's in it.
By testing live network, the DUT becomes the network (or a circuit). And we have no way to know what we influence, if anything This is the way I understand harmful.

That being said. Harmful might be understood in a different way.
The title "Use on Production Networks Considered Harmful" might be a little bit alarming...
While the text in the draft is more realistic
- abstract  "Overload of shared resources would likely be harmful to user traffic performance on a production network"
- section 5 "Overload of shared resources would likely be harmful to user traffic performance on a production network"

Text from Scott Bradner
We feel that use of the 2544 in production networks is harmful in the sense that IETF has been using that term for a very long time. Some of the tests would harm other users of the network and some of the tests would get bad (misleading) results.

Any wording that would soften the word "harmful" would be a step in the right direction IMHO.
Alternatively, your own definition of harmful would also help. For example, "By harmful, we mean that the consequences are unknown/unpredictable/etc ..." or "By harmful, we mean: a form of harm from wrong and misleading results"

If anything needs an "Updates:" header, it seems to me this document does. Is there a reason you have not put in an "Updates: 2544"?

(I wish 2544 were standards track. I wish this one were as well. Sailed ships I expect. No action needed; I'll just go weep in my beer.)

Section 1:

   To directly address this situation, the past and present Chairs of
   the IETF Benchmarking Methodology Working Group (BMWG) have prepared
   this Applicability Statement for RFC 2544.

The mention of the past and present Chairs seems a bit self-aggrandizing. Could you simply say: "This Applicability Statement for RFC 2544 (or even just "This document") directly addresses this situation."

- section 1: "unintended specifications" sound quite odd, I think you
could usefully re-phrase that.

Thank you for your work you have done to address my concerns.