Data Center Benchmarking Methodology
draft-ietf-bmwg-dcbench-methodology-18

[Ballot comment]
Clarifying the differences to the tests in RFC2889 addresses my discuss. Thanks and sorry for the delay! I still think describing these tests in a more general form and then (just) discuss if there are specific things to consider when applying these tests for dc testing would have been the better approach but I do understand well that this work only focussed on dc so far and it would be quite some more work to make it more generic.

-----

Old comments:

Please provide the appropriate references for DSCP and COS!

Also I find the name of section 6 confusing ("Incast Stateful and Stateless Traffic ") because your microburst test (section 4) is basically also incast testing but without TCP cross-traffic. Further the terms stateful and stateless are also confusing to me; I'd usually use adaptive and constant bit rate (CBR)/non-adaptive; or is stateful/stateless the commonly-used term in benchmarking?

[Ballot comment]
I agree with Alvaro's discuss -- you can't cite 2119 and then override it. It may well be that what 2119 does not make sense for this document; if so, don't cite it, and be clear up front that you're using a *different* set of meanings for these specific terms.

I *think* I can suss out the nature of "east-west" versus "north-south" in the introduction, but I'm really not sure. Can you please define these terms or point to a document that does so?

Editorial:

- Something has gone well and truly bonkers with the references section formatting.

- Please fix reference [1] so that it correctly points to draft-ietf-bmwg-dcbench-terminology. This will ensure that it is updated to the correct RFC value at publication.

Nits:

Please expand "SUT" on first use.

  ** The abstract seems to contain references ([1]), which it shouldn't.
    Please replace those with straight textual mentions of the documents in
    question.

[Ballot comment]
Nits:

Please expand "SUT" on first use.

  ** The abstract seems to contain references ([1]), which it shouldn't.
    Please replace those with straight textual mentions of the documents in
    question.

[Ballot comment]
- I support Alvaro's DISCUSS

- I agree with the questions about why this is specific to datacenters.

- Please expand DUT on first use.

[Ballot comment]
Not sure if the WG discussed this, but it would have been interesting to have some energy efficiency related metrics as the draft is datacenter focused and energy efficiency is an important consideration in datacenters.

[Ballot comment]
Not sure if the WG discussed this, but it would have been interesting to have something energy efficiency related metrics (in both the terminology and the methodology documents).

[Ballot comment]
At some point in time, I was wondering if this document was about (benchmarking) terminology or about (benchmarking) performance metric definition?
And I was wondering what the connection was with the RFC 6390 PMOL template and with https://tools.ietf.org/html/draft-ietf-ippm-metric-registry-11?
When I look at the section 3 "jitter", there are obvious elements from the PMOL template.
Thinking about it, whether a performance metric is used for benchmark or not, we need a formal definition.
Is it time for:
1. IPPM to finish up draft-ietf-ippm-metric-registry
2. start populating the registry
3. BMWG to start using/specifying those performance metrics

Thoughts?

Below is Tim Chown's OPS DIR review:
verall, the document is well written, and I believe it to be Ready with minor
nits.

General comment:

It would be interesting to see an Appendix with an example of a recorded test
using the language defined here.

Specific comments:

Section 1:

“- Low amount of buffer (in the Mb range)”
Change to MB?  (given later you refer to KB/MB/GB as the measurement unit for
buffers)

Section 2.1

Expand DUT on first use.

Section 3.1

Perhaps clarify relationship of Delay and Latency, since you focus on Latency
in the document and not Delay?

Last para, you say “If” here, but for Latency the FILO timestamp was a MUST in
Section 2?  This doesn’t seem consistent?

Expand PDV on first use.

Section 6.1.1

“1518 bytes frames” -> “1518 byte frames”

Section 7.1,

Why is ‘and’ in quotes here?  Not sure you can say the balance is defined by
goodput?  Do you mean that goodput is an indication of the balance?  For
standard TCP, a very small loss can have a dramatic effect on application
throughput.

The second para should follow the first, change “[RFC2647].  Goodput…” to
“[RFC2647], i.e., goodput…”

Section 7.3

I don’t understand how the example given correlates to G = S/Ft ?

There’s a few typos in this section; please re-check.

[Ballot comment]
(I'm making the same comment on draft-ietf-bmwg-dcbench-terminology and draft-ietf-bmwg-dcbench-methodology)

I'm looking at the ballot positions on draft-ietf-bmwg-dcbench-terminology and draft-ietf-bmwg-dcbench-methodology that assert these documents aren't specific to "data centers". That wouldn't surprise me, but I'm not seeing a definition of "data center" in either document - did I miss it?

I suspect that the authors have specific technical characteristics in mind, that happen to map onto what data centers look like today, but may not in the future ("RFCs last forever"). Is it possible to tease those characteristics out?

(Full disclosure: my first working group in the IETF could have been called "TCP over cellular links", but it turned out that when we said "cellular links", we meant "low-speed links with high loss rates and asymmetric bandwidth". "Cellular links" in the late 1990s didn't have the same characteristics that "cellular links" have in 2017, but there are other link types with those characteristics, so the documents ended up being useful in places like CORE. I'm not suggesting anything like a restructure of the document(s), only that they be clear about how future readers would know whether they should be reading them in 2027)

[Ballot discuss]
This document contains both the RFC2119 boilerplate and specific definitions for MUST, SHOULD/RECOMMENDED and MAY. Even though the additional definitions are close enough to RFC2119, I think that one of them should be taken off to avoid any type of confusion.

Note that the (new) definitions in this document are focused around "metrics", which would not apply to text such as (from 2.2) "A traffic generator SHOULD be connected to all ports on the DUT. Two tests MUST be conducted...".

[Ballot comment]
I support Mirja's DISCUSS.  The authors recognize that the applicability of the tests described may go beyond the DC, so it should be reflected appropriately (starting with the title of the document).  From the Abstract:

  The purpose of this informational document is to establish test and
  evaluation methodology and measurement techniques for physical
  network equipment in the data center. Many of these terms and methods
  may be applicable beyond this publication's scope as the technologies
  originally applied in the data center are deployed elsewhere.

[Ballot discuss]
Again, I don't really understand why any of these tests are specific for data center equipment. These tests can be applied to any equipment. I think the only point is that these tests are especially important to run for data center equipment but that doesn't change the fact that the test itself are not specific for data center equipment. The reason why it's a discuss this time, is that I further really don't understand why existing tests have to be specified again is this document even though a test for that case already exists that specifies exactly the same thing in another RFC; I'm talking about Section 5 on Head of Line Blocking that specifies the same test as in section 5.5 of RFC2889 (there called Congestion Control which a confusing term because this doesn't address end-to-end congestion control, anyway...). I didn't check if that's also the case for any of the other tests.

[Ballot comment]
Please provide the appropriate references for DSCP and COS!

Also I find the name of section 6 confusing ("Incast Stateful and Stateless Traffic ") because your microburst test (section 4) is basically also incast testing but without TCP cross-traffic. Further the terms stateful and stateless are also confusing to me; I'd usually use adaptive and constant bit rate (CBR)/non-adaptive; or is stateful/stateless the commonly-used term in benchmarking?

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Services Operator has reviewed draft-ietf-bmwg-dcbench-methodology-07.txt, which is currently in Last Call, and has the following comments:

We understand that this document doesn't require any registry actions.

While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, we do not object.

If this assessment is not accurate, please respond as soon as possible.

Thank you,

Sabrina Tanamal
IANA Services Specialist
PTI

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
CC: Sarah Banks , bmwg-chairs@ietf.org, bmwg@ietf.org, draft-ietf-bmwg-dcbench-methodology@ietf.org, sbanks@encrypted.net, warren@kumari.net
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (Data Center Benchmarking Methodology) to Informational RFC


The IESG has received a request from the Benchmarking Methodology WG
(bmwg) to consider the following document:
- 'Data Center Benchmarking Methodology'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2017-06-13. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  The purpose of this informational document is to establish test and
  evaluation methodology and measurement techniques for physical
  network equipment in the data center.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-bmwg-dcbench-methodology/

IESG discussion can be tracked via
https://datatracker.ietf.org/doc/draft-ietf-bmwg-dcbench-methodology/ballot/


No IPR declarations have been submitted directly on this I-D.

This is the Publication Request and document shepherd write-up for   

Data Center Benchmarking Methodology and Terminology
  https://tools.ietf.org/html/draft-ietf-bmwg-dcbench-terminology-08
  https://tools.ietf.org/html/draft-ietf-bmwg-dcbench-methodology-06


This version is dated 11 May 2017.

Sarah Banks is the Document Shepherd, and prepared this form: May 2017.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational, as indicated on the title page. All BMWG RFCs are traditionally Informational, in part because they do not define protocols and the traditional conditions for standards track advancement did not apply.  However, they are specifications and the RFC 2119 terms are applicable to identify the level of requirements.

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:
This draft establishes definitions and techniques for benchmarking within the data center. The draft defines a set of definitions, metrics, methodologies and terminologies for benchmarking in a wide variety of traffic conditions.


Working Group Summary:
There has been an extensive amount of work done on this draft, and progress made on revisions, feedback, and comments. There have been extensive conversations on, for example, "goodput". This is one of the most reviewed drafts we've had in BMWG in quite some time, and the document has undergone 2 WGLCs.


Document Quality:
This document is ready for publication.

Personnel:

Who is the Document Shepherd? Who is the Responsible Area Director?
Sarah Banks is the Shepherd, Warren Kumari is the AD.


(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

I've reviewed this draft at WGLC. Nits check is clean. All comments and feedback by the WG have been addressed by the Author.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No. This document has been thoroughly reviewed.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

No. There was extensive discussion and review and revision of this draft, over a long period of time, and the draft underwent 2 WGLCs.


(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

None.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

No IPR has been filed.


(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No IPR disclosed.

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

A second WGLC was called on September 13, 2016, and closed on September 27, 2016. There has been a significant amount of feedback on this draft, reviewed in meetings and on the list, and the latest revisions of the draft include and account for feedback from IETF98. There is solid consensus that this draft is ready for publication.


(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

No nits.

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, media type, and URI type reviews.

NA

(13) Have all references within this document been identified as either normative or informative?

All references are either Normative or Informative, and marked as such.


(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

No

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

None

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 5226).

There are no IANA actions required.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

None.

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, etc.

NA.