Data Center Benchmarking Methodology
draft-ietf-bmwg-dcbench-methodology-18

I agree with Alvaro's discuss -- you can't cite 2119 and then override it. It may well be that what 2119 does not make sense for this document; if so, don't cite it, and be clear up front that you're using a *different* set of meanings for these specific terms.

I *think* I can suss out the nature of "east-west" versus "north-south" in the introduction, but I'm really not sure. Can you please define these terms or point to a document that does so?

Editorial:

- Something has gone well and truly bonkers with the references section formatting.

- Please fix reference [1] so that it correctly points to draft-ietf-bmwg-dcbench-terminology. This will ensure that it is updated to the correct RFC value at publication.

Nits:

Please expand "SUT" on first use.

  ** The abstract seems to contain references ([1]), which it shouldn't. 
     Please replace those with straight textual mentions of the documents in
     question.

[Thanks for addressing my DISCUSS.]

- I support Alvaro's DISCUSS

- I agree with the questions about why this is specific to datacenters.

- Please expand DUT on first use.

At some point in time, I was wondering if this document was about (benchmarking) terminology or about (benchmarking) performance metric definition?
And I was wondering what the connection was with the RFC 6390 PMOL template and with https://tools.ietf.org/html/draft-ietf-ippm-metric-registry-11?
When I look at the section 3 "jitter", there are obvious elements from the PMOL template.
Thinking about it, whether a performance metric is used for benchmark or not, we need a formal definition.
Is it time for:
1. IPPM to finish up draft-ietf-ippm-metric-registry
2. start populating the registry
3. BMWG to start using/specifying those performance metrics

Thoughts?

Below is Tim Chown's OPS DIR review:
verall, the document is well written, and I believe it to be Ready with minor
nits.

General comment:

It would be interesting to see an Appendix with an example of a recorded test
using the language defined here.

Specific comments:

Section 1:

“- Low amount of buffer (in the Mb range)”
Change to MB?  (given later you refer to KB/MB/GB as the measurement unit for
buffers)

Section 2.1

Expand DUT on first use.

Section 3.1

Perhaps clarify relationship of Delay and Latency, since you focus on Latency
in the document and not Delay?

Last para, you say “If” here, but for Latency the FILO timestamp was a MUST in
Section 2?  This doesn’t seem consistent?

Expand PDV on first use.

Section 6.1.1

“1518 bytes frames” -> “1518 byte frames”

Section 7.1,

Why is ‘and’ in quotes here?  Not sure you can say the balance is defined by
goodput?  Do you mean that goodput is an indication of the balance?  For
standard TCP, a very small loss can have a dramatic effect on application
throughput.

The second para should follow the first, change “[RFC2647].  Goodput…” to
“[RFC2647], i.e., goodput…”

Section 7.3

I don’t understand how the example given correlates to G = S/Ft ?

There’s a few typos in this section; please re-check.

Clarifying the differences to the tests in RFC2889 addresses my discuss. Thanks and sorry for the delay! I still think describing these tests in a more general form and then (just) discuss if there are specific things to consider when applying these tests for dc testing would have been the better approach but I do understand well that this work only focussed on dc so far and it would be quite some more work to make it more generic.

-----

Old comments:

Please provide the appropriate references for DSCP and COS!

Also I find the name of section 6 confusing ("Incast Stateful and Stateless Traffic ") because your microburst test (section 4) is basically also incast testing but without TCP cross-traffic. Further the terms stateful and stateless are also confusing to me; I'd usually use adaptive and constant bit rate (CBR)/non-adaptive; or is stateful/stateless the commonly-used term in benchmarking?

(I'm making the same comment on draft-ietf-bmwg-dcbench-terminology and draft-ietf-bmwg-dcbench-methodology)

I'm looking at the ballot positions on draft-ietf-bmwg-dcbench-terminology and draft-ietf-bmwg-dcbench-methodology that assert these documents aren't specific to "data centers". That wouldn't surprise me, but I'm not seeing a definition of "data center" in either document - did I miss it?

I suspect that the authors have specific technical characteristics in mind, that happen to map onto what data centers look like today, but may not in the future ("RFCs last forever"). Is it possible to tease those characteristics out?

(Full disclosure: my first working group in the IETF could have been called "TCP over cellular links", but it turned out that when we said "cellular links", we meant "low-speed links with high loss rates and asymmetric bandwidth". "Cellular links" in the late 1990s didn't have the same characteristics that "cellular links" have in 2017, but there are other link types with those characteristics, so the documents ended up being useful in places like CORE. I'm not suggesting anything like a restructure of the document(s), only that they be clear about how future readers would know whether they should be reading them in 2027)

Not sure if the WG discussed this, but it would have been interesting to have some energy efficiency related metrics as the draft is datacenter focused and energy efficiency is an important consideration in datacenters.

There are two discusses in place that perfectly capture my thoughts.