Last Call Review of draft-ietf-ippm-model-based-metrics-10
review-ietf-ippm-model-based-metrics-10-genart-lc-sparks-2017-03-13-00

I am the assigned Gen-ART reviewer for this draft. The General Area
Review Team (Gen-ART) reviews all IETF documents being processed
by the IESG for the IETF Chair.  Please treat these comments just
like any other last call comments.

For more information, please see the FAQ at

<https://trac.ietf.org/trac/gen/wiki/GenArtfaq>.

Document: draft-ietf-ippm-model-based-metrics-10
Reviewer: Robert Sparks
Review Date: 2017-03-13
IETF LC End Date: 2017-03-14
IESG Telechat date: Not scheduled for a telechat

Status: Almost ready, but has significant issues to address before
publication as an Informational RFC

I'm guessing the goal for this document changed substantially during its
development. The point and tone if the document changes depending on
where you are in it. I found it very difficult to read (and I did see the
shepherd's report call out that the document has already had a strong
editorial pass).  It really feels to me that the purpose this document is
trying to serve could be better achieved with two much shorter documents.
Please consider pulling out the most important observations into its own
informational document, and an experimental document that succinctly
presents the tests, the framework, and the feedback you want from people
wo try these things.

Major issues:

There are issues with the equations in section 7.2. Please look at the
line containing "Xa < defect_count(n) < Xb". Where is Xb defined? Was
it supposed to be Xr from further up the page? (or perhaps that Xr should
have been Xb?

The document starts using 2119 MUSTs in section 7.3. None of the uses
are appropriate 2119 keywords - they aren't talking about protocol. Please
rewrite the prose without using the 2119 keywords.

Section 8.2.4 seems to be missing an actual description/definition of the 
test it is trying to talk about.


Minor issues:


The abstract says this document does not define tests. The rest of the
document says it does. (See phrases like "The tests described in this
note")

Where the document claims it is neccessary to "suppress the effects of
TCP congestion control", it should say "avoid the effects". The mechanism
of using precalculated traffic patterns discussed in the document avoids,
but does not suppress.

At the end of section 1, you call out "multiple standardized versions of
TCP". Please clarify what you're trying to point to?

In the definition of "traffic patterns", you note that the goal is to
"mimic the range of common patterns". Consider qualifying that to "current
common patterns", and add some discussion later in the document when you
talk about pregenerating traffic patterns to point out that these change
over time. 

At the next to last paragraph of section 4.1, you say "metrics have
entirely thwarted the analytic framework". I don't think that's what you
mean. I suspect you mean to say attempts to create metrics have been
unsuccessful (I suggest you avoid the word "thwarted").

At the 3rd paragraph on page 19, you say "There is no model". Do you mean
no model exists in the world for this, or only that this document does not
provide a model.

I strongly object to the attempt to use "infinitesimal" to describe the
conditions on the test network you are trying to convey. At the very least,
you are dealing with a finite (if arbitrarily large) set of states that
network can be in, not a continuum. You should be looking to raid terms
from discrete mathematics instead, but I don't think even that's the right
thing to do. Please replace the infinitesimal concept here with it's
definition (which you have to call out anyway). Just _say_ "has the
tightest available constraints that allow the tests to pass." 

Consider providing some rational for the numbers you came up with in the
strawman in the second paragraph on page 33. 1mS seems pretty arbitrary
otherwise. I'm not asking for you to _defend_ the choice as the right
number - just talk about why you picked it instead of something else.


Nits:

- Consider pointing to the models you are actually using in the first
  sentence of the first paragraph on page 9

- The third paragraph of the abstract is particularly hard to read. Please
  consider breaking the compound sentences into simpler ones.

- You use "engineering test" before you say what you mean by that (see
  sections 8.2.4 and 8.3.2 for examples)

- There are several places where you talk about tests being inconclusive
  because the precomputed traffic was not accurately generated. Are you
  trying to say "if you have an inconclusive test and can't otherwise figure
  out why, look closely at your precomputed traffic", or "Double-check your
  precomputed traffic before you run your test or risk garbage-in/garbage
  out"?