Telechat Review of draft-ietf-bmwg-sdn-controller-benchmark-term-09
review-ietf-bmwg-sdn-controller-benchmark-term-09-genart-telechat-bryant-2018-04-16-00

I am the assigned Gen-ART reviewer for this draft. The General Area
Review Team (Gen-ART) reviews all IETF documents being processed
by the IESG for the IETF Chair. Please wait for direction from your
document shepherd or AD before posting a new version of the draft.

For more information, please see the FAQ at

<https://trac.ietf.org/trac/gen/wiki/GenArtfaq>.

Document: draft-ietf-bmwg-sdn-controller-benchmark-term-09
Reviewer: Stewart Bryant
Review Date: 2018-04-16
IETF LC End Date: 2018-02-02
IESG Telechat date: 2018-04-19

Summary: Generally a well written document. The various comments I make are
mostly editorial with some on the fringe of being technical. I do not seem to
have received a response to a number of issues raised, and as far as I can see
there is no change to the text in the issues noted below. Apologies if I have
missed your email.

Major issues: None

Minor issues:

I do not seem to have received a response to this issue, and as far as I can
see there is no change to the text. Apologies if I have missed your email.

*rate* implies actions per unit time, but this does not seem to feature in your
definition.

2.3.1.3. Asynchronous Message Processing Rate

Definition:
   The number responses to asynchronous messages (such as new flow
SB> That should be the number of responses per second.

Discussion:
   As SDN assures flexible network and agile provisioning, it is
   important to measure how many network events the controller can
   handle at a time. This benchmark is obtained by sending asynchronous
   messages from every connected Network Device at the rate that the
   controller processes (without dropping them).

SB> So what you are testing here is the control network and the
SB> controller. This is perhaps the only practical way to run the
SB> test, but it seems a pity that you do not deconvolve these
SB> two aspects of the test.
SB>
SB> I suppose this is really network Async Msg Proc rate rather than
SB> controller Async proc rate.
SB>
SB> We may get to this in the companion document, but doesn't there
SB> need to be some standardization of the event message to compare
SB> apple with apples over time?

Nits/editorial comments:

2.2.4. Number of Cluster nodes

Discussion:
   This parameter is relevant when testing the controller performance
   in clustering/teaming mode. The number of nodes in the cluster MUST
   be greater than 1.

SB> I see what you are saying, but you may wish to clarify that this
SB> constraint does not apply all the time. For example one of two nodes
SB> may start later than another, or fail, or maybe I worry over nothing here.

2.3.2.1. Control Sessions Capacity

Measurement Units:
SB> Surely this should be in units of sessions?

2.3.2.2. Network Discovery Size

Measurement Units:

   N/A

SB> How can this be N/A surely it is a number of network units of various type.

2.3.2.3. Forwarding Table Capacity