Last Call Review of draft-ietf-bmwg-sdn-controller-benchmark-meth-07

Request Review of draft-ietf-bmwg-sdn-controller-benchmark-meth
Requested rev. no specific revision (document currently at 09)
Type Last Call Review
Team Ops Directorate (opsdir)
Deadline 2018-02-02
Requested 2018-01-19
Other Reviews Rtgdir Last Call review of -07 by Henning Rogge (diff)
Genart Last Call review of -07 by Stewart Bryant (diff)
Secdir Last Call review of -07 by Russ Housley (diff)
Genart Telechat review of -08 by Stewart Bryant (diff)
Review State Completed
Reviewer Scott Bradner
Review review-ietf-bmwg-sdn-controller-benchmark-meth-07-opsdir-lc-bradner-2018-01-29
Posted at
Reviewed rev. 07 (document currently at 09)
Review result Has Issues
Draft last updated 2018-01-29
Review completed: 2018-01-29


This ID specifies the methodology to be used in testing the performance of SDN controllers. It is a standard BMWG methodology document and thus, cannot have any impact, operational or otherwise, on operational networks - see RFC 6815

follow are some comments on the document itself

section 4.1 - Leaf-Spine used but not defined 
section 4.4 - using old software versions seems to add complexity with little benefit
section 4.7 - would be useful to say that the variance over the repeated tests should be reported in the "test reporting" section - e.g., in section 5.1.1 I would add a row that reports the variance - same for all tests
section 5.1.1 - is the topology discovery process slow enough that a 3 second granularity of the measurement will show differences between systems? 
section 5.1.2 - procedure step 2 - what inserts the timestamp in the response message (R1)?
section 5.1.2 - the ID says "successful messages exchanged" - might it be useful to report the % of unsuccessful exchanges?
section 5.1.6 - the title & objective description do not match - the title says "rate" but the objective is a count - the test seems to try to get the rate
section 5.1.7 - same issue as section 5.1.6
section 5.2.3 - procedure step 1 - this reads as if the addresses are unique but unchanging - if that is not what is meant then it should specifically say that the addresses are changing
section 5.3.1 - it might be useful to establish specific "invalid messages" since I could imagine that the devices could handle different types in different ways that could have an impact on speed of handling
section 5.3.2 - "a huge number" is somewhat undefined 
	do you mean to say TCP SYN messages rather than TCP SYNC messages?
section 6 - RFC 2026 is referenced in the introduction but not listed in the references
section 9 - I have now retired so no affiliation should be listed