Benchmarking Virtual Switches in the Open Platform for NFV (OPNFV)
draft-ietf-bmwg-vswitch-opnfv-04

Note: This ballot was opened for revision 03 and is now closed.

Warren Kumari Yes

Alia Atlas No Objection

Deborah Brungard No Objection

Ben Campbell No Objection

Comment (2017-06-05 for -03)
-General:

It seems a bit odd to me to use an IETF stream RFC to describe the status of an open source project, even when that project is closely related to IETF work. But I do not object strongly enough to get in the way of publication.

-3.4: "It is essential to increase the flow timeout   time on a vswitch before conducting any performance tests that do not   intend to measure the flow setup time."

Does this mean to make allowances for the startup characteristics of virtual network elements, when physical elements might not have such limitations?  That seems sort of like optimizing for the test.

-4, figures:

Figure numbers and cross references would be very helpful for this section.

Benoit Claise No Objection

Comment (2017-06-08 for -03)
For sure, the topic of benchmarking of benchmarking virtual switches is important in the industry today.

Technical Summary: This draft summarizes the progress of the OPNFV project on virtual switch performance characterization, and notes that the OPNFV work builds upon the IETF work, particularly the BMWG work within IETF, with an emphasis on RFC2544.

I like the cross SDO aspects of this draft, as mentioned in the shepherd writeup:
The document itself was reviewed in ETSI NFV, and used as a reference by at least one of the TST WG and IFA WF specifications (2 references in total). The document has also been reviewed by several individuals from the OPNFV community, and NFVRG (IRTF) is aware of the draft.

I would have preferred to see a document that specifies "benchmarking virtual switches" instead of "benchmarking virtual switches in OPNFV".
In other words, just looking at the abstract, this is completely different to write:

   This memo describes the way to benchmark virtual switch performance.
   The specifications are based on the Open Platform for NFV (OPNFV) project on virtual switch performance "VSPERF".  

As opposed to:

   This memo describes the progress of the Open Platform for NFV (OPNFV)
   project on virtual switch performance "VSPERF". 

Regarding the "describes the progress of", sentences such as ...
    - This project intends to build ...
    - Therefore, this memo begins to describe the additional considerations ...
    - This memo summarizes the progress of the Open Platform for NFV (OPNFV) project on virtual switch performance characterization, "VSPERF", through the Brahmaputra (second) release [BrahRel].
... lead me to think that this is unfinished work, or that work will anyway have to be updated.

Again, looking at the scope:

   The primary purpose and scope of the memo is to inform the industry
   of work-in-progress that builds on the body of extensive BMWG
   literature and experience, and describe the extensions needed for
   benchmarking virtual switches.

I was hoping that the primary purpose and scope of the memo would be the specifications of the virtual switch benchmarking, not to inform about the work-in-progress.

Reading further, I see text such as:

    Specifications to be included in future updates of the LTD include:

       o  [RFC3918] Methodology for IP Multicast Benchmarking

       o  [RFC4737] Packet Reordering Metrics

or

   When considering characteristics important to "telco" network
   functions, we must begin to consider additional performance metrics.

or

   Tests have been (or will be) designed to collect the metrics below:


On one side, I appreciate that you're publishing what you learned, which should lead up to revisions in the future.
On the other side, I've not been used to that approach in BMWG.
I'll trust the responsible AD regarding the publication of this document as a kind of intermediary document.

I believe the document would be a better flow (and that some concerns would disappear) if it would have the following structure
    - benchmarking virtual switches is key in the industry today
    - there are the virtual switches benchmarking specification today
    - we worked on this with OPNFV (VSPERF). Great collaboration.
    - this is a complex and evolving field:
      compared to RFC 2544, we learn that ...
      we realize that this is work in progress ...
    - future plan: we will need to integrate more "stuff" in the benchmark, such as RFC3918, RFC4737, additional perf metrics, you-name-it
    - we'll work with OPNFV (and other groups) on this future plan.

Editorial
- Section 3.4. Flow timeout.
You mean the active timeout, section 2.2.23 in RFC 6645?
Should this section refer to RFC6645, or RFC 5470 (section 5.1.1)?

-

   In addition to this, the LTD also re-uses the terminology defined by:

   o  [RFC2285] Benchmarking Terminology for LAN Switching Devices

   o  [RFC5481] Packet Delay Variation Applicability Statement

RFC5481 is already mentioned in the previous bullet list (The base performance tests in the LTD are based on the pre-existing specifications developed by BMWG to test the performance of physical switches. These specifications include). So hopefully, it re-uses the terminology, right?

Alissa Cooper (was Discuss) No Objection

Comment (2017-06-09)
Thanks for addressing my DISCUSS and comments. One nit:

s/for a future draft/for a future document/

Mirja K├╝hlewind No Objection

Comment (2017-05-29 for -03)
One minor question:
sec 3.4: "The recommended flow classification parameters for any vswitch
   performance tests are: the input port (physical or logical), the
   source IP address, the destination IP address and the Ethernet
   protocol type field."
Why does this not include the transport ports? Is it assumed that there is always only one flow between two IP addresses during a test?

Minor editorial comment:
Not sure why it is so important to mention several times that this work is done "by referencing existing literature" (without referencing any additional work beside opnfv)...?

Kathleen Moriarty No Objection

Adam Roach No Objection

Comment (2017-06-07 for -03)
I think this is a variation on the issue that Alissa calls out, but I'm having a hard time reconciling:

   It's unlikely that the virtual switch will be the only application
   running on the System Under Test (SUT), so CPU utilization, Cache
   utilization, and Memory footprint should also be recorded for the
   virtual implementations of internetworking functions.

...with...

   Further, benchmarking is performed on a "black-box" basis, relying
   solely on measurements observable external to the DUT/SUT.

Please add text that clarifies how the metrics that 3.1 says should be recorded in section 3.1 relate to benchmarking.

Terry Manderson Abstain

Comment (2017-06-07 for -03)
I'm not comfortable using the RFC series to describe the _progress_ of the Open Platform for NFV. This is orthogonal to the immutable nature of the RFC series as in a very short period the progress on the Open Platform for NFV will be different, and requiring a BIS or other revisions.  Thus to "inform the industry of work-in-progress" (to quote the draft) is not something that I see as a part of the RFC series. 

I shall not stand in the way of publication and have balloted ABSTAIN. However, my advice is to seek other avenues for publication more in line with the white-paper that is draft-ietf-bmwg-vswitch-opnfv.

==-=-=-=-=- added comment
I should also add that I have seen the response in relation to Alvaro's ABSTAIN, and that does not sway me. If you are interested in documenting the the test specifications that ARE implemented and any associated metrics, then that I think is a different document, especially when this document has such words as "Future/planned test specs include"...

Alvaro Retana Abstain

Comment (2017-06-07 for -03)
I don't see the value of publishing this document as an RFC, specially because it reflects the progress of the VSPERF project "through the Brahmaputra (second) release" -- which would make this document obsolete instantly since the project is already working on their Danube (fourth) release.  I appreciate the fact that the WG has found the discussions around this document useful and that the methodologies created are being referenced elsewhere.

Because the WG/Chairs/Responsible AD find value in publication I am not going to stand in the way; I am then ABSTAINing.