Benchmarking Virtual Switches in the Open Platform for NFV (OPNFV)
draft-ietf-bmwg-vswitch-opnfv-04

[Ballot comment]
For sure, the topic of benchmarking of benchmarking virtual switches is important in the industry today.

Technical Summary: This draft summarizes the progress of the OPNFV project on virtual switch performance characterization, and notes that the OPNFV work builds upon the IETF work, particularly the BMWG work within IETF, with an emphasis on RFC2544.

I like the cross SDO aspects of this draft, as mentioned in the shepherd writeup:
The document itself was reviewed in ETSI NFV, and used as a reference by at least one of the TST WG and IFA WF specifications (2 references in total). The document has also been reviewed by several individuals from the OPNFV community, and NFVRG (IRTF) is aware of the draft.

I would have preferred to see a document that specifies "benchmarking virtual switches" instead of "benchmarking virtual switches in OPNFV".
In other words, just looking at the abstract, this is completely different to write:

  This memo describes the way to benchmark virtual switch performance.
  The specifications are based on the Open Platform for NFV (OPNFV) project on virtual switch performance "VSPERF". 

As opposed to:

  This memo describes the progress of the Open Platform for NFV (OPNFV)
  project on virtual switch performance "VSPERF".

Regarding the "describes the progress of", sentences such as ...
    - This project intends to build ...
    - Therefore, this memo begins to describe the additional considerations ...
    - This memo summarizes the progress of the Open Platform for NFV (OPNFV) project on virtual switch performance characterization, "VSPERF", through the Brahmaputra (second) release [BrahRel].
... lead me to think that this is unfinished work, or that work will anyway have to be updated.

Again, looking at the scope:

  The primary purpose and scope of the memo is to inform the industry
  of work-in-progress that builds on the body of extensive BMWG
  literature and experience, and describe the extensions needed for
  benchmarking virtual switches.

I was hoping that the primary purpose and scope of the memo would be the specifications of the virtual switch benchmarking, not to inform about the work-in-progress.

Reading further, I see text such as:

    Specifications to be included in future updates of the LTD include:

      o  [RFC3918] Methodology for IP Multicast Benchmarking

      o  [RFC4737] Packet Reordering Metrics

or

  When considering characteristics important to "telco" network
  functions, we must begin to consider additional performance metrics.

or

  Tests have been (or will be) designed to collect the metrics below:


On one side, I appreciate that you're publishing what you learned, which should lead up to revisions in the future.
On the other side, I've not been used to that approach in BMWG.
I'll trust the responsible AD regarding the publication of this document as a kind of intermediary document.

I believe the document would be a better flow (and that some concerns would disappear) if it would have the following structure
    - benchmarking virtual switches is key in the industry today
    - there are the virtual switches benchmarking specification today
    - we worked on this with OPNFV (VSPERF). Great collaboration.
    - this is a complex and evolving field:
      compared to RFC 2544, we learn that ...
      we realize that this is work in progress ...
    - future plan: we will need to integrate more "stuff" in the benchmark, such as RFC3918, RFC4737, additional perf metrics, you-name-it
    - we'll work with OPNFV (and other groups) on this future plan.

Editorial
- Section 3.4. Flow timeout.
You mean the active timeout, section 2.2.23 in RFC 6645?
Should this section refer to RFC6645, or RFC 5470 (section 5.1.1)?

-

  In addition to this, the LTD also re-uses the terminology defined by:

  o  [RFC2285] Benchmarking Terminology for LAN Switching Devices

  o  [RFC5481] Packet Delay Variation Applicability Statement

RFC5481 is already mentioned in the previous bullet list (The base performance tests in the LTD are based on the pre-existing specifications developed by BMWG to test the performance of physical switches. These specifications include). So hopefully, it re-uses the terminology, right?

[Ballot comment]
I'm not comfortable using the RFC series to describe the _progress_ of the Open Platform for NFV. This is orthogonal to the immutable nature of the RFC series as in a very short period the progress on the Open Platform for NFV will be different, and requiring a BIS or other revisions.  Thus to "inform the industry of work-in-progress" (to quote the draft) is not something that I see as a part of the RFC series.

I shall not stand in the way of publication and have balloted ABSTAIN. However, my advice is to seek other avenues for publication more in line with the white-paper that is draft-ietf-bmwg-vswitch-opnfv.

==-=-=-=-=- added comment
I should also add that I have seen the response in relation to Alvaro's ABSTAIN, and that does not sway me. If you are interested in documenting the the test specifications that ARE implemented and any associated metrics, then that I think is a different document, especially when this document has such words as "Future/planned test specs include"...

[Ballot comment]
I'm not comfortable using the RFC series to describe the _progress_ of the Open Platform for NFV. This is orthogonal to the immutable nature of the RFC series as in a very short period the progress on the Open Platform for NFV will be different, and requiring a BIS or other revisions.  Thus to "inform the industry of work-in-progress" (to quote the draft) is not something that I see as a part of the RFC series.

I shall not stand in the way of publication and have balloted ABSTAIN. However, my advice is to seek other avenues for publication more in line with the white-paper that is draft-ietf-bmwg-vswitch-opnfv.

[Ballot comment]
I'm not comfortable using the RFC series to describe the _progress_ of the Open Platform for NFV. This is orthogonal to the immutable nature of the RFC series as in a very short period the progress on the Open Platform for NFV will be different, and requiring a BIS or other revisions.  Thus to "inform the industry of work-in-progress" (to quote the draft) is not something that I see as a part of the RFC series.

I shall not stand in the way of publication and have balloted ABSTAIN. However, my advise is to seek other avenues for publication more in line with the white-paper that is draft-ietf-bmwg-vswitch-opnfv.

[Ballot comment]
I think this is a variation on the issue that Alissa calls out, but I'm having a hard time reconciling:

  It's unlikely that the virtual switch will be the only application
  running on the System Under Test (SUT), so CPU utilization, Cache
  utilization, and Memory footprint should also be recorded for the
  virtual implementations of internetworking functions.

...with...

  Further, benchmarking is performed on a "black-box" basis, relying
  solely on measurements observable external to the DUT/SUT.

Please add text that clarifies how the metrics that 3.1 says should be recorded in section 3.1 relate to benchmarking.

[Ballot discuss]
Picking up on the thread that started with the Gen-ART review, I'm not clear on the position of this document when it comes to repeatability. If all of the parameters listed in Section 3.3 (assuming they apply) are configured and documented, is it assumed that the benchmarks will be repeatable and comparable with hardware implementation benchmarks? Or is the position of the document that the benchmarks are not likely to be repeatable/comparable in some (many?) cases, given the increase in complexity? Or that more work needs to be done (outside the specification process) to achieve repeatability? I think this needs to be more clear in the document.

[Ballot comment]
I understand the debate about the publication of this document as an IETF consensus RFC. To me it seems valuable to publish and that this item is within the WG's charter. I think adding the updates necessary to make the document up-to-date with the Danube 3.0 release as suggested in the thread with Alvaro would be valuable.

[Ballot comment]
I don't see the value of publishing this document as an RFC, specially because it reflects the progress of the VSPERF project "through the Brahmaputra (second) release" -- which would make this document obsolete instantly since the project is already working on their Danube (fourth) release.  I appreciate the fact that the WG has found the discussions around this document useful and that the methodologies created are being referenced elsewhere.

Because the WG/Chairs/Responsible AD find value in publication I am not going to stand in the way; I am then ABSTAINing.

[Ballot comment]
-General:

It seems a bit odd to me to use an IETF stream RFC to describe the status of an open source project, even when that project is closely related to IETF work. But I do not object strongly enough to get in the way of publication.

-3.4: "It is essential to increase the flow timeout  time on a vswitch before conducting any performance tests that do not  intend to measure the flow setup time."

Does this mean to make allowances for the startup characteristics of virtual network elements, when physical elements might not have such limitations?  That seems sort of like optimizing for the test.

-4, figures:

Figure numbers and cross references would be very helpful for this section.

[Ballot comment]
One minor question:
sec 3.4: "The recommended flow classification parameters for any vswitch
  performance tests are: the input port (physical or logical), the
  source IP address, the destination IP address and the Ethernet
  protocol type field."
Why does this not include the transport ports? Is it assumed that there is always only one flow between two IP addresses during a test?

Minor editorial comment:
Not sure why it is so important to mention several times that this work is done "by referencing existing literature" (without referencing any additional work beside opnfv)...?

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Services Operator has reviewed draft-ietf-bmwg-vswitch-opnfv-03.txt, which is currently in Last Call, and has the following comments:

We understand that this document doesn't require any registry actions.

While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, we do not object.

If this assessment is not accurate, please respond as soon as possible.

Thank you,

Sabrina Tanamal
IANA Services Specialist
PTI

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
CC: Sarah Banks , bmwg-chairs@ietf.org, draft-ietf-bmwg-vswitch-opnfv@ietf.org, bmwg@ietf.org, sbanks@encrypted.net, warren@kumari.net
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (Benchmarking Virtual Switches in OPNFV) to Informational RFC


The IESG has received a request from the Benchmarking Methodology WG
(bmwg) to consider the following document:
- 'Benchmarking Virtual Switches in OPNFV'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2017-05-15. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This memo describes the progress of the Open Platform for NFV (OPNFV)
  project on virtual switch performance "VSPERF".  This project intends
  to build on the current and completed work of the Benchmarking
  Methodology Working Group in IETF, by referencing existing
  literature.  The Benchmarking Methodology Working Group has
  traditionally conducted laboratory characterization of dedicated
  physical implementations of internetworking functions.  Therefore,
  this memo begins to describe the additional considerations when
  virtual switches are implemented in general-purpose hardware.  The
  expanded tests and benchmarks are also influenced by the OPNFV
  mission to support virtualization of the "telco" infrastructure.





The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-bmwg-vswitch-opnfv/

IESG discussion can be tracked via
https://datatracker.ietf.org/doc/draft-ietf-bmwg-vswitch-opnfv/ballot/


No IPR declarations have been submitted directly on this I-D.

This is the Publication Request and document shepherd write-up for   

Considerations for Benchmarking Virtual Network Functions and Their Infrastructure
  https://tools.ietf.org/html/draft-ietf-bmwg-vswitch-opnfv-02

This version is dated 17 April 2017.

Sarah Banks is the Document Shepherd, and prepared this form: April 2017.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational, as indicated on the title page. All BMWG RFCs are traditionally Informational, in part because they do not define protocols and the traditional conditions for standards track advancement did not apply.  However, they are specifications and the RFC 2119 terms are applicable to identify the level of requirements.

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:
This draft summarizes the progress of the OPNFV project on virtual switch performance characterization, and notes that the OPNFV work builds upon the IETF work, particularly the BMWG work within IETF, with an emphasis on RFC2544.


Working Group Summary:
There has been a fair amount of work done on this draft, and progress made on revisions, feedback, and comments. Several presentations have been made in the room during IETF meetings, and followup and discussion taken to the BMWG list. This draft is particularly useful, given the popularity of VNF's within the industry.


Document Quality:
This document is ready for publication.

Personnel:

Who is the Document Shepherd? Who is the Responsible Area Director?
Sarah Banks is the Shepherd, Warren Kumari is the AD.


(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

I've reviewed this draft at WGLC. Nits check is clean, with 2 warnings that have no impact on publication.
All comments and feedback by the WG have been addressed by the Author.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

No specific reason to call out these reviews here, but the document itself was reviewed in ETSI NFV, and used as a reference by at least one of the TST WG and IFA WF specifications (2 references in total). The document has also been reviewed by several individuals from the OPNFV community, and NFVRG (IRTF) is aware of the draft.


(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

None.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

No IPR has been filed.


(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No IPR disclosed.

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

A WGLC was called on November 2, 2016, and closed on November 16, 2016. There was significant support for this work at IETF 95, and consensus was achieved amongst the WG. It's worth noting that the document is moving through the process slowly due to the document shepherd being late in writing the writeup. The document shepherd wishes to express her apologies.


(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

No nits, 2 errors.

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, media type, and URI type reviews.

NA

(13) Have all references within this document been identified as either normative or informative?

All references are either Normative or Informative, and marked as such.


(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

No

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

Yes, RFC 2679 (Obsoleted by RFC7679)

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 5226).

There are no IANA actions required.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

None.

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, etc.

NA.