Terminology/Methodology for Benchmarking SDN Controller Performance
Changes are expected over time. This version is dated 24 February 2012.
(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)? Why
is this the proper type of RFC? Is this type of RFC indicated in the
title page header?
Informational, all BMWG RFCs to date are Informational.
The status is correctly indicated on the title pages.
(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:
These two memos specify the terminology, benchmark definitions
and methods for characterizing key performance aspects of
Software Defined Network (SDN) Controllers. Considering the
number of benchmarking and performance comparison studies that
have been published prior to standards development, this is an
important area for new specifications and tools to implement them.
These memos focus on the ability of SDN controllers to learn
topology, communicate with switches, and instantiate paths using
both reactive and proactive techniques (including controller clusters).
Further, requirements for test traffic and network emulation
capabilities of the test devices are specified. The memos approach
this problem in a generic way (Openflow specific procedures are
included in an Appendix) for broader applicability and longevity.
Working Group Summary
Consensus for these drafts required several WGLC which prompted
careful review and further comments. However, the process to achieve
consensus was smooth, and at no time was there sustained controversy.
There are existing implementations of the methods described here,
both full and partial, and many are available as Open Source tools.
Tools are tied to specific versions of Northbound and Southbound
protocols, but the memo avoids this dependency (and redundancy with
other industry efforts) by defining benchmarks for generic SDN
controller functions (as suggested in https://mailarchive.ietf.org/arch/msg/bmwg/Glchqbvg6F7vOUc0ug3ACyztcNg ).
The document benefits from review in the NFVRG, and from interactions
with the Open Platform for Network Function Virtualization (OPNFV) "CPerf"
project team (which includes members from Open Daylight Integration
test team, ETSI NFV Test and Open Source WG, and other Linux Foundation
and independent Open Source projects related to SDN performance).
Al Morton is the Document Shepherd
Warren Kumari is the Responsible Area Director
(3) Briefly describe the review of this document that was performed by
the Document Shepherd. If this version of the document is not ready
for publication, please explain why the document is being forwarded to
The Doc Shepherd has reviewed many versions of these drafts, and finds
that the current versions are ready for publication (recognizing that
some editorial suggestions are a certainty of further review, and that
the editors are very willing to implement them).
1. The Shepherd provided editorial suggestions for the authors,
and they were addressed in 07 versions.
2. In the -term draft, it may be possible to avoid page breaks in the
Figures and the Table in Sec 3.
(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?
(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
No additional reviews appear to be needed (see below).
(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
No Specific issues.
(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.
All five authors listed on the drafts have confirmed that they are
unaware of any IPR related to these drafts.
(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
There are currently no IPR disclosures for either draft
officially submitted to the IETF.
(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it?
Over time, most of the working group has participated in discussion
and review of these drafts, so I think it is fair to say that the
majority of the WG understands and agrees with the content.
(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)
No appeals threatened.
(11) Identify any ID nits the Document Shepherd has found in this
document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
The nits check is free of warnings and errors.
(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.
(13) Have all references within this document been identified as
either normative or informative?
(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?
(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in the
Last Call procedure.
(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.
(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).
There are no requests of IANA, as indicated.
(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.
(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.