Internet-Draft Bhuvaneswaran Vengainathan
Network Working Group Anton Basil
Intended Status: Informational Veryx Technologies
Expires: September 2014 Vishwas Manral
Ionos Corp
March 24, 2014
Benchmarking Methodology for OpenFlow SDN Controller Performance
draft-bhuvan-bmwg-of-controller-benchmarking-00
Abstract
This document discusses and defines a number of tests that may be
used to measure the performance characteristics of OpenFlow (OF) SDN
controller. In addition to defining the tests, this document
also describes specific formats for reporting the results of the
tests.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in
progress.
This Internet-Draft will expire on September 20, 2014.
Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved.
Bhuvan Expires September 20, 2014 [Page 1]
Internet Draft OF Controller Benchmarking Methodology March 2014
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4. Test Setup . . . . . . . . . . . . . . . . . . . . . . . . . 4
4.1. Test Setup for Standalone Controller . . . . . . . . . . . 4
4.1.1. Proactive Mode . . . . . . . . . . . . . . . . . . . . . 4
4.1.2. Reactive Mode . . . . . . . . . . . . . . . . . . . . . 5
4.2. Test Setup for Controller Teaming . . . . . . . . . . . . 5
4.2.1. Proactive Mode . . . . . . . . . . . . . . . . . . . . . 5
4.2.2. Reactive Mode . . . . . . . . . . . . . . . . . . . . . . 6
5. Test Considerations . . . . . . . . . . . . . . . . . . . . . 6
5.1. Virtual Switches/Applications . . . . . . . . . . . . . . . 6
5.2. Test Traffic Requirements . . . . . . . . . . . . . . . . . 6
5.3. Southbound Connection Setup Requirements . . . . . . . . . 7
5.3.1. OpenFlow Channel Setup . . . . . . . . . . . . . . . . . 7
5.3.2. Test Recommendations . . . . . . . . . . . . . . . . . . 8
6. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . 8
6.1 Performance . . . . . . . . . . . . . . . . . . . . . . . 8
6.1.1 Flow setup rate . . . . . . . . . . . . . . . . . . . 8
6.1.1.1 Flow setup rate under constant network load . . . 8
6.1.1.2 Flow setup rate under incremental network load . . 10
6.1.2 Flow setup delay . . . . . . . . . . . . . . . . . . . 12
6.1.2.1 Flow setup delay under constant network load . . . 12
6.1.2.2 Flow setup delay under incremental network load . . 13
6.1.2.3 Flow setup delay under stress network load . . . . 14
6.1.3 End-End flow setup duration . . . . . . . . . . . . . 16
6.2 Scalability . . . . . . . . . . . . . . . . . . . . . . . 18
6.2.1 OpenFlow connections capacity . . . . . . . . . . . . 17
6.2.2 Switch scalable limit . . . . . . . . . . . . . . . . 19
6.2.3 Flow scalable limit . . . . . . . . . . . . . . . . . 20
6.3 Reliability . . . . . . . . . . . . . . . . . . . . . . . 21
6.3.1 Errored OpenFlow connections handling . . . . . . . . 21
6.3.2 Denial of service handling . . . . . . . . . . . . . . 23
6.3.3 Controller failover time . . . . . . . . . . . . . . . 24
6.3.4 Data path re-convergence time . . . . . . . . . . . . 24
Bhuvan Expires September 20, 2014 [Page 2]
Internet Draft OF Controller Benchmarking Methodology March 2014
7. References . . . . . . . . . . . . . . . . . . . . . . . . . 26
7.1. Normative References . . . . . . . . . . . . . . . . . . 26
7.2. Informative References . . . . . . . . . . . . . . . . 26
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 26
9. Security Considerations . . . . . . . . . . . . . . . . . . . 26
10. Appendix A - Test setup and test applicability . . . . . . . 26
11. Appendix B - OpenFlow protocol overview . . . . . . . . . . 27
11.1. Protocol Overview . . . . . . . . . . . . . . . . . . . 27
11.2. Messages Overview . . . . . . . . . . . . . . . . . . . 27
11.3. Connection Overview . . . . . . . . . . . . . . . . . . 28
12. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . 28
1. Introduction
This document provides methodologies for benchmarking OpenFlow SDN
controller performance. This includes benchmarking controller
for flow performance, scalability and reliability. In addition to
defining tests, this document also describes specific formats for
reporting test results. The results of these tests will provide
the vendor and user quantitative data with which to evaluate the
controller performance.
Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
this document are to be interpreted as described in RFC 2119.
2. Terminology
Reactive Flow Setup: Controller can add, update or delete flow
entries in flow tables of a switch in response to packets received
from the switch
Proactive Flow Setup: Controller can add, update or delete flow
entries in flow tables of a switch based on trigger from
controller's management plane.
REST: Representational state transfer
OFR: OpenFlow response messages
Bhuvan Expires September 20, 2014 [Page 3]
Internet Draft OF Controller Benchmarking Methodology March 2014
3. Scope
SDN defines logically centralized control plane to enable network
programmability and agility. The logically centralized control
plane can be a composition of single SDN controller (standalone) or
a cluster of SDN controllers (controller teaming) enabling network
programmability through one or more southbound interfaces. OpenFlow
is one among the protocol widely used for southbound communication.
This document defines methodologies for benchmarking OpenFlow based
SDN controller performance independent of composition.
4. Test Setup
Test configurations defined in this document will be confined to
standalone controller and cluster of controllers supporting
proactive and/or reactive mode of flow setup. Not all tests
described in this document are to be performed for all test setups.
These tests are performed by simulating real network scenarios in a
lab environment. Appendix A lists the test applicability for each
test setup. All of the tests that are relevant to an SDN controller
SHOULD be performed.
4.1. Test Setup for Standalone Controller
4.1.1. Proactive Mode
--------------------
| Orchestration |
--------------------
|
| (Northbound interface)
-----------------------
| SDN Controller |
| (DUT) |
-----------------------
| (Southbound interface)
|
---------------------------
| | |
---------- ---------- ----------
| SDN | | SDN |..| SDN |
| Switch 1 | | Switch 2 | | Switch n |
---------- ---------- ----------
Figure 1
Bhuvan Expires September 20, 2014 [Page 4]
Internet Draft OF Controller Benchmarking Methodology March 2014
4.1.2. Reactive Mode
-----------------------
| SDN Controller |
| (DUT) |
-----------------------
| (Southbound interface)
|
---------------------------
| | |
---------- ---------- ----------
| SDN | | SDN |..| SDN |
| Switch 1 | | Switch 2 | | Switch n |
---------- ---------- ----------
Figure 2
4.2. Test Setup for Controller Teaming
In controller clustering, the connectivity and the communication
between controllers are outside the scope of this document.
4.2.1. Proactive Mode
--------------------
| Orchestration |
--------------------
|
|
| (Northbound interface)
---------------------------------------------------------
| ------------------ ------------------ |
| | SDN Controller 1 | <--E/W--> | SDN Controller n | |
| ------------------ ------------------ |
---------------------------------------------------------
| (Southbound interface)
|
---------------------------
| | |
---------- ---------- ----------
| SDN | | SDN |..| SDN |
| Switch 1 | | Switch 2 | | Switch n |
---------- ---------- ----------
Figure 3
Bhuvan Expires September 20, 2014 [Page 5]
Internet Draft OF Controller Benchmarking Methodology March 2014
4.2.2. Reactive Mode
---------------------------------------------------------
| ------------------ ------------------ |
| | SDN Controller 1 | <--E/W--> | SDN Controller n | |
| ------------------ ------------------ |
---------------------------------------------------------
| (Southbound interface)
|
---------------------------
| | |
---------- ---------- ----------
| SDN | | SDN |..| SDN |
| Switch 1 | | Switch 2 | | Switch n |
---------- ---------- ----------
Figure 4
5. Test Considerations
5.1. Virtual Switches/Applications
SDN controller performance testing involves connections and flows
setup from multiple switches, hosts and/or applications. These
entities may not necessarily be a real physical hardware and
can be emulated in a single hardware platform. The test report
MUST indicate the number of switches and hosts used in the
test.
5.2. Test Traffic Requirements
While the key function of an SDN controller is to program the data
plane, the SDN controller may use Layer 2, Layer 3 or Layer 4
fields for programming the data plane. The author understands that
it will take a considerable period of time to perform tests for
various traffic types and packet sizes. We believe that the results
are worth the effort. For the purposes of benchmarking SDN
controller performance, the document references traffic type IP of
size 128 bytes
Bhuvan Expires September 20, 2014 [Page 6]
Internet Draft OF Controller Benchmarking Methodology March 2014
5.3. Southbound Connection Setup Requirements
The methodologies defined in this document are independent of
OpenFlow version and channel setup ways and connection type (e.g.,
TCP/TLS or SSL).However this document describes a number of channel
setup ways, and recommends some of the channel setup ways and
connection type over which the tests MUST be carried out and
measure the performance.
5.3.1. OpenFlow Channel Setup
This section describes three ways of OpenFlow channel setup.
5.3.1.1. Single Connection
By default, a single OpenFlow channel is established between each
OpenFlow switch and controller when a single controller
(standalone) manages a network.
5.3.1.2. Multiple Connections
Multiple OpenFlow channels are established from each OpenFlow
switch to multiple controllers when multiple controllers are
clustered and manage a network In such case, each OpenFlow switch
can employ any one of the below modes.
Active-Passive:
In this mode, one controller takes a MASTER role and other
controllers in the cluster take SLAVE role. The selection process
is out of this document scope. The OpenFlow channel between the
switch and master controller is in active state and others are in
standby state. Any asynchronous communications from the switch have
to be sent on the active channel.
Active-Active:
In this mode, all controllers take role as EQUAL. The OpenFlow
channel between the switch and all controllers are in active state.
Communication between OpenFlow switch and controller cluster can be
sent on any active channels.
5.3.1.3. Auxiliary Connection
Additional connections apart from the primary connection (as
described in section 5.3.1.1. and 5.3.1.2.) can be established
between a switch to a controller in order to improve the switch
processing performance.
Bhuvan Expires September 20, 2014 [Page 7]
Internet Draft OF Controller Benchmarking Methodology March 2014
5.3.2. Test Recommendations
For standalone controller, the performance tests MUST be carried on
'single connection' using 'TCP'. Tests over other connections like
'auxiliary connections' are optional, depending on a specific
support of the product.
For controller teaming, the performance tests MUST be carried on
'multiple connections - active-passive mode' using 'TCP'. Tests
over other connections like 'multiple connections - active-active',
'auxiliary connections' are optional, depending on a specific
support of the product.
6. Benchmarking Tests
6.1 Performance
6.1.1 Flow setup rate
Maximum number of flows setup by a controller, expressed in flows
per second.
This metric is measured in two modes, using constant network load
and incremental network load.
6.1.1.1 Flow setup rate under constant network load
6.1.1.1.1 Objective
To measure the total number of OpenFlow flow responses (Packet-
Out/Flow-Mod) received from the controller under constant network
load (fixed number of switches and flows)
6.1.1.1.2 Setup Parameters
The following parameters MUST be defined:
Network setup parameters:
Number of Switches - Defines the number of OpenFlow switch
connections established with controller.
Number of Flows - Defines unique number of flows setup on each
OpenFlow connections.
Bhuvan Expires September 20, 2014 [Page 8]
Internet Draft OF Controller Benchmarking Methodology March 2014
OpenFlow setup parameters:
OpenFlow Version - Defines OpenFlow protocol version used for
OpenFlow connection setup.
Channel Type - Defines OpenFlow connection setup type used for
OpenFlow connection setup, for example TCP or TLS.
Test setup parameters:
Test Iterations - Defines the number of times the test need to
be repeated.
Test Duration - Defines the duration of test iteration,
expressed in seconds.
6.1.1.1.3 Procedure
Reactive Mode Flow Setup:
Establish OpenFlow connection from all the specified number of
switches. Load the controller with Packet-In messages send from
all switches for a configured number of flows and for a specified
test duration. Single Packet-In message can have one or more
flow headers in its payload.
Proactive Mode Flow Setup:
Establish OpenFlow connection from all the specified number of
switches. Load the controller with configured number of flows
through the northbound interface of controller for a specified
test duration.
The test SHOULD be repeated a number of times for each of the above
procedure and record the results. It is recommended to give
sufficient time for the controller to flush all the pending
response between test iteration. On the completion of the test,
gracefully close all the OpenFlow sessions established with
controller.
Bhuvan Expires September 20, 2014 [Page 9]
Internet Draft OF Controller Benchmarking Methodology March 2014
6.1.1.1.4 Measurement
Total (OFRi)
Flow setup rate (FSRi) = ---------------
Test Duration
SUM (FSR1, FSR2,..FSRn)
Average flow setup rate = ------------------------
Test Iterations
Maximum flow setup rate = Max (FSR1,FSR2,..FSRn)
Minimum flow setup rate = Min (FSR1,FSR2,..FSRn)
Where,
FSR1 is the average OpenFlow responses received in 1st iteration
and FSRn is the average OpenFlow responses received in nth
iteration.
Measurement MUST take into account all the Packet-Outs embedded in
a single OpenFlow response message (Packet-Out/Flow-Mod).
6.1.1.1.5 Reporting Format
The results of the flow setup rate under constant network load test
for each of the test procedure SHOULD be plotted as a graph. If
this is done then the X axis MUST be the test iteration number The
Y axis MUST be the OpenFlow responses per second. The Y axis value
range should be between the minimum and maximum OpenFlow setup
rate. Plot the flow setup rate results obtained on each test
iteration in the graph. The maximum, average and minimum flow setup
rate should be reported at the bottom of the graph. The test report
MUST indicate whether learning was enabled or disabled.
Also report the total number of frames transmitted, number of valid
responses and error responses received for each iteration in a
table.
6.1.1.2 Flow setup rate under incremental network load
6.1.1.2.1 Objective
To measure the total OpenFlow flow responses (Packet-Out/Flow-Mod)
received from the controller for incremental network load
(increasing number of switches and flows).
6.1.1.2.2 Setup Parameters
Use the same setup parameters as defined in section 6.1.1.1.2
Bhuvan Expires September 20, 2014 [Page 10]
Internet Draft OF Controller Benchmarking Methodology March 2014
6.1.1.2.3 Procedure
Reactive Mode Flow Setup:
Establish OpenFlow connection from 10% of the specified number
of switches. Load controller with Packet-In messages send from
10% of switches for a configured number of flows and for a
specified sample interval. Single Packet-In message can have one
or more flow headers in its payload.
The test SHOULD be repeated with 20% of the specified number of
switches in the next iteration and continue the test until the
configured number of switches is included in the test. On the
completion of the test, gracefully close all the OpenFlow sessions
established with controller.
6.1.1.2.4 Measurement
Total (OFRi)
Flow setup rate (FSRi) = ---------------
Test Duration
Maximum flow setup rate = Max (FSR1,FSR2,..FSR10)
Minimum flow setup rate = Min (FSR1,FSR2,..FSR10)
Where,
FSR1 is the average OpenFlow responses received in 1st iteration
and FSR10 is the average OpenFlow responses received in 10th
iteration.
Measurement MUST take into account all the Packet-Outs embedded in
a single OpenFlow response message (Packet-Out/Flow-Mod).
6.1.1.2.5 Reporting Format
The results of the flow setup rate under incremental network load
test SHOULD be plotted as a graph. If this is done then the X axis
MUST be the number of switches per test iteration The Y axis MUST
be the OpenFlow responses per second. The Y axis value range should
be between the minimum and maximum OpenFlow setup rate. Plot the
flow setup rate results obtained on each test iteration in the
graph. The maximum and minimum flow setup rate should be reported
at the bottom of the graph. The test report MUST indicate whether
learning was enabled or disabled.
Bhuvan Expires September 20, 2014 [Page 11]
Internet Draft OF Controller Benchmarking Methodology March 2014
Also report the total number of frames transmitted, number of valid
responses and error responses received for each iteration in a
table.
6.1.2 Flow setup delay
Time taken by the controller to setup a flow, expressed in
milliseconds.
This metric is measured in three modes, using constant network
load, incremental network load and stress network load.
6.1.2.1 Flow setup delay under constant network load
6.1.2.1.1 Objective
To measure the time taken by controller to setup a flow under low
network load.
6.1.2.1.2 Setup Parameters
Use the same setup parameters as defined in section 6.1.1.1.2
6.1.2.1.3 Procedure
Reactive Mode Flow Setup:
Establish OpenFlow connection from all the specified number of
switches. Send Packet-In messages from each of the switches one
at a time and wait for the response before soliciting the next
flow request. Single Packet-In message MUST have one flow
header in its payload. This process is repeated as many times as
possible within the configured sample time interval.
The test SHOULD be repeated a number of times and record the
values. On the completion of the test, gracefully close all the
OpenFlow sessions established with controller.
6.1.2.1.4 Measurement
Test Duration
Flow setup delay (FSDi) = ---------------
Total (OFRi)
SUM (FSD1, FSD2,..FSDn)
Average flow setup delay = -------------------------
Test Iterations
Bhuvan Expires September 20, 2014 [Page 12]
Internet Draft OF Controller Benchmarking Methodology March 2014
Maximum flow setup delay = Max (FSD1,FSD2,..FSDn)
Minimum flow setup delay = Min (FSD1,FSD2,..FSDn)
Where,
FSD1 is the average time taken to setup a flow in first iteration
and
FSDn is the average time taken to setup a flow in nth iteration.
6.1.2.1.5 Reporting Format
The results of the flow setup delay under constant network load
test SHOULD be plotted as a graph. If this is done then the X axis
MUST be the test iteration number The Y axis MUST be the Flow setup
delay in milliseconds. The Y axis value range should be between the
minimum and maximum OpenFlow setup delay. Plot the flow setup delay
results obtained on each test iteration in the graph. The maximum,
average and minimum flow setup delay should also be reported at the
bottom of the graph. The test report MUST indicate whether learning
was enabled or disabled.
Also report the total number of frames transmitted, number of valid
responses and error responses received for each iteration in a
table.
6.1.2.2 Flow setup delay under incremental network load
6.1.2.2.1 Objective
To measure the amount of time taken by controller to setup a flow
under gradual increase of network load.
6.1.2.2.2 Setup Parameters
Use the same setup parameters as defined in section 6.1.1.1.2
6.1.2.2.3 Procedure
Reactive Mode Flow Setup:
Establish OpenFlow connection from 10% of the specified number
of switches. Send Packet-In messages from each of the switches
one at a time and wait for the response before soliciting the
next flow request. Single Packet-In message MUST have one flow
header in its payload. This process is repeated as many times as
possible within the configured sample time interval.
Bhuvan Expires September 20, 2014 [Page 13]
Internet Draft OF Controller Benchmarking Methodology March 2014
The test SHOULD be repeated with 20% of the specified number of
switches in the next iteration and continue the test until the
configured number of switches are included in the test. On the
completion of the test, gracefully close all the OpenFlow sessions
established with controller.
6.1.2.2.4 Measurement
Test Duration
Flow setup delay (FSDi) = ---------------
Total (OFRi)
Maximum flow setup delay = Max (FSD1,FSD2,..FSD10)
Minimum flow setup delay = Min (FSD1,FSD2,..FSD10)
Where,
FSD1 is the average time taken to setup a flow in first iteration
and
FSD10 is the average time taken to setup a flow in 10th iteration.
6.1.2.2.5 Reporting Format
The results of the flow setup delay under incremental network load
test SHOULD be plotted as a graph. If this is done then the X axis
MUST be the number of switches per test iteration. The Y axis MUST
be the Flow setup delay in milliseconds. The Y axis value range
should be between the minimum and maximum OpenFlow setup delay.
Plot the flow setup delay results obtained on each test iteration
in the graph. The maximum and minimum flow setup delay should also
be reported at the bottom of the graph. The test report MUST
indicate whether learning was enabled or disabled.
Also report the total number of frames transmitted, number of valid
responses and error responses received for each iteration in a
table.
6.1.2.3 Flow setup delay under stress network load
6.1.2.3.1 Objective
To measure the amount of time taken by controller to setup a flow
under high network load.
6.1.2.3.2 Setup Parameters
Use the same setup parameters as defined in section 6.1.1.1.2
Bhuvan Expires September 20, 2014 [Page 14]
Internet Draft OF Controller Benchmarking Methodology March 2014
6.1.2.3.3 Procedure
Reactive Mode Flow Setup:
Establish OpenFlow connection from all the specified number of
switches. Reserve one switch as a test switch and send Packet-In
messages from other switches for a configured number of flows
for a specified sample interval. The Packet-In message sent from
other switches can have one or more flow headers in its payload.
From the test switch, send Packet-In message one at a time and
wait for the response before soliciting the next flow request.
At regular frequency, insert time stamp to the Packet-In message
and measure the round trip time it takes for the flow to get
added in the switch. This process is repeated as many times as
possible within the configured sample time interval.
The test SHOULD be repeated a number of times and record the
values. On the completion of the test, gracefully close all the
OpenFlow sessions established with controller.
6.1.2.3.4 Measurement
Test Duration
Flow setup delay (FSDi) = -------------------------
Total (OFRi on test switch)
SUM (FSD1, FSD2,..FSDn)
Average flow setup delay = ------------------------
Test Iterations
Maximum flow setup delay = Max (FSD1,FSD2,..FSDn)
Minimum flow setup delay = Min (FSD1,FSD2,..FSDn)
Where,
FSD1 is the average time taken to setup a flow in first iteration
and
FSDn is the average time taken to setup a flow in nth iteration.
Bhuvan Expires September 20, 2014 [Page 15]
Internet Draft OF Controller Benchmarking Methodology March 2014
6.1.2.3.5 Reporting Format
The results of the flow setup delay under constant network load
test SHOULD be plotted as a graph. If this is done then the X axis
MUST be the test iteration number The Y axis MUST be the Flow setup
delay in milliseconds. The Y axis value range should be between the
minimum and maximum OpenFlow setup delay. Plot the flow setup delay
results obtained on each test iteration in the graph. The maximum,
average and minimum flow setup delay should also be reported at the
bottom of the graph. The test report MUST indicate whether learning
was enabled or disabled.
Also report the total number of frames transmitted, number of valid
responses and error responses received for each iteration in a
table.
6.1.3 End-End flow setup duration
6.1.3.1 Objective
To determine the time taken by the controller to setup a flow
between a source to a destination, expressed in milliseconds.
6.1.3.2 Setup Parameters
The following parameters MUST be defined:
Network setup parameters:
Number of switches - Total number of switches inter-connected in
a topology.
OpenFlow setup parameters:
OpenFlow Version - Defines OpenFlow protocol version used for
OpenFlow connection setup.
Channel Type - Defines OpenFlow connection setup type used for
OpenFlow connection setup, for example TCP or TLS.
Bhuvan Expires September 20, 2014 [Page 16]
Internet Draft OF Controller Benchmarking Methodology March 2014
6.1.3.4 Procedure
Inter connect the specified number of switches in linear or tree
fashion. Establish OpenFlow connection from all switches. To ensure
that the controller completes the network discovery process, send
traffic stream from a source connected on one end of the network to
the destination connected on the other end of network and verify
that the traffic reaches the destination properly. Prior sending
the traffic, send gratuitous ARP to the controller for the
destination.
On completion of above verification, send a traffic stream to a
different destination with the timestamp inserted on each frame.
On completion of the test, gracefully close all the OpenFlow
connection established with the controller.
The test SHOULD be repeated by varying the number of inter-
connected switches.
6.1.3.5 Measurement
Flow setup delay = Flow transmit time at the source - Flow
reception time at the destination.
6.1.3.6 Reporting Format
The test results MUST be reported in a table format with a row for
number of switches and a column for Flow setup delay.
6.2 Scalability
6.2.1 OpenFlow connections capacity
6.2.1.1 Objective
Maximum number of concurrent OpenFlow connections supported by a
controller.
6.2.1.2 Setup Parameters
The following parameters MUST be defined:
Network setup parameters:
Connection Setup Rate - Maximum number of TCP connection requests
accept per second.
Bhuvan Expires September 20, 2014 [Page 17]
Internet Draft OF Controller Benchmarking Methodology March 2014
OpenFlow setup parameters:
OpenFlow Version - Defines OpenFlow protocol version used for
OpenFlow connection setup.
Channel Type - Defines OpenFlow connection setup type used for
OpenFlow connection setup, for example TCP or TLS.
Test setup parameters:
Test Iterations - Defines the number of times the test need to be
repeated.
6.2.1.3 Procedure
Establish OpenFlow connection with the controller at the specified
connection setup rate until the controller drops the OpenFlow
connection request.
To validate each connection, send a Packet-In message after the
OpenFlow connection has been established. On the completion of the
test, gracefully close all the OpenFlow sessions established with
controller.
The test SHOULD be repeated a number of times. Each iteration,
record total number of responses received from the controller and
the test duration, in milliseconds.
6.2.1.4 Measurement
Maximum Concurrent OpenFlow Connections = Max (OFR1, OFR2, .. OFRn)
Minimum OpenFlow Connection Establishment Time, in milliseconds
Test duration of Min (OFR1, OFR2, .. OFRn)
= ------------------------------------------
Min (OFR1, OFR2, .. OFRn)
6.2.1.5 Reporting Format
The results of the OpenFlow connection capacity test SHOULD be
plotted as a graph. If this is done then the X axis MUST be the
test iteration number The Y axis MUST be the OpenFlow connection
capacity. Plot the OpenFlow responses obtained on each test
iteration in the graph. The Maximum Concurrent OpenFlow Connections
and Minimum OpenFlow Connection Establishment Time should be
reported at the bottom of the graph.
Bhuvan Expires September 20, 2014 [Page 18]
Internet Draft OF Controller Benchmarking Methodology March 2014
6.2.2 Switch scalable limit
6.2.2.1 Objective
To determine the number of switches a controller can optimally
manage.
6.2.2.2 Setup Parameters
The following parameters MUST be defined:
Network setup parameters:
Acceptable Flow Setup Delay - Acceptable flow response time,
expressed in milliseconds.
Connection Setup Rate - Maximum number of TCP connection requests
accept per second.
Test setup parameters:
Test Iterations - Defines the number of times the test need to be
repeated.
OpenFlow setup parameters:
OpenFlow Version - Defines OpenFlow protocol version used for
OpenFlow connection setup.
Channel Type - Defines OpenFlow connection setup type used for
OpenFlow connection setup, for example TCP or TLS.
6.2.2.3 Procedure
Establish OpenFlow connection with the controller at the specified
connection setup rate. Reserve one switch as a test switch and send
Packet-In messages from other switches at a constant load. The
Packet-In message sent from other switches can have one or more
flow headers in its payload.
From the test switch, send Packet-In message one at a time and wait
for the response before soliciting the next flow request. At
regular frequency, insert time stamp to the Packet-In message and
measure the round trip time it takes for the flow to get added in
the switch. Compare the measured flow round trip time with the
configured acceptable flow setup delay value at each step.
Bhuvan Expires September 20, 2014 [Page 19]
Internet Draft OF Controller Benchmarking Methodology March 2014
The above step should be continued until the measured response time
is lower or equal to the configured response time. On the
completion of the test, gracefully close all the OpenFlow sessions
established with controller.
The test SHOULD be repeated with varying number of network loads.
6.2.2.4 Measurement
Switch Scalable Limit = Number of switch connections active at the
step when the measured value starts exceeding the acceptable value
6.2.2.5 Reporting Format
The test report MUST note the switch scalable limit factor and the
offered network load in terms of flows for every iteration. The
results SHOULD be reported in the format of a table with a row for
offered network load and columns for number of switch connections
attempted and number of active switch connections (Switch Scalable
Limit).
6.2.3 Flow scalable limit
6.2.3.1 Objective
To determine the controller OpenFlow table capacity
6.2.3.2 Setup Parameters
The following parameters MUST be defined:
Network setup parameters:
Number of Flows - Defines unique number of flows send on the
established OpenFlow connection at the beginning of the test.
OpenFlow setup parameters:
OpenFlow Version - Defines OpenFlow protocol version used for
OpenFlow connection setup.
Channel Type - Defines OpenFlow connection setup type used for
OpenFlow connection setup, for example TCP or TLS.
Bhuvan Expires September 20, 2014 [Page 20]
Internet Draft OF Controller Benchmarking Methodology March 2014
6.2.3.3 Procedure
Establish OpenFlow connection from a switch with the controller.
Send Packet-In messages from the established connection for a
constant number of flows and measure the Flow-Mod messages received
from the controller. Make sure that the controller has learnt all
flows destination prior the test. One way to achieve this is by
sending Packet-In messages with gratuitous ARP frame for all flows.
The Packet-In message sent on each connection can have one or more
flow headers in its payload.
Repeat the iteration by varying the number of flows until the
maximum flow request, at which no flow response loss occurs, is
found. Since the controller may employ aging time mechanism for the
flow, the controller flow aging time SHOULD be set to a maximum
possible value
On the completion of the test, gracefully close the OpenFlow
session established with controller.
6.2.3.4 Measurement
Flow Scalable Limit:
Maximum flow request, expressed in packets per second, at which
no flow response loss is detected.
6.2.3.5 Reporting Format
The test report MUST note the offered load and the responses
received on each iteration. The results SHOULD be reported in the
format of a table with a row for the offered load and a column for
the number of responses received.
6.3 Reliability
6.3.1 Errored OpenFlow connections handling
6.3.1.1 Objective
To characterize the behavior of the controller when presented with
a combination of both legal and Illegal OpenFlow messages.
Bhuvan Expires September 20, 2014 [Page 21]
Internet Draft OF Controller Benchmarking Methodology March 2014
6.3.1.2 Setup Parameters
The following parameters MUST be defined:
Network setup parameters:
Number of Switches - Defines the number of OpenFlow switch
connections established with controller.
Number of Flows - Defines unique number of flows setup on each
OpenFlow connections.
OpenFlow setup parameters:
OpenFlow Version - Defines OpenFlow protocol version used for
OpenFlow connection setup.
Channel Type - Defines OpenFlow connection setup type used for
OpenFlow connection setup, for example TCP or TLS.
Test setup parameters:
Test Iterations - Defines the number of times the test need to be
repeated.
Test Duration - Defines the duration of test iteration, expressed
in seconds.
6.3.1.3 Procedure
Establish OpenFlow connection from all the specified number of
switches. Load controller with Packet-In messages send from all
switches for a configured number of flows and for a specified test
duration. On each connection, after every 5% of flows transmission,
send an invalid Packet-In message (message with wrong header field
value). Single Packet-In message can have one or more flow headers
in its payload.
The test SHOULD be repeated a number of times by varying the
offered network load (number of switches) and record the results.
On the completion of the test, gracefully close all the OpenFlow
sessions established with controller.
6.3.1.4 Measurement
Average response rate:
Total number of OpenFlow responses received divided by the test
duration, expressed in responses per second.
Bhuvan Expires September 20, 2014 [Page 22]
Internet Draft OF Controller Benchmarking Methodology March 2014
6.3.1.5 Reporting Format
The test report MUST note the number of switches, average offered
load (correct and incorrect messages) and the average responses
received. The results SHOULD be reported in the format of a table
with a row for the number of switches and columns for the average
offered load and average responses received.
6.3.2 Denial of service handling
6.3.2.1 Objective
To determine the effect of a denial of service attack on a
controller OpenFlow connection establishment rates. The denial of
service handling test MUST be run after obtaining baseline
measurements from section 6.3.
6.3.2.2 Setup Parameters
Use the same setup parameters defined in section 6.2.1.2
In addition, the following setup parameters MUST be defined:
OpenFlow connection attack rate - Rate at which the TCP connections
are established without sending any OpenFlow messages.
6.3.2.3 Procedure
Use the same procedure as defined in section 6.2.1.3. In addition,
establish TCP connections at the specified OpenFlow attack rate
without performing any Hello exchanges.
6.3.2.4 Measurement
Perform the same measurements as defined in section 6.2.1.4. In
addition, track the number of TCP connections established without
performing any Hello exchanges.
6.3.2.5 Reporting Format
The test SHOULD use the same format as defined in section 6.2.1.5. In
addition, the test SHOULD report the number of OpenFlow connections
attempted without performing any Hello exchanges at the bottom of
the report.
Bhuvan Expires September 20, 2014 [Page 23]
Internet Draft OF Controller Benchmarking Methodology March 2014
6.3.3 Controller failover time
6.3.3.1 Objective
To determine the time taken to switch from one controller to
another when the controllers are teamed and the master controller
fails.
6.3.3.2 Setup Parameters
Use the same setup parameters defined in section 6.1.3.2.
6.3.3.3 Procedure
Establish OpenFlow connection with MASTER and SLAVE controllers.
Duplicate Packet-In message and send on both connections one at a
time. Insert timestamp to each Packet-In message sent on both
connections. During the test, bring down the master controller.
The test SHOULD be continued until first flow response is received
from the new master. The test SHOULD be repeated a number of times
by varying the number of switches.
6.3.3.4 Measurement
Failover Time = (Time at which last response received from the old
master) - (Time at which the first response received from the new
master)
Packet Losses = Total number of unique requests sent - Total number
of unique responses received.
6.3.3.5 Reporting Format
The test MUST note the number of flow requests sent on each
connection and the number of responses received. The results SHOULD
be reported in the format of a table with a row for the number of
switches and columns for number of requests sent, responses
received, packet losses and failover time.
6.3.4 Data path re-convergence time
6.3.4.1 Objective
To determine the time taken to re-route a flow by the controller
when there is a failure in the existing flow path.
Bhuvan Expires September 20, 2014 [Page 24]
Internet Draft OF Controller Benchmarking Methodology March 2014
6.3.4.2 Setup Parameters
Use the same setup parameters defined in section 6.1.3.2
6.3.4.3 Procedure
Inter connect the specified number of switches in tree fashion.
Establish OpenFlow connection from all switches. To ensure that the
controller completes the network discovery process, send traffic
stream from a source connected on one end of the network to the
destination connected on the other end of network and verify that
the traffic reaches the destination properly. Prior sending the
traffic, send gratuitous ARP to the controller for learning the
destination.
In addition, ensure that the topology has a redundant path from a
source to destination. Send a traffic stream for different
destination with sequence number inserted in each of the frame.
During the course of the test, bring down the link of the traffic
path.
On completion of the test, gracefully close all the OpenFlow
connection established with the controller. Repeat the test with
varying number of inter-connected switches.
6.3.4.4 Measurement
Failover time:
Delta between the receive timestamp of the errored sequence
packet (first packet received after re-convergence) and the last
valid packet, expressed in milliseconds.
Packet Loss:
Difference between the sequence number of errored sequence packet
and the last valid packet.
6.3.4.5 Reporting Format
The test results MUST be reported in a table format with a row for
number of switches and a column for failover time and packet
losses.
Bhuvan Expires September 20, 2014 [Page 25]
Internet Draft OF Controller Benchmarking Methodology March 2014
7. References
7.1. Normative References
[1] OpenFlow Switch Specification, Version 1.4.0 (Wire Protocol
0x05), October 14, 2013
7.2. Informative References
8. IANA Considerations
This document does not have any IANA requests.
9. Security Considerations
The primary goal of this document is to provide methodologies in
benchmarking OpenFlow controller's performance with respect to
southbound communication. There may arise some performance and
security issues while interacting through northbound interfaces,
assessment of such issues are outside the scope of this document.
10. Appendix A - Test setup and test applicability
+-------------------------------------------------------------------+
|Benchmarking Tests |Standalone Controller|Controller Teaming |
| |---------------------|---------------------|
| |Proactive |Reactive |Proactive |Reactive |
+-------------------------------------------------------------------+
|1. Flow Setup Rate | | | | |
| - Constant Loadi |Applicable|Applicable|Applicable|Applicable|
| - Incremental Load | NA |Applicable| NA |Applicable|
| | | | | |
|2. Flow Setup Delay | | | | |
| - Constant Load | NA |Applicable| NA |Applicable|
| - Incremental Load | NA |Applicable| NA |Applicable|
| - Stress Load | NA |Applicable| NA |Applicable|
| | | | |
|3. OpenFlow Connections| | | | |
| Capacity | NA |Applicable| NA |Applicable|
| | | | | |
|4. Switch Scalable | | | | |
| Limit | NA |Applicable| NA |Applicable|
| | | | | |
|5. Flow Scalable Limit | NA |Applicable| NA |Applicable|
| | | | | |
|6. Errored OpenFlow | | | | |
| Connections Handling| NA |Applicable| NA |Applicable|
| | | | | |
Bhuvan Expires September 20, 2014 [Page 26]
Internet Draft OF Controller Benchmarking Methodology March 2014
|7. Denial of Service | | | | |
| Handling | NA |Applicable| NA |Applicable|
| | | | | |
|8. End-End Flow Setup | | | | |
| Duration | NA |Applicable| NA |Applicable|
| | | | | |
|9. Controller Failover | | | | |
| Time | NA |Applicable| NA |Applicable|
| | | | | |
|10. Datapath | | | | |
| Re-convergence Time| NA |Applicable| NA |Applicable|
+-------------------------------------------------------------------+
11. Appendix B - OpenFlow protocol overview
11.1. Protocol Overview
OpenFlow is an open standard protocol defined by Open Networking
Foundation (ONF), used for programming the forwarding plane of
network switches or routers via a centralized controller.
11.2. Messages Overview
OpenFlow protocol supports three messages types namely controller-
to-switch, asynchronous and symmetric.
Controller-to-switch messages are initiated by the controller and
used to directly manage or inspect the state of the switch. These
messages allow controllers to query/configure the switch (Features,
Configuration messages), collect information from switch (Read-
State message), send packets on specified port of switch (Packet-
out message), and modify switch forwarding plane and state (Modify-
State, Role-Request messages etc.).
Asynchronous messages are generated by the switch without a
controller soliciting them. These messages allow switches to update
controllers to denote an arrival of new flow (Packet-in), switch
state change (Flow-Removed, Port-status) and error (Error).
Symmetric messages are generated in either direction without
solicitation. These messages allow switches and controllers to set
up connection (Hello), verify for liveness (Echo) and offer
additional functionalities (Experimenter).
Bhuvan Expires September 20, 2014 [Page 27]
Internet Draft OF Controller Benchmarking Methodology March 2014
11.3. Connection Overview
OpenFlow channel is used to exchange OpenFlow message between an
OpenFlow switch and an OpenFlow controller. The OpenFlow channel
connection can be setup using plain TCP or TLS. By default, a
switch establishes single connection with SDN controller. A switch
may establish multiple parallel connections to single controller
(auxiliary connection) or multiple controllers to handle controller
failures and load balancing.
12. Authors' Addresses
Bhuvaneswaran Vengainathan
Veryx Technologies Inc.
1 International Plaza, Suite 550
Philadelphia
PA 19113
Email: bhuvaneswaran.vengainathan@veryxtech.com
Anton Basil
Veryx Technologies Inc.
1 International Plaza, Suite 550
Philadelphia
PA 19113
Email: anton.basil@veryxtech.com
Vishwas Manral
Ionos Corp,
4100 Moorpark Ave,
San Jose, CA
Email: vishwas@ionosnetworks.com
Bhuvan Expires September 20, 2014 [Page 28]