Network Working Group                                          A. Morton
Internet-Draft                                                 AT&T Labs
Intended status: Informational                          October 26, 2014
Expires: April 29, 2015

  Considerations for Benchmarking Virtual Network Functions and Their


   Benchmarking Methodology Working Group has traditionally conducted
   laboratory characterization of dedicated physical implementations of
   internetworking functions.  This memo investigates additional
   considerations when network functions are virtualized and performed
   in commodity off-the-shelf hardware.

Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   document are to be interpreted as described in RFC 2119 [RFC2119].

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on April 29, 2015.

Copyright Notice

   Copyright (c) 2014 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents

Morton                   Expires April 29, 2015                 [Page 1]

Internet-Draft     Benchmarking VNFs and Related Inf.       October 2014

   ( in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Considerations for Hardware and Testing . . . . . . . . . . .   3
     3.1.  Hardware Components . . . . . . . . . . . . . . . . . . .   3
     3.2.  Configuration Parameters  . . . . . . . . . . . . . . . .   4
     3.3.  Testing Strategies  . . . . . . . . . . . . . . . . . . .   4
   4.  Benchmarking Considerations . . . . . . . . . . . . . . . . .   5
     4.1.  Comparison with Physical Network Functions  . . . . . . .   5
     4.2.  Continued Emphasis on Black-Box Benchmarks  . . . . . . .   5
     4.3.  New Benchmarks  . . . . . . . . . . . . . . . . . . . . .   6
     4.4.  Assessment of Benchmark Coverage  . . . . . . . . . . . .   7
   5.  Security Considerations . . . . . . . . . . . . . . . . . . .   8
   6.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   8
   7.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   8
   8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   9
     8.1.  Normative References  . . . . . . . . . . . . . . . . . .   9
     8.2.  Informative References  . . . . . . . . . . . . . . . . .  10
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  10

1.  Introduction

   Benchmarking Methodology Working Group (BMWG) has traditionally
   conducted laboratory characterization of dedicated physical
   implementations of internetworking functions.  The Black-box
   Benchmarks of Throughput, Latency, Forwarding Rates and others have
   served our industry for many years.  [RFC1242] and [RFC2544] are the
   cornerstones of the work.

   An emerging set of service provider and vendor development goals is
   to reduce costs while increasing flexibility of network devices, and
   drastically accelerate their deployment.  Network Function
   Virtualization (NFV) has the promise to achieve these goals, and
   therefore has garnered much attention.  It now seems certain that
   some network functions will be virtualized following the success of
   cloud computing and virtual desktops supported by sufficient network
   path capacity, performance, and widespread deployment; many of the
   same techniques will help achieve NFV.

Morton                   Expires April 29, 2015                 [Page 2]

Internet-Draft     Benchmarking VNFs and Related Inf.       October 2014

   See for
   more background, for example, the white papers there may be a useful
   starting place.  The Performance and Portability Best Practices
   [NFV.PER001] are particularly relevant to BMWG.  There are currently
   work-in-progress documents available in the Open Area including drafts
   describing Infrastructure aspects and service quality.

2.  Scope

   BMWG will consider the new topic of Virtual Network Functions and
   related Infrastructure to ensure that common issues are recognized
   from the start, using background materials from industry and SDOs
   (e.g., IETF, ETSI NFV).

   This memo investigates additional methodological considerations
   necessary when benchmarking VNF instantiated and hosted in commodity
   off-the-shelf (COTS) hardware.  An essential consideration is
   benchmarking both physical and virtual network functions, thereby
   allowing direct comparison.

   A clearly related goal: the benchmarks for the capacity of COTS to
   host a plurality of VNF instances should be investigated.  Existing
   networking technology benchmarks will also be considered for
   adaptation to NFV and closely associated technologies.

   A non-goal is any overlap with traditional computer benchmark
   development and their specific metrics (SPECmark suites such as

   A colossal non-goal is any form of architecture development related
   to NFV and associated technologies in BMWG, as has been the case
   since BMWG began work in 1989.

3.  Considerations for Hardware and Testing

   This section lists the new considerations which must be addressed to
   benchmark VNF(s) and their supporting infrastructure.

3.1.  Hardware Components

   New Hardware devices will become part of the test set-up.

   1.  High volume server platforms (COTS, possibly with virtual
       technology enhancements).

   2.  Storage systems with large capacity, high speed, and high

Morton                   Expires April 29, 2015                 [Page 3]

Internet-Draft     Benchmarking VNFs and Related Inf.       October 2014

   3.  Network Interface ports specially designed for efficient service
       of many virtual NICs.

   4.  High capacity Ethernet Switches.

   Labs conducting comparisons of different VNFs may be able to use the
   same hardware platform over many studies, until the steady march of
   innovations overtakes their capabilities (as happens with the lab's
   traffic generation and testing devices today).

3.2.  Configuration Parameters

   It will be necessary to configure and document the settings for the
   entire COTS platform, including:

   o  number of server blades (shelf occupation)

   o  CPUs

   o  caches

   o  storage system

   o  I/O

   as well as configurations that support the devices which host the VNF

   o  Hypervisor

   o  Virtual Machine

   o  Infrastructure Virtual Network

   and finally, the VNF itself, with items such as:

   o  specific function being implemented in VNF

   o  number of VNF components in the service function chain

   o  number of physical interfaces and links transited in the service
      function chain

3.3.  Testing Strategies

   The concept of characterizing performance at capacity limits may
   change.  For example:

Morton                   Expires April 29, 2015                 [Page 4]

Internet-Draft     Benchmarking VNFs and Related Inf.       October 2014

   1.  It may be more representative of system capacity to characterize
       the case where Virtual Machines (VM, hosting the VNF) are
       operating at 50% Utilization, and therefore sharing the "real"
       processing power across many VMs.

   2.  Another important case stems from the need for partitioning
       functions.  A noisy neighbor (VM hosting a VNF in an infinite
       loop) would ideally be isolated and the performance of other VMs
       would continue according to their specifications.

   3.  System errors will likely occur as transients, implying a
       distribution of performance characteristics with a long tail
       (like latency), leading to the need for longer-term tests of each
       set of configuration and test parameters.

   4.  The desire for Elasticity and flexibility among network functions
       will include tests where there is constant flux in the VM
       instances.  Requests for new VMs and Releases for VMs hosting
       VNFs no longer needed would be an normal operational condition.

   5.  All physical things can fail, and benchmarking efforts can also
       examine recovery aided by the virtual architecture with different
       approaches to resiliency.

4.  Benchmarking Considerations

   This section discusses considerations related to Benchmarks
   applicable to VNFs and their associated technologies.

4.1.  Comparison with Physical Network Functions

   In order to compare the performance of virtual designs and
   implementations with their physical counterparts, identical
   benchmarks must be used.  Since BMWG has developed specifications for
   many network functions already, there will be re-use of existing
   benchmarks through references, while allowing for the possibility of
   benchmark curation during development of new methodologies.
   Consideration should be given to quantifying the number of parallel
   VNFs required to achieve comparable performance with a given physical
   device, or whether some limit of scale was reached before the VNFs
   could achieve the comparable level.

4.2.  Continued Emphasis on Black-Box Benchmarks

   When the network functions under test are based on Open Source code,
   there may be a tendency to rely on internal measurements to some
   extent, especially when the externally-observable phenomena only
   support an inference of internal events (such as routing protocol

Morton                   Expires April 29, 2015                 [Page 5]

Internet-Draft     Benchmarking VNFs and Related Inf.       October 2014

   convergence).  However, external observations remain essential as the
   basis for Benchmarks.  Internal observations with fixed specification
   and interpretation may be provided in parallel, to assist the
   development of operations procedures when the technology is deployed,
   for example.  Internal metrics and measurements from Open Source
   implementations may be the only direct source of performance results
   in a desired dimension, but corroborating external observations are
   still required to assure the integrity of measurement discipline was
   maintained for all reported results.

   A related aspect of benchmark development is where the scope includes
   multiple approaches to a common function under the same benchmark.
   For example, there are many ways to arrange for activation of a
   network path between interface points and the activation times can be
   compared if the start-to-stop activation interval has a generic and
   unambiguous definition.  Thus, generic benchmark definitions are
   preferred over technology/protocol specific definitions where

4.3.  New Benchmarks

   There will be new classes of benchmarks needed for network design and
   assistance when developing operational practices (possibly automated
   management and orchestration of deployment scale).  Examples follow
   in the paragraphs below, many of which are prompted by the goals of
   increased elasticity and flexibility of the network functions, along
   with accelerated deployment times.

   Time to deploy VNFs: In cases where the COTS hardware is already
   deployed and ready for service, it is valuable to know the response
   time when a management system is tasked with "standing-up" 100's of
   virtual machines and the VNFs they will host.

   Time to migrate VNFs: In cases where a rack or shelf of hardware must
   be removed from active service, it is valuable to know the response
   time when a management system is tasked with "migrating" some number
   of virtual machines and the VNFs they currently host to alternate
   hardware that will remain in-service.

   Time to create a virtual network in the COTS infrastructure: This is
   a somewhat simplified version of existing benchmarks for convergence
   time, in that the process is initiated by a request from (centralized
   or distributed) control, rather than inferred from network events
   (link failure).  The successful response time would remain dependent
   on dataplane observations to confirm that the network is ready to

Morton                   Expires April 29, 2015                 [Page 6]

Internet-Draft     Benchmarking VNFs and Related Inf.       October 2014

4.4.  Assessment of Benchmark Coverage

   It can be useful to organize benchmarks according to their applicable
   lifecycle stage and the performance criteria they intend to assess.
   The table below provides a way to organize benchmarks such that there
   is a clear indication of coverage for the intersection of lifecycle
   stages and performance criteria.

   |               |             |            |               |
   |               |   SPEED     |  ACCURACY  |  RELIABILITY  |
   |               |             |            |               |
   |               |             |            |               |
   |  Activation   |             |            |               |
   |               |             |            |               |
   |               |             |            |               |
   |  Operation    |             |            |               |
   |               |             |            |               |
   |               |             |            |               |
   | De-activation |             |            |               |
   |               |             |            |               |

   For example, the "Time to deploy VNFs" benchmark described above
   would be placed in the intersection of Activation and Speed, making
   it clear that there are other potential performance criteria to
   benchmark, such as the "percentage of unsuccessful VM/VNF stand-ups"
   in a set of 100 attempts.  This example emphasizes that the
   Activation and De-activation lifecycle stages are key areas for NFV
   and related infrastructure, and encourage expansion beyond
   traditional benchmarks for normal operation.  Thus, reviewing the
   benchmark coverage using this table (sometimes called the 3x3 matrix)
   can be a worthwhile exercise in BMWG.


   In one of the first applications of the 3x3 matrix on BMWG, we
   discovered that metrics on measured size, capacity, or scale do not
   easily match one of the three columns above.  There are three
   alternatives to resolve this:

   1.  Add a column, Scaleability, but then it would be expected to have
       metrics in most of the Activation, Operation, and De-activation
       functions (which may not be the case).

Morton                   Expires April 29, 2015                 [Page 7]

Internet-Draft     Benchmarking VNFs and Related Inf.       October 2014

   2.  Include Scalability under Reliability: This fits the user
       perspective of the 3x3 matrix because the size or capacity of a
       device contributes to the likelihood that a request will be
       blocked, or that operation will be un-reliable when operating in
       an overload state.

   3.  Keep size, capacity, and scale metrics separate from the 3x3
       matrix, and present the results for key benchmarks in different
       versions of the matrix, and the titles of each matrix provide the
       details of configuration and scale.

   Alternative 3 would address a discussion comment from IETF-90, so it
   seems to cover a range of wanted features.

5.  Security Considerations

   Benchmarking activities as described in this memo are limited to
   technology characterization of a Device Under Test/System Under Test
   (DUT/SUT) using controlled stimuli in a laboratory environment, with
   dedicated address space and the constraints specified in the sections

   The benchmarking network topology will be an independent test setup
   and MUST NOT be connected to devices that may forward the test
   traffic into a production network, or misroute traffic to the test
   management network.

   Further, benchmarking is performed on a "black-box" basis, relying
   solely on measurements observable external to the DUT/SUT.

   Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
   benchmarking purposes.  Any implications for network security arising
   from the DUT/SUT SHOULD be identical in the lab and in production

6.  IANA Considerations

   No IANA Action is requested at this time.

7.  Acknowledgements

   The author acknowledges an encouraging conversation on this topic
   with Mukhtiar Shaikh and Ramki Krishnan in November 2013.
   Bhuvaneswaran Vengainathan, Bhavani Parise, and Ilya Varlashkin have
   provided useful suggestions to expand these considerations.

Morton                   Expires April 29, 2015                 [Page 8]

Internet-Draft     Benchmarking VNFs and Related Inf.       October 2014

8.  References

8.1.  Normative References

              "Network Function Virtualization: Performance and
              Portability Best Practices", Group Specification ETSI GS
              NFV-PER 001 V1.1.1 (2014-06), June 2014.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC2330]  Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
              "Framework for IP Performance Metrics", RFC 2330, May

   [RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
              Network Interconnect Devices", RFC 2544, March 1999.

   [RFC2679]  Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
              Delay Metric for IPPM", RFC 2679, September 1999.

   [RFC2680]  Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
              Packet Loss Metric for IPPM", RFC 2680, September 1999.

   [RFC2681]  Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip
              Delay Metric for IPPM", RFC 2681, September 1999.

   [RFC3393]  Demichelis, C. and P. Chimento, "IP Packet Delay Variation
              Metric for IP Performance Metrics (IPPM)", RFC 3393,
              November 2002.

   [RFC3432]  Raisanen, V., Grotefeld, G., and A. Morton, "Network
              performance measurement with periodic streams", RFC 3432,
              November 2002.

   [RFC4737]  Morton, A., Ciavattone, L., Ramachandran, G., Shalunov,
              S., and J. Perser, "Packet Reordering Metrics", RFC 4737,
              November 2006.

   [RFC5357]  Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
              Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
              RFC 5357, October 2008.

   [RFC5905]  Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network
              Time Protocol Version 4: Protocol and Algorithms
              Specification", RFC 5905, June 2010.

Morton                   Expires April 29, 2015                 [Page 9]

Internet-Draft     Benchmarking VNFs and Related Inf.       October 2014

8.2.  Informative References

   [RFC1242]  Bradner, S., "Benchmarking terminology for network
              interconnection devices", RFC 1242, July 1991.

   [RFC5481]  Morton, A. and B. Claise, "Packet Delay Variation
              Applicability Statement", RFC 5481, March 2009.

   [RFC6248]  Morton, A., "RFC 4148 and the IP Performance Metrics
              (IPPM) Registry of Metrics Are Obsolete", RFC 6248, April

   [RFC6390]  Clark, A. and B. Claise, "Guidelines for Considering New
              Performance Metric Development", BCP 170, RFC 6390,
              October 2011.

Author's Address

   Al Morton
   AT&T Labs
   200 Laurel Avenue South
   Middletown,, NJ  07748

   Phone: +1 732 420 1571
   Fax:   +1 732 368 1192

Morton                   Expires April 29, 2015                [Page 10]