Network Working Group                                          A. Morton
Internet-Draft                                                 AT&T Labs
Intended status: Informational                         February 14, 2014
Expires: August 18, 2014


  Considerations for Benchmarking Virtual Network Functions and Their
                             Infrastructure
                    draft-morton-bmwg-virtual-net-00

Abstract

   BMWG has traditionally conducted laboratory characterization of
   dedicated physical implementations of internetworking functions.
   This memo investigates additional considerations when network
   functions are virtualized and performed in commodity off-the-shelf
   hardware.

Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on August 18, 2014.

Copyright Notice

   Copyright (c) 2014 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents



Morton                   Expires August 18, 2014                [Page 1]


Internet-Draft               Round-trip Loss               February 2014


   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  New Considerations  . . . . . . . . . . . . . . . . . . . . .   3
     3.1.  Hardware Components . . . . . . . . . . . . . . . . . . .   3
     3.2.  Configuration Parameters  . . . . . . . . . . . . . . . .   3
     3.3.  Testing Strategies  . . . . . . . . . . . . . . . . . . .   4
   4.  Security Considerations . . . . . . . . . . . . . . . . . . .   5
   5.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   5
   6.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   5
   7.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   5
     7.1.  Normative References  . . . . . . . . . . . . . . . . . .   5
     7.2.  Informative References  . . . . . . . . . . . . . . . . .   6
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .   7

1.  Introduction

   BMWG has traditionally conducted laboratory characterization of
   dedicated physical implementations of internetworking functions.  The
   Black-box Benchmarks of Throughput, Latency, Forwarding Rates and
   others have served our industry for many years.  [RFC1242] and
   [RFC2544] are the cornerstones of the work.

   A set of development goals is to reduce costs while increasing
   flexibility of network devices, and drastically accelerate their
   deployment.  Network Function Virtualization has the promise to
   achieve these goals, and therefore has garnered much attention.  It
   now seems certain that some network functions will be virtualized
   following the success of cloud computing and virtual desktops
   supported by sufficient network path capacity, performance,and
   widespread deployment; many of the same techniques will be brought to
   bear.

   See http://www.etsi.org/technologies-clusters/technologies/nfv for
   more background, for example, the white papers there may be a useful
   starting place.






Morton                   Expires August 18, 2014                [Page 2]


Internet-Draft               Round-trip Loss               February 2014


2.  Scope

   This memo investigates additional methodological considerations
   necessary when benchmarking Virtual Network Functions (VNF)
   instantiated and hosted in commodity off-the-shelf hardware (COTS).

   A clearly related goal: the benchmarks for the capacity of COTS to
   host a plurality of VNF instances should be investigated.

   A non-goal is any overlap with traditional computer benchmark
   development and their specific metrics (SPECmark suites such as
   SPECCPU).

3.  New Considerations

   This section lists the new considerations which must be addressed to
   benchmark VNF(s) and their supporting infrastructure.

3.1.  Hardware Components

   New Hardware devices will become part of the test set-up.

   1.  High volume server platforms (COTS, possibly with virtual
       technology enhancements).

   2.  Large capacity, and high speed, high reliability storage systems.

   3.  Network Interface ports specially designed for efficient service
       of many virtual NICs.

   4.  High capacity Ethernet Switches.

   Labs conducting comparisons of different VNFs may be able to use the
   same hardware platform over many studies, until the steady march of
   innovations overtakes their capabilities (as happens with the lab's
   traffic generation and testing devices today).

3.2.  Configuration Parameters

   It will be necessary to configure and document the settings for the
   entire COTS platform, including:

   o  number of server blades (shelf occupation)

   o  CPUs

   o  caches




Morton                   Expires August 18, 2014                [Page 3]


Internet-Draft               Round-trip Loss               February 2014


   o  storage system

   o  I/O

   as well as configurations that support the devices which host the VNF
   itself:

   o  Hypervisor

   o  Virtual Machine

   o  Infrastructure Virtual Network

   and finally, the VNF itself, with items such as:

   o  specific function being implemented in VNF

   o  number of VNF components in the service function chain

   o  number of physical interfaces and links transited in the service
      function chain

3.3.  Testing Strategies

   The concept of characterizing performance at capacity limits may
   change.  For example:

   1.  It may be more representative of system capacity to characterize
       the case where Virtual Machines (VM, hosting the VNF) are
       operating at 50% Utilization, and therefore sharing the "real"
       processing power across many VMs.

   2.  Another important case stems from the need for partitioning
       functions.  A noisy neighbor (VM hosting a VNF in an infinite
       loop) would ideally be isolated and the performance of other VMs
       would continue according to their specifications.

   3.  System errors will likely occur as transients, implying a
       distribution of performance characteristics with a long tail
       (like latency), leading to the need for longer-term tests of each
       set of configuration and test parameters.

   4.  The desire for Elasticity and flexibility among network functions
       will include tests where there is constant flux in the VM
       instances.  Requests for new VMs and Releases for VMs hosting
       VNFs no longer needed would be an normal operational condition.





Morton                   Expires August 18, 2014                [Page 4]


Internet-Draft               Round-trip Loss               February 2014


   5.  All physical things can fail, and benchmarking efforts can also
       examine recovery aided by the virtual architecture with different
       approaches to resiliency.

4.  Security Considerations

   Benchmarking activities as described in this memo are limited to
   technology characterization using controlled stimuli in a laboratory
   environment, with dedicated address space and the constraints
   specified in the sections above.

   The benchmarking network topology will be an independent test setup
   and MUST NOT be connected to devices that may forward the test
   traffic into a production network, or misroute traffic to the test
   management network.

   Further, benchmarking is performed on a "black-box" basis, relying
   solely on measurements observable external to the DUT/SUT.

   Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
   benchmarking purposes.  Any implications for network security arising
   from the DUT/SUT SHOULD be identical in the lab and in production
   networks.

5.  IANA Considerations

   No IANA Action is requested at this time.

6.  Acknowledgements

   The author acknowledges an encouraging conversation on this topic
   with Mukhtiar Shaikh and Ramki Krishnan in November 2013.

7.  References

7.1.  Normative References

   [RFC1242]  Bradner, S., "Benchmarking terminology for network
              interconnection devices", RFC 1242, July 1991.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC2330]  Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
              "Framework for IP Performance Metrics", RFC 2330, May
              1998.





Morton                   Expires August 18, 2014                [Page 5]


Internet-Draft               Round-trip Loss               February 2014


   [RFC2544]  Bradner, S. and J. McQuaid, "Benchmarking Methodology for
              Network Interconnect Devices", RFC 2544, March 1999.

   [RFC2679]  Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
              Delay Metric for IPPM", RFC 2679, September 1999.

   [RFC2680]  Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
              Packet Loss Metric for IPPM", RFC 2680, September 1999.

   [RFC2681]  Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip
              Delay Metric for IPPM", RFC 2681, September 1999.

   [RFC3393]  Demichelis, C. and P. Chimento, "IP Packet Delay Variation
              Metric for IP Performance Metrics (IPPM)", RFC 3393,
              November 2002.

   [RFC3432]  Raisanen, V., Grotefeld, G., and A. Morton, "Network
              performance measurement with periodic streams", RFC 3432,
              November 2002.

   [RFC4737]  Morton, A., Ciavattone, L., Ramachandran, G., Shalunov,
              S., and J. Perser, "Packet Reordering Metrics", RFC 4737,
              November 2006.

   [RFC5357]  Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
              Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
              RFC 5357, October 2008.

   [RFC5905]  Mills, D., Martin, J., Burbank, J., and W. Kasch, "Network
              Time Protocol Version 4: Protocol and Algorithms
              Specification", RFC 5905, June 2010.

7.2.  Informative References

   [RFC1242]  Bradner, S., "Benchmarking terminology for network
              interconnection devices", RFC 1242, July 1991.

   [RFC5481]  Morton, A. and B. Claise, "Packet Delay Variation
              Applicability Statement", RFC 5481, March 2009.

   [RFC6248]  Morton, A., "RFC 4148 and the IP Performance Metrics
              (IPPM) Registry of Metrics Are Obsolete", RFC 6248, April
              2011.

   [RFC6390]  Clark, A. and B. Claise, "Guidelines for Considering New
              Performance Metric Development", BCP 170, RFC 6390,
              October 2011.




Morton                   Expires August 18, 2014                [Page 6]


Internet-Draft               Round-trip Loss               February 2014


Author's Address

   Al Morton
   AT&T Labs
   200 Laurel Avenue South
   Middletown,, NJ  07748
   USA

   Phone: +1 732 420 1571
   Fax:   +1 732 368 1192
   Email: acmorton@att.com
   URI:   http://home.comcast.net/~acmacm/







































Morton                   Expires August 18, 2014                [Page 7]