BMWG R. Rosa, Ed.
Internet-Draft Unicamp
Intended status: Informational R. Szabo
Expires: September 22, 2016 Ericsson
March 21, 2016
VNF Benchmarking Methodology
draft-rosa-bmwg-vnfbench-00
Abstract
This document describes VNF benchmarking methodologies.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 22, 2016.
Copyright Notice
Copyright (c) 2016 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Rosa & Szabo Expires September 22, 2016 [Page 1]
Internet-Draft VNFBench March 2016
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 2
3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4. Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . 4
5. VNF Benchmarking Considerations . . . . . . . . . . . . . . . 5
6. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 5
6.1. Benchmarking . . . . . . . . . . . . . . . . . . . . . . . 6
6.1.1. Throughput . . . . . . . . . . . . . . . . . . . . . . . 6
6.1.2. Latency . . . . . . . . . . . . . . . . . . . . . . . . . 7
6.1.3. Frame Loss Rate . . . . . . . . . . . . . . . . . . . . . 7
7. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8
9. Security Considerations . . . . . . . . . . . . . . . . . . . 8
10. Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . 8
11. Informative References . . . . . . . . . . . . . . . . . . . 8
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 9
1. Introduction
New paradigms of network services envisioned by NFV bring VNFs as
software based entities, which can be deployed in virtualized
environments [ETS14a]. In order to be managed/orchestrated or
compared with physical network functions, VNF Descriptors can specify
performance profiles containing metrics (e.g., throughput) associated
with allocated resources (e.g., vCPU). This document describes
benchmarking methodologies to obtain VNF profiles (resource -
performance figures).
2. Terminology
The reader is assumed to be familiar with the terminology as defined
in the European Telecommunications Standards Institute (ETSI) NFV
document [ETS14b]. Some of these terms, and others commonly used in
this document, are defined below.
NFV: Network Function Virtualization - The principle of separating
network functions from the hardware they run on by using virtual
hardware abstraction.
NFVI PoP: NFV Infrastructure Point of Presence - Any combination of
virtualized compute, storage and network resources.
NFVI: NFV Infrastructure - Collection of NFVI PoPs under one
orchestrator.
Rosa & Szabo Expires September 22, 2016 [Page 2]
Internet-Draft VNFBench March 2016
VIM: Virtualized Infrastructure Manager - functional block that is
responsible for controlling and managing the NFVI compute, storage
and network resources, usually within one operator's
Infrastructure Domain (e.g. NFVI-PoP).
NFVO: NFV Orchestrator - functional block that manages the Network
Service (NS) life-cycle and coordinates the management of NS life-
cycle, VNF life-cycle (supported by the VNFM) and NFVI resources
(supported by the VIM) to ensure an optimized allocation of the
necessary resources and connectivity
VNF: Virtualized Network Function - a software-based network
function.
VNFD: Virtualised Network Function Descriptor - configuration
template that describes a VNF in terms of its deployment and
operational behaviour, and is used in the process of VNF on-
boarding and managing the life cycle of a VNF instance.
VNF-FG: Virtualized Network Function Forwarding Graph - an ordered
list of VNFs creating a service chain.
MANO: Management and Orchestration - In the ETSI NFV framework
[ETS14a], this is the global entity responsible for management and
orchestration of NFV life-cycle.
Network Service: composition of Network Functions and defined by its
functional and behavioural specification.
Additional terminology not defined by ETSI NFV ISG.
VNF-BP: VNF Benchmarking Profile - the specification how to measure
a VNF Profile. VNF-BP may be specific to a VNF or applicable to
several VNF types. The specification includes structural and
functional instructions, and variable parameters (metrics) at
different abstractions (e.g., vCPU, memory, throughput, latency;
session, transaction, tenants, etc.).
VNF Profile: is a mapping between virtualized resources (e.g., vCPU,
memory) and VNF performance (e.g., throughput, latency between in/
out ports) at a given NFVI PoP. An orchestration function can use
the VNF Profile to select a host (NFVI PoP) for a VNF and to
allocate necessary resources to deliver the required performance
characteristics.
Customer: A user/subscriber/consumer of ETSI's Network Service.
Rosa & Szabo Expires September 22, 2016 [Page 3]
Internet-Draft VNFBench March 2016
Agents: Network Functions performing benchmarking tasks (e.g.,
synthetic traffic sources and sinks; measurement and observation
functions, etc.).
SUT: System Under Test comprises the VNF under test.
3. Scope
This document assumes VNFs as black boxes when defining VNF
performance benchmarking methodologies. White box benchmarking of
VNFs are left for further studies and may be added later.
4. Assumptions
We assume a VNF benchmarking set-up as shown in Figure 1. Customers
can request Network Services (NS) from an NFVO with associated
service level specifications (e.g., throughput and delay). The NFVO,
in turn, must select hosts and software resource allocations for the
VNFs and build the necessary network overlay to meet the
requirements. Therefore, the NFVO must know VNF Profiles per target
hosts to perform location and resource assignments.
In a highly dynamic environment, where both the VNF instances (e.g.,
revised VM image) and the NFVI resources (e.g., hw upgrades) are
changing, the NFVO should be able to create VNF Profiles on-demand.
We assume, that based on VNF Benchmarking Profile definitions NFVOs
can run benchmarking evaluations to learn VNF Profiles per target
hosts.
In a virtualization environment, however, not only the SUT but all
the other benchmarking agents may be software defined (physical or
virtualized network functions).
Figure 1 shows an example, where the NFVO can use PoPa and PoPb to
set-up benchmarking functions to test VNFs hosted in PoP 1, 2, 3
domains corresponding to VIM 1, 2 and 3. The NFVO uses the VNF
Benchmarking Profiles to deploy agents according to the SUT VNF. The
VNF Benchmarking Profile is defined by the VNF Developer. The
results of the VNF benchmarking is stored in a VNF Profile.
,----.
,----. ( VNF2 )
{VNF1: {10Mbps,200ms}{ ( VNF1 ) `----'
{{2CPU, 8GB}@PoP1} `----'
{{8CPU, 16GB}@PoP2} +---------+ +--------------+
{{4CPU, 4GB}@PoP3}}} |Customers| |VNF Developers|
{20Mbps,300ms}...} +-----+---+ +------.-------+
Rosa & Szabo Expires September 22, 2016 [Page 4]
Internet-Draft VNFBench March 2016
{VNF2:{10Mbps,200ms}{ | |
{{8CPU, 16GB}@PoP1} | |
...}} +-----+-------+ ,------+--------.
,---------------. | |<->(VNF Benchmarking )
( VNF-Profiles )<--->| NFVO / VNFM | \ Profiles /
`---------------' | | `-------------'
+-+----+----+-+
____....----'/ | \---..__
...----''' V V V ```--...__
+----+-+ +------+ +------+ +------+ +-+----+
| VIMa | | VIM1 | | VIM2 | | VIM3 | | VIMb |
+-----++ +-+----+ +-+----+ +-+----+ +-----++
| | | | NFVI |
+------+--+ *-------+--------+--------+--------* +------+--+
|PoPa | | | | | | |PoPb |
|+------+ |SAP | +-----+-+ +---+---+ +-+-----+ | SAP| +------+|
||Agents|=|>O--+-| PoP 1 |--| PoP 2 |--| PoP 3 |--+--O>|=|Agents||
|+------+ | | +-------+ +-------+ +-------+ | | +------+|
| | | PoP1 PoP2 PoP3 | | |
| | | Container Enhanced Baremetal | | |
| | | OS Hypervisor | | |
+---------+ *----------------------------------* +---------+
Figure 1: VNF Testing Scenario
5. VNF Benchmarking Considerations
VNF benchmarking considerations are defined in [Mor15].
Additionally, VNF pre-deployment testing considerations are well
explored in [ETS14c].
This document list further considerations:
Black-Box SUT with Black-Box Benchmarking Agents: In virtualization
environments neither the VNF instance nor the underlying
virtualization environment nor the agents specifics may be known
by the entity managing abstract resources. This implies black box
testing with black box functional components, which are configured
by opaque configuration parameters defined by the VNF developers
or alike for the benchmarking entity (e.g., NFVO).
6. Methodology
Following the ETSI's model ([ETS14c]), we distinguish three methods
for VNF evaluation:
Benchmarking: Where resource {cpu, memory, storage} parameters are
provided and the corresponding {latency, throughput} performance
Rosa & Szabo Expires September 22, 2016 [Page 5]
Internet-Draft VNFBench March 2016
parameters are obtained. Note, such request might create multiple
reports, for example, with minimal latency or maximum throughput
results.
Verification: Both resources {cpu, memory, storage} and performance
{latency, throughput} parameters are provided and agents verifies
if the given association is correct or not.
Dimensioning: Where performance parameters {latency, throughput} are
provided and the corresponding {cpu, memory, storage} resource
parameters obtained. Note, multiple deployment interactions may
be required, or if possible, underlying allocated resources need
to be dynamically altered.
Note: Verification and Dimensioning can be reduced to Benchmarking.
Therefore, we detail Benchmarking in what follows.
6.1. Benchmarking
All benchmarking methodologies described in this section consider the
definition of VNF-BPs for each testing procedure. Information about
Benchmarking Methodology for Network Interconnect Devices, defined in
[rfc2544], is considered in all subsections below. Besides, the
tests are defined based on notions introduced and discussed in the IP
Performance Metrics (IPPM) Framework document [rfc2330].
6.1.1. Throughput
Objective: Provide, for a particular set of resources allocated, the
throughput among two or more VNF ports, expressed in VNF-BP.
Prerequisite: VNF (SUT) must be deployed and stable and its
allocated resources collected. VNF must be reachable by agents.
The frame size to be used for agents must be defined in the VNF-
BP.
Procedure:
1. Establish connectivity between agents and VNF ports.
2. Agents initiate source of traffic, specifically designed for
VNF test, increasing rate periodically.
3. Throughput is measured when traffic rate is achieved without
frame losses.
Reporting Format: report must contain VNF allocated resources and
throughput measured (aka throughput in [rfc2544]).
Rosa & Szabo Expires September 22, 2016 [Page 6]
Internet-Draft VNFBench March 2016
6.1.2. Latency
Objective: Provide, for a particular set of resources allocated, the
latency among two or more VNF ports, expressed in VNF-BP.
Prerequisite: VNF (SUT) must be deployed and stable and its
allocated resources collected. VNF must be reachable by agents.
The frame size and respective throughput to be used for agents
must be defined in the VNF-BP.
Procedure:
1. Establish connectivity between agents and VNF ports.
2. Agents initiate source of traffic, throughput and frame size
specifically designed for VNF test.
3. Latency is measured when throughput is achieved for the period
of time specified in VNF-BP.
Reporting Format: report must contain VNF allocated resources,
throughput used for stimulus and latency measurement (aka latency
in [rfc2544]).
6.1.3. Frame Loss Rate
Objective: Provide, for a particular set of resources allocated, the
frame loss rate among two or more VNF ports, expressed in VNF-BP.
Prerequisite: VNF (SUT) must be deployed and stable, its allocated
resources collected specifying any particular feature of the
underlying VNF virtualized environment, provided by NFVO/VIM or
independently extracted. VNF must be reachable by agents. Rate
of source traffic and frame type used for agents stimulus must be
defined in VNF-BP.
Procedure:
1. Establish connectivity between agents and VNF ports.
2. Agents initiate source of traffic, specifically designed for
VNF test, achieving rate of source traffic defined in VNF-BP.
3. Frame loss rate is measured when pre-defined traffic rate is
achieved for period of time established in VNF-BP.
Rosa & Szabo Expires September 22, 2016 [Page 7]
Internet-Draft VNFBench March 2016
Reporting Format: report must contain VNF allocated resources, rate
of source traffic used as stimulus and frame loss rate measurement
(aka frame loss rate in [rfc2544]).
7. Summary
This document describes black-box benchmarking methodologies for
black-box VNFs in virtualization environments (e.g., ETSI NFV
framework) to create VNF Profiles containing the association of
resources and performance metrics of a given VNF at a given host
(e.g., NFVI PoP).
The authors see the following next steps:
VNF Scaling: Two scaling options: single instance with more
resources or multiple instances. Questions: What is the maximum
performance of a single instance VNF at a given host with
increasing resources? How many independent VNF instances (or
components) can be run with maximum performance at a given host?
On the other hand, what is the performance of the smallest
resource footprint VNF allocation?
VNF instantiation time: this metric concerns at least three
components: VNF bootstraping (SUT), execution environment and the
orchestration process.
8. IANA Considerations
This memo includes no request to IANA.
9. Security Considerations
TBD
10. Acknowledgement
The authors would like to thank the support of Ericsson Research,
Brazil.
This work is partially supported by FP7 UNIFY, a research project
partially funded by the European Community under the Seventh
Framework Program (grant agreement no. 619609). The views expressed
here are those of the authors only. The European Commission is not
liable for any use that may be made of the information in this
document.
11. Informative References
Rosa & Szabo Expires September 22, 2016 [Page 8]
Internet-Draft VNFBench March 2016
[ETS14a] ETSI, "Architectural Framework - ETSI GS NFV 002 V1.2.1",
Dec 2014, <http://www.etsi.org/deliver/etsi\_gs/NFV/
001\_099/002/01.02.01-\_60/gs\_NFV002v010201p.pdf>.
[ETS14b] ETSI, "Terminology for Main Concepts in NFV - ETSI GS NFV
003 V1.2.1", Dec 2014, <http://www.etsi.org/deliver/
etsi_gs/NFV/001_099-/003/01.02.01_60/
gs_NFV003v010201p.pdf>.
[ETS14c] ETSI, "NFV Pre-deployment Testing - ETSI GS NFV TST001
V0.0.15", February 2016, <http://docbox.etsi.org/ISG/NFV/
Open/DRAFTS/TST001_-_Pre-deployment_Validation/NFV-
TST001v0015.zip>.
[Mor15] A. Morton, "Considerations for Benchmarking Virtual
Network Functions and Their Infrastructure", February
2015, <http://tools.ietf.org/html/draft-morton-bmwg-
virtual-net-03>.
[rfc2330] V. Paxson, G. Almes, J. Mahdavi, M. Mathis, "Framework for
IP Performance Metrics", May 1998, <https://tools.ietf.org
/html/rfc2330>.
[rfc2544] S. Bradner and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", March 1999, <https://
tools.ietf.org/html/rfc2544>.
Authors' Addresses
Raphael Vicente Rosa (editor)
University of Campinas
Av. Albert Einstein 300
Campinas, Sao Paulo 13083-852
Brazil
Email: raphaelvrosa@gmail.com
URI: http://www.intrig.dca.fee.unicamp.br/
Robert Szabo
Ericsson Research, Hungary
Irinyi Jozsef u. 4-20
Budapest 1117
Hungary
Email: robert.szabo@ericsson.com
URI: http://www.ericsson.com/
Rosa & Szabo Expires September 22, 2016 [Page 9]