NFVRG                                                      CJ. Bernardos
Internet-Draft                                                      UC3M
Intended status: Informational                                 A. Rahman
Expires: September 14, 2017                                 InterDigital
                                                              JC. Zuniga
                                                           LM. Contreras
                                                               P. Aranda
                                                                P. Lynch
                                                          March 13, 2017

               Network Virtualization Research Challenges


   This document describes open research challenges for network
   virtualization.  Network virtualization is following a similar path
   as previously taken by cloud computing.  Specifically, Cloud
   computing popularized migration of computing functions (e.g.,
   applications) and storage from local, dedicated, physical resources
   to remote virtual functions accessible through the Internet.  In a
   similar manner, network virtualization is encouraging migration of
   networking functions from dedicated physical hardware nodes to a
   virtualized pool of resources.  However, network virtualization can
   be considered to be a more complex problem than cloud computing as it
   not only involves virtualization of computing and storage functions
   but also involves abstraction of the network itself.  This document
   describes current research challenges in network virtualization
   including guaranteeing quality-of-service, performance improvement,
   supporting multiple domains, network slicing, service composition,
   device virtualization, privacy and security.  In addition, some
   proposals are made for new activities in IETF/IRTF that could address
   some of these challenges.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at

Bernardos, et al.      Expires September 14, 2017               [Page 1]

Internet-Draft Network Virtualization Research Challenges     March 2017

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on September 14, 2017.

Copyright Notice

   Copyright (c) 2017 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   ( in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Background  . . . . . . . . . . . . . . . . . . . . . . . . .   5
     3.1.  Network Function Virtualization . . . . . . . . . . . . .   5
     3.2.  Software Defined Networking . . . . . . . . . . . . . . .   7
     3.3.  Mobile Edge Computing . . . . . . . . . . . . . . . . . .  11
     3.4.  IEEE 802.1CF (OmniRAN)  . . . . . . . . . . . . . . . . .  12
     3.5.  Distributed Management Task Force . . . . . . . . . . . .  12
     3.6.  Open Source initiatives . . . . . . . . . . . . . . . . .  12
     3.7.  Internet of Things (IoT)  . . . . . . . . . . . . . . . .  14
   4.  Network Virtualization Challenges . . . . . . . . . . . . . .  14
     4.1.  Introduction  . . . . . . . . . . . . . . . . . . . . . .  14
     4.2.  Guaranteeing quality-of-service . . . . . . . . . . . . .  15
       4.2.1.  Virtualization Technologies . . . . . . . . . . . . .  15
       4.2.2.  Metrics for NFV characterization  . . . . . . . . . .  15
       4.2.3.  Predictive analysis . . . . . . . . . . . . . . . . .  16
       4.2.4.  Portability . . . . . . . . . . . . . . . . . . . . .  17
     4.3.  Performance improvement . . . . . . . . . . . . . . . . .  17
       4.3.1.  Energy Efficiency . . . . . . . . . . . . . . . . . .  17
       4.3.2.  Improved link usage . . . . . . . . . . . . . . . . .  18
     4.4.  Multiple Domains  . . . . . . . . . . . . . . . . . . . .  18
     4.5.  5G and Network Slicing  . . . . . . . . . . . . . . . . .  18
       4.5.1.  Virtual Network Operators . . . . . . . . . . . . . .  19
       4.5.2.  Extending Virtual Networks and Systems to the

Bernardos, et al.      Expires September 14, 2017               [Page 2]

Internet-Draft Network Virtualization Research Challenges     March 2017

               Internet of Things  . . . . . . . . . . . . . . . . .  20
     4.6.  Service Composition . . . . . . . . . . . . . . . . . . .  21
     4.7.  End-user device virtualization  . . . . . . . . . . . . .  22
     4.8.  Security and Privacy  . . . . . . . . . . . . . . . . . .  22
     4.9.  Separation of control concerns  . . . . . . . . . . . . .  24
     4.10. Testing . . . . . . . . . . . . . . . . . . . . . . . . .  24
       4.10.1.  Changes in methodology . . . . . . . . . . . . . . .  24
       4.10.2.  New functionality  . . . . . . . . . . . . . . . . .  26
       4.10.3.  Opportunities  . . . . . . . . . . . . . . . . . . .  26
   5.  Technology Gaps and Potential IETF Efforts  . . . . . . . . .  27
   6.  Mapping to NFVRG Near-Term work items . . . . . . . . . . . .  27
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  28
   8.  Security Considerations . . . . . . . . . . . . . . . . . . .  28
   9.  Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .  28
   10. Informative References  . . . . . . . . . . . . . . . . . . .  28
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  32

1.  Introduction

   The telecommunications sector is experiencing a major revolution that
   will shape the way networks and services are designed and deployed
   for the next few decades.  In order to cope with continuously
   increasing demand and cost, network operators are taking lessons from
   the IT paradigm of cloud computing.  This new approach of
   virtualizing network functions will enable multi-fold advantages by
   outsourcing communication services from bespoke hardware in the
   operator's core network to Commercial off-the-shelf (COTS) equipment
   distributed across datacenters.

   Some of the network virtualization mechanisms that are being
   considered include: sharing of network infrastructure to reduce
   costs, virtualization of core servers running in data centers as a
   way of supporting their load- aware elastic dimensioning, and dynamic
   energy policies to reduce the electricity consumption.

   This document presents research challenges in Network Function
   Virtualization (NFV) that need to be addressed in order to achieve
   these goals.  The objective of this memo is to document the technical
   challenges and corresponding current approaches and to expose
   requirements that should be addressed by future research and
   standards work.

2.  Terminology

   The following terms used in this document are defined by the ETSI NVF
   ISG [etsi_gs_nfv_003], the ONF [onf_tr_521] and the IETF [RFC7665]:

Bernardos, et al.      Expires September 14, 2017               [Page 3]

Internet-Draft Network Virtualization Research Challenges     March 2017

      Application Plane - The collection of applications and services
      that program network behavior.

      Control Plane (CP) - The collection of functions responsible for
      controlling one or more network devices.  CP instructs network
      devices with respect to how to process and forward packets.  The
      control plane interacts primarily with the forwarding plane and,
      to a lesser extent, with the operational plane.

      Forwarding Plane (FP) - The collection of resources across all
      network devices responsible for forwarding traffic.

      Management Plane (MP) - The collection of functions responsible
      for monitoring, configuring, and maintaining one or more network
      devices or parts of network devices.  The management plane is
      mostly related to the operational plane (it is related less to the
      forwarding plane).

      NFV Infrastructure (NFVI): totality of all hardware and software
      components which build up the environment in which VNFs are

      NFV Management and Orchestration (NFV-MANO): functions
      collectively provided by NFVO, VNFM, and VIM.

      NFV Orchestrator (NFVO): functional block that manages the Network
      Service (NS) lifecycle and coordinates the management of NS
      lifecycle, VNF lifecycle (supported by the VNFM) and NFVI
      resources (supported by the VIM) to ensure an optimized allocation
      of the necessary resources and connectivity.

      OpenFlow protocol (OFP): allowing vendor independent programming
      of control functions in network nodes.

      Operational Plane (OP) - The collection of resources responsible
      for managing the overall operation of individual network devices.

      Physical Network Function (PNF): Physical implementation of a
      Network Function in a monolithic realization.

      Service Function Chain (SFC): for a given service, the abstracted
      view of the required service functions and the order in which they
      are to be applied.  This is somehow equivalent to the Network
      Function Forwarding Graph (NF-FG) at ETSI.

      Service Function Path (SFP): the selection of specific service
      function instances on specific network nodes to form a service
      graph through which an SFC is instantiated.

Bernardos, et al.      Expires September 14, 2017               [Page 4]

Internet-Draft Network Virtualization Research Challenges     March 2017

      Virtualized Infrastructure Manager (VIM): functional block that is
      responsible for controlling and managing the NFVI compute, storage
      and network resources, usually within one operator's
      Infrastructure Domain.

      Virtualized Network Function (VNF): implementation of a Network
      Function that can be deployed on a Network Function Virtualization
      Infrastructure (NFVI).

      Virtualized Network Function Manager (VNFM): functional block that
      is responsible for the lifecycle management of VNF.

3.  Background

3.1.  Network Function Virtualization

   The ETSI ISG NFV is a working group which, since 2012, aims to evolve
   quasi-standard IT virtualization technology to consolidate many
   network equipment types into industry standard high volume servers,
   switches, and storage.  It enables implementing network functions in
   software that can run on a range of industry standard server hardware
   and can be moved to, or loaded in, various locations in the network
   as required, without the need to install new equipment.  To date,
   ETSI NFV is by far the most accepted NFV reference framework and
   architectural footprint [etsi_nvf_whitepaper_2].  The ETSI NFV
   framework architecture framework is composed of three domains
   (Figure 1):

   o  Virtualized Network Function, running over the NFVI.

   o  NFV Infrastructure (NFVI), including the diversity of physical
      resources and how these can be virtualized.  NFVI supports the
      execution of the VNFs.

   o  NFV Management and Orchestration, which covers the orchestration
      and life-cycle management of physical and/or software resources
      that support the infrastructure virtualization, and the life-cycle
      management of VNFs.  NFV Management and Orchestration focuses on
      all virtualization specific management tasks necessary in the NFV

Bernardos, et al.      Expires September 14, 2017               [Page 5]

Internet-Draft Network Virtualization Research Challenges     March 2017

   +-------------------------------------------+  +---------------+
   |   Virtualized Network Functions (VNFs)    |  |               |
   |  -------   -------   -------   -------    |  |               |
   |  |     |   |     |   |     |   |     |    |  |               |
   |  | VNF |   | VNF |   | VNF |   | VNF |    |  |               |
   |  |     |   |     |   |     |   |     |    |  |               |
   |  -------   -------   -------   -------    |  |               |
   +-------------------------------------------+  |               |
                                                  |               |
   +-------------------------------------------+  |               |
   |         NFV Infrastructure (NFVI)         |  |      NFV      |
   | -----------    -----------    ----------- |  |  Management   |
   | | Virtual |    | Virtual |    | Virtual | |  |      and      |
   | | Compute |    | Storage |    | Network | |  | Orchestration |
   | -----------    -----------    ----------- |  |               |
   | +---------------------------------------+ |  |               |
   | |         Virtualization Layer          | |  |               |
   | +---------------------------------------+ |  |               |
   | +---------------------------------------+ |  |               |
   | | -----------  -----------  ----------- | |  |               |
   | | | Compute |  | Storage |  | Network | | |  |               |
   | | -----------  -----------  ----------- | |  |               |
   | |          Hardware resources           | |  |               |
   | +---------------------------------------+ |  |               |
   +-------------------------------------------+  +---------------+

                       Figure 1: ETSI NFV framework

   The NFV architectural framework identifies functional blocks and the
   main reference points between such blocks.  Some of these are already
   present in current deployments, whilst others might be necessary
   additions in order to support the virtualization process and
   consequent operation.  The functional blocks are (Figure 2):

   o  Virtualized Network Function (VNF).

   o  Element Management (EM).

   o  NFV Infrastructure, including: Hardware and virtualized resources,
      and Virtualization Layer.

   o  Virtualized Infrastructure Manager(s) (VIM).

   o  NFV Orchestrator.

   o  VNF Manager(s).

   o  Service, VNF and Infrastructure Description.

Bernardos, et al.      Expires September 14, 2017               [Page 6]

Internet-Draft Network Virtualization Research Challenges     March 2017

   o  Operations and Business Support Systems (OSS/BSS).

   +-------------------------------------------+  | ----------------   |
   |                 OSS/BSS                   |  | | NFV          |   |
   +-------------------------------------------+  | | Orchestrator +-- |
                                                  | ---+------------ | |
   +-------------------------------------------+  |    |             | |
   |  ---------     ---------     ---------    |  |    |             | |
   |  | EM 1  |     | EM 2  |     | EM 3  |    |  |    |             | |
   |  ----+----     ----+----     ----+----    |  | ---+----------   | |
   |      |             |             |        |--|-|    VNF     |   | |
   |  ----+----     ----+----     ----+----    |  | | manager(s) |   | |
   |  | VNF 1 |     | VNF 2 |     | VNF 3 |    |  | ---+----------   | |
   |  ----+----     ----+----     ----+----    |  |    |             | |
   +------|-------------|-------------|--------+  |    |             | |
          |             |             |           |    |             | |
   +------+-------------+-------------+--------+  |    |             | |
   |         NFV Infrastructure (NFVI)         |  |    |             | |
   | -----------    -----------    ----------- |  |    |             | |
   | | Virtual |    | Virtual |    | Virtual | |  |    |             | |
   | | Compute |    | Storage |    | Network | |  |    |             | |
   | -----------    -----------    ----------- |  | ---+------       | |
   | +---------------------------------------+ |  | |        |       | |
   | |         Virtualization Layer          | |--|-| VIM(s) +-------- |
   | +---------------------------------------+ |  | |        |         |
   | +---------------------------------------+ |  | ----------         |
   | | -----------  -----------  ----------- | |  |                    |
   | | | Compute |  | Storage |  | Network | | |  |                    |
   | | | hardware|  | hardware|  | hardware| | |  |                    |
   | | -----------  -----------  ----------- | |  |                    |
   | |          Hardware resources           | |  |  NFV Management    |
   | +---------------------------------------+ |  | and Orchestration  |
   +-------------------------------------------+  +--------------------+

                 Figure 2: ETSI NFV reference architecture

3.2.  Software Defined Networking

   The Software Defined Networking (SDN) paradigm pushes the
   intelligence currently residing in the network elements to a central
   controller implementing the network functionality through software.
   In contrast to traditional approaches, in which the network's control
   plane is distributed throughout all network devices, with SDN the
   control plane is logically centralized.  In this way, the deployment
   of new characteristics in the network no longer requires of complex
   and costly changes in equipment or firmware updates, but only a
   change in the software running in the controller.  The main advantage

Bernardos, et al.      Expires September 14, 2017               [Page 7]

Internet-Draft Network Virtualization Research Challenges     March 2017

   of this approach is the flexibility it provides operators with to
   manage their network, i.e., an operator can easily change its
   policies on how traffic is distributed throughout the network.

   The most visible of the SDN protocol stacks is the OpenFlow protocol
   (OFP), which is maintained and extended by the Open Network
   Foundation (ONF:  Originally this
   protocol was developed specifically for IEEE 802.1 switches
   conforming to the ONF OpenFlow Switch specification.  As the benefits
   of the SDN paradigm have reached a wider audience, its application
   has been extended to more complex scenarios such as Wireless and
   Mobile networks.  Within this area of work, the ONF is actively
   developing new OFP extensions addressing three key scenarios: (i)
   Wireless backhaul, (ii) Cellular Evolved Packet Core (EPC), and (iii)
   Unified access and management across enterprise wireless and fixed

Bernardos, et al.      Expires September 14, 2017               [Page 8]

Internet-Draft Network Virtualization Research Challenges     March 2017

   | -------  |
   | |Oper.|  |            O
   | |Mgmt.|  |<........> -+- Network Operator
   | |Iface|  |            ^
   | -------  |      +----------------------------------------+
   |          |      | +------------------------------------+ |
   |          |      | | ---------  ---------     --------- | |
   |--------- |      | | | App 1 |  | App 2 | ... | App n | | |
   ||Plugins| |<....>| | ---------  ---------     --------- | |
   |--------- |      | | Plugins                            | |
   |          |      | +------------------------------------+ |
   |          |      | Application Plane                      |
   |          |      +----------------------------------------+
   |          |                         A
   |          |                         |
   |          |                         V
   |          |      +----------------------------------------+
   |          |      | +------------------------------------+ |
   |--------- |      | |     ------------  ------------     | |
   || Netw. | |      | |     | Module 1 |  | Module 2 |     | |
   ||Engine | |<....>| |     ------------  ------------     | |
   |--------- |      | | Network Engine                     | |
   |          |      | +------------------------------------+ |
   |          |      | Controller Plane                       |
   |          |      +----------------------------------------+
   |          |                         A
   |          |                         |
   |          |                         V
   |          |      +----------------------------------------+
   |          |      |  +--------------+   +--------------+   |
   |          |      |  | ------------ |   | ------------ |   |
   |----------|      |  | | OpenFlow | |   | | OpenFlow | |   |
   ||OpenFlow||<....>|  | ------------ |   | ------------ |   |
   |----------|      |  | NE           |   | NE           |   |
   |          |      |  +--------------+   +--------------+   |
   |          |      | Data Plane                             |
   |Management|      +----------------------------------------+

                 Figure 3: High level SDN ONF architecture

   Figure 3 shows the blocks and the functional interfaces of the ONF
   architecture, which comprises three planes: Data, Controller, and
   Application.  The Data plane comprehends several Network Entities
   (NE), which expose their capabilities toward the Controller plane via
   a Southbound API.  The Controller plane includes several cooperating
   modules devoted to the creation and maintenance of an abstracted

Bernardos, et al.      Expires September 14, 2017               [Page 9]

Internet-Draft Network Virtualization Research Challenges     March 2017

   resource model of the underneath network.  Such model is exposed to
   the applications via a Northbound API where the Application plane
   comprises several applications/services, each of which has exclusive
   control of a set of exposed resources.

   The Management plane spans its functionality across all planes
   performing the initial configuration of the network elements in the
   Data plane, the assignment of the SDN controller and the resources
   under its responsibility.  In the Controller plane, the Management
   needs to configure the policies defining the scope of the control
   given to the SDN applications, to monitor the performance of the
   system, and to configure the parameters required by the SDN
   controller modules.  In the Application plane, Management configures
   the parameters of the applications and the service level agreements.
   In addition to the these interactions, the Management plane exposes
   several functions to network operators which can easily and quickly
   configure and tune the network at each layer.

   The IRTF Software-Defined Networking Research Group (SDNRG)
   documented in RFC7426 [RFC7426], a layer model of an SDN
   architecture, since this has been a controvertial discussion topic:
   what is exactly SDN? what is the layer structure of the SDN
   architecture? how do layers interface with each oter? etc.

   Figure 4 reproduces the figure included in RFC7426 [RFC7426] to
   summarize the SDN architecture abstractions in the form of a
   detailed, high-level schematic.  In a particular implementation,
   planes can be collocated with other planes or can be physically

   In SDN, a controller manipulates controlled entities via an
   interface.Interfaces, when local, are mostly API invocations through
   some library or system call.  However, such interfaces may be
   extended via some protocol definition, which may use local inter-
   process communication (IPC) or a protocol that could also act
   remotely; the protocol may be defined as an open standard or in a
   proprietary manner.

   SDN expands multiple planes: Forwarding, Operational, Control,
   Management and Applications.  All planes mentioned above are
   connected via interfaces.  Additionally, RFC7426 [RFC7426] considers
   four abstraction layers: the Device and resource Abstraction Layer
   (DAL), the Control Abstraction Layer (CAL), the Management
   Abstraction Layer (MAL) and the Network Services Abstraction Layer

Bernardos, et al.      Expires September 14, 2017              [Page 10]

Internet-Draft Network Virtualization Research Challenges     March 2017

                  |                                |
                  | +-------------+   +----------+ |
                  | | Application |   |  Service | |
                  | +-------------+   +----------+ |
                  |       Application Plane        |
    |           Network Services Abstraction Layer (NSAL)           |
           |                                                |
           |               Service Interface                |
           |                                                |
    o------Y------------------o       o---------------------Y------o
    |      |    Control Plane |       | Management Plane    |      |
    | +----Y----+   +-----+   |       |  +-----+       +----Y----+ |
    | | Service |   | App |   |       |  | App |       | Service | |
    | +----Y----+   +--Y--+   |       |  +--Y--+       +----Y----+ |
    |      |           |      |       |     |               |      |
    | *----Y-----------Y----* |       | *---Y---------------Y----* |
    | | Control Abstraction | |       | | Management Abstraction | |
    | |     Layer (CAL)     | |       | |      Layer (MAL)       | |
    | *----------Y----------* |       | *----------Y-------------* |
    |            |            |       |            |               |
    o------------|------------o       o------------|---------------o
                 |                                 |
                 | CP                              | MP
                 | Southbound                      | Southbound
                 | Interface                       | Interface
                 |                                 |
    |         Device and resource Abstraction Layer (DAL)           |
    |            |                                 |                |
    |    o-------Y----------o   +-----+   o--------Y----------o     |
    |    | Forwarding Plane |   | App |   | Operational Plane |     |
    |    o------------------o   +-----+   o-------------------o     |
    |                       Network Device                          |

                     Figure 4: SDN Layer Architecture

3.3.  Mobile Edge Computing

   Mobile Edge Computing capabilities deployed in the edge of the mobile
   network can facilitate the efficient and dynamic provision of
   services to mobile users.  The ETSI ISG MEC working group, operative

Bernardos, et al.      Expires September 14, 2017              [Page 11]

Internet-Draft Network Virtualization Research Challenges     March 2017

   from end of 2014, intends to specify an open environment for
   integrating MEC capabilities with service providers networks,
   including also applications from 3rd parties.  These distributed
   computing capabilities will make available IT infrastructure as in a
   cloud environment for the deployment of functions in mobile access
   networks.  It can be seen then as a complement to both NFV and SDN.

3.4.  IEEE 802.1CF (OmniRAN)

   The IEEE 802.1CF Recommended Practice [omniran] specifies an access
   network, which connects terminals to their access routers, utilizing
   technologies based on the family of IEEE 802 Standards (e.g., 802.3
   Ethernet, 802.11 Wi-Fi, etc.).  The specification defines an access
   network reference model, including entities and reference points
   along with behavioral and functional descriptions of communications
   among those entities.

   The goal of this project is to help unifying the support of different
   interfaces, enabling shared network control and use of software
   defined network (SDN) principles, thereby lowering the barriers to
   new network technologies, to new network operators, and to new
   service providers.

3.5.  Distributed Management Task Force

   The DMTF is an industry standards organization working to simplify
   the manageability of network-accessible technologies through open and
   collaborative efforts by some technology companies.  The DMTF is
   involved in the creation and adoption of interoperable management
   standards, supporting implementations that enable the management of
   diverse traditional and emerging technologies including cloud,
   virtualization, network and infrastructure.

   There are several DMTF initiatives that are relevant to the network
   virtualization area, such as the Open Virtualization Format (OVF),
   for VNF packaging; the Cloud Infrastructure Management Interface
   (CIM), for cloud infrastructure management; the Network Management
   (NETMAN), for VNF management; and, the Virtualization Management
   (VMAN), for virtualization infrastructure management.

3.6.  Open Source initiatives

   The Open Source community is especially active in the area of network
   virtualization.  We next summarize some of the active efforts:

   o  OpenStack.  OpenStack is a free and open-source cloud-computing
      software platform.  OpenStack software controls large pools of

Bernardos, et al.      Expires September 14, 2017              [Page 12]

Internet-Draft Network Virtualization Research Challenges     March 2017

      compute, storage, and networking resources throughout a
      datacenter, managed through a dashboard or via the OpenStack API.

   o  OpenDayLight.  OpenDaylight (ODL) is a highly available, modular,
      extensible, scalable and multi-protocol controller infrastructure
      built for SDN deployments on modern heterogeneous multi-vendor
      networks.  It provides a model-driven service abstraction platform
      that allows users to write apps that easily work across a wide
      variety of hardware and southbound protocols.

   o  ONOS.  The ONOS (Open Network Operating System) project is an open
      source community hosted by The Linux Foundation.  The goal of the
      project is to create a software-defined networking (SDN) operating
      system for communications service providers that is designed for
      scalability, high performance and high availability.

   o  OpenContrail.  OpenContrail is an Apache 2.0-licensed project that
      is built using standards-based protocols and provides all the
      necessary components for network virtualization-SDN controller,
      virtual router, analytics engine, and published northbound APIs.
      It has an extensive REST API to configure and gather operational
      and analytics data from the system.

   o  OPNFV.  OPNFV is a carrier-grade, integrated, open source platform
      to accelerate the introduction of new NFV products and services.
      By integrating components from upstream projects, the OPNFV
      community aims at conducting performance and use case-based
      testing to ensure the platform's suitability for NFV use cases.
      The scope of OPNFV's initial release is focused on building NFV
      Infrastructure (NFVI) and Virtualized Infrastructure Management
      (VIM) by integrating components from upstream projects such as
      OpenDaylight, OpenStack, Ceph Storage, KVM, Open vSwitch, and
      Linux.  These components, along with application programmable
      interfaces (APIs) to other NFV elements form the basic
      infrastructure required for Virtualized Network Functions (VNF)
      and Management and Network Orchestration (MANO) components.
      OPNFV's goal is to increase performance and power efficiency;
      improve reliability, availability, and serviceability; and deliver
      comprehensive platform instrumentation.

   o  OSM.  Open Source Mano (OSM) is an ETSI-hosted project to develop
      an Open Source NFV Management and Orchestration (MANO) software
      stack aligned with ETSI NFV.  OSM is based on components from
      previous projects, such Telefonica's OpenMANO or Canonical's Juju,
      among others.

   o  OpenBaton.  OpenBaton is a ETSI NFV compliant Network Function
      Virtualization Orchestrator (NFVO).  OpenBaton was part of the

Bernardos, et al.      Expires September 14, 2017              [Page 13]

Internet-Draft Network Virtualization Research Challenges     March 2017

      OpenSDNCore project started with the objective of providing a
      compliant implementation of the ETSI NFV specification.

   Among the main areas that are being developed by the former open
   source activities that related to network virtualization research, we
   can highlight: policy-based resource management, analytics for
   visibility and orchestration, service verification with regards to
   security and resiliency.

3.7.  Internet of Things (IoT)

   The Internet of Things (IoT) refers to the vision of connecting a
   multitude of automated devices (e.g. lights, environmental sensors,
   traffic lights, parking meters, health and security systems, etc.) to
   the Internet for purposes of reporting, and remote command and
   control of the device.  This vision is being realized by a multi-
   pronged approach of standardization in various forums and
   complementary open source activities.  For example, in IETF, support
   of IoT web services has been defined by an HTTP-like protocol adapted
   for IoT called CoAP [RFC7252], and lately a group has been studying
   the need to develop a new network layer to support IP applications
   over Low Power Wide Area Networks (LPWAN).

   Elsewhere, for 5G cellular evolution there is much discussion on the
   need for supporting virtual "network slices" for the expected massive
   numbers of IoT devices.  A separate virtual network slice is
   considered necessary for different 5G IoT use cases because devices
   will have very different characteristics than typical cellular
   devices like smart phones [ngmn_5G_whitepaper], and the number of IoT
   devices is expected to be at least one or two orders of magnitude
   higher than other 5G devices.

4.  Network Virtualization Challenges

4.1.  Introduction

   Network Virtualization is changing the way the telecommunications
   sector will deploy, extend and operate their networks.  These new
   technologies aim at reducing the overall costs by outsourcing
   communication services from specific hardware in the operators' core
   to server farms scattered in datacenters (i.e.  compute and storage
   virtualization).  In addition, the connecting networks are
   fundamentally affected in the way they route, process and control
   traffic (i.e.  network virtualization).

Bernardos, et al.      Expires September 14, 2017              [Page 14]

Internet-Draft Network Virtualization Research Challenges     March 2017

4.2.  Guaranteeing quality-of-service

   Guaranteeing a given quality-of-service in an NFV environment is not
   an easy task.  For example, ensuring a guaranteed and stable
   forwarding data rate has proven not to be straightforward when the
   forwarding function is virtualized and runs on top of COTS server
   hardware [openmano_dataplane]
   [I-D.mlk-nfvrg-nfv-reliability-using-cots] [etsi_nvf_whitepaper_3].
   We next identify some of the challenges that this poses.

4.2.1.  Virtualization Technologies

   The issue of guaranteeing a network quality-of-service is less of an
   issue for "traditional cloud computing" because the workloads that
   are treated there are servers or clients in the networking sense and
   hardly ever process packets.  Cloud computing provides hosting for
   applications on shared servers in a highly separated way.  Its main
   advantage is that the infrastructure costs are shared among tenants
   and that the Cloud infrastructure provides levels of reliability that
   can not be achieved on individual premises in a cost-efficient way
   [intel_10_differences_nfv_cloud].  NFV poses very strict requirements
   posed in terms of performance, stability and consistency.  Although
   there are some tools and mechanisms to improve this, such as Enhanced
   Performance Awareness (EPA), SR-IOV, NUMA, DPDK, etc, these are still
   unsolved challenges.  One open research issue is finding out
   technologies that are different from VM and more suitable for dealing
   with network functionalities.

   Lately, a number of light-weight virtualization technologies
   including containers, unikernels (specialized VMs) and minimalistic
   distributions of general-purpose OSes have appeared as virtualization
   approaches that can be used when constructing an NFV platform.
   [I-D.natarajan-nfvrg-containers-for-nfv] describes the challenges in
   building such a platform and discusses to what extent these
   technologies, as well as traditional VMs, are able to address them.

4.2.2.  Metrics for NFV characterization

   Another relevant aspect is the need for tools for diagnostics and
   measurement suited for NFV.  There is a pressing need to define
   metrics and associated protocols to measure the performance of NFV.
   Specifically, since NFV is based on the concept of taking centralized
   functions and evolving it to highly distributed SW functions, there
   is a commensurate need to fully understand and measure the baseline
   performance of such systems.

   The IP Performance Metrics (IPPM) WG defines metrics that can be used
   to measure the quality and performance of Internet services and

Bernardos, et al.      Expires September 14, 2017              [Page 15]

Internet-Draft Network Virtualization Research Challenges     March 2017

   applications running over transport layer protocols (e.g., TCP, UPD)
   over IP.  It also develops and maintains protocols for the
   measurement of these metrics.  While the IPPM WG is a long running WG
   that started in 1997 it does not have a charter item or active drafts
   related to the topic of network virtualization.  In addition to using
   IPPM metrics to evaluate the QoS, there is a need for specific
   metrics for assessing the performance of network virtualization

   The Benchmarking Methodology Working Group (BMWG) is also performing
   work related to NFV metrics.  For example,
   [I-D.ietf-bmwg-virtual-net] investigates additional methodological
   considerations necessary when benchmarking VNFs instantiated and
   hosted in general- purpose hardware, using bare-metal hypervisors or
   other isolation environments such as Linux containers.  An essential
   consideration is benchmarking physical and virtual network functions
   in the same way when possible, thereby allowing direct comparison.

   As stated in the document [I-D.ietf-bmwg-virtual-net], there is a
   clear motivation for the work on performance metrics for NFV
   [etsi_gs_nfv_per_001], that is worth replicating here: "I'm designing
   and building my NFV Infrastructure platform.  The first steps were
   easy because I had a small number of categories of VNFs to support
   and the VNF vendor gave HW recommendations that I followed.  Now I
   need to deploy more VNFs from new vendors, and there are different
   hardware recommendations.  How well will the new VNFs perform on my
   existing hardware?  Which among several new VNFs in a given category
   are most efficient in terms of capacity they deliver?  And, when I
   operate multiple categories of VNFs (and PNFs) *concurrently* on a
   hardware platform such that they share resources, what are the new
   performance limits, and what are the software design choices I can
   make to optimize my chosen hardware platform?  Conversely, what
   hardware platform upgrades should I pursue to increase the capacity
   of these concurrently operating VNFs?"

   Lately, there are also some efforts lately lookinh into VNF
   benchmarking.  The selection of an NFV Infrastructure Point of
   Presence to host a VNF or allocation of resources (e.g., virtual
   CPUs, memory) needs to be done over virtualized (abstracted and
   simplified) resource views [vnf_benchmarking]

4.2.3.  Predictive analysis

   On top of diagnostic tools that enable an assessment of the QoS,
   predictive analyses are required to react before anomalies occur.
   Due to the SW characteristics of VNFs, a reliable diagnosis framework
   could potentially enable the prevention of issues by a proper

Bernardos, et al.      Expires September 14, 2017              [Page 16]

Internet-Draft Network Virtualization Research Challenges     March 2017

   diagnosis and then a reaction in terms of acting on the potentially
   impacted service (e.g., migration to a different compute node,
   scaling in/out, up/down, etc).

4.2.4.  Portability

   Portability in NFV refers to the ability to run a given VNF on
   multiple NFVIs, that is, that it is possible to guarantee that the
   VNF would be able to perform its functions with a high and
   predictable performance given that a set of requirements on the NFVI
   resources is met.  Therefore, portability is a key feature that, if
   fully enabled, would contribute to making the NFV environment achieve
   a better reliability than a traditional system.  The fact of running
   functionality in SW over "commodity" infrastructure should make much
   easier to port/move functions from one place to another.  However
   this is not yet as ideal as it sounds and there are aspects not fully
   tackled.  The existence of different hypervisors, specific hardware
   dependencies (e.g., EPA related) or state synchronization aspects are
   just some examples of trouble-makers for portability purposes.

   The ETSI NFV ISG is doing work in relation to portability.
   [etsi_gs_nfv_per_001] provides a list of minimal features which the
   VM Descriptor and Compute Host Descriptor should contain for the
   appropriate deployment of VM Images over an NFVI (i.e. a "telco
   datacentre"), in order to guarantee high and predictable performance
   of data plane workloads while assuring their portability.  In
   addition, the document provides a set of recommendations on the
   minimum requirements which HW and hypervisor should have for a "telco
   datacentre" suitable for different workloads (data-plane, control-
   plane, etc.) present in VNFs.  The purpose of this document is to
   provide the list of VM requirements that should be included in the VM
   Descriptor template, and the list of HW capabilities that should be
   included in the Compute Host Descriptor (CHD) to assure predictable
   high performance.  ETSI NFV assumes that the MANO Functions will make
   the mix & match.  There are therefore still quite several research
   challenges to be addressed here.

4.3.  Performance improvement

4.3.1.  Energy Efficiency

   Virtualization is typically seen as a direct enabler of energy
   savings.  Some of the enablers for this that are often mentioned
   [nfv_sota_research_challenges] are: (i) the multiplexing gains
   achieved by centralizing functions in data centers reduce overall the
   energy consumed, (ii) the flexibility brought by network
   programmability enables to switch off infrastructure as needed in a
   much easier way.  However there is still a lot of room for

Bernardos, et al.      Expires September 14, 2017              [Page 17]

Internet-Draft Network Virtualization Research Challenges     March 2017

   improvement in terms of virtualization techniques to reduce the power
   consumption, such as enhanced hypervisor technologies.

4.3.2.  Improved link usage

   The use of NFV and SDN technologies can help improving link usage.
   SDN has shown already that it can greatly increase average link usage
   (e.g., Google example [google_sdn_wan]).  NFV adds more complexity
   (e.g., due to service function chaining / VNF forwarding drafts)
   which need to be considered.  Aspects like the ones described in
   [I-D.bagnulo-nfvrg-topology] on NFV data center topology design have
   to be carefully looked as well.

4.4.  Multiple Domains

   Market fragmentation has resulted in a multitude of network operators
   each focused on different countries and regions.  This makes it
   difficult to create infrastructure services spanning multiple
   countries, such as virtual connectivity or compute resources, as no
   single operator has a footprint everywhere.  Cross-domain
   orchestration of services over multiple administrations or over
   multi-domain single administrations will allow end-to-end network and
   service elements to mix in multi-vendor, heterogeneous technology and
   resource environments.

   For the specific use case of 'Network as a Service', it becomes even
   more important to ensure, that Cross Domain Orchestration also takes
   care of hierarchy of networks and their association, with respect to
   provisioning tunnels and overlays.

   Multi-domain orchestration is currently an active research topic,
   which is being tackled, among others, by ETSI NFV ISG and the 5GEx
   project ( [I-D.bernardos-nfvrg-multidomain].

4.5.  5G and Network Slicing

   From the beginning of all 5G discussions in the research and industry
   fora, it has been agreed that 5G will have to address much more use
   cases than the preceding wireless generations, which first focused on
   voice services, and then on voice and high speed packet data
   services.  In this case, 5G should be able to handle not only the
   same (or enhanced) voice and packet data services, but also new
   emerging services like tactile Internet and IoT.  These use cases
   take the requirements to opposite extremes, as some of them require
   ultra-low latency and higher-speed, whereas some others require
   ultra-low power consumption and high delay tolerance.

Bernardos, et al.      Expires September 14, 2017              [Page 18]

Internet-Draft Network Virtualization Research Challenges     March 2017

   Because of these very extreme 5G use cases, it is envisioned that
   different radio access networks are needed to better address the
   specific requirements of each one of the use cases.  However, on the
   core network side, virtualization techniques can allow tailoring the
   network resources on separate slices, specifically for each radio
   access network and use case, in an efficient manner.

   Network slicing techniques can also allow dedicating resources for
   even more specific use cases within the major 5G categories.  For
   example, within the major IoT category, which is perhaps the most
   disrupting one, some autonomous IoT devices will have very low
   throughput, will have much longer sleep cycles (and therefore high
   latency), and a battery life thousands of times longer compared to
   smart phones or some other connected IoT devices that will have
   almost continuous control and data communications.  Hence, it is
   envisioned that a single virtual core network could be used by
   slicing separate resources to dedicated radio access networks (RANs)
   that are better suited for specific use cases.

   The actual definition of network slicing is still a sensitive
   subject, currently under heavy discussion
   [I-D.defoy-netslices-3gpp-network-slicing] [ngmn_5G_whitepaper].
   Network slicing is a key for introducing new actors in existing
   market at low cost -- by letting new players rent "blocks" of
   capacity, if this new market provides performance that are adequate
   with the application needs (e.g., broadcasting updates to many
   sensors with satellite broadcasting capabilities).  However, more
   work needs to be done to define how network slicing will impact
   existing architectures like ETSI NFV, and to define the impacts of
   network slicing to guaranteeing quality-of-service as described in
   Section 4.2.

4.5.1.  Virtual Network Operators

   The widespread of system and network virtualization technologies has
   conducted to new business opportunities, enlarging the offer of IT
   resources with virtual network and computing resources, among others.
   As a consequence, the network ecosystem now differentiates between
   the owner of physical resources, the Infrastructure Provider (InP),
   and the intermediary that conforms and delivers network services to
   the final customers, the Virtual Network Operator (VNO).

   VNOs aim to exploit the virtualized infrastructures to deliver new
   and improved services to their customers.  However, current network
   virtualization techniques offer poor support for VNOs to control
   their resources.  It has been considered that the InP is responsible
   of the reliability of the virtual resources but there are several

Bernardos, et al.      Expires September 14, 2017              [Page 19]

Internet-Draft Network Virtualization Research Challenges     March 2017

   situations in which an VNO requires to gain a finer control on its
   resources.  For instance, dynamic events, such as the identification
   of new requirements or the detection of incidents within the virtual
   system, might urge a VNO to quickly reform its virtual infrastructure
   and resource allocation.  However, the interfaces offered by current
   virtualization platforms do not offer the necessary functions for
   VNOs to perform the elastic adaptations they require to tackle with
   their dynamic operation environments.

   Beyond their heterogeneity, which can be resolved by software
   adapters, current virtualization platforms do not have common methods
   and functions, so it is difficult for the virtual network controllers
   used by the VNOs to actually manage and control virtual resources
   instantiated on different platforms, not even considering different
   InPs.  Therefore it is necessary to reach a common definition of the
   functions that should be offered by underlying platforms to enable
   such overlay controllers with the possibility of allocate and
   deallocate resources dynamically and get monitoring data about them.

   Such common methods should be offered by all underlying controllers,
   regardless of being network-oriented (e.g.  ODL, ONOS, Ryu) or
   computing-oriented (e.g.  OpenStack, OpenNebula, Eucalyptus).
   Furthermore, it is also important for those platforms to offer some
   "PUSH" function to report resource state, avoiding the need for the
   VNO's controller to "POLL" for such data.  A starting point to get
   proper notifications within current REST APIs could be to consider
   the protocol proposed by the WEBPUSH WG.

   Finally, in order to establish a proper order and allow the
   coexistence and collaboration of different systems, a common ontology
   regarding network and system virtualization should be defined and
   agreed, so different and heterogeneous systems can understand each
   other without requiring to rely on specific adatpation mechanisms
   that might break with any update on any side of the relation.

4.5.2.  Extending Virtual Networks and Systems to the Internet of Things

   The specific nature of the Internet of Things (IoT) ecosystem,
   particularly reflected in the Machine-to-Machine (M2M)
   communications, conducts to the creation of new and highly
   distributed systems which demand location-based network and computing
   services.  An specific example can be represented by a set of
   "things" that suddenly require to set-up a firewall to allow external
   entities to access their data while outsourcing some computation
   requirements to more powerful systems relying on Cloud-based
   services.  This representative use case exposes important
   requirements for both NFV and the underlying Cloud infrastructures.

Bernardos, et al.      Expires September 14, 2017              [Page 20]

Internet-Draft Network Virtualization Research Challenges     March 2017

   In order to provide the aforementioned location-based functions
   integrated with highly distributed systems, the so called FOG
   infrastructures should be able to instantiate VNFs, placing them in
   the required place, e.g. close to their consumers.  This requirement
   implies that the interfaces offered by virtualization platforms must
   support the specification of location-based resources, which is a key
   function in those scenarios.  Moreover, those platforms must also be
   able to interpret and understand the references used by IoT systems
   to their location (e.g., "My-AP", "5BLDG+2F") and also the
   specification of identifiers linked to other resources, such as the
   case of requiring the infrastructure to establish a link between a
   specific AP and a specific virtual computing node.

4.6.  Service Composition

   Current network services deployed by operators often involve the
   composition of several individual functions (such as packet
   filtering, deep packet inspection, load balancing).  These services
   are typically implemented by the ordered combination of a number of
   service functions that are deployed at different points within a
   network, not necessary on the direct data path.  This requires
   traffic to be steered through the required service functions,
   wherever they are deployed [RFC7498].

   For a given service, the abstracted view of the required service
   functions and the order in which they are to be applied is called a
   Service Function Chain (SFC), which is called Network Function
   Forwarding Graph (NF-FG) in ETSI.  An SFC is instantiated through
   selection of specific service function instances on specific network
   nodes to form a service graph: this is called a Service Function Path
   (SFP).  The service functions may be applied at any layer within the
   network protocol stack (network layer, transport layer, application
   layer, etc.).

   Service composition is a powerful tool which can provide significant
   benefits when applied in a softwarized network environment.  There
   are however many research challenges in this area, as for example the
   ones related to composition mechanisms and algorithms to enable load
   balancing and improve reliability.  The service composition should
   also act as an enabler to gather information across all hierarchies
   (underlays and overlays) of network deployments which may span across
   multiple operators, for faster serviceability thus facilitating in
   accomplishing aforementioned goals of "load balancing and improve

   The SFC working group is working on an architecture for service
   function chaining [RFC7665] that includes the necessary protocols or
   protocol extensions to convey the Service Function Chain and Service

Bernardos, et al.      Expires September 14, 2017              [Page 21]

Internet-Draft Network Virtualization Research Challenges     March 2017

   Function Path information to nodes that are involved in the
   implementation of service functions and Service Function Chains, as
   well as mechanisms for steering traffic through service functions.

   In terms of actual work items, the SFC WG is has not yet considered
   working on the management and configuration of SFC components related
   to the support of Service Function Chaining.  This part is of special
   interest for operators and would be required in order to actually put
   SFC mechanisms into operation.  Similarly, redundancy and reliability
   mechanisms are currently not dealt with by any WG in the IETF.  While
   this was the main goal of the VNFpool BoF efforts, it still remains

4.7.  End-user device virtualization

   So far, most of the network softwarization efforts have focused on
   virtualizing functions of network elements.  While virtualization of
   network elements started with the core, mobile networks architectures
   are now heavily switching to also virtualize radio access network
   (RAN) functions.  The next natural step is to get virtualization down
   at the level of the end-user device (i.e., virtualizing a smartphone)
   [virtualization_mobile_device].  The cloning of a device in the cloud
   (central or local) bears attractive benefits to both the device and
   network operations alike (e.g., power saving at the device by
   offloading computational-heaving functions to the cloud, optimized
   networking -- both device-to-device and device-to-infrastructure) for
   service delivery through tighter integration of the device (via its
   clone in the networking infrastructure).  This is being explored for
   example by the European H2020 ICIRRUS project (

4.8.  Security and Privacy

   Similar to any other situation where resources are shared, security
   and privacy are two important aspects that need to be taken into

   In the case of security, there are situations where multiple vendors
   will need to coexist in a virtual or hybrid physical/virtual
   environment.  This requires attestation procedures amongst different
   virtual/physical functions and resources, as well as ongoing external
   monitoring.  Similarly, different network slices operating on the
   same infrastructure can present security problems, for instance if
   one slice running critical applications (e.g. support for a safety
   system) is affected by another slice running a less critical
   application.  In general, the minimum common denominator for security
   measures on a shared system should be equal or higher than the one
   required by the most critical application.  Multiple and continuous

Bernardos, et al.      Expires September 14, 2017              [Page 22]

Internet-Draft Network Virtualization Research Challenges     March 2017

   threat model analysis, as well as DevOps model are required to
   maintain certain level of security in an NFV system.

   On the other hand, privacy in its strictest interpretation, refers to
   concerns about exposing users of the system to individual threats
   such as surveillance, identification, stored data compromise,
   secondary use, intrusion, etc.  In this case, the storage,
   transmission, collection, and potential correlation of information in
   the NFV system, for purposes not originally intended or not known by
   the user, should be avoided.  This is particularly challenging, as
   future intentions and threats cannot be easily predicted, and still
   can be applied for instance on data collected in the past.
   Therefore, well-known techniques such as data minimization, using
   privacy features as default, and allowing users to opt in/out should
   be used to prevent potential privacy issues.

   Compared to traditional networks, NFV will result in networks that
   are much more dynamic (in function distribution and topology) and
   elastic (in size and boundaries).  NFV will thus require network
   operators to evolve their operational and administrative security
   solutions to work in this new environment.  For example, in NFV the
   network orchestrator will become a key node to provide security
   policy orchestration across the different physical and virtual
   components of the virtualized network.  For highly confidential data,
   for example, the network orchestrator should take into account if
   certain physical HW of the network is considered more secure (e.g.,
   because it is located in secure premises) than other HW.

   Traditional telecom networks typically run under a single
   administrative domain controlled by an operator.  With NFV, it is
   expected that in many cases, the telecom operator will now become a
   tenant (running the VNFs), and the infrastructure (NFVI) may be run
   by a different operator and/or cloud service provider (see also
   Section 4.4).  Thus, there will be multiple administrative domains
   which will make coordination of security policy more complex.  For
   example, who will be in charge of provisioning and maintaining
   security credentials such as public and private keys?  Also, should
   private keys be allowed to be replicated across the NFV for
   redundancy reasons?

   On a positive note, NFV will allow better defense against Denial of
   Service (DoS) attacks because of the distributed nature of the
   network (i.e. no single point of failure) and the ability to steer
   (undesirable) traffic quickly [etsi_gs_nfv_sec_001].  Also, NFVs
   which have physical HW which is distributed across multiple data
   centers will also provide better fault isolation environments.
   Especially, if each data center is protected separately via fire
   walls, DMZs and other network protection techniques.

Bernardos, et al.      Expires September 14, 2017              [Page 23]

Internet-Draft Network Virtualization Research Challenges     March 2017

4.9.  Separation of control concerns

   NFV environments offer two possible levels of SDN control.  One level
   is the need for controlling the NFVI to provide connectivity end-to-
   end among VNFs or among VNFs and PNFs (Physical Network Functions).
   A second level is the control and configuration of the VNFs
   themselves (in other words, the configuration of the network service
   implemented by those VNFs), taking profit of the programmability
   brought by SDN.  Both control concerns are separated in nature.
   However, interaction between both could be expected in order to
   optimize, scale or influence each other.

   Clear mechanisms for such interaction are needed in order to avoid
   mal-functioning or interference among concerns.  These ideas are
   considered in [etsi_gs_nfv_eve005] and [I-D.irtf-sdnrg-layered-sdn]

4.10.  Testing

   The impacts of network virtualization on testing can be divided into
   3 groups:

   1.  Changes in methodology.

   2.  New functionality.

   3.  Opportunities.

4.10.1.  Changes in methodology

   The largest impact of NFV is the ability to isolate the System Under
   Test (SUT).  When testing Physical Network Functions (PNF), isolating
   the SUT means that all the other devices that the SUT communicates
   with are replaced with simulations (or controlled executions) in
   order to place the SUT under test by itself.  The SUT may be
   comprised of one or more devices.  The simulations use the
   appropriate traffic type and protocols in order to execute test
   cases.  See Figure 5.

    +--------+      +-----------+     +--------+
    |        |      |           |     |        |
    |  Sim A |      |    SUT    |     | Sim B  |
    |        +------+           +-----+        |
    |        |      |           |     |        |
    +--------+      +-----------+     +--------+

                       Figure 5: Testing methodology

Bernardos, et al.      Expires September 14, 2017              [Page 24]

Internet-Draft Network Virtualization Research Challenges     March 2017

   As shown in Figure 2, NFV provides a common architecture for all
   functions to use.  A VNF is executed using resources offered by the
   NFVI, which have been allocated using the MANO function.  It is not
   possible to test a VNF by itself, without the entire supporting
   environment present.  This fundamentally changes how to consider the
   SUT.  In the case of a VNF (or multiple VNFs), the SUT is part of a
   larger architecture which is necessary in order to run the SUTs.

   Isolation of the SUT therefore becomes controlling the environment in
   a disciplined manner.  The components of the environment necessary to
   run the SUTs that are not part of the SUT become the test
   environment.  In the case of VNFs which are the SUT, then the NFVI
   and MANO become the test environment.  The configurations and
   policies that guide the test environment should remain constant
   during the execution of the tests, and also from test to test.
   Configurations such as CPU pinning, NUMA configuration, the SW
   versions and configurations of the hypervisor, vSwitch and NICs
   should remain constant.  The only variables in the testing should be
   those controlling the SUT itself.  If any configuration in the test
   environment is changed from test to test, then the results become
   very difficult, if not impossible, to compare since the test
   environment behavior may change the results as a consequence of the
   configuration change.

   Testing the NFVI itself also presents new considerations.  With a
   PNF, the dedicated hardware supporting it is optimized for the
   particular workload of the function.  Routing hardware is specially
   built to support packet forwarding functions, while the hardware to
   support a purely control plane application (say, a DNS server, or a
   Diameter function) will not have this specialized capability.  In
   NFV, the NFVI is required to support all types of potentially
   different workload types.

   Testing the NFVI therefore requires careful consideration to what
   types of metrics are sought.  This, in turn, depends on the workload
   type the expected VNF will be.  Examples of different workload types
   are data forwarding, control plane, encryption, and authentication.
   All these types of expected workloads will determine the types of
   metrics that should be sought.  For example, if the workload is
   control plane, then a metric such as jitter is not useful, but
   dropped packets is critical.  In a multi-tenant environment, then the
   NFVI could support various types of workloads.  In this case, testing
   with a variety of traffic types while measuring the corresponding
   metrics simultaneously becomes necessary.

Bernardos, et al.      Expires September 14, 2017              [Page 25]

Internet-Draft Network Virtualization Research Challenges     March 2017

4.10.2.  New functionality

   NFV presents a collection of new functionality in order to support
   the goal of software networking.  Each component on the architecture
   shown in Figure 2 has an associated set of functionality that allows
   VNFs to run: onboarding, lifecycle management for VNFs and Networks
   Services (NS), resource allocation, hypervisor functions, etc.

   One of the new capabilities enabled by NFV is VNFFG (VNF Forwarding
   Graphs).  This refers to the graph that represents a Network Service
   by chaining together VNFs into a forwarding path.  In practice, the
   forwarding path can be implemented in a variety of ways using
   different networking capabilities: vSwitch, SDN, SDN with a
   northbound application, and the VNFFG might use tunneling protocols
   like VXLAN.  The dynamic allocation and implementation of these
   networking paths will have different performance characteristics
   depending on the methods used.  The path implementation mechanism
   becomes a variable in the network testing of the NSs.  The
   methodology used to test the various mechanisms should largely remain
   the same, and as usual, the test environment should remain constant
   for each of the tests, focusing on varying the path establishment

   Scaling refers to the change in allocation of resources to a VNF or
   NS.  It happens dynamically at run-time, based on defined policies
   and triggers.  The triggers can be network, compute or storage based.
   Scaling can allocate more resources in times of need, or reduce the
   amount of resources allocated when the demand is reduced.  The SUT in
   this case becomes much larger than the VNF itself: MANO controls how
   scaling is done based on policies, and then allocates the resources
   accordingly in the NFVI.  Essentially, the testing of scaling
   includes the entire NFV architecture components into the SUT.

4.10.3.  Opportunities

   Softwarization of networking functionality leads to softwarization of
   test as well.  As Physical Network Functions (PNF) are being
   transformed into VNFs, so have the test tools.  This leads to the
   fact that test tools are also being controlled and executed in the
   same environment as the VNFs are.  This presents an opportunity to
   include VNF-based test tools along with the deployment of the VNFs
   supporting the services of the service provider into the host data
   centers.  Tests can therefore be automatically executed upon
   deployment in the target environment, for each deployment, and each
   service.  With PNFs, this was very difficult to achieve.

   This new concept helps to enable modern concepts like DevOps and CI/
   CD in the NFV environment.  Simplistically, DevOps is a process that

Bernardos, et al.      Expires September 14, 2017              [Page 26]

Internet-Draft Network Virtualization Research Challenges     March 2017

   combines multiple functions into single cohesive teams in order to
   quickly produce quality software.  It typically relies on also
   applying the Agile development process, which focusses on (among many
   things) dividing large features into multiple, smaller deliveries.
   One part of this is to immediately test the new smaller features in
   order to get immediate feedback on errors so that if present, they
   can be immediately fixed and redeployed.  The CI/CD (Continuous
   Integration and Continuous Deployment) pipeline supports this
   concept.  It consists of a series of tools, among which immediate
   testing is an integral part, to deliver software from source to
   deployment.  The ability to deploy the test tools themselves into the
   production environment stretches the CI/CD pipeline all the way to
   production deployment, allowing a range of tests to be executed.  The
   tests can be simple, with a goal of verifying the correct deployment
   and networking establishment, to the more complex like testing VNF

5.  Technology Gaps and Potential IETF Efforts

   Table 1 correlates the open network virtualization research areas
   identified in this document to potential IETF groups that could
   address some aspects of them.  An example of a specific gap that the
   group could potentially address is identified in parenthetical beside
   the group name.

   | Open Research Area               | Potential IETF/IRTF Group      |
   | 1-Guaranteeing QoS               | IPPM WG (Measurements of NFVI) |
   | 2-Performance improvement        | VNFPOOL BoF (NFV resilience)   |
   | 3-Multiple Domains               | NFVRG                          |
   | 4-Network Slicing                | NVO3 WG, NETSLICES bar BoF     |
   | 5-Service Composition            | SFC WG (SFC Mgmt and Config)   |
   | 6-End-user device virtualization | N/A                            |
   | 7-Security                       | N/A                            |
   | 8-Separation of control concerns | NFVRG                          |

     Table 1: Mapping of Open Research Areas to Potential IETF Groups

6.  Mapping to NFVRG Near-Term work items

   Table 2 correlates the currently identified NFVRG near-work items to
   the open network virtualization research areas enumerated in this
   document.  This can help the NFVRG in identifying and prioritizing
   research topics.

Bernardos, et al.      Expires September 14, 2017              [Page 27]

Internet-Draft Network Virtualization Research Challenges     March 2017

    | NFVRG Near-Term work item            | Open Research Area      |
    | 1-Policy-based resource management   | - Performance improvem. |
    |                                      | - Network Slicing       |
    | 2-Analytics for visibility & orches. | - Guaranteeing QoS      |
    | 3-Security and service verification  | - Security              |
    | 4-Reliability and fault detection    | - Guaranteeing QoS      |
    | 5-Service orchestration & lifecycle  | - Multiple Domains      |
    |                                      | - Network Slicing       |
    |                                      | - Service Composition   |
    | 6-Real-time properties               | - Guaranteeing QoS      |
    |                                      |                         |
    | (other)                              | - End-user device virt. |
    |                                      | - Separation of control |

   Table 2: Mapping of NFVRG Near-Term work items to Open Research Areas

7.  IANA Considerations


8.  Security Considerations

   This is an informational document, which therefore does not introduce
   any security threat.  Research challenges and gaps related to
   security and privacy have been included in Section 4.8.

9.  Acknowledgments

   The authors want to thank Dirk von Hugo, Rafa Marin, Diego Lopez,
   Ramki Krishnan, Kostas Pentikousis, Rana Pratap Sircar, Alfred
   Morton, Nicolas Kuhn and Saumya Dikshit for their very useful reviews
   and comments to the document.  Special thanks to Pedro Martinez-
   Julia, who provided text for the network slicing section.

   The work of Carlos J.  Bernardos and Luis M.  Contreras is partially
   supported by the H2020-ICT-2014 project 5GEx (Grant Agreement no.

10.  Informative References

Bernardos, et al.      Expires September 14, 2017              [Page 28]

Internet-Draft Network Virtualization Research Challenges     March 2017

              ETSI NFV ISG, "Network Functions Virtualisation (NFV);
              Terminology for Main Concepts in NFV", ETSI GS NFV 003
              V1.2.1 NFV 003, December 2014,

              ETSI NFV ISG, "Network Functions Virtualisation (NFV);
              Ecosystem; Report on SDN Usage in NFV Architectural
              Framework", ETSI GS NFV-EVE 005 V1.1.1 NFV-EVE 005,
              December 2015, <

              ETSI NFV ISG, "Network Functions Virtualisation (NFV); NFV
              Performance & Portability Best Practises", ETSI GS NFV-PER
              001 V1.1.2 NFV-PER 001, December 2014,

              ETSI NFV ISG, "Network Functions Virtualisation (NFV); NFV
              Security; Problem Statement", ETSI GS NFV-SEC 001 V1.1.1
              NFV-SEC 001, October 2014,

              "Network Functions Virtualisation (NFV). White Paper 2",
              October 2013.

              "Network Functions Virtualisation (NFV). White Paper 3",
              October 2014.

              "B4: experience with a globally-deployed Software Defined
              WAN", Proceedings of the ACM SIGCOMM 2013 , August 2013.

              Bagnulo, M. and D. Dolson, "NFVI PoP Network Topology:
              Problem Statement", draft-bagnulo-nfvrg-topology-01 (work
              in progress), March 2016.

Bernardos, et al.      Expires September 14, 2017              [Page 29]

Internet-Draft Network Virtualization Research Challenges     March 2017

              Bernardos, C., Contreras, L., and I. Vaishnavi, "Multi-
              domain Network Virtualization", draft-bernardos-nfvrg-
              multidomain-01 (work in progress), October 2016.

              Foy, X. and A. Rahman, "Network Slicing - 3GPP Use Case",
              draft-defoy-netslices-3gpp-network-slicing-00 (work in
              progress), March 2017.

              Galis, A., Dong, J.,, k.,
              Bryant, S., Boucadair, M., and P. Martinez-Julia, "Network
              Slicing - Introductory Document and Revised Problem
              Statement", draft-gdmb-netslices-intro-and-ps-02 (work in
              progress), February 2017.

              Morton, A., "Considerations for Benchmarking Virtual
              Network Functions and Their Infrastructure", draft-ietf-
              bmwg-virtual-net-04 (work in progress), August 2016.

              Contreras, L., Bernardos, C., Lopez, D., Boucadair, M.,
              and P. Iovanna, "Cooperating Layered Architecture for
              SDN", draft-irtf-sdnrg-layered-sdn-01 (work in progress),
              October 2016.

              Mo, L. and B. Khasnabish, "NFV Reliability using COTS
              Hardware", draft-mlk-nfvrg-nfv-reliability-using-cots-01
              (work in progress), October 2015.

    , n., Krishnan, R., Ghanwani,
              A., Krishnaswamy, D., Willis, P., Chaudhary, A., and F.
              Huici, "An Analysis of Lightweight Virtualization
              Technologies for NFV", draft-natarajan-nfvrg-containers-
              for-nfv-03 (work in progress), July 2016.

              Rosa, R., Rothenberg, C., and R. Szabo, "VNF Benchmark-as-
              a-Service", draft-rorosz-nfvrg-vbaas-00 (work in
              progress), October 2015.

Bernardos, et al.      Expires September 14, 2017              [Page 30]

Internet-Draft Network Virtualization Research Challenges     March 2017

              Intel, "Discover the Top 10 Differences Between NFV and
              Cloud Environments", November 2015,

              , , , , , and , "Network Function Virtualization: State-
              of-the-art and Research Challenges", IEEE Communications
              Surveys & Tutorials Volume: 18, Issue: 1, September 2015.

              "NGMN 5G. White Paper", February 2015.

   [omniran]  IEEE, "802.1CF Network Reference Model and Functional
              Description of IEEE 802 Access Network", 802.1cf, Draft
              0.4 802.1cf, February 2017.

              ONF, "SDN Architecture, Issue 1.1", ONF TR-521 TR-521,
              February 2016,

              Telefonica I+D, "OpenMANO: The Dataplane Ready Open Source
              NFV MANO Stack", March 2015,

   [RFC7252]  Shelby, Z., Hartke, K., and C. Bormann, "The Constrained
              Application Protocol (CoAP)", RFC 7252,
              DOI 10.17487/RFC7252, June 2014,

   [RFC7426]  Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S.,
              Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software-
              Defined Networking (SDN): Layers and Architecture
              Terminology", RFC 7426, DOI 10.17487/RFC7426, January
              2015, <>.

   [RFC7498]  Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for
              Service Function Chaining", RFC 7498,
              DOI 10.17487/RFC7498, April 2015,

Bernardos, et al.      Expires September 14, 2017              [Page 31]

Internet-Draft Network Virtualization Research Challenges     March 2017

   [RFC7665]  Halpern, J., Ed. and C. Pignataro, Ed., "Service Function
              Chaining (SFC) Architecture", RFC 7665,
              DOI 10.17487/RFC7665, October 2015,

              "Virtualization of Mobile Device User Experience", Patent
              US 9.542.062 B2 , January 2017.

              FEEC/UNICAMP, FEEC/UNICAMP, and Ericsson, "A VNF Testing
              Framework Design, Implementation and Partial Results",
              November 2016,

Authors' Addresses

   Carlos J. Bernardos
   Universidad Carlos III de Madrid
   Av. Universidad, 30
   Leganes, Madrid  28911

   Phone: +34 91624 6236

   Akbar Rahman
   InterDigital Communications, LLC
   1000 Sherbrooke Street West, 10th floor
   Montreal, Quebec  H3A 3G4


   Juan Carlos Zuniga
   425 rue Jean Rostand
   Labege  31670


Bernardos, et al.      Expires September 14, 2017              [Page 32]

Internet-Draft Network Virtualization Research Challenges     March 2017

   Luis M. Contreras
   Telefonica I+D
   Ronda de la Comunicacion, S/N
   Madrid  28050


   Pedro Aranda
   Universidad Carlos III de Madrid
   Av. Universidad, 30
   Leganes, Madrid  28911


   Pierre Lynch


Bernardos, et al.      Expires September 14, 2017              [Page 33]