Network Virtualization Research Challenges
draft-irtf-nfvrg-gaps-network-virtualization-09
This document is an Internet-Draft (I-D) that has been submitted to the Internet Research Task Force (IRTF) stream.
This I-D is not endorsed by the IETF and has no formal standing in the
IETF standards process.
The information below is for an old version of the document.
Document | Type |
This is an older version of an Internet-Draft that was ultimately published as RFC 8568.
Expired & archived
|
|
---|---|---|---|
Authors | Carlos J. Bernardos , Akbar Rahman , Juan-Carlos Zúñiga , Luis M. Contreras , Pedro Andres Aranda , Pierre Lynch | ||
Last updated | 2018-09-01 (Latest revision 2018-02-28) | ||
Replaces | draft-bernardos-nfvrg-gaps-network-virtualization | ||
RFC stream | Internet Research Task Force (IRTF) | ||
Formats | |||
IETF conflict review | conflict-review-irtf-nfvrg-gaps-network-virtualization, conflict-review-irtf-nfvrg-gaps-network-virtualization, conflict-review-irtf-nfvrg-gaps-network-virtualization, conflict-review-irtf-nfvrg-gaps-network-virtualization, conflict-review-irtf-nfvrg-gaps-network-virtualization, conflict-review-irtf-nfvrg-gaps-network-virtualization | ||
Stream | IRTF state | In IRSG Poll | |
Consensus boilerplate | Unknown | ||
Document shepherd | (None) | ||
IESG | IESG state | Became RFC 8568 (Informational) | |
Telechat date | (None) | ||
Responsible AD | (None) | ||
Send notices to | (None) |
draft-irtf-nfvrg-gaps-network-virtualization-09
NFVRG CJ. Bernardos Internet-Draft UC3M Intended status: Informational A. Rahman Expires: September 1, 2018 InterDigital JC. Zuniga SIGFOX LM. Contreras TID P. Aranda UC3M P. Lynch Ixia February 28, 2018 Network Virtualization Research Challenges draft-irtf-nfvrg-gaps-network-virtualization-09 Abstract This document describes open research challenges for network virtualization. Network virtualization is following a similar path as previously taken by cloud computing. Specifically, cloud computing popularized migration of computing functions (e.g., applications) and storage from local, dedicated, physical resources to remote virtual functions accessible through the Internet. In a similar manner, network virtualization is encouraging migration of networking functions from dedicated physical hardware nodes to a virtualized pool of resources. However, network virtualization can be considered to be a more complex problem than cloud computing as it not only involves virtualization of computing and storage functions but also involves abstraction of the network itself. This document describes current research and engineering challenges in network virtualization including guaranteeing quality-of-service, performance improvement, supporting multiple domains, network slicing, service composition, device virtualization, privacy and security, separation of control concerns, network function placement and testing. In addition, some proposals are made for new activities in IETF/IRTF that could address some of these challenges. This document is a product of the Network Function Virtualization Research Group (NFVRG). Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Bernardos, et al. Expires September 1, 2018 [Page 1] Internet-Draft Network Virtualization Research Challenges February 2018 Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on September 1, 2018. Copyright Notice Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction and scope . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Background . . . . . . . . . . . . . . . . . . . . . . . . . 5 3.1. Network Function Virtualization . . . . . . . . . . . . . 5 3.2. Software Defined Networking . . . . . . . . . . . . . . . 7 3.3. ITU-T functional architecture of SDN . . . . . . . . . . 12 3.4. Multi-access Edge Computing . . . . . . . . . . . . . . . 13 3.5. IEEE 802.1CF (OmniRAN) . . . . . . . . . . . . . . . . . 14 3.6. Distributed Management Task Force . . . . . . . . . . . . 14 3.7. Open Source initiatives . . . . . . . . . . . . . . . . . 14 4. Network Virtualization Challenges . . . . . . . . . . . . . . 16 4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . 16 4.2. Guaranteeing quality-of-service . . . . . . . . . . . . . 16 4.2.1. Virtualization Technologies . . . . . . . . . . . . . 17 4.2.2. Metrics for NFV characterization . . . . . . . . . . 17 4.2.3. Predictive analysis . . . . . . . . . . . . . . . . . 18 4.2.4. Portability . . . . . . . . . . . . . . . . . . . . . 19 4.3. Performance improvement . . . . . . . . . . . . . . . . . 19 4.3.1. Energy Efficiency . . . . . . . . . . . . . . . . . . 19 Bernardos, et al. Expires September 1, 2018 [Page 2] Internet-Draft Network Virtualization Research Challenges February 2018 4.3.2. Improved link usage . . . . . . . . . . . . . . . . . 20 4.4. Multiple Domains . . . . . . . . . . . . . . . . . . . . 20 4.5. 5G and Network Slicing . . . . . . . . . . . . . . . . . 21 4.5.1. Virtual Network Operators . . . . . . . . . . . . . . 22 4.5.2. Extending Virtual Networks and Systems to the Internet of Things . . . . . . . . . . . . . . . . . 23 4.6. Service Composition . . . . . . . . . . . . . . . . . . . 24 4.7. End-user device virtualization . . . . . . . . . . . . . 25 4.8. Security and Privacy . . . . . . . . . . . . . . . . . . 26 4.9. Separation of control concerns . . . . . . . . . . . . . 27 4.10. Network Function placement . . . . . . . . . . . . . . . 28 4.11. Testing . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.11.1. Changes in methodology . . . . . . . . . . . . . . . 28 4.11.2. New functionality . . . . . . . . . . . . . . . . . 30 4.11.3. Opportunities . . . . . . . . . . . . . . . . . . . 31 5. Technology Gaps and Potential IETF Efforts . . . . . . . . . 31 6. NFVRG focus areas . . . . . . . . . . . . . . . . . . . . . . 32 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 33 8. Security Considerations . . . . . . . . . . . . . . . . . . . 33 9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 33 10. Informative References . . . . . . . . . . . . . . . . . . . 34 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39 1. Introduction and scope The telecommunications sector is experiencing a major revolution that will shape the way networks and services are designed and deployed for the next few decades. In order to cope with continuously increasing demand and cost, network operators are taking lessons from the IT paradigm of cloud computing. This new approach of virtualizing network functions will enable multi-fold advantages by moving communication services from bespoke hardware in the operator's core network to Commercial off-the-shelf (COTS) equipment distributed across datacenters. Some of the network virtualization mechanisms that are being considered include: sharing of network infrastructure to reduce costs, virtualization of core and edge servers/services running in data centers as a way of supporting their load-aware elastic dimensioning, and dynamic energy policies to reduce the electricity consumption. This document presents research and engineering challenges in network virtualization that need to be addressed in order to achieve these goals, spanning from pure research and engineering/standards space. The objective of this memo is to document the technical challenges and corresponding current approaches and to expose requirements that should be addressed by future research and standards work. Bernardos, et al. Expires September 1, 2018 [Page 3] Internet-Draft Network Virtualization Research Challenges February 2018 This document represents the consensus of the NFV Research Group. It has been reviewed by the Research Group members active in the specific areas of work covered by the document. 2. Terminology The following terms used in this document are defined by the ETSI Network Function Virtualization (NVF) Industrial Study Group (ISG) [etsi_gs_nfv_003], the ONF [onf_tr_521] and the IETF [RFC7426] [RFC7665]: Application Plane - The collection of applications and services that program network behavior. Control Plane (CP) - The collection of functions responsible for controlling one or more network devices. CP instructs network devices with respect to how to process and forward packets. The control plane interacts primarily with the forwarding plane and, to a lesser extent, with the operational plane. Forwarding Plane (FP) - The collection of resources across all network devices responsible for forwarding traffic. Management Plane (MP) - The collection of functions responsible for monitoring, configuring, and maintaining one or more network devices or parts of network devices. The management plane is mostly related to the operational plane (it is related less to the forwarding plane). NFV Infrastructure (NFVI): totality of all hardware and software components which build up the environment in which VNFs are deployed. NFV Management and Orchestration (NFV-MANO): functions collectively provided by NFVO, VNFM, and VIM. NFV Orchestrator (NFVO): functional block that manages the Network Service (NS) lifecycle and coordinates the management of NS lifecycle, VNF lifecycle (supported by the VNFM) and NFVI resources (supported by the VIM) to ensure an optimized allocation of the necessary resources and connectivity. Operational Plane (OP) - The collection of resources responsible for managing the overall operation of individual network devices. Physical Network Function (PNF): Physical implementation of a Network Function in a monolithic realization. Bernardos, et al. Expires September 1, 2018 [Page 4] Internet-Draft Network Virtualization Research Challenges February 2018 Service Function Chain (SFC): for a given service, the abstracted view of the required service functions and the order in which they are to be applied. This is somehow equivalent to the Network Function Forwarding Graph (NF-FG) at ETSI. Service Function Path (SFP): the selection of specific service function instances on specific network nodes to form a service graph through which an SFC is instantiated. Virtualized Infrastructure Manager (VIM): functional block that is responsible for controlling and managing the NFVI compute, storage and network resources, usually within one infrastructure operator's Domain. Virtualized Network Function (VNF): implementation of a Network Function that can be deployed on a Network Function Virtualization Infrastructure (NFVI). Virtualized Network Function Manager (VNFM): functional block that is responsible for the lifecycle management of VNF. 3. Background This section briefly describes some basic background technologies, as well as other standards developing organizations and open source initiatives working on network virtualization or related topics. 3.1. Network Function Virtualization The ETSI ISG NFV is a working group which, since 2012, aims to evolve quasi-standard IT virtualization technology to consolidate many network equipment types into industry standard high volume servers, switches, and storage. It enables implementing network functions in software that can run on a range of industry standard server hardware and can be moved to, or loaded in, various locations in the network as required, without the need to install new equipment. The ETSI NFV is one of the predominant NFV reference framework and architectural footprints [nfv_sota_research_challenges]. The ETSI NFV framework architecture framework is composed of three domains (Figure 1): o Virtualized Network Function, running over the NFVI. o NFV Infrastructure (NFVI), including the diversity of physical resources and how these can be virtualized. NFVI supports the execution of the VNFs. o NFV Management and Orchestration, which covers the orchestration and life-cycle management of physical and/or software resources Bernardos, et al. Expires September 1, 2018 [Page 5] Internet-Draft Network Virtualization Research Challenges February 2018 that support the infrastructure virtualization, and the life-cycle management of VNFs. NFV Management and Orchestration focuses on all virtualization specific management tasks necessary in the NFV framework. +-------------------------------------------+ +---------------+ | Virtualized Network Functions (VNFs) | | | | ------- ------- ------- ------- | | | | | | | | | | | | | | | | | VNF | | VNF | | VNF | | VNF | | | | | | | | | | | | | | | | | ------- ------- ------- ------- | | | +-------------------------------------------+ | | | | +-------------------------------------------+ | | | NFV Infrastructure (NFVI) | | NFV | | ----------- ----------- ----------- | | Management | | | Virtual | | Virtual | | Virtual | | | and | | | Compute | | Storage | | Network | | | Orchestration | | ----------- ----------- ----------- | | | | +---------------------------------------+ | | | | | Virtualization Layer | | | | | +---------------------------------------+ | | | | +---------------------------------------+ | | | | | ----------- ----------- ----------- | | | | | | | Compute | | Storage | | Network | | | | | | | ----------- ----------- ----------- | | | | | | Hardware resources | | | | | +---------------------------------------+ | | | +-------------------------------------------+ +---------------+ Figure 1: ETSI NFV framework The NFV architectural framework identifies functional blocks and the main reference points between such blocks. Some of these are already present in current deployments, whilst others might be necessary additions in order to support the virtualization process and consequent operation. The functional blocks are (Figure 2): o Virtualized Network Function (VNF). o Element Management (EM). o NFV Infrastructure, including: Hardware and virtualized resources, and Virtualization Layer. o Virtualized Infrastructure Manager(s) (VIM). Bernardos, et al. Expires September 1, 2018 [Page 6] Internet-Draft Network Virtualization Research Challenges February 2018 o NFV Orchestrator. o VNF Manager(s). o Service, VNF and Infrastructure Description. o Operations and Business Support Systems (OSS/BSS). +--------------------+ +-------------------------------------------+ | ---------------- | | OSS/BSS | | | NFV | | +-------------------------------------------+ | | Orchestrator +-- | | ---+------------ | | +-------------------------------------------+ | | | | | --------- --------- --------- | | | | | | | EM 1 | | EM 2 | | EM 3 | | | | | | | ----+---- ----+---- ----+---- | | ---+---------- | | | | | | |--|-| VNF | | | | ----+---- ----+---- ----+---- | | | manager(s) | | | | | VNF 1 | | VNF 2 | | VNF 3 | | | ---+---------- | | | ----+---- ----+---- ----+---- | | | | | +------|-------------|-------------|--------+ | | | | | | | | | | | +------+-------------+-------------+--------+ | | | | | NFV Infrastructure (NFVI) | | | | | | ----------- ----------- ----------- | | | | | | | Virtual | | Virtual | | Virtual | | | | | | | | Compute | | Storage | | Network | | | | | | | ----------- ----------- ----------- | | ---+------ | | | +---------------------------------------+ | | | | | | | | Virtualization Layer | |--|-| VIM(s) +-------- | | +---------------------------------------+ | | | | | | +---------------------------------------+ | | ---------- | | | ----------- ----------- ----------- | | | | | | | Compute | | Storage | | Network | | | | | | | | hardware| | hardware| | hardware| | | | | | | ----------- ----------- ----------- | | | | | | Hardware resources | | | NFV Management | | +---------------------------------------+ | | and Orchestration | +-------------------------------------------+ +--------------------+ Figure 2: ETSI NFV reference architecture 3.2. Software Defined Networking The Software Defined Networking (SDN) paradigm pushes the intelligence currently residing in the network elements to a central controller implementing the network functionality through software. Bernardos, et al. Expires September 1, 2018 [Page 7] Internet-Draft Network Virtualization Research Challenges February 2018 In contrast to traditional approaches, in which the network's control plane is distributed throughout all network devices, with SDN the control plane is logically centralized. In this way, the deployment of new characteristics in the network no longer requires complex and costly changes in equipment or firmware updates, but only a change in the software running in the controller. The main advantage of this approach is the flexibility it provides operators to manage their network, i.e., an operator can easily change its policies on how traffic is distributed throughout the network. One of the most well known protocols for the SDN control plane between the central controller and the networking elements is the OpenFlow protocol (OFP), which is maintained and extended by the Open Network Foundation (ONF: https://www.opennetworking.org/). Originally this protocol was developed specifically for IEEE 802.1 switches conforming to the ONF OpenFlow Switch specification. As the benefits of the SDN paradigm have reached a wider audience, its application has been extended to more complex scenarios such as Wireless and Mobile networks. Within this area of work, the ONF is actively developing new OFP extensions addressing three key scenarios: (i) Wireless backhaul, (ii) Cellular Evolved Packet Core (EPC), and (iii) Unified access and management across enterprise wireless and fixed networks. Bernardos, et al. Expires September 1, 2018 [Page 8] Internet-Draft Network Virtualization Research Challenges February 2018 +----------+ | ------- | | |Oper.| | O | |Mgmt.| |<........> -+- Network Operator | |Iface| | ^ | ------- | +----------------------------------------+ | | | +------------------------------------+ | | | | | --------- --------- --------- | | |--------- | | | | App 1 | | App 2 | ... | App n | | | ||Plugins| |<....>| | --------- --------- --------- | | |--------- | | | Plugins | | | | | +------------------------------------+ | | | | Application Plane | | | +----------------------------------------+ | | A | | | | | V | | +----------------------------------------+ | | | +------------------------------------+ | |--------- | | | ------------ ------------ | | || Netw. | | | | | Module 1 | | Module 2 | | | ||Engine | |<....>| | ------------ ------------ | | |--------- | | | Network Engine | | | | | +------------------------------------+ | | | | Controller Plane | | | +----------------------------------------+ | | A | | | | | V | | +----------------------------------------+ | | | +--------------+ +--------------+ | | | | | ------------ | | ------------ | | |----------| | | | OpenFlow | | | | OpenFlow | | | ||OpenFlow||<....>| | ------------ | | ------------ | | |----------| | | NE | | NE | | | | | +--------------+ +--------------+ | | | | Data Plane | |Management| +----------------------------------------+ +----------+ Figure 3: High level SDN ONF architecture Figure 3 shows the blocks and the functional interfaces of the ONF architecture, which comprises three planes: Data, Controller, and Application. The Data plane comprehends several Network Entities (NE), which expose their capabilities toward the Controller plane via a Southbound API. The Controller plane includes several cooperating modules devoted to the creation and maintenance of an abstracted Bernardos, et al. Expires September 1, 2018 [Page 9] Internet-Draft Network Virtualization Research Challenges February 2018 resource model of the underlying network. Such model is exposed to the applications via a Northbound API where the Application plane comprises several applications/services, each of which has exclusive control of a set of exposed resources. The Management plane spans its functionality across all planes performing the initial configuration of the network elements in the Data plane, the assignment of the SDN controller and the resources under its responsibility. In the Controller plane, the Management needs to configure the policies defining the scope of the control given to the SDN applications, to monitor the performance of the system, and to configure the parameters required by the SDN controller modules. In the Application plane, Management configures the parameters of the applications and the service level agreements. In addition to these interactions, the Management plane exposes several functions to network operators which can easily and quickly configure and tune the network at each layer. In RFC7426 [RFC7426], the IRTF Software-Defined Networking Research Group (SDNRG) documented a layer model of an SDN architecture, since this has been a controversial discussion topic: what exactly is SDN? what is the layer structure of the SDN architecture? how do layers interface with each other? etc. Figure 4 reproduces the figure included in RFC7426 [RFC7426] to summarize the SDN architecture abstractions in the form of a detailed, high-level schematic. In a particular implementation, planes can be collocated with other planes or can be physically separated. In SDN, a controller manipulates controlled entities via an interface. Interfaces, when local, are mostly API invocations through some library or system call. However, such interfaces may be extended via some protocol definition, which may use local inter- process communication (IPC) or a protocol that could also act remotely; the protocol may be defined as an open standard or in a proprietary manner. SDN expands multiple planes: Forwarding, Operational, Control, Management and Applications. All planes mentioned above are connected via interfaces. Additionally, RFC7426 [RFC7426] considers four abstraction layers: the Device and resource Abstraction Layer (DAL), the Control Abstraction Layer (CAL), the Management Abstraction Layer (MAL) and the Network Services Abstraction Layer (NSAL). Bernardos, et al. Expires September 1, 2018 [Page 10] Internet-Draft Network Virtualization Research Challenges February 2018 o--------------------------------o | | | +-------------+ +----------+ | | | Application | | Service | | | +-------------+ +----------+ | | Application Plane | o---------------Y----------------o | *-----------------------------Y---------------------------------* | Network Services Abstraction Layer (NSAL) | *------Y------------------------------------------------Y-------* | | | Service Interface | | | o------Y------------------o o---------------------Y------o | | Control Plane | | Management Plane | | | +----Y----+ +-----+ | | +-----+ +----Y----+ | | | Service | | App | | | | App | | Service | | | +----Y----+ +--Y--+ | | +--Y--+ +----Y----+ | | | | | | | | | | *----Y-----------Y----* | | *---Y---------------Y----* | | | Control Abstraction | | | | Management Abstraction | | | | Layer (CAL) | | | | Layer (MAL) | | | *----------Y----------* | | *----------Y-------------* | | | | | | | o------------|------------o o------------|---------------o | | | CP | MP | Southbound | Southbound | Interface | Interface | | *------------Y---------------------------------Y----------------* | Device and resource Abstraction Layer (DAL) | *------------Y---------------------------------Y----------------* | | | | | o-------Y----------o +-----+ o--------Y----------o | | | Forwarding Plane | | App | | Operational Plane | | | o------------------o +-----+ o-------------------o | | Network Device | +---------------------------------------------------------------+ Figure 4: SDN Layer Architecture While SDN is often directly associated to OpenFlow, this is just one (relevant) example of a southbound protocol between the central controller and the network entities. Other relevant examples of protocols in the SDN family are NETCONF [RFC6241], RESTCONF [RFC8040] and ForCES [RFC5810]. Bernardos, et al. Expires September 1, 2018 [Page 11] Internet-Draft Network Virtualization Research Challenges February 2018 3.3. ITU-T functional architecture of SDN The Telecommunication standardization sector of the International Telecommunication Union (ITU) -- the ITU-T -- has also looked into SDN architectures, defining a slightly modified one from what other SDOs have done. ITU-T provides in the recommendation ITU-T Y.3302 [itu-t-y.3302] a functional architecture of SDN with descriptions of functional components and reference points. The described functional architecture is intended to be used as an enabler for further studies on other aspects such as protocols and security as well as being used to customize SDN in support of appropriate use cases (e.g., cloud computing, mobile networks). This recommendation is based on ITU-T Y.3300 [itu-t-y.3300] and ITU-T Y.3301 [itu-t-y.3301]. While the first describes the framework of SDN (including definitions, objectives, high-level capabilities, requirements and the high-level architecture of SDN), the second describes more detailed requirements. Figure 5 shows the SDN functional architecture defined by the ITU-T. It is a layered architecture composed of the SDN application layer (SDN-AL), the SDN control layer (SDN-CL) and the SDN resource layer (SDN-RL). It also has multi-layer management functions (MMF), which provides functionalities for managing the functionalities of SDN layers, i.e., SDN-AL, SDN-CL and SDN-RL. MMF interacts with these layers using MMFA, MMFC, and MMFR reference points. The SDN-AL enables a service-aware behavior of the underlying network in a programmatic manner. The SDN-CL provides programmable means to control the behavior of SDN-RL resources (such as data transport and processing), following requests received from the SDN-AL according to MMF policies. The SDN-RL is where the physical or virtual network elements perform transport and/or processing of data packets according to SDN-CL decisions. Bernardos, et al. Expires September 1, 2018 [Page 12] Internet-Draft Network Virtualization Research Challenges February 2018 MMFO MMFA +-----+ . +---------------------+ . +--------------------+ | | . |+---+ +---+ +-------+| . |+---------+ +-----+ | | | . || | | | | || . || AL. | | | | | | . || E | | | | App. || . || mngmt. | | SDN | | SDN-AL | | . || x | | M | | layer || . || support | | app | | | | . || t.| | u | | Mngmt.|| . || & orch. | | | | | | . || | | l | +-------+| . |+---------+ +-----+ | | | . || r | | t | | . +--------------------+ | | . || e | | i | |MMFC ..................... ACI | | . || l | | | | . +--------------------+ | | . || a | | l | +-------+| . |+------+ +---------+| | OSS/| . || t | | a | | || . || | | App. || | BSS | . || i | | y | | || . || | | support || | | . || o | | e | | || . || | +---------+| | | . || n | | r | | || . || CL | +---------+| | | . || s | | | |Control|| . ||mngmt.| | Control || | | . || h | | m | | layer || . || supp.| | layer || SDN-CL | | . || i | | a | | mngmt.|| . || and | | serv. || | | . || p | | n | | || . || orch.| +---------+| | | . || | | a | | || . || | +---------+| | | . || m | | g | | || . || | | Resource|| | | . || n | | e | | || . || | | abstrac.|| | | . || g | | m | +-------+| . |+------+ +---------+| | | . || m | | e | | . +--------------------+ | | . || t.| | n | |MMFR ..................... RCI | | . || | | t | | . +--------------------+ +-----+ . |+---+ | | +-------+| . |+------++----------+| | | o | | || . || ||RL control|| | | r | |Resour.|| . || RL |+----------+| MMF | | c | | layer || . ||mngmt.|+----++----+| SDN-RL | | h.| | mngmt.|| . || supp.||Data||Data|| | | | | || . || ||tran||proc|| | +---+ +-------+| . |+------++----++----+| +---------------------+ . +--------------------+ Figure 5: ITU-T SDN functional architecture 3.4. Multi-access Edge Computing Multi-access Edge Computing (MEC) -- formerly known as Mobile Edge Computing -- capabilities deployed in the edge of the mobile network can facilitate the efficient and dynamic provision of services to mobile users. The ETSI ISG MEC working group, operative from end of 2014, intends to specify an open environment for integrating MEC capabilities with service providers' networks, including also applications from 3rd parties. These distributed computing capabilities will make available IT infrastructure as in a cloud Bernardos, et al. Expires September 1, 2018 [Page 13] Internet-Draft Network Virtualization Research Challenges February 2018 environment for the deployment of functions in mobile access networks. It can be seen then as a complement to both NFV and SDN. 3.5. IEEE 802.1CF (OmniRAN) The IEEE 802.1CF Recommended Practice [omniran] specifies an access network, which connects terminals to their access routers, utilizing technologies based on the family of IEEE 802 Standards (e.g., 802.3 Ethernet, 802.11 Wi-Fi, etc.). The specification defines an access network reference model, including entities and reference points along with behavioral and functional descriptions of communications among those entities. The goal of this project is to help unifying the support of different interfaces, enabling shared network control and use of SDN principles, thereby lowering the barriers to new network technologies, to new network operators, and to new service providers. 3.6. Distributed Management Task Force The DMTF (https://www.dmtf.org/) is an industry standards organization working to simplify the manageability of network- accessible technologies through open and collaborative efforts by some technology companies. The DMTF is involved in the creation and adoption of interoperable management standards, supporting implementations that enable the management of diverse traditional and emerging technologies including cloud, virtualization, network and infrastructure. There are several DMTF initiatives that are relevant to the network virtualization area, such as the Open Virtualization Format (OVF), for VNF packaging; the Cloud Infrastructure Management Interface (CIM), for cloud infrastructure management; the Network Management (NETMAN), for VNF management; and, the Virtualization Management (VMAN), for virtualization infrastructure management. 3.7. Open Source initiatives The Open Source community is especially active in the area of network virtualization and orchestration. We next summarize some of the active efforts: o OpenStack. OpenStack is a free and open-source cloud-computing software platform. OpenStack software controls large pools of compute, storage, and networking resources throughout a datacenter, managed through a dashboard or via the OpenStack API. Bernardos, et al. Expires September 1, 2018 [Page 14] Internet-Draft Network Virtualization Research Challenges February 2018 o Kubernetes. Kubernetes is an open-source system for automating deployment, scaling and management of containerized applications. Kubernetes can schedule and run application containers on clusters of physical or virtual machines. Kubernetes allows: (i) Scale on the fly, (ii) Limit hardware usage to required resources only, (iii) Load balancing Monitoring, and (iv) Efficient lifecycle management. o OpenDayLight. OpenDaylight (ODL) is a highly available, modular, extensible and scalable multi-protocol controller infrastructure built for SDN deployments on modern heterogeneous multi-vendor networks. It provides a model-driven service abstraction platform that allows users to write apps that easily work across a wide variety of hardware and southbound protocols. o ONOS. The ONOS (Open Network Operating System) project is an open source community hosted by The Linux Foundation. The goal of the project is to create a SDN operating system for communications service providers that is designed for scalability, high performance and high availability. o OpenContrail. OpenContrail is an Apache 2.0-licensed project that is built using standards-based protocols and provides all the necessary components for network virtualization-SDN controller, virtual router, analytics engine, and published northbound APIs. It has an extensive REST API to configure and gather operational and analytics data from the system. o OPNFV. OPNFV is a carrier-grade, integrated, open source platform to accelerate the introduction of new NFV products and services. By integrating components from upstream projects, the OPNFV community aims at conducting performance and use case-based testing to ensure the platform's suitability for NFV use cases. The scope of OPNFV's initial release is focused on building NFV Infrastructure (NFVI) and Virtualized Infrastructure Management (VIM) by integrating components from upstream projects such as OpenDaylight, OpenStack, Ceph Storage, KVM, Open vSwitch, and Linux. These components, along with application programmable interfaces (APIs) to other NFV elements form the basic infrastructure required for Virtualized Network Functions (VNF) and Management and Network Orchestration (MANO) components. OPNFV's goal is to (i) increase performance and power efficiency, (ii) improve reliability, availability, and serviceability, and (iii) deliver comprehensive platform instrumentation. o OSM. Open Source Mano (OSM) is an ETSI-hosted project to develop an Open Source NFV Management and Orchestration (MANO) software stack aligned with ETSI NFV. OSM is based on components from Bernardos, et al. Expires September 1, 2018 [Page 15] Internet-Draft Network Virtualization Research Challenges February 2018 previous projects, such Telefonica's OpenMANO or Canonical's Juju, among others. o OpenBaton. OpenBaton is a ETSI NFV compliant Network Function Virtualization Orchestrator (NFVO). OpenBaton was part of the OpenSDNCore project started with the objective of providing a compliant implementation of the ETSI NFV specification. o ONAP. ONAP (Open Network Automation Platform) is an open source software platform that delivers capabilities for the design, creation, orchestration, monitoring, and life cycle management of: (i) Virtual Network Functions (VNFs), (ii) The carrier-scale Software Defined Networks (SDNs) that contain them, and (iii) Higher-level services that combine the above. ONAP (derived from the AT&T's ECOMP) provides for automatic, policy-driven interaction of these functions and services in a dynamic, real- time cloud environment. o SONA. SONA (Simplified Overlay Network Architecture) is an extension to ONOS to have a almost full SDN network control in OpenStack for virtual tenant network provisioning. Basically, SONA is an SDN-based network virtualization solution for cloud DC. Among the main areas that are being developed by the former open source activities that relate to network virtualization research, we can highlight: policy-based resource management, analytics for visibility and orchestration, service verification with regards to security and resiliency. 4. Network Virtualization Challenges 4.1. Introduction Network Virtualization is changing the way the telecommunications sector will deploy, extend and operate their networks. These new technologies aim at reducing the overall costs by moving communication services from specific hardware in the operators' core to server farms scattered in datacenters (i.e. compute and storage virtualization). In addition, the networks interconnecting the functions that compose a network service are fundamentally affected in the way they route, process and control traffic (i.e. network virtualization). 4.2. Guaranteeing quality-of-service Achieving a given quality-of-service in an NFV environment with virtualized and distributed computing, storage and networking functions is more challenging than providing the equivalent in Bernardos, et al. Expires September 1, 2018 [Page 16] Internet-Draft Network Virtualization Research Challenges February 2018 discrete non-virtualized components. For example, ensuring a guaranteed and stable forwarding data rate has proven not to be straightforward when the forwarding function is virtualized and runs on top of COTS server hardware [openmano_dataplane] [I-D.mlk-nfvrg-nfv-reliability-using-cots] [etsi_nvf_whitepaper_3]. Again, the comparison point is against a router or forwarder built on optimized hardware. We next identify some of the challenges that this poses. 4.2.1. Virtualization Technologies The issue of guaranteeing a network quality-of-service is less of an issue for "traditional cloud computing" because the workloads that are treated there are servers or clients in the networking sense and hardly ever process packets. Cloud computing provides hosting for applications on shared servers in a highly separated way. Its main advantage is that the infrastructure costs are shared among tenants and that the cloud infrastructure provides levels of reliability that can not be achieved on individual premises in a cost-efficient way [intel_10_differences_nfv_cloud]. NFV has very strict requirements posed in terms of performance, stability and consistency. Although there are some tools and mechanisms to improve this, such as Enhanced Performance Awareness (EPA), Single Root I/O Virtualization (SR-IOV), Non-Uniform Memory Access (NUMA), Data Plane Development Kit (DPDK), etc, these are still unsolved challenges. One open research issue is finding out technologies that are different from VM and more suitable for dealing with network functionalities. Lately, a number of light-weight virtualization technologies including containers, unikernels (specialized VMs) and minimalistic distributions of general-purpose OSes have appeared as virtualization approaches that can be used when constructing an NFV platform. [I-D.natarajan-nfvrg-containers-for-nfv] describes the challenges in building such a platform and discusses to what extent these technologies, as well as traditional VMs, are able to address them. 4.2.2. Metrics for NFV characterization Another relevant aspect is the need for tools for diagnostics and measurement suited for NFV. There is a pressing need to define metrics and associated protocols to measure the performance of NFV. Specifically, since NFV is based on the concept of taking centralized functions and evolving it to highly distributed SW functions, there is a commensurate need to fully understand and measure the baseline performance of such systems. The IP Performance Metrics (IPPM) WG defines metrics that can be used to measure the quality and performance of Internet services and Bernardos, et al. Expires September 1, 2018 [Page 17] Internet-Draft Network Virtualization Research Challenges February 2018 applications running over transport layer protocols (e.g., TCP, UDP) over IP. It also develops and maintains protocols for the measurement of these metrics. While the IPPM WG is a long running WG that started in 1997, at the time of writing it does not have a charter item or active drafts related to the topic of network virtualization. In addition to using IPPM metrics to evaluate the QoS, there is a need for specific metrics for assessing the performance of network virtualization techniques. The Benchmarking Methodology Working Group (BMWG) is also performing work related to NFV metrics. For example, [RFC8172] investigates additional methodological considerations necessary when benchmarking VNFs instantiated and hosted in general-purpose hardware, using bare- metal hypervisors or other isolation environments such as Linux containers. An essential consideration is benchmarking physical and virtual network functions in the same way when possible, thereby allowing direct comparison. As stated in the document [RFC8172], there is a clear motivation for the work on performance metrics for NFV [etsi_gs_nfv_per_001], that is worth replicating here: "I'm designing and building my NFV Infrastructure platform. The first steps were easy because I had a small number of categories of VNFs to support and the VNF vendor gave HW recommendations that I followed. Now I need to deploy more VNFs from new vendors, and there are different hardware recommendations. How well will the new VNFs perform on my existing hardware? Which among several new VNFs in a given category are most efficient in terms of capacity they deliver? And, when I operate multiple categories of VNFs (and PNFs) *concurrently* on a hardware platform such that they share resources, what are the new performance limits, and what are the software design choices I can make to optimize my chosen hardware platform? Conversely, what hardware platform upgrades should I pursue to increase the capacity of these concurrently operating VNFs?" Lately, there are also some efforts looking into VNF benchmarking. The selection of an NFV Infrastructure Point of Presence to host a VNF or allocation of resources (e.g., virtual CPUs, memory) needs to be done over virtualized (abstracted and simplified) resource views [vnf_benchmarking] [I-D.rorosz-nfvrg-vbaas]. 4.2.3. Predictive analysis On top of diagnostic tools that enable an assessment of the QoS, predictive analyses are required to react before anomalies occur. Due to the SW characteristics of VNFs, a reliable diagnosis framework could potentially enable the prevention of issues by a proper diagnosis and then a reaction in terms of acting on the potentially Bernardos, et al. Expires September 1, 2018 [Page 18] Internet-Draft Network Virtualization Research Challenges February 2018 impacted service (e.g., migration to a different compute node, scaling in/out, up/down, etc). 4.2.4. Portability Portability in NFV refers to the ability to run a given VNF on multiple NFVIs, that is, guaranteeing that the VNF would be able to perform its functions with a high and predictable performance given that a set of requirements on the NFVI resources is met. Therefore, portability is a key feature that, if fully enabled, would contribute to making the NFV environment achieve a better reliability than a traditional system. Implementing functionality in SW over "commodity" infrastructure should make it much easier to port/move functions from one place to another. However this is not yet as ideal as it sounds, and there are aspects that are not fully tackled. The existence of different hypervisors, specific hardware dependencies (e.g., EPA related) or state synchronization aspects are just some examples of trouble-makers for portability purposes. The ETSI NFV ISG is doing work in relation to portability. [etsi_gs_nfv_per_001] provides a list of minimal features which the VM Descriptor and Compute Host Descriptor should contain for the appropriate deployment of VM images over an NFVI (i.e. a "telco datacenter"), in order to guarantee high and predictable performance of data plane workloads while assuring their portability. In addition, the document provides a set of recommendations on the minimum requirements which HW and hypervisor should have for a "telco datacenter" suitable for different workloads (data-plane, control- plane, etc.) present in VNFs. The purpose of this document is to provide the list of VM requirements that should be included in the VM Descriptor template, and the list of HW capabilities that should be included in the Compute Host Descriptor (CHD) to assure predictable high performance. ETSI NFV assumes that the MANO Functions will make the mix & match. There are therefore still several research challenges to be addressed here. 4.3. Performance improvement 4.3.1. Energy Efficiency Virtualization is typically seen as a direct enabler of energy savings. Some of the enablers for this that are often mentioned [nfv_sota_research_challenges] are: (i) the multiplexing gains achieved by centralizing functions in data centers reduce the overall energy consumed, (ii) the flexibility brought by network programmability enables to switch off infrastructure as needed in a much easier way. However there is still a lot of room for Bernardos, et al. Expires September 1, 2018 [Page 19] Internet-Draft Network Virtualization Research Challenges February 2018 improvement in terms of virtualization techniques to reduce the power consumption, such as enhanced hypervisor technologies. Some additional examples of research topics that could enable energy savings are [nfv_sota_research_challenges]: o Energy aware scaling (e.g., reductions in CPU speeds and partially turning off some hardware com- ponents to meet a given energy consumption target. o Energy-aware function placement. o Scheduling and chaining algorithms, for example adapting the network topology and operating parameters to minimize the operation cost (e.g., tracking energy costs to identify the cheapest prices). Note that it is also important to analyze the trade-off between energy efficiency and network performance. 4.3.2. Improved link usage The use of NFV and SDN technologies can help improve link usage. SDN has already shown that it can greatly increase average link utilization (e.g., Google example [google_sdn_wan]). NFV adds more complexity (e.g., due to service function chaining / VNF forwarding graphs) which need to be considered. Aspects like the ones described in [I-D.bagnulo-nfvrg-topology] on NFV data center topology design have to be carefully looked at as well. 4.4. Multiple Domains Market fragmentation has resulted in a multitude of network operators each focused on different countries and regions. This makes it difficult to create infrastructure services spanning multiple countries, such as virtual connectivity or compute resources, as no single operator has a footprint everywhere. Cross-domain orchestration of services over multiple administrations or over multi-domain single administrations will allow end-to-end network and service elements to mix in multi-vendor, heterogeneous technology and resource environments [multi-domain_5GEx]. For the specific use case of 'Network as a Service', it becomes even more important to ensure that Cross Domain Orchestration also takes care of hierarchy of networks and their association, with respect to provisioning tunnels and overlays. Bernardos, et al. Expires September 1, 2018 [Page 20] Internet-Draft Network Virtualization Research Challenges February 2018 Multi-domain orchestration is currently an active research topic, which is being tackled, among others, by ETSI NFV ISG and the 5GEx project (https://www.5gex.eu/) [I-D.bernardos-nfvrg-multidomain] [multi-domain_5GEx]. Another side of the multi-domain problem is the integration/ harmonization of different management domains. A key example comes from Multi-access Edge Computing, which, according to ETSI, comes with its own MANO system, and would require to be integrated if interconnected to a generic NFV system. 4.5. 5G and Network Slicing From the beginning of all 5G discussions in the research and industry fora, it has been agreed that 5G will have to address much more use cases than the preceding wireless generations, which first focused on voice services, and then on voice and high speed packet data services. In this case, 5G should be able to handle not only the same (or enhanced) voice and packet data services, but also new emerging services like tactile Internet and IoT. These use cases take the requirements to opposite extremes, as some of them require ultra-low latency and higher-speed, whereas some others require ultra-low power consumption and high delay tolerance. Because of these very extreme 5G use cases, it is envisioned that selective combinations of radio access networks and core network components will have to be combined into a given network slice to address the specific requirements of each use case. For example, within the major IoT category, which is perhaps the most disrupting one, some autonomous IoT devices will have very low throughput, will have much longer sleep cycles (and therefore high latency), and a battery life time exceeding by a factor of thousands that of smart phones or some other devices that will have almost continuous control and data communications. Hence, it is envisioned that a customized network slice will have to be stitched together from virtual resources or sub-slices to meet these requirements. The actual definition of network slice from an IP infrastructure viewpoint is currently undergoing intense debate [I-D.geng-coms-problem-statement] [I-D.gdmb-netslices-intro-and-ps] [I-D.defoy-netslices-3gpp-network-slicing] [ngmn_5G_whitepaper]. Network slicing is a key for introducing new actors in existing market at low cost -- by letting new players rent "blocks" of capacity, if the new business model enables performance that meets the application needs (e.g., broadcasting updates to many sensors with satellite broadcasting capabilities). However, more work needs to be done to define the basic architectural approach of how network Bernardos, et al. Expires September 1, 2018 [Page 21] Internet-Draft Network Virtualization Research Challenges February 2018 slices will be defined and formed. For example, is it mostly a matter of defining the appropriate network models (e.g. YANG) to stitch the network slice from existing components. Or do end-to-end timing, synchronization and other low level requirements mean that more fundamental research has to be done. 4.5.1. Virtual Network Operators The widespread use/discussion/practice of system and network virtualization technologies has led to new business opportunities, enlarging the offer of IT resources with virtual network and computing resources, among others. As a consequence, the network ecosystem now differentiates between the owner of physical resources, the Infrastructure Provider (InP), and the intermediary that conforms and delivers network services to the final customers, the Virtual Network Operator (VNO). VNOs aim to exploit the virtualized infrastructures to deliver new and improved services to their customers. However, current network virtualization techniques offer poor support for VNOs to control their resources. It has been considered that the InP is responsible for the reliability of the virtual resources but there are several situations in which a VNO requires to gain a finer control on its resources. For instance, dynamic events, such as the identification of new requirements or the detection of incidents within the virtual system, might urge a VNO to quickly reform its virtual infrastructure and resource allocation. However, the interfaces offered by current virtualization platforms do not offer the necessary functions for VNOs to perform the elastic adaptations they require to tackle with their dynamic operation environments. Beyond their heterogeneity, which can be resolved by software adapters, current virtualization platforms do not have common methods and functions, so it is difficult for the virtual network controllers used by the VNOs to actually manage and control virtual resources instantiated on different platforms, not even considering different InPs. Therefore it is necessary to reach a common definition of the functions that should be offered by underlying platforms to give such overlay controllers the possibility to allocate and deallocate resources dynamically and get monitoring data about them. Such common methods should be offered by all underlying controllers, regardless of being network-oriented (e.g. ODL, ONOS, Ryu) or computing-oriented (e.g. OpenStack, OpenNebula, Eucalyptus). Furthermore, it is also important for those platforms to offer some "PUSH" function to report resource state, avoiding the need for the VNO's controller to "POLL" for such data. A starting point to get Bernardos, et al. Expires September 1, 2018 [Page 22] Internet-Draft Network Virtualization Research Challenges February 2018 proper notifications within current REST APIs could be to consider the protocol proposed by the WEBPUSH WG [RFC8030]. Finally, in order to establish a proper order and allow the coexistence and collaboration of different systems, a common ontology regarding network and system virtualization should be defined and agreed, so different and heterogeneous systems can understand each other without requiring to rely on specific adaptation mechanisms that might break with any update on any side of the relation. 4.5.2. Extending Virtual Networks and Systems to the Internet of Things The Internet of Things (IoT) refers to the vision of connecting a multitude of automated devices (e.g. lights, environmental sensors, traffic lights, parking meters, health and security systems, etc.) to the Internet for purposes of reporting, and remote command and control of the device. This vision is being realized by a multi- pronged approach of standardization in various forums and complementary open source activities. For example, in the IETF, support of IoT web services has been defined by an HTTP-like protocol adapted for IoT called CoAP [RFC7252], and lately a group has been studying the need to develop a new network layer to support IP applications over Low Power Wide Area Networks (LPWAN). Elsewhere, for 5G cellular evolution there is much discussion on the need for supporting virtual "network slices" for the expected massive numbers of IoT devices. A separate virtual network slice is considered necessary for different 5G IoT use cases because devices will have very different characteristics than typical cellular devices like smart phones [ngmn_5G_whitepaper], and the number of IoT devices is expected to be at least one or two orders of magnitude higher than other 5G devices (see Section 4.5). The specific nature of the IoT ecosystem, particularly reflected in the Machine-to-Machine (M2M) communications, leads to the creation of new and highly distributed systems which demand location-based network and computing services. A specific example can be represented by a set of "things" that suddenly require to set-up a firewall to allow external entities to access their data while outsourcing some computation requirements to more powerful systems relying on cloud-based services. This representative use case exposes important requirements for both NFV and the underlying cloud infrastructures. In order to provide the aforementioned location-based functions integrated with highly distributed systems, the so called fog infrastructures should be able to instantiate VNFs, placing them in the required place, e.g. close to their consumers. This requirement Bernardos, et al. Expires September 1, 2018 [Page 23] Internet-Draft Network Virtualization Research Challenges February 2018 implies that the interfaces offered by virtualization platforms must support the specification of location-based resources, which is a key function in those scenarios. Moreover, those platforms must also be able to interpret and understand the references used by IoT systems to their location (e.g., "My-AP", "5BLDG+2F") and also the specification of identifiers linked to other resources, such as the case of requiring the infrastructure to establish a link between a specific AP and a specific virtual computing node. In summary, the research gap is exact localization of VNFs at far network edge infrastructure which is highly distributed and dynamic. 4.6. Service Composition Current network services deployed by operators often involve the composition of several individual functions (such as packet filtering, deep packet inspection, load balancing). These services are typically implemented by the ordered combination of a number of service functions that are deployed at different points within a network, not necessarily on the direct data path. This requires traffic to be steered through the required service functions, wherever they are deployed [RFC7498]. For a given service, the abstracted view of the required service functions and the order in which they are to be applied is called a Service Function Chain (SFC) [sfc_challenges], which is called Network Function Forwarding Graph (NF-FG) in ETSI. An SFC is instantiated through selection of specific service function instances on specific network nodes to form a service graph: this is called a Service Function Path (SFP). The service functions may be applied at any layer within the network protocol stack (network layer, transport layer, application layer, etc.). Service composition is a powerful means which can provide significant benefits when applied in a softwarized network environment. There are however many research challenges in this area, as for example the ones related to composition mechanisms and algorithms to enable load balancing and improve reliability. The service composition should also act as an enabler to gather information across all hierarchies (underlays and overlays) of network deployments which may span across multiple operators, for faster serviceability thus facilitating accomplishing aforementioned goals of "load balancing and improve reliability". As described in [dynamic_chaining], different algorithms can be used to enable dynamic service composition that optimizes a QoS-based utility function (e.g., minimizing the latency per-application traffic flows) for a given composition plan. Such algorithms can consider the computation capabilities and load status of resources Bernardos, et al. Expires September 1, 2018 [Page 24] Internet-Draft Network Virtualization Research Challenges February 2018 executing the VNF instances, either deduced through estimations from historical usage data or collected through real-time monitoring (i.e., context-aware selection). For this reason, selections should include references to dynamic information on the status of the service instance and its constituent elements, i.e., monitoring information related to individual VNF instances and links connecting them as well as derived monitoring information at the chain level (e.g., end-to-end delay). At runtime, if one or more VNF instances are no more available or QoS degrades below a given threshold, the service selection task can be rerun to perform service substitution. There are different research directions that relate to the previous point. For example, the use of Integer Linear Programming (ILP) techniques can be explored to optimize the management of diverse traffic flows. Deep machine learning can also be applied to optimize service chains using information parameters such as some of the ones mentioned above. Newer scheduling paradigms, like co-flows, can also be used. The SFC working group is working on an architecture for service function chaining [RFC7665] that includes the necessary protocols or protocol extensions to convey the Service Function Chain and Service Function Path information to nodes that are involved in the implementation of service functions and Service Function Chains, as well as mechanisms for steering traffic through service functions. In terms of actual work items, the SFC WG is has not yet considered working on the management and configuration of SFC components related to the support of Service Function Chaining. This part is of special interest for operators and would be required in order to actually put SFC mechanisms into operation. Similarly, redundancy and reliability mechanisms for service function chaining are currently not dealt with by any WG in the IETF. While this was the main goal of the VNFpool BoF efforts, it still remains unaddressed. 4.7. End-user device virtualization So far, most of the network softwarization efforts have focused on virtualizing functions of network elements. While virtualization of network elements started with the core, mobile networks architectures are now heavily switching to also virtualize radio access network (RAN) functions. The next natural step is to get virtualization down at the level of the end-user device (e.g., virtualizing a smartphone) [virtualization_mobile_device]. The cloning of a device in the cloud (central or local) bears attractive benefits to both the device and network operations alike (e.g., power saving at the device by offloading computational-heaving functions to the cloud, optimized networking -- both device-to-device and device-to-infrastructure) for Bernardos, et al. Expires September 1, 2018 [Page 25] Internet-Draft Network Virtualization Research Challenges February 2018 service delivery through tighter integration of the device (via its clone in the networking infrastructure). This is, for example, being explored by the European H2020 ICIRRUS project (www.icirrus- 5gnet.eu). 4.8. Security and Privacy Similar to any other situation where resources are shared, security and privacy are two important aspects that need to be taken into account. In the case of security, there are situations where multiple service providers will need to coexist in a virtual or hybrid physical/ virtual environment. This requires attestation procedures amongst different virtual/physical functions and resources, as well as ongoing external monitoring. Similarly, different network slices operating on the same infrastructure can present security problems, for instance if one slice running critical applications (e.g. support for a safety system) is affected by another slice running a less critical application. In general, the minimum common denominator for security measures on a shared system should be equal or higher than the one required by the most critical application. Multiple and continuous threat model analysis, as well as DevOps model are required to maintain a certain level of security in an NFV system. Simplistically, DevOps is a process that combines multiple functions into single cohesive teams in order to quickly produce quality software. It typically relies on also applying the Agile development process, which focuses on (among many things) dividing large features into multiple, smaller deliveries. One part of this is to immediately test the new smaller features in order to get immediate feedback on errors so that if present, they can be immediately fixed and redeployed. On the other hand, privacy refers to concerns about the control of personal data and the decision of what to reveal to whom. In this case, the storage, transmission, collection, and potential correlation of information in the NFV system, for purposes not originally intended or not known by the user, should be avoided. This is particularly challenging, as future intentions and threats cannot be easily predicted, and still can be applied on data collected in the past. Therefore, well-known techniques such as data minimization, using privacy features as default, and allowing users to opt in/out should be used to prevent potential privacy issues. Compared to traditional networks, NFV will result in networks that are much more dynamic (in function distribution and topology) and elastic (in size and boundaries). NFV will thus require network operators to evolve their operational and administrative security Bernardos, et al. Expires September 1, 2018 [Page 26] Internet-Draft Network Virtualization Research Challenges February 2018 solutions to work in this new environment. For example, in NFV the network orchestrator will become a key node to provide security policy orchestration across the different physical and virtual components of the virtualized network. For highly confidential data, for example, the network orchestrator should take into account if certain physical hardware (HW) of the network is considered more secure (e.g., because it is located in secure premises) than other HW. Traditional telecom networks typically run under a single administrative domain controlled by (exactly) one operator. With NFV, it is expected that in many cases, the telecom operator will now become a tenant (running the VNFs), and the infrastructure (NFVI) may be run by a different operator and/or cloud service provider (see also Section 4.4). Thus, there will be multiple administrative domains involved, making security policy coordination more complex. For example, who will be in charge of provisioning and maintaining security credentials such as public and private keys? Also, should private keys be allowed to be replicated across the NFV for redundancy reasons? Alternatively, it can be investigated how to develop a mechanism that avoid such a security policy coordination, this making the system more robust. On a positive note, NFV may better defense against Denial of Service (DoS) attacks because of the distributed nature of the network (i.e. no single point of failure) and the ability to steer (undesirable) traffic quickly [etsi_gs_nfv_sec_001]. Also, NFVs which have physical HW which is distributed across multiple data centers will also provide better fault isolation environments. This holds true in particular if each data center is protected separately via firewalls, DMZs and other network protection techniques. SDN can also be used to help improve security by facilitating the operation of existing protocols, such as Authentication, Authorization and Accounting (AAA). The management of AAA infrastructures, namely the management of AAA routing and the establishment of security associations between AAA entities, can be performed using SDN, as analyzed in [I-D.marin-sdnrg-sdn-aaa-mng]. 4.9. Separation of control concerns NFV environments offer two possible levels of SDN control. One level is the need for controlling the NFVI to provide connectivity end-to- end among VNFs or among VNFs and PNFs (Physical Network Functions). A second level is the control and configuration of the VNFs themselves (in other words, the configuration of the network service implemented by those VNFs), taking advantage of the programmability brought by SDN. Both control concerns are separated in nature. Bernardos, et al. Expires September 1, 2018 [Page 27] Internet-Draft Network Virtualization Research Challenges February 2018 However, interaction between both could be expected in order to optimize, scale or influence each other. Clear mechanisms for such interaction are needed in order to avoid malfunctioning or interference concerns. These ideas are considered in [etsi_gs_nfv_eve005] and [I-D.irtf-sdnrg-layered-sdn] 4.10. Network Function placement Network function placement is a problem in any kind of network telecommunications infrastructure. Moreover, the increased degree of freedom added by network virtualization makes this problem even more important, and also harder to tackle. Deciding where to place virtual network functions is a resource allocation problem which needs to (or may) take into consideration quite a few aspects: resiliency, (anti-)affinity, security, privacy, energy efficiency, etc. When several functions are chained (typical scenario), placement algorithms become more complex and important (as described in Section 4.6). While there has been research on the topic [nfv_piecing] [dynamic_placement][vnf-p], this still remains an open challenges that requires more attention. Multi-domain also adds another component of complexity to this problem that has to be considered. 4.11. Testing The impacts of network virtualization on testing can be divided into 3 groups: 1. Changes in methodology. 2. New functionality. 3. Opportunities. 4.11.1. Changes in methodology The largest impact of NFV is the ability to isolate the System Under Test (SUT). When testing Physical Network Functions (PNF), isolating the SUT means that all the other devices that the SUT communicates with are replaced with simulations (or controlled executions) in order to place the SUT under test by itself. The SUT may be comprised of one or more devices. The simulations use the appropriate traffic type and protocols in order to execute test cases. Bernardos, et al. Expires September 1, 2018 [Page 28] Internet-Draft Network Virtualization Research Challenges February 2018 As shown in Figure 2, NFV provides a common architecture for all functions to use. A VNF is executed using resources offered by the NFVI, which have been allocated using the MANO function. It is not possible to test a VNF by itself, without the entire supporting environment present. This fundamentally changes how to consider the SUT. In the case of a VNF (or multiple VNFs), the SUT is part of a larger architecture which is necessary in order to run the SUTs. Isolation of the SUT therefore becomes controlling the environment in a disciplined manner. The components of the environment necessary to run the SUTs that are not part of the SUT become the test environment. In the case of VNFs which are the SUT, the NFVI and MANO become the test environment. The configurations and policies that guide the test environment should remain constant during the execution of the tests, and also from test to test. Configurations such as CPU pinning, NUMA configuration, the SW versions and configurations of the hypervisor, vSwitch and NICs should remain constant. The only variables in the testing should be those controlling the SUT itself. If any configuration in the test environment is changed from test to test, the results become very difficult, if not impossible, to compare since the test environment behavior may change the results as a consequence of the configuration change. Testing the NFVI itself also presents new considerations. With a PNF, the dedicated hardware supporting it is optimized for the particular workload of the function. Routing hardware is specially built to support packet forwarding functions, while the hardware to support a purely control plane application (say, a DNS server, or a Diameter function) will not have this specialized capability. In NFV, the NFVI is required to support all types of potentially different workload types. Testing the NFVI therefore requires careful consideration about what types of metrics are sought. This, in turn, depends on the workload type the expected VNF will be. Examples of different workload types are data forwarding, control plane, encryption, and authentication. All these types of expected workloads will determine the types of metrics that should be sought. For example, if the workload is control plane, then a metric such as jitter is not useful, but dropped packets are critical. In a multi-tenant environment, the NFVI could support various types of workloads. In this case, testing with a variety of traffic types while measuring the corresponding metrics simultaneously becomes necessary. Test beds for any type of testing for an NFV-based system will be largely similar to previously used test architectures. The methods are impacted by virtualization, as described above, but the design of Bernardos, et al. Expires September 1, 2018 [Page 29] Internet-Draft Network Virtualization Research Challenges February 2018 test beds are similar as in the past. There are two main new considerations: o Since networking is based on software, which has lead to greater automation in deployment, the test system should also be deployable with the rest of the system in order to fully automate the system. This is especially relevant in a DevOps environment supported by a CI/CD tool chain (see Section 4.11.3 below). o In any performance test bed, the test system should not share the same resources as the System Under Test (SUT). While multi- tenenacy is a reality in virtualization, having the test system share resources with the SUT will impact the measured results in a performance test bed. The test system should be deployed on a separate platform in order to not to impact the resources available to the SUT. 4.11.2. New functionality NFV presents a collection of new functionality in order to support the goal of software networking. Each component on the architecture shown in Figure 2 has an associated set of functionality that allows VNFs to run: onboarding, lifecycle management for VNFs and Networks Services (NS), resource allocation, hypervisor functions, etc. One of the new capabilities enabled by NFV is VNFFG (VNF Forwarding Graphs). This refers to the graph that represents a Network Service by chaining together VNFs into a forwarding path. In practice, the forwarding path can be implemented in a variety of ways using different networking capabilities: vSwitch, SDN, SDN with a northbound application, and the VNFFG might use tunneling protocols like VXLAN. The dynamic allocation and implementation of these networking paths will have different performance characteristics depending on the methods used. The path implementation mechanism becomes a variable in the network testing of the NSs. The methodology used to test the various mechanisms should largely remain the same, and as usual, the test environment should remain constant for each of the tests, focusing on varying the path establishment method. Scaling refers to the change in allocation of resources to a VNF or NS. It happens dynamically at run-time, based on defined policies and triggers. The triggers can be network, compute or storage based. Scaling can allocate more resources in times of need, or reduce the amount of resources allocated when the demand is reduced. The SUT in this case becomes much larger than the VNF itself: MANO controls how scaling is done based on policies, and then allocates the resources Bernardos, et al. Expires September 1, 2018 [Page 30] Internet-Draft Network Virtualization Research Challenges February 2018 accordingly in the NFVI. Essentially, the testing of scaling includes the entire NFV architecture components into the SUT. 4.11.3. Opportunities Softwarization of networking functionality leads to softwarization of test as well. As Physical Network Functions (PNF) are being transformed into VNFs, so have the test tools. This leads to the fact that test tools are also being controlled and executed in the same environment as the VNFs are. This presents an opportunity to include VNF-based test tools along with the deployment of the VNFs supporting the services of the service provider into the host data centers. Tests can therefore be automatically executed upon deployment in the target environment, for each deployment, and each service. With PNFs, this was very difficult to achieve. This new concept helps to enable modern concepts like DevOps and Continuous Integration and Continuous Deployment in the NFV environment. The CI/CD pipeline supports this concept. It consists of a series of tools, among which immediate testing is an integral part, to deliver software from source to deployment. The ability to deploy the test tools themselves into the production environment stretches the CI/CD pipeline all the way to production deployment, allowing a range of tests to be executed. The tests can be simple, with a goal of verifying the correct deployment and networking establishment, but can also be more complex, like testing VNF functionality. 5. Technology Gaps and Potential IETF Efforts Table 1 correlates the open network virtualization research areas identified in this document to potential IETF and IRTF groups that could address some aspects of them. An example of a specific gap that the group could potentially address is identified in parenthetical beside the group name. Bernardos, et al. Expires September 1, 2018 [Page 31] Internet-Draft Network Virtualization Research Challenges February 2018 +-------------------------+-----------------------------------------+ | Open Research Area | Potential IETF/IRTF Group | +-------------------------+-----------------------------------------+ | 1-Guaranteeing QoS | IPPM WG (Measurements of NFVI) | | 2-Performance | SFC WG, NFVRG (energy driven | | improvement | orchestration) | | 3-Multiple Domains | NFVRG (multi-domain orchestration) | | 4-Network Slicing | NVO3 WG, NETSLICES bar BoF (multi- | | | tenancy support) | | 5-Service Composition | SFC WG (SFC Mgmt and Config) | | 6-End-user device | N/A | | virtualization | | | 7-Security | N/A | | 8-Separation of control | NFVRG (separation between transport | | concerns | control and services) | | 9-Testing | NFVRG (testing of scaling) | | 10-Function placement | NFVRG, SFC WG (VNF placement algorithms | | | and protocols) | +-------------------------+-----------------------------------------+ Table 1: Mapping of Open Research Areas to Potential IETF Groups 6. NFVRG focus areas Table 2 correlates the currently identified NFVRG topics of interests/focus areas to the open network virtualization research areas enumerated in this document. This can help the NFVRG in identifying and prioritizing research topics. The current list of NFVRG focus points is the following: o Re-architecting functions, including aspects such as new architectural and design patterns (e.g., containerization, statelessness, serverless, control/data plane separation), SDN integration, and proposals on programmability. o New management frameworks, considering aspects related to new OAM mechanisms (e.g., configuration control, hybrid descriptors) and lightweight MANO proposals. o Techniques to guarantee low latency, resource isolation, and other dataplane features, including hardware acceleration, functional offloading to dataplane elements (including NICs), and related approaches. o Measurement and benchmarking, addressing both internal measurements and external applications. Bernardos, et al. Expires September 1, 2018 [Page 32] Internet-Draft Network Virtualization Research Challenges February 2018 +----------------------------------------+-------------------------+ | NFVRG Focus Point | Open Research Area | +----------------------------------------+-------------------------+ | 1-Re-architecting functions | - Performance improvem. | | | - Network Slicing | | | - Guaranteeing QoS | | | - Security | | | - End-user device virt. | | | - Separation of control | | 2-New management frameworks | - Multiple Domains | | | - Service Composition | | | - End-user device virt. | | 3-Low latency, resource isolation, etc | - Performance improvem. | | | - Separation of control | | 4-Measurement and benchmarking | - Guaranteeing QoS | | | - Testing | +----------------------------------------+-------------------------+ Table 2: Mapping of NFVRG Focus Points to Open Research Areas 7. IANA Considerations N/A. 8. Security Considerations This is an informational document, which therefore does not introduce any security threat. Research challenges and gaps related to security and privacy have been included in Section 4.8. 9. Acknowledgments The authors want to thank Dirk von Hugo, Rafa Marin, Diego Lopez, Ramki Krishnan, Kostas Pentikousis, Rana Pratap Sircar, Alfred Morton, Nicolas Kuhn, Saumya Dikshit, Fabio Giust, Evangelos Haleplidis, Angeles Vazquez-Castro, Barbara Martini, Jose Saldana and Gino Carrozzo for their very useful reviews and comments to the document. Special thanks to Pedro Martinez-Julia, who provided text for the network slicing section. The authors want to also thank Dave Oran and Michael Welzl for their very detailed IRSG reviews. The work of Carlos J. Bernardos and Luis M. Contreras is partially supported by the H2020 5GEx (Grant Agreement no. 671636) and 5G- TRANSFORMER (Grant Agreement no. 761536) projects. Bernardos, et al. Expires September 1, 2018 [Page 33] Internet-Draft Network Virtualization Research Challenges February 2018 10. Informative References [dynamic_chaining] Martini, B. and F. Paganelli, "A Service-Oriented Approach for Dynamic Chaining of Virtual Network Functions over Multi-Provider Software-Defined Networks", Future Internet vol. 8, no. 2, June 2016. [dynamic_placement] Clayman, S., Maini, E., and A. Galis, "The dynamic placement of virtual network functions", 2014 IEEE Network Operations and Management Symposium (NOMS) pp. 1-9, May 2014. [etsi_gs_nfv_003] ETSI NFV ISG, "Network Functions Virtualisation (NFV); Terminology for Main Concepts in NFV", ETSI GS NFV 003 V1.2.1 NFV 003, December 2014, <http://www.etsi.org/deliver/etsi_gs/ NFV/001_099/003/01.02.01_60/gs_NFV003v010201p.pdf>. [etsi_gs_nfv_eve005] ETSI NFV ISG, "Network Functions Virtualisation (NFV); Ecosystem; Report on SDN Usage in NFV Architectural Framework", ETSI GS NFV-EVE 005 V1.1.1 NFV-EVE 005, December 2015, <http://www.etsi.org/deliver/etsi_gs/NFV- EVE/001_099/005/01.01.01_60/gs_NFV-EVE005v010101p.pdf>. [etsi_gs_nfv_per_001] ETSI GS NFV-PER 001 V1.1.2, "Network Functions Virtualisation (NFV); NFV Performance & Portability Best Practises", ETSI GS NFV-PER 001 V1.1.2 NFV-PER 001, December 2014, <http://www.etsi.org/deliver/etsi_gs/NFV- PER/001_099/001/01.01.02_60/gs_NFV-PER001v010102p.pdf>. [etsi_gs_nfv_sec_001] ETSI GS NFV-SEC 001 V1.1.1, "Network Functions Virtualisation (NFV); NFV Security; Problem Statement", ETSI GS NFV-SEC 001 V1.1.1 NFV-SEC 001, October 2014, <http://www.etsi.org/deliver/etsi_gs/NFV- SEC/001_099/001/01.01.01_60/gs_NFV-SEC001v010101p.pdf>. [etsi_nvf_whitepaper_3] "Network Functions Virtualisation (NFV). White Paper 3", October 2014. Bernardos, et al. Expires September 1, 2018 [Page 34] Internet-Draft Network Virtualization Research Challenges February 2018 [google_sdn_wan] Sushant Jain et al., "B4: experience with a globally- deployed Software Defined WAN", Proceedings of the ACM SIGCOMM 2013 , August 2013. [I-D.bagnulo-nfvrg-topology] Bagnulo, M. and D. Dolson, "NFVI PoP Network Topology: Problem Statement", draft-bagnulo-nfvrg-topology-01 (work in progress), March 2016. [I-D.bernardos-nfvrg-multidomain] Bernardos, C., Contreras, L., Vaishnavi, I., and R. Szabo, "Multi-domain Network Virtualization", draft-bernardos- nfvrg-multidomain-03 (work in progress), September 2017. [I-D.defoy-netslices-3gpp-network-slicing] Foy, X. and A. Rahman, "Network Slicing - 3GPP Use Case", draft-defoy-netslices-3gpp-network-slicing-02 (work in progress), October 2017. [I-D.gdmb-netslices-intro-and-ps] Galis, A., Dong, J., kiran.makhijani@huawei.com, k., Bryant, S., Boucadair, M., and P. Martinez-Julia, "Network Slicing - Introductory Document and Revised Problem Statement", draft-gdmb-netslices-intro-and-ps-02 (work in progress), February 2017. [I-D.geng-coms-problem-statement] 67, 4., Wang, L., Slawomir, S., Qiang, L., Matsushima, S., Galis, A., and L. Contreras, "Problem Statement of Supervised Heterogeneous Network Slicing", draft-geng- coms-problem-statement-01 (work in progress), October 2017. [I-D.irtf-sdnrg-layered-sdn] Contreras, L., Bernardos, C., Lopez, D., Boucadair, M., and P. Iovanna, "Cooperating Layered Architecture for SDN", draft-irtf-sdnrg-layered-sdn-01 (work in progress), October 2016. [I-D.marin-sdnrg-sdn-aaa-mng] Lopez, R. and G. Lopez-Millan, "Software-Defined Networking (SDN)-based AAA Infrastructures Management", draft-marin-sdnrg-sdn-aaa-mng-00 (work in progress), November 2015. Bernardos, et al. Expires September 1, 2018 [Page 35] Internet-Draft Network Virtualization Research Challenges February 2018 [I-D.mlk-nfvrg-nfv-reliability-using-cots] Mo, L. and B. Khasnabish, "NFV Reliability using COTS Hardware", draft-mlk-nfvrg-nfv-reliability-using-cots-01 (work in progress), October 2015. [I-D.natarajan-nfvrg-containers-for-nfv] natarajan.sriram@gmail.com, n., Krishnan, R., Ghanwani, A., Krishnaswamy, D., Willis, P., Chaudhary, A., and F. Huici, "An Analysis of Lightweight Virtualization Technologies for NFV", draft-natarajan-nfvrg-containers- for-nfv-03 (work in progress), July 2016. [I-D.rorosz-nfvrg-vbaas] Rosa, R., Rothenberg, C., and R. Szabo, "VNF Benchmark-as- a-Service", draft-rorosz-nfvrg-vbaas-00 (work in progress), October 2015. [intel_10_differences_nfv_cloud] Torre, P., "Discover the Top 10 Differences Between NFV and Cloud Environments", November 2015, <https://software.intel.com/en-us/videos/discover-the-top- 10-differences-between-nfv-and-cloud-environments>. [itu-t-y.3300] ITU-T, "Y.3300: Framework of software-defined networking", ITU-T Recommendation Y.3300 (06/14), June 2014, <http://www.itu.int/rec/T-REC-Y.3300-201406-I/en>. [itu-t-y.3301] ITU-T, "Y.3301: Functional requirements of software- defined networking", ITU-T Recommendation Y.3301 (09/16), September 2016, <http://www.itu.int/rec/T-REC-Y.3301-201609-I/en>. [itu-t-y.3302] ITU-T, "Y.3302: Functional architecture of software- defined networking", ITU-T Recommendation Y.3302 (01/17), January 2017, <http://www.itu.int/rec/T-REC-Y.3302-201701-I/en>. [multi-domain_5GEx] Bernardos, C., Geroe, B., Di Girolamo, M., Kern, A., Martini, B., and I. Vaishnavi, "5GEx: Realizing a Europe wide Multi-domain Framework for Software Defined Infrastructures", Transactions on Emerging Telecommunications Technologies vol. 27, no. 9, pp. 1271-1280, September 2016. Bernardos, et al. Expires September 1, 2018 [Page 36] Internet-Draft Network Virtualization Research Challenges February 2018 [nfv_piecing] Luizelli, M., Bays, L., and L. Buriol, "Piecing together the NFV provisioning puzzle: Efficient placement and chaining of virtual network functions", 2015 IFIP/IEEE International Symposium on Integrated Network Management (IM) pp. 98-106, May 2015. [nfv_sota_research_challenges] Mijumbi, R., Serrat, J., Gorricho, J-L., Bouten, N., De Turck, F., and R. Boutaba, "Network Function Virtualization: State-of-the-art and Research Challenges", IEEE Communications Surveys & Tutorials Volume: 18, Issue: 1, September 2015. [ngmn_5G_whitepaper] "NGMN 5G. White Paper", February 2015. [omniran] IEEE 802.1CF, "Recommended Practice for Network Reference Model and Functional Description of IEEE 802 Access Network", Draft 1.0 , December 2017. [onf_tr_521] ONF, "SDN Architecture, Issue 1.1", ONF TR-521 TR-521, February 2016, <https://www.opennetworking.org/images/stories/downloads/ sdn-resources/technical-reports/ TR-521_SDN_Architecture_issue_1.1.pdf>. [openmano_dataplane] Lopez, D., "OpenMANO: The Dataplane Ready Open Source NFV MANO Stack", March 2015, <https://www.ietf.org/proceedings/92/slides/ slides-92-nfvrg-7.pdf>. [RFC5810] Doria, A., Ed., Hadi Salim, J., Ed., Haas, R., Ed., Khosravi, H., Ed., Wang, W., Ed., Dong, L., Gopal, R., and J. Halpern, "Forwarding and Control Element Separation (ForCES) Protocol Specification", RFC 5810, DOI 10.17487/RFC5810, March 2010, <https://www.rfc-editor.org/info/rfc5810>. [RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed., and A. Bierman, Ed., "Network Configuration Protocol (NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011, <https://www.rfc-editor.org/info/rfc6241>. Bernardos, et al. Expires September 1, 2018 [Page 37] Internet-Draft Network Virtualization Research Challenges February 2018 [RFC7252] Shelby, Z., Hartke, K., and C. Bormann, "The Constrained Application Protocol (CoAP)", RFC 7252, DOI 10.17487/RFC7252, June 2014, <https://www.rfc-editor.org/info/rfc7252>. [RFC7426] Haleplidis, E., Ed., Pentikousis, K., Ed., Denazis, S., Hadi Salim, J., Meyer, D., and O. Koufopavlou, "Software- Defined Networking (SDN): Layers and Architecture Terminology", RFC 7426, DOI 10.17487/RFC7426, January 2015, <https://www.rfc-editor.org/info/rfc7426>. [RFC7498] Quinn, P., Ed. and T. Nadeau, Ed., "Problem Statement for Service Function Chaining", RFC 7498, DOI 10.17487/RFC7498, April 2015, <https://www.rfc-editor.org/info/rfc7498>. [RFC7665] Halpern, J., Ed. and C. Pignataro, Ed., "Service Function Chaining (SFC) Architecture", RFC 7665, DOI 10.17487/RFC7665, October 2015, <https://www.rfc-editor.org/info/rfc7665>. [RFC8030] Thomson, M., Damaggio, E., and B. Raymor, Ed., "Generic Event Delivery Using HTTP Push", RFC 8030, DOI 10.17487/RFC8030, December 2016, <https://www.rfc-editor.org/info/rfc8030>. [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, <https://www.rfc-editor.org/info/rfc8040>. [RFC8172] Morton, A., "Considerations for Benchmarking Virtual Network Functions and Their Infrastructure", RFC 8172, DOI 10.17487/RFC8172, July 2017, <https://www.rfc-editor.org/info/rfc8172>. [sfc_challenges] Medhat, A., Taleb, T., Elmangoush, A., Carella, G., Covaci, S., and T. Magedanz, "Service Function Chaining in Next Generation Networks: State of the Art and Research Challenges", IEEE Communications Magazine vol. 55, no. 2, pp. 216-223, February 2017. [virtualization_mobile_device] William D. Sproule, "Virtualization of Mobile Device User Experience", Patent US 9.542.062 B2 , January 2017. Bernardos, et al. Expires September 1, 2018 [Page 38] Internet-Draft Network Virtualization Research Challenges February 2018 [vnf-p] Moens, H. and Filip De Turck, "VNF-P: A model for efficient placement of virtualized network functions", 10th International Conference on Network and Service Management (CNSM) and Workshop pp. 418-423, 2014. [vnf_benchmarking] Rosa, R., Rothenberg, C., and R. Szabo, "A VNF Testing Framework Design, Implementation and Partial Results", November 2016, <https://www.ietf.org/proceedings/97/slides/ slides-97-nfvrg-06-vnf-benchmarking-00.pdf>. Authors' Addresses Carlos J. Bernardos Universidad Carlos III de Madrid Av. Universidad, 30 Leganes, Madrid 28911 Spain Phone: +34 91624 6236 Email: cjbc@it.uc3m.es URI: http://www.it.uc3m.es/cjbc/ Akbar Rahman InterDigital Communications, LLC 1000 Sherbrooke Street West, 10th floor Montreal, Quebec H3A 3G4 Canada Email: Akbar.Rahman@InterDigital.com URI: http://www.InterDigital.com/ Juan Carlos Zuniga SIGFOX 425 rue Jean Rostand Labege 31670 France Email: j.c.zuniga@ieee.org URI: http://www.sigfox.com/ Bernardos, et al. Expires September 1, 2018 [Page 39] Internet-Draft Network Virtualization Research Challenges February 2018 Luis M. Contreras Telefonica I+D Ronda de la Comunicacion, S/N Madrid 28050 Spain Email: luismiguel.contrerasmurillo@telefonica.com Pedro Aranda Universidad Carlos III de Madrid Av. Universidad, 30 Leganes, Madrid 28911 Spain Email: pedroandres.aranda@uc3m.es Pierre Lynch Ixia Email: plynch@ixiacom.com Bernardos, et al. Expires September 1, 2018 [Page 40]