Multi-domain Network Virtualization
draft-bernardos-nfvrg-multidomain-03

Versions: 00 01 02 03                                                   
NFV RG                                                     CJ. Bernardos
Internet-Draft                                                      UC3M
Intended status: Informational                             LM. Contreras
Expires: March 14, 2018                                              TID
                                                            I. Vaishnavi
                                                                  Huawei
                                                                R. Szabo
                                                                Ericsson
                                                      September 10, 2017


                  Multi-domain Network Virtualization
                  draft-bernardos-nfvrg-multidomain-03

Abstract

   This draft analyzes the problem of multi-provider multi-domain
   orchestration, by first scoping the problem, then looking into
   potential architectural approaches, and finally describing the
   solutions being developed by the European 5GEx project.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on March 14, 2018.

Copyright Notice

   Copyright (c) 2017 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect



Bernardos, et al.        Expires March 14, 2018                 [Page 1]


Internet-Draft     Multi-domain Network Virtualization    September 2017


   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Background: the ETSI NFV
       architecture  . . . . . . . . . . . . . . . . . . . . . . . .   4
   4.  Multidomain problem statement . . . . . . . . . . . . . . . .   6
   5.  Multi-domain architectural approaches . . . . . . . . . . . .   7
     5.1.  ETSI NFV approaches . . . . . . . . . . . . . . . . . . .   8
     5.2.  Hierarchical  . . . . . . . . . . . . . . . . . . . . . .  11
     5.3.  Cascading . . . . . . . . . . . . . . . . . . . . . . . .  13
   6.  Virtualization and Control for
       Multi-Provider Multi-Domain . . . . . . . . . . . . . . . . .  13
     6.1.  Interworking interfaces . . . . . . . . . . . . . . . . .  15
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  16
   8.  Security Considerations . . . . . . . . . . . . . . . . . . .  16
   9.  Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .  16
   10. Informative References  . . . . . . . . . . . . . . . . . . .  16
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  16

1.  Introduction

   The telecommunications sector is experiencing a major revolution that
   will shape the way networks and services are designed and deployed
   for the next decade.  We are witnessing an explosion in the number of
   applications and services demanded by users, which are now really
   capable of accessing them on the move.  In order to cope with such a
   demand, some network operators are looking at the cloud computing
   paradigm, which enables a potential reduction of the overall costs by
   outsourcing communication services from specific hardware in the
   operator's core to server farms scattered in datacenters.  These
   services have different characteristics if compared with conventional
   IT services that have to be taken into account in this cloudification
   process.  Also the transport network is affected in that it is
   evolving to a more sophisticated form of IP architecture with trends
   like separation of control and data plane traffic, and more fine-
   grained forwarding of packets (beyond looking at the destination IP
   address) in the network to fulfill new business and service goals.

   Virtualization of functions also provides operators with tools to
   deploy new services much faster, as compared to the traditional use
   of monolithic and tightly integrated dedicated machinery.  As a
   natural next step, mobile network operators need to re-think how to



Bernardos, et al.        Expires March 14, 2018                 [Page 2]


Internet-Draft     Multi-domain Network Virtualization    September 2017


   evolve their existing network infrastructures and how to deploy new
   ones to address the challenges posed by the increasing customers'
   demands, as well as by the huge competition among operators.  All
   these changes are triggering the need for a modification in the way
   operators and infrastructure providers operate their networks, as
   they need to significantly reduce the costs incurred in deploying a
   new service and operating it.  Some of the mechanisms that are being
   considered and already adopted by operators include: sharing of
   network infrastructure to reduce costs, virtualization of core
   servers running in data centers as a way of supporting their load-
   aware elastic dimensioning, and dynamic energy policies to reduce the
   monthly electricity bill.  However, this has proved to be tough to
   put in practice, and not enough.  Indeed, it is not easy to deploy
   new mechanisms in a running operational network due to the high
   dependency on proprietary (and sometime obscure) protocols and
   interfaces, which are complex to manage and often require configuring
   multiple devices in a decentralized way.

   Network Function Virtualization (NFV) and Software Defined Networking
   (SDN) are changing the way the telecommunications sector will deploy,
   extend and operate their networks.  TBD: add multi-domain.

2.  Terminology

   The following terms used in this document are defined by the ETSI NVF
   ISG, and the ONF and the IETF:

      NFV Infrastructure (NFVI): totality of all hardware and software
      components which build up the environment in which VNFs are
      deployed

      NFV Management and Orchestration (NFV-MANO): functions
      collectively provided by NFVO, VNFM, and VIM.

      NFV Orchestrator (NFVO): functional block that manages the Network
      Service (NS) lifecycle and coordinates the management of NS
      lifecycle, VNF lifecycle (supported by the VNFM) and NFVI
      resources (supported by the VIM) to ensure an optimized allocation
      of the necessary resources and connectivity.

      Network Service Orchestration (NSO): function responsible for the
      Network Service lifecycle management, including operations such
      as: On-board Network Service, Instantiate Network Service, Scale
      Network Service, Update Network Service, etc.

      OpenFlow protocol (OFP): allowing vendor independent programming
      of control functions in network nodes.




Bernardos, et al.        Expires March 14, 2018                 [Page 3]


Internet-Draft     Multi-domain Network Virtualization    September 2017


      Resource Orchestration (RO): subset of NFV Orchestrator functions
      that are responsible for global resource management governance.

      Service Function Chain (SFC): for a given service, the abstracted
      view of the required service functions and the order in which they
      are to be applied.  This is somehow equivalent to the Network
      Function Forwarding Graph (NF-FG) at ETSI.

      Service Function Path (SFP): the selection of specific service
      function instances on specific network nodes to form a service
      graph through which an SFC is instantiated.

      Virtualized Infrastructure Manager (VIM): functional block that is
      responsible for controlling and managing the NFVI compute, storage
      and network resources, usually within one operator's
      Infrastructure Domain.

      Virtualized Network Function (VNF): implementation of a Network
      Function that can be deployed on a Network Function Virtualization
      Infrastructure (NFVI).

      Virtualized Network Function Manager (VNFM): functional block that
      is responsible for the lifecycle management of VNF.

3.  Background: the ETSI NFV architecture

   The ETSI ISG NFV is a working group which, since 2012, aims to evolve
   quasi-standard IT virtualization technology to consolidate many
   network equipment types into industry standard high volume servers,
   switches, and storage.  It enables implementing network functions in
   software that can run on a range of industry standard server hardware
   and can be moved to, or loaded in, various locations in the network
   as required, without the need to install new equipment.  To date,
   ETSI NFV is by far the most accepted NFV reference framework and
   architectural footprint [etsi_nvf_whitepaper].  The ETSI NFV
   framework architecture framework is composed of three domains
   (Figure 1):

   o  Virtualized Network Function, running over the NFVI.

   o  NFV Infrastructure (NFVI), including the diversity of physical
      resources and how these can be virtualized.  NFVI supports the
      execution of the VNFs.

   o  NFV Management and Orchestration, which covers the orchestration
      and life-cycle management of physical and/or software resources
      that support the infrastructure virtualization, and the life-cycle
      management of VNFs.  NFV Management and Orchestration focuses on



Bernardos, et al.        Expires March 14, 2018                 [Page 4]


Internet-Draft     Multi-domain Network Virtualization    September 2017


      all virtualization specific management tasks necessary in the NFV
      framework.

   +-------------------------------------------+  +---------------+
   |   Virtualized Network Functions (VNFs)    |  |               |
   |  -------   -------   -------   -------    |  |               |
   |  |     |   |     |   |     |   |     |    |  |               |
   |  | VNF |   | VNF |   | VNF |   | VNF |    |  |               |
   |  |     |   |     |   |     |   |     |    |  |               |
   |  -------   -------   -------   -------    |  |               |
   +-------------------------------------------+  |               |
                                                  |               |
   +-------------------------------------------+  |               |
   |         NFV Infrastructure (NFVI)         |  |      NFV      |
   | -----------    -----------    ----------- |  |  Management   |
   | | Virtual |    | Virtual |    | Virtual | |  |      and      |
   | | Compute |    | Storage |    | Network | |  | Orchestration |
   | -----------    -----------    ----------- |  |               |
   | +---------------------------------------+ |  |               |
   | |         Virtualization Layer          | |  |               |
   | +---------------------------------------+ |  |               |
   | +---------------------------------------+ |  |               |
   | | -----------  -----------  ----------- | |  |               |
   | | | Compute |  | Storage |  | Network | | |  |               |
   | | -----------  -----------  ----------- | |  |               |
   | |          Hardware resources           | |  |               |
   | +---------------------------------------+ |  |               |
   +-------------------------------------------+  +---------------+

                       Figure 1: ETSI NFV framework

   The NFV architectural framework identifies functional blocks and the
   main reference points between such blocks.  Some of these are already
   present in current deployments, whilst others might be necessary
   additions in order to support the virtualization process and
   consequent operation.  The functional blocks are (Figure 2):

   o  Virtualized Network Function (VNF).

   o  Element Management (EM).

   o  NFV Infrastructure, including: Hardware and virtualized resources,
      and Virtualization Layer.

   o  Virtualized Infrastructure Manager(s) (VIM).

   o  NFV Orchestrator.




Bernardos, et al.        Expires March 14, 2018                 [Page 5]


Internet-Draft     Multi-domain Network Virtualization    September 2017


   o  VNF Manager(s).

   o  Service, VNF and Infrastructure Description.

   o  Operations and Business Support Systems (OSS/BSS).

                                                  +--------------------+
   +-------------------------------------------+  | ----------------   |
   |                 OSS/BSS                   |  | | NFV          |   |
   +-------------------------------------------+  | | Orchestrator +-- |
                                                  | ---+------------ | |
   +-------------------------------------------+  |    |             | |
   |  ---------     ---------     ---------    |  |    |             | |
   |  | EM 1  |     | EM 2  |     | EM 3  |    |  |    |             | |
   |  ----+----     ----+----     ----+----    |  | ---+----------   | |
   |      |             |             |        |--|-|    VNF     |   | |
   |  ----+----     ----+----     ----+----    |  | | manager(s) |   | |
   |  | VNF 1 |     | VNF 2 |     | VNF 3 |    |  | ---+----------   | |
   |  ----+----     ----+----     ----+----    |  |    |             | |
   +------|-------------|-------------|--------+  |    |             | |
          |             |             |           |    |             | |
   +------+-------------+-------------+--------+  |    |             | |
   |         NFV Infrastructure (NFVI)         |  |    |             | |
   | -----------    -----------    ----------- |  |    |             | |
   | | Virtual |    | Virtual |    | Virtual | |  |    |             | |
   | | Compute |    | Storage |    | Network | |  |    |             | |
   | -----------    -----------    ----------- |  | ---+------       | |
   | +---------------------------------------+ |  | |        |       | |
   | |         Virtualization Layer          | |--|-| VIM(s) +-------- |
   | +---------------------------------------+ |  | |        |         |
   | +---------------------------------------+ |  | ----------         |
   | | -----------  -----------  ----------- | |  |                    |
   | | | Compute |  | Storage |  | Network | | |  |                    |
   | | | hardware|  | hardware|  | hardware| | |  |                    |
   | | -----------  -----------  ----------- | |  |                    |
   | |          Hardware resources           | |  |  NFV Management    |
   | +---------------------------------------+ |  | and Orchestration  |
   +-------------------------------------------+  +--------------------+

                 Figure 2: ETSI NFV reference architecture

4.  Multidomain problem statement

   Market fragmentation results from having a multitude of
   telecommunications network and cloud operators each with a footprint
   focused to a specific region.  This makes it difficult to deploy cost
   effective infrastructure services, such as virtual connectivity or
   compute resources, spanning multiple countries as no single operator



Bernardos, et al.        Expires March 14, 2018                 [Page 6]


Internet-Draft     Multi-domain Network Virtualization    September 2017


   has a big enough footprint.  Even if operators largely aim to provide
   the same infrastructure services (VPN connectivity, compute resources
   based on virtual machines and block storage), inter-operator
   collaboration tools for providing a service spanning several
   administrative boundaries are very limited and cumbersome.  This
   makes service development and provisioning very time consuming.  For
   example, having a VPN with end-points in several countries, in order
   to connect multiple sites of a business (such as a hotel chain),
   requires contacting several network operators.  Such an approach is
   possible only with significant effort and integration work from the
   side of the business.  This is not only slow, but also inefficient
   and expensive, since the business also needs to employ networking
   specialists to do the integration instead of focusing on its core
   business

   Technology fragmentation also represents a major bottleneck
   internally for an operator.  Different networks and different parts
   of a network may be built as different domains using separate
   technologies, such as optical or packet switched (with different
   packet switching paradigms included); having equipment from different
   vendors; having different control paradigms, etc.  Managing and
   integrating these separate technology domains requires substantial
   amount of effort, expertise, and time.  The associated costs are paid
   by both network operators and vendors alike, who need to design
   equipment and develop complex integration features.  In addition to
   technology domains, there are other reasons for having multiple
   domains within an operator, such as, different geographies, different
   performance characteristics, scalability, policy or simply historic
   (e.g., result of a merge or an acquisition).  Multiple domains in a
   network are a necessary and permanent feature however, these should
   not be a roadblock towards service development and provisioning,
   which should be fast and efficient.

   A solution is needed to deal with both the multi-operator
   collaboration issue, and address the multi-domain problem within a
   single network operator.  While these two problems are quite
   different, they also share a lot of common aspects and can benefit
   from having a number of common tools to solve them.

5.  Multi-domain architectural approaches

   This section summarizes different architectural options that can be
   considered to tackle the multi-domain orchestration problem.








Bernardos, et al.        Expires March 14, 2018                 [Page 7]


Internet-Draft     Multi-domain Network Virtualization    September 2017


5.1.  ETSI NFV approaches

   Recently, the ETSI NFV ISG has started to look into viable
   architectural options supporting the placement of functions in
   different administrative domains.  In the document [etsi_nvf_ifa009],
   different approaches are considered, which we summarize next.

   The first option (shown in Figure 3) is based on a split of the NFVO
   into Network Service Orchestrator (NSO) and Resource Orchestrator
   (RO).  A use case that this separation could enable is the following:
   a network operator offering its infrastructure to different
   departments within the same operator, as well as to a different
   network operator like in cases of network sharing agreements.  In
   this scenario, an administrative domain can be defined as one or more
   data centers and VIMs, providing an abstracted view of the resources
   hosted in it.

   A service is orchestrated out of VNFs that can run on infrastructure
   provided and managed by another Service Provider.  The NSO manages
   the lifecycle of network services, while the RO provides an overall
   view of the resources present in the administrative domain to which
   it provides access and hides the interfaces of the VIMs present below
   it.




























Bernardos, et al.        Expires March 14, 2018                 [Page 8]


Internet-Draft     Multi-domain Network Virtualization    September 2017


                             -------
                             | NSO |
                            /-------\
                           /         \
               --------   /  -------- \    --------
               | VNFM |   |  | VNFM |  |   | VNFM |
               --------  /   --------   \  --------
                  / ____/      /  \      \____ \
                 / / _________/    \_________ \ \
                / / /                        \ \ \
   +-----------/-/-/---------+     +----------\-\-\----------+
   |        ---------        |     |        ---------        |
   |        |  RO   |        |     |        |  RO   |        |
   |        ---------        |     |        ---------        |
   |       /    |    \       |     |       /    |    \       |
   |      /     |     \      |     |      /     |     \      |
   |     /      |      \     |     |     /      |      \     |
   | ------- ------- ------- |     | ------- ------- ------- |
   | |VIM 1| |VIM 2| |VIM 3| |     | |VIM 1| |VIM 2| |VIM 3| |
   | ------- ------- ------- |     | ------- ------- ------- |
   | Administrative domain A |     | Administrative domain B |
   +-------------------------+     +-------------------------+

      Figure 3: Infrastructure provided using multiple administrative
                 domains (from ETSI GS NFV-IFA 009 V1.1.1)

   The second option (shown in Figure 4) is based on having an umbrella
   NFVO.  A use case enabled by this is the following: a Network
   Operator offers Network Services to different departments within the
   same operator, as well as to a different network operator like in
   cases of network sharing agreements.  In this scenario, an
   administrative domain is compose of one or more Datacentres, VIMs,
   VNFMs (together with their related VNFs) and NFVO, allowing distinct
   specific sets of network services to be hosted and offered on each.

   A top Network Service can include another Network Service.  A Network
   Service containing other Network Services might also contain VNFs.
   The NFVO in each admin domain provides visibility of the Network
   Services specific to this admin domain.  The umbrella NFVO is
   providing the lifecycle management of umbrella network services
   defined in this NFVO.  In each admin domain, the NFVO is providing
   standard NFVO functionalities, with a scope limited to the network
   services, VNFs and resources that are part of its admin domain.








Bernardos, et al.        Expires March 14, 2018                 [Page 9]


Internet-Draft     Multi-domain Network Virtualization    September 2017


                           ------------
                           | Umbrella |
                           |   NFVO   |
                           ------------
                              / |  \
                            /   |    \
                           / -------- \
                          /  | VNFM |  \
                        /    --------    \
                       /        |         \
                     /       -------        \
                    /        |VIM 1|         \
                  /          -------           \
   --------------/------------     -------------\-------------
   |        --------         |     |        --------         |
   |        | NFVO |         |     |        | NFVO |         |
   |        --------         |     |        --------         |
   |          | | |          |     |          | | |          |
   | -------- | | | -------- |     | -------- | | | -------- |
   | | VNFM | | | | | VNFM | |     | | VNFM | | | | | VNFM | |
   | -------- | | | -------- |     | -------- | | | -------- |
   |   |  \__/__|__\_/_   |  |     |   |  \__/__|__\_/_   |  |
   |   |  __/___|___/\ \  |  |     |   |  __/___|___/\ \  |  |
   |   | / /    |     \ \ |  |     |   | / /    |     \ \ |  |
   | ------- ------- ------- |     | ------- ------- ------- |
   | |VIM 1| |VIM 2| |VIM 3| |     | |VIM 1| |VIM 2| |VIM 3| |
   | ------- ------- ------- |     | ------- ------- ------- |
   | Administrative domain A |     | Administrative domain B |
   +-------------------------+     +-------------------------+

     Figure 4: Network services provided using multiple administrative
                 domains (from ETSI GS NFV-IFA 009 V1.1.1)

   More recently, ETSI NFV has released a new whitepaper, titled
   "Network Operator Perspectives on NFV priorities for 5G"
   [etsi_nvf_whitepaper_5g], which provides network operator
   perspectives on NFV priorities for 5G and identifies common technical
   features in terms of NFV.  This whitepaper identifies multi-site/
   multi-tenant orchestration as one key priority.  ETSI highlights the
   support of Infrastructure as a Service (IaaS), NFV as a Service
   (NFVaaS) and Network Service (NS) composition in different
   administrative domains (for example roaming scenarios in wireless
   networks) as critical for the 5G work.

   Related to this, a new Work Item, IFA028, and titled as "Report on
   architecture options to support multiple administrative domains" has
   been approved.




Bernardos, et al.        Expires March 14, 2018                [Page 10]


Internet-Draft     Multi-domain Network Virtualization    September 2017


5.2.  Hierarchical

   Considering the potential split of the NFVO into a Network Service
   Orchestrator (NSO) and a Resource Orchestrator (RO), multi-provider
   hierarchical interfaces may exist at their northbound APIs.  Figure 5
   illustrates the various interconnection options, namely:

      E/NSO (External NSO): an evolved NFVO northbound API based on
      Network Service (NS).

      E/RO (External RO): VNF-FG oriented resource embedding service.  A
      received VNF-FG that is mapped to the northbound resource view is
      embedded into the distributed resources collected from southbound,
      i.e., VNF-FG_in = VNF-FG_out_1 + VNF-FG_out_2 + ... + VNF-
      FG_out_N, where VNF-FG_out_j corresponds to a spatial embedding to
      subordinate domain "j".  For example, Provider 3's MP-NFVO/RO
      creates VNF-FG corresponding to its E/RO and E/VIM sub-domains.

      E/VIM (External VIM): a generic VIM interface offered to an
      external consumer.  In this case the NFVI-PoP may be shared for
      multiple consumers, each seeing a dedicated NFVI-PoP.  This
      corresponds to IaaS interface.

      I/NSO (Internal NSO): if a Multi-provider NSO (MP-NSO) is
      separated from the provider's operational NSO, e.g., due to
      different operational policies, the MP-NSO may need this interface
      to realize its northbound E/NSO requests.  Provider 1 illustrates
      a scenario the MP-NSO and the NSO are logically separated.
      Observe that Provider 1's tenants connect to the NSO and MP-NSO
      corresponds to "wholesale" services.

      I/RO (Internal RO): VNF-FG oriented resource embedding service.  A
      received VNF-FG that is mapped to the northbound resource view is
      embedded into the distributed resources collected from southbound,
      i.e., VNF-FG_in = VNF-FG_out_1 + VNF-FG_out_2 + ... + VNF-
      FG_out_N, where VNF-FG_out_j corresponds to a spatial embedding to
      subordinate domain "j".  For example, Provider 1's MP-NFVO/RO
      creates VNF-FG corresponding to its I/RO and I/VIM sub-domains.

      I/VIM (Internal VIM): a generic VIM interface at an NFVI-PoP.

      Nfvo-Vim: a generic VIM interface between a (monolithic) NFVO and
      a VIM.

   We would like to explore use-cases and potential benefits for the
   above multi-provider interfaces as well as to learn how much they may
   differ from their existing counterparts.  For example, are (E/RO, I/
   RO), (E/NSO, I/NSO), (E/VIM, I/VIM) pairs different?



Bernardos, et al.        Expires March 14, 2018                [Page 11]


Internet-Draft     Multi-domain Network Virtualization    September 2017


                                             Tenants
                             *    Provider      |
      *                       *   Domain 4   +--+-----------+
       *                       *             |MP-NFVO/NSO:  |
        *                       *            |Network Serv. |
         *   Provider            *           |Orchestrator  |
          *  Domain 3             *          +--+-----------+
           *              Tenants  *            |E/RO
            *                |      ************|*************
             *              ++-------------+    |
              *             |MP-NFVO/NSO:  |    |
      Provider *            |Network Serv. |    |
      Domain 1  *           |Orchestrator  |    |
                 *          +-+-----+------+    |
                  *      E/NSO|     | I/RO     /
                   *.---------'   +-+---------+--+
                   /*             |MP-NFVO/RO:   |
                  /  *            |Rersource     |
   Tenants       /    *           |Orchestrator  |
   |             |     *          +--+---+-------+
   | +-----------+--+   *************|***|********************
   | |MP-NFVO/NSO:  |                |  * \           Provider
   | |Network Serv. |         E/RO  /   *  \ E/VIM    Domain 2
   | |Orchestrator  |  .-----------'    *   `-------.
   | +-+------+-----+  |                *           |
   |   |I/NSO |I/RO    |                *           |
   |   |   +--+--------+--+             *           |
   |   |   |MP-NFVO/RO:   |             *           |
   |   |   |Rersource     |             *           |
    \  |   |Orchestrator  |             *    +------+-------+
     \ |   +----+---- --+-+             *    |VIM:          |
    +--+-----+  |I/RO   |I/VIM          *    |Virtualized   |
    |NFVO/NSO|  |       |               *    |Pys mapping   |
    +------+-+  |       |               *    +--------------+
       I/RO|    |       |               *
    +------+----+---+   |               *
    |    NFVO/RO    |   |               *
    ++-------------++   |               *
     |Nfvo-Vim     |    |               *
    ++-------+    ++----+--+            *
    |WIM|VIM ||   |VIM|WIM |            *
    +--------+|   +--------+            *
     +--------+                         *

         Figure 5: NSO-RO Split: possible multi-provider APIs - an
                               illustration





Bernardos, et al.        Expires March 14, 2018                [Page 12]


Internet-Draft     Multi-domain Network Virtualization    September 2017


5.3.  Cascading

   Cascading is an alternative way of relationship among providers, from
   the network service point of view.  In this case, service
   decomposition is implemented in a paired basis.  This can be extended
   in a recursive manner, then allowing for a concatenation of cascaded
   relations between providers.

   As a complement to this, from a service perspective, the cascading of
   two remote providers (i.e., providers not directly interconnected)
   could require the participation of a third provider (or more)
   facilitating the necessary communication among the other two.  In
   that sense, the final service involves two providers while the
   connectivity imposes the participation of more parties at resource
   level.

6.  Virtualization and Control for Multi-Provider Multi-Domain

   Orchestration operation in multi-domain is somewhat different from
   that in a single domain as the assumption in single domain single
   provider orchestration is that the orchestrator is aware of the
   entire topology and resource availability within its domain as well
   as has complete control over those resources.  This assumption of
   technical control cannot be made in a multi domain scenario,
   furthermore the assumption of the knowledge of the resources and
   topologies cannot be made across providers.  In such a scenario
   solutions are required that enable the exchange of relevant
   information across these orchestrators.  This exchange needs to be
   standardized as shown in Figure 6.

                   |                               |
                   + IF1                           +
              _____|____                       ____|_____
             |   Multi  |        IF2          |   Multi  |
             | Provider |<--------+---------->| Provider |
             |___Orch___|                     |___Orch___|
                  /\                               /\
                 /  \                             /  \
                /    \ IF3                       /    \
        _______/__   _\_________        ________/_    _\________
       |  Domain  | |  Domain  |       |  Domain  |  |  Domain  |
       |___Orch___| |___Orch___|       |___Orch___|  |___Orch___|

       Figure 6: Multi Domain Multi Provider reference architecture

   The figure shows the Multi Provider orchestrator exposing an
   interface 1 (IF1) to the tenant, interface 2 (IF2) to other Multi
   Provider Orchestrator (MPO) and an interface 3 (IF3) to individual



Bernardos, et al.        Expires March 14, 2018                [Page 13]


Internet-Draft     Multi-domain Network Virtualization    September 2017


   domain orchestratrators.  Each one of these interfaces could be a
   possible standardization candidate.  Interface 1 is exposed to the
   tennnt who could request his specific services and/or slices to be
   deployed.  Interface 2 is between the orchestrator and is a key
   interface to enable multi-provider operation.  Interface 3 focuses on
   abstracting the technology or vendor dependent implementation details
   to support orchestration.

   The proposed operation of the MPO follows three main technical steps.
   First, over interface 2 various functions such as abstracted topology
   discovery, pricing and service details are detected.  Second, once a
   request for deploying a service is received over interface 1 the
   Multi Provider Orchestrator evaluates the best orchestrators to
   implement parts of this request.  The request to deploy these parts
   are sent to the different domain orchestrators over IF2 and IF3 and
   the acknowledgement that these are deployed in different domain are
   received back over those interfaces.  Third, on receipt of the
   acknowledgement the slice specific assurance management is started
   within the MPO.  This assurance function collects the appropriate
   information over IF2 and IF3 and reports the performance back to the
   tenant over IF1.  The assurance is also responsible for detecting any
   failures in the service and violations in the SLA and recomending to
   the orchestration engine the reconfiguration of the service or slice
   which again needs to performed over IF2 and IF3.

   Each of the three steps is assigned to a specific block in our high
   level architecture shown in Figure 7.

                   |                                    |
                   + IF1                                +
     ______________|______________                  ____|_____
    |      Multi Provider Orch    |                |  Multi   |
    | ______   ________   _______ |<------+------->| Provider |
    ||Assur-| |        | | Catal-||      IF2       |___Orch___|
    ||-ance | |  NFVO  | | logue ||
    || Mgmt.| |        | | Topo. ||
    ||______| |________| |_Mgmt._||
    |_____________________________|
                  /\
                 /  \ IF3

               Figure 7: Detailed MPO reference architecture

   The catalogue and topology management system is responsible for step
   1.  It discovers the service as well as the resources exposed by the
   other domains both on IF2 and IF3.  The combination of these services
   with coverage over the detected topology is provided to the user over
   IF1.  In turn the catalogue and topology management system is also



Bernardos, et al.        Expires March 14, 2018                [Page 14]


Internet-Draft     Multi-domain Network Virtualization    September 2017


   responsible for exposing the topology and service deployment
   capabilities to the other domain.  The exposure over interface 2 to
   other MPO maybe abstracted and the mapping of this abstracted view to
   the real view when requested byt he NFVO.

   The NFVO (Network Function Virtualization Orchestrator) is
   responsible for the second step.  It deploys the service or slice as
   is received from the tenant over IF2 and IF3.  It then hands over the
   deployment decisions to the Assurance managmeent subsystem which use
   this information to collect the periodic monitoring tickets in step
   3.  On the other end it is responsible for receiving the request over
   IF2 to deploy a part of the service, consult with the catalogue and
   topology management system on the translation of the abstraction to
   the received request and then for the actual deployment over the
   domains using IF3.  The result of this deployment and the management
   and control handles to access the deployed slice or service is then
   returned to the requesting MPO.

   The assurance management component periodically studies the collected
   results to report the overall service performance to the tenant or
   the requesting MPO as well as to ensure that the service is
   functioning within the specified parameters.  In case of failures or
   violations the Assurance management system recomends reconfigurations
   to the NFVO.

6.1.  Interworking interfaces

   In this section we provide more details on the interworking
   interfaces of the MPO reference architecture.  Each interface IF1,
   IF2 and IF3 is broken down into several sub-interfaces.  Each of them
   has a clear scope and functionality.  [Ed.  note: more details will
   be added in future releases of this document]

   For multi provider Network Service orchestration, the Multi-domain
   Orchestrator (MdO) offers Network Services by exposing an OSS/BSS -
   NFVO interface to other MPOs belonging to other providers.  For
   multi-provider resource orchestration, the MPO presents a VIM-like
   view and exposes an extended NFVO - VIM interface to other MPOs.  The
   MPO exposes a northbound sub-interface (IF1-S) through which an MPO
   customer sends the initial request for services.  It handles command
   and control functions to instantiate network services.  Such
   functions include requesting the instantiation and interconnection of
   Network Functions (NFs).  A sub-interface IF2-S is defined to perform
   similar operations between MPOs of different administrative domains.
   A set of sub-interfaces -- IF3-R and IF2-R -- are used to keep an
   updated global view of the underlying infrastructure topology exposed
   by domain orchestrators.  The service catalogue exposes available
   services to customers on a sub-interface IF1-C and to other MPO



Bernardos, et al.        Expires March 14, 2018                [Page 15]


Internet-Draft     Multi-domain Network Virtualization    September 2017


   service operators on sub-interface IF2-C.  Resource orchestration
   related interfaces are broken up to IF2-RC, IF2-RT, IF2-RMon to
   reflect resource control, resource topology and resource monitoring
   respectively.  Furthermore, the sub-interfaces introduced before are
   generalised and also used for interfaces IF3 and IF1.

7.  IANA Considerations

   N/A.

8.  Security Considerations

   TBD.

9.  Acknowledgments

   This work is supported by 5G-PPP 5GEx, an innovation action project
   partially funded by the European Community under the H2020 Program
   (grant agreement no.  671636).  The views expressed here are those of
   the authors only.  The European Commission is not liable for any use
   that may be made of the information in this presentation.

10.  Informative References

   [etsi_nvf_ifa009]
              "Report on Architectural Options, ETSI GS NFV-IFA 009
              V1.1.1", July 2016.

   [etsi_nvf_whitepaper]
              "Network Functions Virtualisation (NFV). White Paper 2",
              October 2014.

   [etsi_nvf_whitepaper_5g]
              "Network Functions Virtualisation (NFV). White Paper on
              "Network Operator Perspectives on NFV priorities for 5G"",
              February 2017.

Authors' Addresses

   Carlos J. Bernardos
   Universidad Carlos III de Madrid
   Av. Universidad, 30
   Leganes, Madrid  28911
   Spain

   Phone: +34 91624 6236
   Email: cjbc@it.uc3m.es
   URI:   http://www.it.uc3m.es/cjbc/



Bernardos, et al.        Expires March 14, 2018                [Page 16]


Internet-Draft     Multi-domain Network Virtualization    September 2017


   Luis M. Contreras
   Telefonica I+D
   Ronda de la Comunicacion, S/N
   Madrid  28050
   Spain

   Email: luismiguel.conterasmurillo@telefonica.com


   Ishan Vaishnavi
   Huawei Technologies Dusseldorf GmBH
   Riesstrasse 25,
   Munich  80992
   Germany

   Email: Ishan.vaishnavi@huawei.com


   Robert Szabo
   Ericsson
   Konyves Kaman krt. 11
   Budapest, EMEA  1097
   Hungary

   Phone: +36703135738
   Email: robert.szabo@ericsson.com

























Bernardos, et al.        Expires March 14, 2018                [Page 17]