Skip to main content

Towards recursive virtualization and programming for network and cloud resources
draft-unify-nfvrg-recursive-programming-01

The information below is for an old version of the document.
Document Type
This is an older version of an Internet-Draft whose latest revision state is "Replaced".
Authors Robert Szabo , Zu Qiang , Mario Kind
Last updated 2015-07-06
Replaced by draft-irtf-nfvrg-unify-recursive-programming
RFC stream (None)
Formats
Stream Stream state (No stream defined)
Consensus boilerplate Unknown
RFC Editor Note (None)
IESG IESG state I-D Exists
Telechat date (None)
Responsible AD (None)
Send notices to (None)
draft-unify-nfvrg-recursive-programming-01
NFVRG                                                           R. Szabo
Internet-Draft                                                  Z. Qiang
Intended status: Informational                                  Ericsson
Expires: January 7, 2016                                         M. Kind
                                                     Deutsche Telekom AG
                                                            July 6, 2015

 Towards recursive virtualization and programming for network and cloud
                               resources
               draft-unify-nfvrg-recursive-programming-01

Abstract

   The introduction of Network Function Virtualization (NFV) in carrier-
   grade networks promises improved operations in terms of flexibility,
   efficiency, and manageability.  NFV is an approach to combine network
   and compute virtualizations together.  However, network and compute
   resource domains expose different virtualizations and programmable
   interfaces.  In [I-D.unify-nfvrg-challenges] we argued for a joint
   compute and network virtualization by looking into different compute
   abstractions.

   In this document we analyze different approaches to orchestrate a
   service graph with transparent network functions into a commodity
   data center.  We show that a recursive compute and network joint
   virtualization and programming has clear advantages compared to other
   approaches with separated control between compute and network
   resources.  The discussion of the problems and the proposed solution
   is generic for any data center use case; however, we use NFV as an
   example.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on January 7, 2016.

Szabo, et al.            Expires January 7, 2016                [Page 1]
Internet-Draft        Toward recursive programming             July 2015

Copyright Notice

   Copyright (c) 2015 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Terms and Definitions . . . . . . . . . . . . . . . . . . . .   3
   3.  Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . .   4
   3.1.  Black Box DC  . . . . . . . . . . . . . . . . . . . . . . .   4
   3.1.1.  Black Box DC with L3 tunnels  . . . . . . . . . . . . . .   4
   3.1.2.  Black Box DC with external steering . . . . . . . . . . .   6
   3.2.  White Box DC  . . . . . . . . . . . . . . . . . . . . . . .   8
   4.  Recursive approach  . . . . . . . . . . . . . . . . . . . . .   9
   4.1.  Virtualization  . . . . . . . . . . . . . . . . . . . . . .  11
   4.1.1.  The virtualizer's data model  . . . . . . . . . . . . . .  13
   5.  Examples  . . . . . . . . . . . . . . . . . . . . . . . . . .  20
   5.1.  Infrastructure reports  . . . . . . . . . . . . . . . . . .  20
   5.2.  Simple requests . . . . . . . . . . . . . . . . . . . . . .  24
   6.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  27
   7.  Security Considerations . . . . . . . . . . . . . . . . . . .  27
   8.  Acknowledgement . . . . . . . . . . . . . . . . . . . . . . .  27
   9.  Informative References  . . . . . . . . . . . . . . . . . . .  27
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  28

1.  Introduction

   To a large degree there is agreement in the research community that
   rigid network control limits the flexibility of service creation.  In
   [I-D.unify-nfvrg-challenges]

   o  we analyzed different compute domain abstractions to argue that
      joint compute and network virtualization and programming is needed
      for efficient combination of these resource domains;

Szabo, et al.            Expires January 7, 2016                [Page 2]
Internet-Draft        Toward recursive programming             July 2015

   o  we described challenges associated with the combined handling of
      compute and network resources for a unified production
      environment.

   Our goal here is to analyze different approaches to instantiate a
   service graph with transparent network functions into a commodity
   Data Center (DC).  More specifically, we analyze

   o  two black box DC set-ups, where the intra-DC network control is
      limited to some generic compute only control programming
      interface;

   o  a white box DC set-up, where the intra-DC network control is
      exposed directly to for a DC external control to coordinate
      forwarding configurations;

   o  a recursive approach, which illustrates potential benefits of a
      joint compute and network virtualization and control.

   The discussion of the problems and the proposed solution is generic
   for any data center use case; however, we use NFV as an example.

2.  Terms and Definitions

   We use the terms compute and "compute and storage" interchangeably
   throughout the document.  Moreover, we use the following definitions,
   as established in [ETSI-NFV-Arch]:

   NFV:  Network Function Virtualization - The principle of separating
      network functions from the hardware they run on by using virtual
      hardware abstraction.

   NFVI:  NFV Infrastructure - Any combination of virtualized compute,
      storage and network resources.

   VNF:  Virtualized Network Function - a software-based network
      function.

   MANO:  Management and Orchestration - In the ETSI NFV framework
      [ETSI-NFV-MANO], this is the global entity responsible for
      management and orchestration of NFV lifecycle.

   Further, we make use of the following terms:

   NF:  a network function, either software-based (VNF) or appliance-
      based.

Szabo, et al.            Expires January 7, 2016                [Page 3]
Internet-Draft        Toward recursive programming             July 2015

   SW:  a (routing/switching) network element with a programmable
      control plane interface.

   DC:  a data center is an interconnection of Compute Nodes (see below)
      with a data center controller, which offers programmatic resource
      control interface to its clients.

   CN:  a server, which is controlled by a DC control plane and provides
      execution environment for virtual machine (VM) images such as
      VNFs.

3.  Use Cases

   Service Function Chaining (SFC) looks into the problem how to deliver
   end-to-end services through the chain of network functions (NFs).
   Many of such NFs are envisioned to be transparent to the client,
   i.e., they intercept the client connection for adding value to the
   services without the knowledge of the client.  However, deploying
   network function chains in DCs with Virtualized Network Functions
   (VNFs) are far from trivial [I-D.ietf-sfc-dc-use-cases].  For
   example, different exposures of the internals of the DC will imply
   different dynamisms in operations, different orchestration
   complexities and may yield for different business cases with regards
   to infrastructure sharing.

   We investigate different scenarios with a simple NF forwarding graph
   of three VNFs (o->VNF1->VNF2->VNF3->o), where all VNFs are deployed
   within the same DC.  We assume that the DC is a multi-tier leaf and
   spine (CLOS) and that all VNFs of the forwarding graph are bump-in-
   the-wire NFs, i.e., the client cannot explicitly access them.

3.1.  Black Box DC

   In Black Bock DC set-ups, we assume that the compute domain is an
   autonomous domain with legacy (e.g., OpenStack) orchestration APIs.
   Due to the lack of direct forwarding control within the DC, no native
   L2 forwarding can be used to insert VNFs running in the DC into the
   forwarding graph.  Instead, explicit tunnels (e.g., VxLAN) must be
   used, which need termination support within the deployed VNFs.
   Therefore, VNFs must be aware of the previous and the next hops of
   the forwarding graph to receive and forward packets accordingly.

3.1.1.  Black Box DC with L3 tunnels

   Figure 1 illustrates a set-up where an external VxLAN termination
   point in the SDN domain is used to forward packets to the first NF
   (VNF1) of the chain within the DC.  VNF1, in turn, is configured to

Szabo, et al.            Expires January 7, 2016                [Page 4]
Internet-Draft        Toward recursive programming             July 2015

   forward packets to the next SF (VNF2) in the chain and so forth with
   VNF2 and VNF3.

   In this set-up VNFs must be capable of handling L3 tunnels (e.g.,
   VxLAN) and must act as forwarders themselves.  Additionally, an
   operational L3 underlay must be present so that VNFs can address each
   other.

   Furthermore, VNFs holding chain forwarding information could be
   untrusted user plane functions from 3rd party developers.
   Enforcement of proper forwarding is problematic.

   Additionally, compute only orchestration might result in sub-optimal
   allocation of the VNFs with regards to the forwarding overlay, for
   example, see back-forth use of a core switch in Figure 1.

   In [I-D.unify-nfvrg-challenges] we also pointed out that within a
   single Compute Node (CN) similar VNF placement and overlay
   optimization problem may reappear in the context of network interface
   cards and CPU cores.

Szabo, et al.            Expires January 7, 2016                [Page 5]
Internet-Draft        Toward recursive programming             July 2015

                                   |                         A     A
                                 +---+                       | S   |
                                 |SW1|                       | D   |
                                 +---+                       | N   | P
                                /     \                      V     | H
                               /       \                           | Y
                              |         |                    A     | S
                            +---+      +-+-+                 |     | I
                            |SW |      |SW |                 |     | C
                           ,+--++.._  _+-+-+                 |     | A
                        ,-"   _|,,`.""-..+                   | C   | L
                      _,,,--"" |    `.   |""-.._             | L   |
                 +---+      +--++     `+-+-+    ""+---+      | O   |
                 |SW |      |SW |      |SW |      |SW |      | U   |
                 +---+    ,'+---+    ,'+---+    ,'+---+      | D   |
                 |   | ,-"  |   | ,-"  |   | ,-"  |   |      |     |
               +--+ +--+  +--+ +--+  +--+ +--+  +--+ +--+    |     |
               |CN| |CN|  |CN| |CN|  |CN| |CN|  |CN| |CN|    |     |
               +--+ +--+  +--+ +--+  +--+ +--+  +--+ +--+    V     V
                 |          |                          |
                +-+        +-+                        +-+          A
                |V|        |V|                        |V|          | L
                |N|        |N|                        |N|          | O
                |F|        |F|                        |F|          | G
                |1|        |3|                        |2|          | I
                +-+        +-+                        +-+          | C
     +---+ --1>-+ |        | +--<3---------------<3---+ |          | A
     |SW1|        +-2>-----------------------------2>---+          | L
     +---+ <4--------------+                                       V

         <<=============================================>>
                IP tunnels, e.g., VxLAN

             Figure 1: Black Box Data Center with VNF Overlay

3.1.2.  Black Box DC with external steering

   Figure 2 illustrates a set-up where an external VxLAN termination
   point in the SDN domain is used to forward packets among all the SFs
   (VNF1-VNF3) of the chain within the DC.  VNFs in the DC need to be
   configured to receive and send packets between only the SDN endpoint,
   hence are not aware of the next hop VNF address.  Shall any VNFs need
   to be relocated, e.g., due to scale in/out as described in
   [I-D.zu-nfvrg-elasticity-vnf], the forwarding overlay can be
   transparently re-configured at the SDN domain.

Szabo, et al.            Expires January 7, 2016                [Page 6]
Internet-Draft        Toward recursive programming             July 2015

   Note however, that traffic between the DC internal SFs (VNF1, VNF2,
   VNF3) need to exit and re-enter the DC through the external SDN
   switch.  This, certainly, is sub-optimal an results in ping-pong
   traffic similar to the local and remote DC case discussed in
   [I-D.zu-nfvrg-elasticity-vnf].

                                   |                         A     A
                                 +---+                       | S   |
                                 |SW1|                       | D   |
                                 +---+                       | N   | P
                                /     \                      V     | H
                               /       \                           | Y
                              |         |   ext port         A     | S
                            +---+      +-+-+                 |     | I
                            |SW |      |SW |                 |     | C
                           ,+--++.._  _+-+-+                 |     | A
                        ,-"   _|,,`.""-..+                   | C   | L
                      _,,,--"" |    `.   |""-.._             | L   |
                 +---+      +--++     `+-+-+    ""+---+      | O   |
                 |SW |      |SW |      |SW |      |SW |      | U   |
                 +---+    ,'+---+    ,'+---+    ,'+---+      | D   |
                 |   | ,-"  |   | ,-"  |   | ,-"  |   |      |     |
               +--+ +--+  +--+ +--+  +--+ +--+  +--+ +--+    |     |
               |CN| |CN|  |CN| |CN|  |CN| |CN|  |CN| |CN|    |     |
               +--+ +--+  +--+ +--+  +--+ +--+  +--+ +--+    V     V
                 |          |                          |
                +-+        +-+                        +-+          A
                |V|        |V|                        |V|          | L
                |N|        |N|                        |N|          | O
                |F|        |F|                        |F|          | G
                |1|        |3|                        |2|          | I
                +-+        +-+                        +-+          | C
     +---+ --1>-+ |        | |                        | |          | A
     |SW1| <2-----+        | |                        | |          | L
     |   | --3>---------------------------------------+ |          |
     |   | <4-------------------------------------------+          |
     |   | --5>------------+ |                                     |
     +---+ <6----------------+                                     V

          <<=============================================>>
                          IP tunnels, e.g., VxLAN

             Figure 2: Black Box Data Center with ext Overlay

Szabo, et al.            Expires January 7, 2016                [Page 7]
Internet-Draft        Toward recursive programming             July 2015

3.2.  White Box DC

   Figure 3 illustrates a set-up where the internal network of the DC is
   exposed in full details through an SDN Controller for steering
   control.  We assume that native L2 forwarding can be applied all
   through the DC until the VNFs' port, hence IP tunneling and tunnel
   termination at the VNFs are not needed.  Therefore, VNFs need not be
   forwarding graph aware but transparently receive and forward packets.
   However, the implications are that the network control of the DC must
   be handed over to an external forwarding controller (see that the SDN
   domain and the DC domain overlaps in Figure 3).  This most probably
   prohibits clear operational separation or separate ownerships of the
   two domains.

Szabo, et al.            Expires January 7, 2016                [Page 8]
Internet-Draft        Toward recursive programming             July 2015

                                   |                     A         A
                                 +---+                   | S       |
                                 |SW1|                   | D       |
                                 +---+                   | N       | P
                                /     \                  |         | H
                               /       \                 |         | Y
                              |         |   ext port     |   A     | S
                            +---+      +-+-+             |   |     | I
                            |SW |      |SW |             |   |     | C
                           ,+--++.._  _+-+-+             |   |     | A
                        ,-"   _|,,`.""-..+               |   | C   | L
                      _,,,--"" |    `.   |""-.._         |   | L   |
                 +---+      +--++     `+-+-+    ""+---+  |   | O   |
                 |SW |      |SW |      |SW |      |SW |  |   | U   |
                 +---+    ,'+---+    ,'+---+    ,'+---+  V   | D   |
                 |   | ,-"  |   | ,-"  |   | ,-"  |   |      |     |
               +--+ +--+  +--+ +--+  +--+ +--+  +--+ +--+    |     |
               |CN| |CN|  |CN| |CN|  |CN| |CN|  |CN| |CN|    |     |
               +--+ +--+  +--+ +--+  +--+ +--+  +--+ +--+    V     V
                 |          |                          |
                +-+        +-+                        +-+          A
                |V|        |V|                        |V|          | L
                |N|        |N|                        |N|          | O
                |F|        |F|                        |F|          | G
                |1|        |3|                        |2|          | I
                +-+        +-+                        +-+          | C
     +---+ --1>-+ |        | +--<3---------------<3---+ |          | A
     |SW1|        +-2>-----------------------------2>---+          | L
     +---+ <4--------------+                                       V

         <<=============================================>>
                       L2 overlay

              Figure 3: White Box Data Center with L2 Overlay

4.  Recursive approach

   We argued in [I-D.unify-nfvrg-challenges] for a joint software and
   network programming interface.  Consider that such joint software and
   network abstraction (virtualization) exists around the DC with a
   corresponding resource programmatic interface.  A software and
   network programming interface could include VNF requests and the
   definition of the corresponding network overlay.  However, such
   programming interface is similar to the top level services
   definition, for example, by the means of a VNF Forwarding Graph.

Szabo, et al.            Expires January 7, 2016                [Page 9]
Internet-Draft        Toward recursive programming             July 2015

   Figure 4 illustrates a joint domain virtualization and programming
   setup.  In Figure 4 "[x]" denotes ports of the virtualized data plane
   while "x" denotes port created dynamically as part of the VNF
   deployment request.  Over the joint software and network
   virtualization VNF placement and the corresponding traffic steering
   could be defined in an atomic, which is orchestrated, split and
   handled to the next levels (see Figure 5) in the hierarchy for
   further orchestration.  Such setup allows clear operational
   separation, arbitrary domain virtualization (e.g., topology details
   could be omitted) and constraint based optimization of domain wide
   resources.

                                   |
          +-----------------------[x]--------------------+  A
          |Domain 0                |                     |  |O
          |              +--------[x]----------+         |  |V
          |              |        / \          |         |  |E
          |Big Switch    |   -<---   --->--    |         |  |R
          |with          |  /    BiS-BiS   \   |         |  |A
          |Big Software  |  | +-->-+ +-->-+ |  |         |  |R
          |(BiS-BiS)     |  | |    | |    | |  |         |  |C
          |              +--x-x----x-x----x-x--+         |  |H
          |                 | |    | |    | |            |  |I
          |                 +-+    +-+    +-+            |  |N
          |                 |V|    |V|    |V|            |  |G V
          |                 |N|    |N|    |N|            |  |  N
          |                 |F|    |F|    |F|            |  |  F
          |                 |1|    |2|    |3|            |  |
          |                 +-+    +-+    +-+            |  |  F
          |                                              |  |  G
          +----------------------------------------------+  V

        Figure 4: Recursive Domain Virtualization and Joint VNF FG
                       programming: Overarching View

Szabo, et al.            Expires January 7, 2016               [Page 10]
Internet-Draft        Toward recursive programming             July 2015

       +-------------------------|-----------------------------+ A
       | +----------------------[x]---------------------+  AV  | |
       | | Domain 1             /  \                    |  |N  | |
       | |                     |    A                   |  |F  | |
       | | Big Switch (BS)     |    |                   |  |   | |O
       | |                     V    |                   |  |F  | |V
       | |                    /      \                  |  |G  | |E
       | +-----------------[x]--------[x]---------------+  V1  | |R
       |                    |          |                       | |A
       | +------------------|----------|----------------+  A   | |R
       | |Domain 2          |          A                |  |   | |C
       | |                  V          |                |  |   | |H
       | |             +---[x]--------[x]----+          |  |V  | |I
       | |Big Switch   |   /   BiS-BiS  \    |          |  |N  | |N
       | |with         |  /              \   |          |  |F  | |G
       | |Big Software |  | +-->-+ +-->-+ |  |          |  |   | |
       | |(BiS-BiS)    |  | |    | |    | |  |          |  |F  | |V
       | |             +--x-x----x-x----x-x--+          |  |G  | |N
       | |                | |    | |    | |             |  |2  | |F
       | |                +-+    +-+    +-+             |  |   | |
       | |                |V|    |V|    |V|             |  |   | |F
       | |                |N|    |N|    |N|             |  |   | |G
       | |                |F|    |F|    |F|             |  |   | |
       | |                |1|    |2|    |3|             |  |   | |
       | |                +-+    +-+    +-+             |  |   | |
       | +----------------------------------------------+  V   | |
       +-------------------------------------------------------+ V

        Figure 5: Recursive Domain Virtualization and Joint VNF FG
                         programming: Domain Views

4.1.  Virtualization

   Let us first define the joint software and network abstraction
   (virtualization) as a Big Switch with Big Software (BiS-BiS).  A BiS-
   BiS is a node abstraction, which incorporates both software and
   networking resources with an associated joint software and network
   control API (see Figure 6).

Szabo, et al.            Expires January 7, 2016               [Page 11]
Internet-Draft        Toward recursive programming             July 2015

                   API o         __
                       |           \
                Software Ctrler     \
     API O-------------+     \       \
         |                    \       \
     Compute Ctrler            \      |
         |                      \     |
         | +---------------------+    |
         | |                     |    |           Joint Software &
         | |   {vCPU             |    |           Network Ctrl API
         | |    memory           |    |                   o
         | |    storage}         |    |                   |
         | |                     |    |        +---------------------+
         | |                     |    |        | {{vCPU              |
         | |Compute Node         |    \       [1   memory            3]
         | |                     |     ==>     |   storage}          |
         | +----------x----------+    /       [2  {port rate         4]
          \           |               |        |   switching delay}} |
           +----------x----------+    |        +---------------------+
           |                     |    |             Big Switch &
          [1  {port rate         3]   |         Big Software (BiS-BiS)
           |   switching delay}  |    |             with joint
          [2                     4]   /       Software & Network Ctrler
           |  Network Element    |   /
           +---------------------+  /
                                 __/

             Figure 6: Big Switch with Big Software definition

   The configuration over a BiS-BiS allows the atomic definition of NF
   placements and the corresponding forwarding overlay as a Network
   Function - Forwarding Graph (NF-FG).  The embedment of NFs into a
   BiS-BiS allows the inclusion of NF ports into the forwarding overlay
   definition (see ports a, b, ...,f in Figure 7).  Ports 1,2, ..., 4
   are seen as infrastructure ports while NF ports are created and
   destroyed with NF placements.

Szabo, et al.            Expires January 7, 2016               [Page 12]
Internet-Draft        Toward recursive programming             July 2015

      Step 1: Placement of NFs
      Step 2: Interconnect NFs __         Step 1: Placement of NFs
                                 \                with the forwarding
          Compute Node            \               overlay definition
         +---------------------+   \
         |  +-+    +-+    +-+  |    \           +-+    +-+    +-+
         |  |V|    |V|    |V|  |    |           |V|    |V|    |V|
         |  |N|    |N|    |N|  |    |           |N|    |N|    |N|
         |  |F|    |F|    |F|  |    |           |F|    |F|    |F|
         |  |1|    |2|    |3|  |    |           |1|    |2|    |3|
         |  +-+    +-+    +-+  |    |           +-+    +-+    +-+
         |  | +---.| |.---+ |  |    \           | |    | |    | |
         |  +------\ /------+  |     ==>     +--a-b----c-d----e-f--+
         +----------x----------+    /        |  | |    | |    | |  |
                    |               |       [1->+ +-->-+ +-->-+ |  3]
         +----------x----------+    |        |                  |  |
         |         / \         |    |       [2                  +->4]
        [1->----->-   -->---+  3]   |        |                     |
         |                  |  |    |        +---------------------+
        [2                  +->4]   /            Big Switch with
         |   Network Element   |   /          Big Software (BiS-BiS)
         +---------------------+  /
                               __/

     Figure 7: Big Switch with Big Software definition with a Network
                    Function - Forwarding Graph (NF-FG)

4.1.1.  The virtualizer's data model

4.1.1.1.  Tree view

          module: virtualizer
             +--rw virtualizer
                +--rw id?      string
                +--rw name?    string
                +--rw nodes
                |  +--rw node* [id]
                |     +--rw id              string
                |     +--rw name?           string
                |     +--rw type            string
                |     +--rw ports
                |     |  +--rw port* [id]
                |     |     +--rw id           string
                |     |     +--rw name?        string
                |     |     +--rw port_type    string
                |     |     +--rw port_data?   string
                |     +--rw links
                |     |  +--rw link* [src dst]

Szabo, et al.            Expires January 7, 2016               [Page 13]
Internet-Draft        Toward recursive programming             July 2015

                |     |     +--rw id?          string
                |     |     +--rw name?        string
                |     |     +--rw src          port-ref
                |     |     +--rw dst          port-ref
                |     |     +--rw resources
                |     |        +--rw delay?       string
                |     |        +--rw bandwidth?   string
                |     +--rw resources
                |     |  +--rw cpu        string
                |     |  +--rw mem        string
                |     |  +--rw storage    string
                |     +--rw NF_instances
                |     |  +--rw node* [id]
                |     |     +--rw id           string
                |     |     +--rw name?        string
                |     |     +--rw type         string
                |     |     +--rw ports
                |     |     |  +--rw port* [id]
                |     |     |     +--rw id           string
                |     |     |     +--rw name?        string
                |     |     |     +--rw port_type    string
                |     |     |     +--rw port_data?   string
                |     |     +--rw links
                |     |     |  +--rw link* [src dst]
                |     |     |     +--rw id?          string
                |     |     |     +--rw name?        string
                |     |     |     +--rw src          port-ref
                |     |     |     +--rw dst          port-ref
                |     |     |     +--rw resources
                |     |     |        +--rw delay?       string
                |     |     |        +--rw bandwidth?   string
                |     |     +--rw resources
                |     |        +--rw cpu        string
                |     |        +--rw mem        string
                |     |        +--rw storage    string
                |     +--rw capabilities
                |     |  +--rw supported_NFs
                |     |     +--rw node* [id]
                |     |        +--rw id           string
                |     |        +--rw name?        string
                |     |        +--rw type         string
                |     |        +--rw ports
                |     |        |  +--rw port* [id]
                |     |        |     +--rw id           string
                |     |        |     +--rw name?        string
                |     |        |     +--rw port_type    string
                |     |        |     +--rw port_data?   string
                |     |        +--rw links

Szabo, et al.            Expires January 7, 2016               [Page 14]
Internet-Draft        Toward recursive programming             July 2015

                |     |        |  +--rw link* [src dst]
                |     |        |     +--rw id?          string
                |     |        |     +--rw name?        string
                |     |        |     +--rw src          port-ref
                |     |        |     +--rw dst          port-ref
                |     |        |     +--rw resources
                |     |        |        +--rw delay?       string
                |     |        |        +--rw bandwidth?   string
                |     |        +--rw resources
                |     |           +--rw cpu        string
                |     |           +--rw mem        string
                |     |           +--rw storage    string
                |     +--rw flowtable
                |        +--rw flowentry* [port match action]
                |           +--rw port         port-ref
                |           +--rw match        string
                |           +--rw action       string
                |           +--rw resources
                |              +--rw delay?       string
                |              +--rw bandwidth?   string
                +--rw links
                   +--rw link* [src dst]
                      +--rw id?          string
                      +--rw name?        string
                      +--rw src          port-ref
                      +--rw dst          port-ref
                      +--rw resources
                         +--rw delay?       string
                         +--rw bandwidth?   string

            Figure 8: Virtualizer's YANG data model: tree view

4.1.1.2.  YANG Module

    <CODE BEGINS> file "virtualizer.yang"

      module virtualizer {
      namespace "http://fp7-unify.eu/framework/virtualizer";
      prefix virt;
      organization "EU-FP7-UNIFY";
      contact "Robert Szabo <robert.szabo@ericsson.com>";
      description "data model for joint software and network
      virtualization and resource control";

      revision 2015-06-27 {
        reference "Initial version";
      }

Szabo, et al.            Expires January 7, 2016               [Page 15]
Internet-Draft        Toward recursive programming             July 2015

      // REUSABLE GROUPS
      grouping id-name {
        description "used for key (id) and naming";
        leaf id {
          type string;
          description "For unique key id";}
        leaf name {
          type string;
          description "Descriptive name";}
      }

      grouping node-type {
        description "For node type defintion";
        leaf type{
          type string;
          mandatory true;
          description "to identify nodes (infrastructure or NFs)";
        }
      }

      // PORTS
      typedef port-ref {
        type string;
        description "path to a port; can refer to ports at multiple
        levels in the hierarchy";
      }

      grouping port {
        description "Port definition: used for infrastructure and NF
        ports";
        uses id-name;
        leaf port_type {
          type string;
          mandatory true;
          description "Port type identification: abstract is for
          technology independent ports and SAPs for technology specific
          ports";}
        leaf port_data{
          type string;
          description "Opaque data for port specific types";
        }
      }

      grouping ports {
        description "Collection of ports";
        container ports {
          description "see above";
          list port{

Szabo, et al.            Expires January 7, 2016               [Page 16]
Internet-Draft        Toward recursive programming             July 2015

            key "id";
            uses port;
        description "see above";
          }
        }
      }
      // FORWARDING BEHAVIOR
      grouping flowentry {
        leaf port {
          type port-ref;
          mandatory true;
          description "path to the port";
        }
        leaf match {
          type string;
          mandatory true;
          description "matching rule";
        }
        leaf action {
          type string;
          mandatory true;
          description "forwarding action";
        }
        container resources{
          uses link-resource;
          description "network resources assigned to forwarding entry";
        }
        description "SDN forwarding entry";
      }

      grouping flowtable {
        container flowtable {
          description "Collection of flowentries";
          list flowentry {
            key "port match action";
        description "Index list of flowentries";
            uses flowentry;
          }
        }
        description "See container description";
      }

      // LINKS
      grouping link-resource {
        description "Core networking characteristics / resources
        (bandwidth, delay)";
        leaf delay {
          type string;

Szabo, et al.            Expires January 7, 2016               [Page 17]
Internet-Draft        Toward recursive programming             July 2015

          description "Delay value with unit; e.g. 5ms";
        }
        leaf bandwidth {
          type string;
          description "Bandwithd value with unit; e.g. 10Mbps";
        }
      }

      grouping link {
        description "Link between src and dst ports with attributes";
        uses id-name;
        leaf src {
          type port-ref;
          description "relative path to the source port";
        }
        leaf dst {
          type port-ref;
          description "relative path to the destination port";
        }
        container resources{
          uses link-resource;
          description "Link resources (attributes)";
        }
      }

      grouping links {
        description "Collection of links in a virtualizer or a node";
        container links {
          description "See above";
          list link {
            key "src dst";
        description "Indexed list of links";
            uses link;
          }
        }
      }

      // CAPABILITIES
      grouping capabilities {
        description "For capability reporting: currently supported NF
        types";
        container supported_NFs { // supported NFs are enumerated
          description "Collecction of nodes as supported NFs";
          list node{
            key "id";
        description "see above";
            uses node;
          }

Szabo, et al.            Expires January 7, 2016               [Page 18]
Internet-Draft        Toward recursive programming             July 2015

        }
        // TODO: add other capabilities
      }

      // NODE

      grouping software-resource {
        description "Core software resources";
        leaf cpu {
          type string;
          mandatory true;
          description "In virtual CPU (vCPU) units";
        }
        leaf mem {
          type string;
          mandatory true;
          description "Memory with units, e.g., 1Gbyte";
        }
        leaf storage {
          type string;
          mandatory true;
          description "Storage with units, e.g., 10Gbyte";
        }
      }

      grouping node {
        description "Any node: infrastructure or NFs";
        uses id-name;
        uses node-type;
        uses ports;
        uses links;
        container resources{
          description "Software resources offer/request of the node";
          uses software-resource;
        }
      }

      grouping infra-node {
        description "Infrastructure nodes wich can contain other nodes
        as NFs";
        uses node;
        container NF_instances {
          description "Hosted NFs";
          list node{
            key "id";
            uses node;
        description "see above";
          }

Szabo, et al.            Expires January 7, 2016               [Page 19]
Internet-Draft        Toward recursive programming             July 2015

        }
        container capabilities {
          description "Supported NFs as capability reports";
          uses capabilities;
        }
        uses flowtable;
      }

      //======== Virtualizer  ====================

      container virtualizer {
        description "Definition of a virtualizer instance";
        uses id-name;

        container nodes{
          description "infra nodes, which embeds NFs and report
          capabilities";
          list node{
            key "id";
            uses infra-node;
        description "see above";
          }
        }
        uses links;
      }
    }
    <CODE ENDS>

                  Figure 9: Virtualizer's YANG data model

5.  Examples

5.1.  Infrastructure reports

   Figure 10 show a single node infrastructure report.  The example
   shows a BiS-BiS with two ports, out of which Port 0 is also a Service
   Access Point 0 (SAP0).

Szabo, et al.            Expires January 7, 2016               [Page 20]
Internet-Draft        Toward recursive programming             July 2015

      <virtualizer xmlns="http://fp7-unify.eu/framework/virtualizer">
          <id>UUID001</id>
          <name>Single node simple infrastructure report</name>
          <nodes>
              <node>
                  <id>UUID11</id>
                  <name>single Bis-Bis node</name>
                  <type>BisBis</type>
                  <ports>
                      <port>
                          <id>0</id>
                          <name>SAP0 port</name>
                          <port_type>port-sap</port_type>
                          <vxlan>...</vxlan>
                      </port>
                      <port>
                          <id>1</id>
                          <name>North port</name>
                          <port_type>port-abstract</port_type>
                          <capability>...</capability>
                      </port>
                      <port>
                          <id>2</id>
                          <name>East port</name>
                          <port_type>port-abstract</port_type>
                          <capability>...</capability>
                      </port>
                  </ports>
                  <resources>
                      <cpu>20</cpu>
                      <mem>64 GB</mem>
                      <storage>100 TB</storage>
                  </resources>
              </node>
          </nodes>
      </virtualizer>

           Figure 10: Single node infrastructure report example

   Figure 11 shows a 3-node infrastructure report with 3 BiS-BiS nodes.
   Infrastructure links are inserted into the virtualization view
   between the ports of the BiS-BiS nodes.

    <virtualizer xmlns="http://fp7-unify.eu/framework/virtualizer">
        <id>UUID002</id>
        <name>3-node simple infrastructure report</name>
        <nodes>
            <node>

Szabo, et al.            Expires January 7, 2016               [Page 21]
Internet-Draft        Toward recursive programming             July 2015

                <id>UUID11</id>
                <name>West Bis-Bis node</name>
                <type>BisBis</type>
                <ports>
                    <port>
                        <id>0</id>
                        <name>SAP0 port</name>
                        <port_type>port-sap</port_type>
                        <vxlan>...</vxlan>
                    </port>
                    <port>
                        <id>1</id>
                        <name>North port</name>
                        <port_type>port-abstract</port_type>
                        <capability>...</capability>
                    </port>
                    <port>
                        <id>2</id>
                        <name>East port</name>
                        <port_type>port-abstract</port_type>
                        <capability>...</capability>
                    </port>
                </ports>
                <resources>
                    <cpu>20</cpu>
                    <mem>64 GB</mem>
                    <storage>100 TB</storage>
                </resources>
            </node>
            <node>
                <id>UUID12</id>
                <name>East Bis-Bis node</name>
                <type>BisBis</type>
                <ports>
                    <port>
                        <id>1</id>
                        <name>SAP1 port</name>
                        <port_type>port-sap</port_type>
                        <vxlan>...</vxlan>
                    </port>
                    <port>
                        <id>0</id>
                        <name>North port</name>
                        <port_type>port-abstract</port_type>
                        <capability>...</capability>
                    </port>
                    <port>
                        <id>2</id>

Szabo, et al.            Expires January 7, 2016               [Page 22]
Internet-Draft        Toward recursive programming             July 2015

                        <name>West port</name>
                        <port_type>port-abstract</port_type>
                        <capability>...</capability>
                    </port>
                </ports>
                <resources>
                    <cpu>10</cpu>
                    <mem>32 GB</mem>
                    <storage>100 TB</storage>
                </resources>
            </node>
            <node>
                <id>UUID13</id>
                <name>North Bis-Bis node</name>
                <type>BisBis</type>
                <ports>
                    <port>
                        <id>0</id>
                        <name>SAP2 port</name>
                        <port_type>port-sap</port_type>
                        <vxlan>...</vxlan>
                    </port>
                    <port>
                        <id>1</id>
                        <name>East port</name>
                        <port_type>port-abstract</port_type>
                        <capability>...</capability>
                    </port>
                    <port>
                        <id>2</id>
                        <name>West port</name>
                        <port_type>port-abstract</port_type>
                        <capability>...</capability>
                    </port>
                </ports>
                <resources>
                    <cpu>20</cpu>
                    <mem>64 GB</mem>
                    <storage>1 TB</storage>
                </resources>
            </node>
        </nodes>
        <links>
            <link>
                <id>0</id>
                <name>Horizontal link</name>
                <src>../../nodes/node[id=UUID11]/ports/port[id=2]</src>
                <dst>../../nodes/node[id=UUID12]/ports/port[id=2]</dst>

Szabo, et al.            Expires January 7, 2016               [Page 23]
Internet-Draft        Toward recursive programming             July 2015

                <resources>
                    <delay>2 ms</delay>
                    <bandwidth>10 Gb</bandwidth>
                </resources>
            </link>
            <link>
                <id>1</id>
                <name>West link</name>
                <src>../../nodes/node[id=UUID11]/ports/port[id=1]</src>
                <dst>../../nodes/node[id=UUID13]/ports/port[id=2]</dst>
                <resources>
                    <delay>5 ms</delay>
                    <bandwidth>10 Gb</bandwidth>
                </resources>
            </link>
            <link>
                <id>2</id>
                <name>East link</name>
                <src>../../nodes/node[id=UUID12]/ports/port[id=0]</src>
                <dst>../../nodes/node[id=UUID13]/ports/port[id=1]</dst>
                <resources>
                    <delay>2 ms</delay>
                    <bandwidth>5 Gb</bandwidth>
                </resources>
            </link>
        </links>
    </virtualizer>

              Figure 11: 3-node infrastructure report example

5.2.  Simple requests

   Figure 12 shows the allocation request for 3 NFs (Parental control
   B.4, Http Cache 1.2 and Stateful firewall C) as instrumented over a
   BiS-BiS node.  It can be seen that the configuration request contains
   both the NF placement and the forwarding overlay definition as a
   joint request.

    <virtualizer xmlns="http://fp7-unify.eu/framework/virtualizer">
        <id>UUID001</id>
        <name>Single node simple request</name>
        <nodes>
            <node>
                <id>UUID11</id>
                <NF_instances>
                    <node>
                        <id>NF1</id>
                        <name>first NF</name>

Szabo, et al.            Expires January 7, 2016               [Page 24]
Internet-Draft        Toward recursive programming             July 2015

                        <type>Parental control B.4</type>
                        <ports>
                            <port>
                                <id>2</id>
                                <name>in</name>
                                <port_type>port-abstract</port_type>
                                <capability>...</capability>
                            </port>
                            <port>
                                <id>3</id>
                                <name>out</name>
                                <port_type>port-abstract</port_type>
                                <capability>...</capability>
                            </port>
                        </ports>
                    </node>
                    <node>
                        <id>NF2</id>
                        <name>cache</name>
                        <type>Http Cache 1.2</type>
                        <ports>
                            <port>
                                <id>4</id>
                                <name>in</name>
                                <port_type>port-abstract</port_type>
                                <capability>...</capability>
                            </port>
                            <port>
                                <id>5</id>
                                <name>out</name>
                                <port_type>port-abstract</port_type>
                                <capability>...</capability>
                            </port>
                        </ports>
                    </node>
                    <node>
                        <id>NF3</id>
                        <name>firewall</name>
                        <type>Stateful firewall C</type>
                        <ports>
                            <port>
                                <id>6</id>
                                <name>in</name>
                                <port_type>port-abstract</port_type>
                                <capability>...</capability>
                            </port>
                            <port>
                                <id>7</id>

Szabo, et al.            Expires January 7, 2016               [Page 25]
Internet-Draft        Toward recursive programming             July 2015

                                <name>out</name>
                                <port_type>port-abstract</port_type>
                                <capability>...</capability>
                            </port>
                        </ports>
                    </node>
                </NF_instances>
                <flowtable>
                    <flowentry>
                        <port>../../ports/port[id=0]</port>
                        <match>*</match>
                        <action>output:../../NF_instances/node[id=NF1]
                  /ports/port[id=2]</action>
                    </flowentry>
                    <flowentry>
                        <port>../../NF_instances/node[id=NF1]
                  /ports/port[id=3]</port>
                        <match>fr-a</match>
                        <action>output:../../NF_instances/node[id=NF2]
                  /ports/port[id=4]</action>
                    </flowentry>
                    <flowentry>
                        <port>../../NF_instances/node[id=NF1]
                  /ports/port[id=3]</port>
                        <match>fr-b</match>
                        <action>output:../../NF_instances/node[id=NF3]
                  /ports/port[id=6]</action>
                    </flowentry>
                    <flowentry>
                        <port>../../NF_instances/node[id=NF2]
                  /ports/port[id=5]</port>
                        <match>*</match>
                        <action>output:../../ports/port[id=1]</action>
                    </flowentry>
                    <flowentry>
                        <port>../../NF_instances/node[id=NF3]
                  /ports/port[id=7]</port>
                        <match>*</match>
                        <action>output:../../ports/port[id=1]</action>
                    </flowentry>
                </flowtable>
            </node>
        </nodes>
    </virtualizer>

          Figure 12: Simple request of 3 NFs on a single BiS-BiS

Szabo, et al.            Expires January 7, 2016               [Page 26]
Internet-Draft        Toward recursive programming             July 2015

6.  IANA Considerations

   This memo includes no request to IANA.

7.  Security Considerations

   TBD

8.  Acknowledgement

   The research leading to these results has received funding from the
   European Union Seventh Framework Programme (FP7/2007-2013) under
   grant agreement no. 619609 - the UNIFY project.  The views expressed
   here are those of the authors only.  The European Commission is not
   liable for any use that may be made of the information in this
   document.

   We would like to thank in particular David Jocha and Janos Elek from
   Ericsson for the useful discussions.

9.  Informative References

   [ETSI-NFV-Arch]
              ETSI, "Architectural Framework v1.1.1", Oct 2013,
              <http://www.etsi.org/deliver/etsi_gs/
              NFV/001_099/002/01.01.01_60/gs_NFV002v010101p.pdf>.

   [ETSI-NFV-MANO]
              ETSI, "Network Function Virtualization (NFV) Management
              and Orchestration V0.6.1 (draft)", Jul. 2014,
              <http://docbox.etsi.org/ISG/NFV/Open/Latest_Drafts/
              NFV-MAN001v061-%20management%20and%20orchestration.pdf>.

   [I-D.ietf-sfc-dc-use-cases]
              Surendra, S., Tufail, M., Majee, S., Captari, C., and S.
              Homma, "Service Function Chaining Use Cases In Data
              Centers", draft-ietf-sfc-dc-use-cases-02 (work in
              progress), January 2015.

   [I-D.unify-nfvrg-challenges]
              Szabo, R., Csaszar, A., Pentikousis, K., Kind, M., and D.
              Daino, "Unifying Carrier and Cloud Networks: Problem
              Statement and Challenges", draft-unify-nfvrg-challenges-00
              (work in progress), October 2014.

   [I-D.zu-nfvrg-elasticity-vnf]
              Qiang, Z. and R. Szabo, "Elasticity VNF", draft-zu-nfvrg-
              elasticity-vnf-01 (work in progress), March 2015.

Szabo, et al.            Expires January 7, 2016               [Page 27]
Internet-Draft        Toward recursive programming             July 2015

Authors' Addresses

   Robert Szabo
   Ericsson Research, Hungary
   Irinyi Jozsef u. 4-20
   Budapest  1117
   Hungary

   Email: robert.szabo@ericsson.com
   URI:   http://www.ericsson.com/

   Zu Qiang
   Ericsson
   8400, boul. Decarie
   Ville Mont-Royal, QC  8400
   Canada

   Email: zu.qiang@ericsson.com
   URI:   http://www.ericsson.com/

   Mario Kind
   Deutsche Telekom AG
   Winterfeldtstr. 21
   10781 Berlin
   Germany

   Email: mario.kind@telekom.de

Szabo, et al.            Expires January 7, 2016               [Page 28]