Internet Engineering Task Force                            Florin Balus
Internet Draft                                        Dimitri Stiliadis
Intended status: standards track                         Nuage Networks
Expires: April 2013
                                                            Nabil Bitar
Wim Henderickx                                                  Verizon
Marc Lasserre
Alcatel-Lucent                                            Kenichi Ogaki
                                                                   KDDI


                                                       October 22, 2012




                 Federated SDN-based Controllers for NVO3
                    draft-sb-nvo3-sdn-federation-01.txt




Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other documents
   at any time.  It is inappropriate to use Internet-Drafts as
   reference material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html

   This Internet-Draft will expire on April 22, 2013.

Copyright Notice

   Copyright (c) 2012 IETF Trust and the persons identified as the
   document authors. All rights reserved.




Balus, Stiliadis et al.  Expires April 22, 2013                [Page 1]


Internet-Draft        Federated SDN controllers            October 2012


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with
   respect to this document.

Abstract

   The familiar toolset of VPN and IP/MPLS protocols defined in IETF
   has been discussed as a good starting point for addressing several
   of the problems described in the NVO3 problem statement and
   requirements drafts. However, there is a significant gap between the
   VPN technologies and the scale and complexity required when the NVEs
   are running in server hypervisors.

   This draft proposes a solution that bridges the gap between the
   concepts familiar to the data center IT community with some of the
   best concepts developed by the networking industry.

   The proposed solution is based on the understanding that in a cloud
   environment NVEs may reside in end-devices that should not be
   burdened with the complexities of control plane protocols. The
   complex control plane functionalities are decoupled from the
   forwarding plane to minimize the required NVE processing,
   recognizing that major hypervisor distributions already employ
   Openflow to address this issue.

   The solution also defines the mechanisms for scaling data center
   network control horizontally and interoperates seamlessly with
   existing L2 and L3 VPN devices.

Table of Contents

   1. Introduction...................................................3
   2. Conventions used in this document..............................4
      2.1. General terminology.......................................4
   3. Solution Overview..............................................4
   4. Control plane details..........................................6
      4.1. Tenant System State Discovery.............................6
         4.1.1. Tenant System states and related information.........7
         4.1.2. Tracking local TS events.............................8
            4.1.2.1. NVE to Controller signaling of TS events........8
      4.2. Address advertisement and FIB population..................9
         4.2.1. Push versus Pull.....................................9
      4.3. Underlay awareness........................................9
      4.4. Controllers federation...................................10


Balus, Stiliadis et al.  Expires April 22, 2013                [Page 2]


Internet-Draft        Federated SDN controllers            October 2012


   5. Data plane considerations.....................................11
      5.1. L2 and L3 services.......................................11
      5.2. NVO3 encapsulations......................................11
   6. Resiliency considerations.....................................12
      6.1. Controller resiliency....................................12
      6.2. Data plane resiliency....................................12
   7. Practical deployment considerations...........................12
      7.1. Controller distribution..................................12
      7.2. Hypervisor NVE processing................................13
      7.3. Interoperating with non-NVO3 domains.....................13
      7.4. VM Mobility..............................................13
      7.5. Openflow and Open vSwitch................................13
   8. Security Considerations.......................................14
   9. IANA Considerations...........................................14
   10. References...................................................14
      10.1. Normative References....................................14
      10.2. Informative References..................................14
   11. Acknowledgments..............................................15

1. Introduction

   Several data center networking challenges are described in the NVO3
   problem statement and requirements drafts. A number of documents
   propose extensions to or re-use of existing IETF protocols to
   address these challenges.

   The data center environment though, is dominated by the presence of
   software networking components (vswitches) in server hypervisors,
   which may outnumber by several orders of magnitude the physical
   networking nodes. Limited resources are available for software
   networking as hypervisor software is designed to maximize the
   revenue generating compute resources rather than expending them in
   network protocol processing.

   More importantly the cloud environment is driven by the need to
   innovate and bring new IT services fast to market, so network
   automation and network flexibility is of paramount importance.

   This document proposes for NVO3 control plane a combination between
   IETF VPN mechanisms and the Software Defined Network (SDN) concepts
   developed by the Open Networking Foundation and already in use in a
   number of cloud deployments. Existing routing mechanisms are
   employed when a scaled out data center deployment is required to
   federate a number of SDN controllers in a multi-vendor environment
   or to interoperate with non-NVO3 domains.




Balus, Stiliadis et al.  Expires April 22, 2013                [Page 3]


Internet-Draft        Federated SDN controllers            October 2012


   The proposed solution can be implemented with minimal extensions to
   existing protocols and to a certain extent is already operational in
   several cloud environments. It also proposes a simple mechanism that
   enables NVO3 domains to seamlessly interoperate with VPN deployments
   without requiring new functionality in the existing VPN networks.

   The focus of this draft is on the NVO3 controller; it describes how
   the SDN concepts defined in [ONF] can be used and extended to
   perform the NVO3 control plane functions and how SDN controller
   domains can be federated using existing routing protocols. The
   handling and support for different NVO3 encapsulations is described
   briefly in the later sections. The terminology of NVO3 controller
   may be subject to further changes in the framework draft [NVO3-FWK].

2. Conventions used in this document

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC-2119 [RFC2119].

   In this document, these words will appear with that interpretation
   only when in ALL CAPS. Lower case uses of these words are not to be
   interpreted as carrying RFC-2119 significance.

2.1. General terminology

   This document uses the terminology defined in NVO3 framework
   document [NVO3-FWK].

3. Solution Overview

   The concept of a NVO3 generic architecture is discussed in the NVO3
   framework draft [NVO3-FWK].

   This section describes how NVO3 control plane functions can be
   implemented starting from the SDN concepts defined in [ONF]. The
   proposed architecture is depicted in the following diagram.












Balus, Stiliadis et al.  Expires April 22, 2013                [Page 4]


Internet-Draft        Federated SDN controllers            October 2012


                               +-----------+
                       +-------|   NVO3    |-------Y
                       |       |Controller |       |
                       |       | Function  |       |
                       |       +-----------+       |
                    +--+-+                       +-+--+
                    | NV |   Controller Domain   | NV |
                TS--|Edge|                       |Edge|--TS
                    +--+-+                       +-+--+
                       |           +----+          |
                       |           | NV |          |
                       '`''''''''''|Edge|'''''''''''
                                   +----+
                                     |
                                     |
                                     TS
              Figure 1 Controller-based architecture for NVO3

   NVO3 controller function is implemented using the concept of a SDN
   controller [ONF]. The Controller is a software component residing in
   a logically centralized location, responsible for a specific NVE
   domain [NVO3-FWK]. Each NVE has a control session to the Controller
   which in turn may run external control plane sessions to communicate
   to the outside world or to other controllers. All the NVEs sharing a
   controller represent a controller domain.

   The Controller provides a generic API for learning about the Tenant
   System profile and related events. The mechanism can be through a
   north bound API of the Controller itself, or through an event
   tracking mechanism.

   As soon as a Tenant System is configured and attached to a
   Controller domain, the cloud management system informs the
   Controller about this activation. Existing cloud management systems
   [Openstack, Cloudstack] assume the Controller provides an API that
   will enable the cloud management system to notify the Controller
   about the new service activation. The Quantum plugin in Openstack is
   an example of an API interface between cloud management and
   Controller.

   Alternative mechanisms can rely on a direct notification of a TS
   (VM) attachment to the NVE that can be done at the hypervisor layer.
   The NVE can then pass the TS information to the controller using the
   procedures described in section 4.1.2.1. For Tenant Systems located
   in a remote Controller domain, (MP)-BGP is used to advertise and
   learn the related information to/from the remote controller.



Balus, Stiliadis et al.  Expires April 22, 2013                [Page 5]


Internet-Draft        Federated SDN controllers            October 2012


   Once the Controller is aware of a new Tenant System attached to a
   NVE, the TS IP and/or MAC addresses are populated in the
   controller's routing database. Since the Controller has full
   visibility of all Tenant Systems belonging to the same tenant within
   its domain, it will generate the related FIBs and populate the
   required entries on the NVEs that have a corresponding tenant VNI.
   The controller can rely on the Openflow protocol [Openflow] for this
   function, since it is only populating FIB entries, and can use
   either a pull or a push method or a combination based on the scale
   and dynamics of the FIB entries that need to be populated in the
   NVEs, and/or based on a FIB population policy.

   When the TS is deleted, the reverse process takes place. The
   Controller, either from API calls or based on events received from
   the NVEs, will learn about the TS removal and it will proceed to
   remove the TS IP and/or MAC from its routing and FIB database. BGP
   will be used to withdraw these addresses from the remote Controllers
   to which those addresses were previously advertised and/or managing
   NVEs with VNIs belonging to the same corresponding tenant VN, if any
   are present. These changes will also be communicated then to the
   impacted NVEs by the corresponding remote Controller.

4. Control plane details

   The controller is the central component for the NVO3 control plane
   architecture. In the proposed solution the NVO3 controller utilizes
   mechanisms to monitor events and create the required connectivity.

   This section discusses how the current SDN model [ONF] with
   appropriate extensions can be used to address the requirements
   outlined in [NVO3-CPREQ].

   To automatically instantiate the required data plane connectivity
   the NVO3 controller has to perform the following functions:

      - Learning of TS profiles and current state (TS State Discovery).
      - Auto-instantiation of NVO3 service.
          o Address advertisement and associated tunnel encapsulation
             mapping
          o FIB population.
      - Underlay aware routing.

4.1. Tenant System State Discovery

   This section sets the stage for a generic set of events and actions
   that MUST be supported by any NVO3 controller if automatic
   instantiation of the service is desired. The goal is to converge


Balus, Stiliadis et al.  Expires April 22, 2013                [Page 6]


Internet-Draft        Federated SDN controllers            October 2012


   first on a common information model between the cloud management
   system and the NVO3 controller. It is also highly desirable to
   converge to a common protocol and information model that will allow
   cloud management systems and Controllers from different vendors to
   interoperate. At the current time the integration of a controller
   with different cloud management systems requires customization work
   and this will lead to interoperability problems and duplicated
   development efforts to interoperate a matrix of cloud management
   systems and Controllers.

4.1.1. Tenant System states and related information

   There is a large variety of Tenant Systems that may need to receive
   service from a NVO3 domain. For example in the Cloud networking
   space a TS may be a VM or an appliance instance (Firewall, LB)
   assigned to a particular tenant. There are a number of possible
   states for the Tenant Systems that need to be signaled to and
   interpreted by the Controller. The following is an initial set of TS
   states that is mainly derived by mirroring the model of virtual
   machine lifecycle management encountered in several cloud management
   tools:

      - Not-deployed: TS exists in management system but is not
        instantiated. The NVE does not need to know about these
        instances.
      - Running: TS is instantiated, active and ready to send traffic.
        The appropriate NVEs with instances of the corresponding tenant
        VNs, must have all required functionality to send and receive
        traffic for the particular TS.
      - Suspended: TS is instantiated, but currently in a suspended
        state. Traffic from/to the TS can be ignored. Routes to this TS
        may be withdrawn from the corresponding tenant VN.
      - Shutdown: TS is in the process of shutting down. A complete
        shutdown is not known though, and it will depend on the
        capabilities of the TS. Traffic from/to the TS must be
        forwarded.
      - Shut off: TS is in the power-off mode and not attached to the
        NVE. This is similar to the suspend state. Traffic from/to the
        TS can be ignored. Routes corresponding to the TS must be
        withdrawn from corresponding tenant VN and the forwarding state
        at the local NVE must be removed.
      - Moving: TS is active but a TS Move command was originated. The
        Controller must participate in any state transfer functions.
        The goal is to directly forward traffic to the TS  at the new
        location and possibly tunnel traffic in transit to the old
        location from the old location to the new one.



Balus, Stiliadis et al.  Expires April 22, 2013                [Page 7]


Internet-Draft        Federated SDN controllers            October 2012


      - Other: Opaque state that refers to additional states defined by
        a specialized TS.

   Even though, the states above are often related to virtual machines,
   the model or a subset can cover the physical appliance states as
   well. Depending on the TS, some of these states might not be easily
   identifiable (additional mechanisms, liveliness check, are required
   to detect a crashed or shutdown physical machine).

4.1.2. Tracking local TS events

  The controller must have full information about the TS state. This
  can be achieved in one of two ways:

  1. The cloud management system utilizes an API exposed by the
     Controller to update the TS state. This is the model deployed by
     the Openstack Quantum API or the Cloudstack Network Guru API.
  2. The NVE tracks the above events when it is co-located with the
     hypervisor using internal mechanisms and reports it to the
     Controller using a signaling mechanism. When the NVE is not
     implemented in the hypervisor hosting the TS, a tracking protocol
     between the hypervisor and NVE can allow the tracking of TS state
     events. A standard protocol for this tracking function can
     significantly assist in the interoperability between different
     hypervisors and NVEs. One such mechanism is discussed in
     [Server2NVE].

  The following section discusses the NVE to controller signaling
  procedure for the second scenario.

4.1.2.1. NVE to Controller signaling of TS events

   In the case where the NVE directly tracks VM events, there is also a
   need for a standard signaling mechanism between the NVE and the
   Controller. This proposal utilizes extensions to Openflow protocol
   [Openflow] in order to accommodate this signaling. Openflow is
   supported by a number of hypervisor distributions and is already
   active in some large cloud deployments. Moreover it has already the
   messaging base required to perform this function.

   In the current Openflow specification, the first packet of an
   unknown flow can be forwarded to the Controller when no match is
   found in the local openflow switch. One possible method to implement
   TS event signaling is to extend this functionality and use the TS
   event as a trigger for a generic "flow request" from the NVE that
   will carry sufficient information to the controller to allow for
   service initialization. However, the flow request is extended here


Balus, Stiliadis et al.  Expires April 22, 2013                [Page 8]


Internet-Draft        Federated SDN controllers            October 2012


   as it does not contain parts of a TS packet, but information of a TS
   event. Alternatively a new request type can be defined. Details of
   this procedure will be added in a future revision.

4.2. Address advertisement and FIB population

   Once the controller learns about the TS state event from North bound
   API or from the NVE it performs the following actions:

      - Identify the required NVO3 service attributes. Service
        attributes could include acces lists and policies for certain
        actions.
      - Populate the VN routing and FIB tables with the TS address(es).
      - If a push-model is used, it downloads the required FIB updates
        and service attributes to the NVEs that participate in the
        related VN (pre-population).
      - If a pull-model is selected, it waits for the first packets of
        the corresponding flows or potentially other requests triggered
        by some events before establishing flow state, as per the
        Openflow specification, or some FIB state in the NVE
      - A combination of push and pull models may be beneficial. For
        example flows that require consistent latency and/or no packet
        loss may be pre-populated using a push model while other less
        important flows may be populated using a pull model. Similarly,
        ARP entries may be pre-populated or pulled-in when the NVE sees
        an ARP request with no corresponding ARP entry in its local
        cache.

4.2.1. Push versus Pull

   The Openflow model described in the previous section enables both a
   push and a pull model. The selection of a model will depend on the
   controller and NVE capabilities and deployment requirements.
   Different implementations might choose alternative methods or a
   combination of pull/push. This is though an implementation detail
   that is supported by the basic solution framework.

4.3. Underlay awareness

   The Underlay network consists of a core often running IP routing
   protocols to control the topology and to provide reachability among
   NVEs. A routing module in the Controller may participate in the
   underlay IP routing to maintain a view of the network underlay. The
   routing information exchanged may be used to control path selection
   or to make forward/drop decisions in the ingress NVEs, so that
   network bandwidth resources are not unnecessarily wasted.



Balus, Stiliadis et al.  Expires April 22, 2013                [Page 9]


Internet-Draft        Federated SDN controllers            October 2012


4.4. Controllers federation

   To address a scale-out NVO3 deployment multiple Controllers may be
   required. The Controllers need to exchange their routing information
   to allow for NVO3 services to extend across NVO3 domains managed by
   individual Controllers. The following diagram depicts this scenario.

                    +-----------+        +-----------+
                    | Controller| ..,,.. | Controller|
                    |  Domain 2 |"      "|  Domain n |
                    +----+------+        +-------+---+
                         "   Controller Federation "
                       '        (IP Routing)       ]
                        `-..                   ,,.-'
                            `''+-----------+'''
                               | Controller|
                               |  Domain 1 |
                               +-----------+
                      Figure 2 Controller Federation

   The fundamental requirement in any such function is a system that
   enables state distribution between the different instances. When
   Controllers from the same vendors are used this can be achieved
   through a pub/sub mechanism or through a distributed database. For
   example an ActiveMQ mechanism such as the one deployed by Openstack
   [Openstack] or a DHT based database like Cassandra.

   When Controllers from multiple vendors are in place or when
   interoperability with existing WAN services is required a standard
   way of distributing TS information is required.

   One can envision that existing VPN mechanisms can be utilized for
   this function, distributing private reachability information
   pertaining to the tenant VNs. A Routing module implementing the
   related MP-BGP procedures can be used as follows:

      - If L3 (IP) services need to be implemented across domains, the
        procedures described in [BGP-VPN], specifically the IP VPN SAFI
        and related NLRI can be used to exchange the Tenant IP
        addresses.
      - For L2 services the procedures described in [BGP-EVPN],
        specifically the EVPN NLRI(s) can be employed to advertise
        Tenant L2 addresses and related information among multiple
        Controllers.
      - For both service types, mechanisms like Route Distinguisher and
        Route Targets provide support for overlapping addressing space
        across VPNs (tenant VNs), and respectively for controlled


Balus, Stiliadis et al.  Expires April 22, 2013               [Page 10]


Internet-Draft        Federated SDN controllers            October 2012


        tenant service topology and route distribution only to the NVEs
        that have at least one TS in that VPN (tenant VN)

   Each Controller can then consolidate the external and internal
   routing tables and generate required FIB entries that can be
   downloaded as required to the NVEs.

   The advantage of utilizing (MP)-BGP to federate Controllers is that
   it enables interoperability between Controllers of different vendors
   as well as interoperability with existing WAN services, including
   Internet and L2/L3 VPNs. BGP is a proven mechanism that can be used
   to achieve the required scale and secure isolation for multi-tenant,
   multi-domain and multi-provider environments.

5. Data plane considerations

   The Controller can be used to control different data planes for
   different solution options.

5.1. L2 and L3 services

   MP-BGP VPN and Openflow can be used to program the required NVE data
   plane for both Layer 2 and Layer 3 services: Openflow is able/can be
   extended to handle both L2 and L3 FIB entries and multiple tunnel
   encapsulations while MP-BGP support for multiple address families
   ([BGP-VPN] and [BGP-EVPN]) allows the extension of both L2 and L3
   services across Controller domains.

   After the local TS discovery and MP-BGP exchanges the L2 and/or L3
   forwarding entries are computed first in the Controller and mapped
   to different types of tunnel encapsulations based on the type of
   core network, addressing type and the negotiated service
   encapsulation type.

   The resulting FIB updates can be downloaded using Openflow to NVEs
   that have VNIs corresponding to tenant VNs associated with FIB
   entries. The NVEs make use of these entries to forward packets at L2
   or L3 depending on the type of service and the desired processing
   sequence.

5.2. NVO3 encapsulations

   A number of vSwitch distributions already provide support for some
   of the encapsulations that had been proposed in some IETF drafts and
   allow the mapping of FIB entries to tunnel encapsulations based on
   these protocols. Opensource code for these encapsulations is also
   available. FIB entries with associated tunneling encapsulations can


Balus, Stiliadis et al.  Expires April 22, 2013               [Page 11]


Internet-Draft        Federated SDN controllers            October 2012


   be communicated from the Controller to the NVE using Openflow where
   supported or via Openflow extensions as required. Openflow supports
   also the MPLS VPN encapsulations easing the way for interoperability
   between NVE and VPN domains. Moreover the use of BGP for MAC and IP
   advertisement for different NVO3 encapsulations has been proposed in
   [EVPN-NVO3].

6. Resiliency considerations

   This section will discuss resiliency for a Controller domain
   implementing the control framework in this document.

6.1. Controller resiliency

   For a large domain, controller resiliency may be required. Non Stop
   control-plane schemes may be extended to cover the Controller
   Openflow component in addition to the BGP component. Alternatively
   the controller resiliency schemes proposed in [Openflow] may be
   employed in conjunction with Graceful Restart (for example [BGP
   Graceful-Restart]) for the routing modules.

6.2. Data plane resiliency

   From a data plane perspective, there are a number of ways to ensure
   the NVE can be multi-homed towards the IP core. Existing mechanisms
   like ECMP may be used to ensure load distribution across multiple
   paths.

   The access multi-homing of Tenant System to NVEs on the other hand
   is applicable only when the NVE and TS are on different physical
   devices. Several mechanisms can be utilized for this function
   depending on whether the TS to NVE communication is over L2 or L3.
   These use cases will be addressed in a future revision of this
   document.

7. Practical deployment considerations

7.1. Controller distribution

   The Controller is providing the control plane functionality for all
   the NVEs in its domain. The number of NVEs per Controller domain may
   vary depending on different scaling factors: for example the number
   of tenant systems, VAPs, VNs and tunnel endpoints. The concept of
   Controller federation allows for modular growth enabling a highly
   distributed deployment where the Controller may be deployed even
   down to server rack/ToR level in cases where scalable control plane



Balus, Stiliadis et al.  Expires April 22, 2013               [Page 12]


Internet-Draft        Federated SDN controllers            October 2012


   may be required or if a BGP-based operational environment is the
   preferred option.

7.2. Hypervisor NVE processing

   The proposed solution is designed to minimize the required
   processing on the hypervisor NVEs. Complex control-plane policy and
   routing functions are delegated to the Controller that can be
   deployed in dedicated processors. Hypervisors only run the Openflow
   agents required to download the FIB entries and process events in
   some models, as well as perform the NVo3 forwarding function.

7.3. Interoperating with non-NVO3 domains

   The routing module of the Controller enables interoperability with
   existing non-NVO3 VPN domains where the whole Controller domain
   appears as just another PE to those domains. A VPN interworking
   function may need, depending on the encapsulation used, to be
   implemented in the data plane of the gateway NVEs to existing non-
   NVO3 VPN domains.

7.4. VM Mobility

   VM mobility is handled by a tight interaction between the cloud
   management system and the Controller. When a VM move is
   instantiated, the cloud management system will notify the Controller
   about the move. The VM will transition from a running state in one
   hypervisor to a running state in another hypervisor. The transition
   of these events will instruct the Controller to properly update the
   routes in each of the NVEs.

7.5. Openflow and Open vSwitch

   Utilizing Openflow as the basic mechanism for controller to NVE
   communication offers several advantages:

   1. Openflow is a binary protocol that is optimized for fast FIB
     updates. Since it relies on a binary format it minimizes the
     amount of data that needs to be transferred and the required
     processing to provide increased scalability.
   2. Openflow is already implemented in multiple hypervisors and is
     deployed in some large cloud environments. The current
     specification supports L2, L3 service FIB and flexible flow
     definition providing a good starting point for future extensions.

   From a practical and deployment perspective, the Open vSwitch [OVS]
   is already part of the latest Linux kernel and most major Linux


Balus, Stiliadis et al.  Expires April 22, 2013               [Page 13]


Internet-Draft        Federated SDN controllers            October 2012


   distributions for the major hypervisors (KVM and Xen). Minimizing
   the new protocols that need to be deployed into servers and relying
   on the existing hypervisor capabilities can significantly simplify
   and accelerate the adoption of NVO3 technologies.

8. Security Considerations

   The tenant to overlay mapping function can introduce significant
   security risks if appropriate protocols are not used that can
   support mutual authentication. Proper configuration of Controller
   and NVEs, and a mutual authentication mechanism is required. The
   Openflow specification includes a TLS option for the controller to
   NVE communication that can address the mutual authentication
   requirement.

   No other new security issues are introduced beyond those described
   already in the related L2VPN and L3VPN RFCs.

9. IANA Considerations

   IANA does not need to take any action for this draft.

10. References

10.1. Normative References

   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
             Requirement Levels", BCP 14, RFC 2119, March 1997.

   [BGP-VPN] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
             Networks (VPNs)", RFC 4364, February 2006.

   [BGP Graceful-Restart] Sangli, S. et al, "Graceful Restart Mechanism
             for BGP", RFC 4724, January 2007

10.2. Informative References

   [NVO3-FWK] Lasserre, M et.al "Framework for DC Network
   Virtualization", draft-ietf-nvo3-framework (work in progress)

   [NVO3-CPREQ] Kreeger, L. et al, "Network Virtualization Overlay
             Control Protocol Requirements", draft-kreeger-nvo3-
             overlay-cp (work in progress)

   [Server2NVE] Kompella, K. et al, "Using signaling to simplify
   network virtualization provisioning", draft-kompella-nv03-server2nve
   (work in progress)


Balus, Stiliadis et al.  Expires April 22, 2013               [Page 14]


Internet-Draft        Federated SDN controllers            October 2012


   [EVPN-NVO3] Drake, J. et al, "A Control Plane for Network
             Virtualized Overlays", draft-drake-nvo3-evpn-control-plane
             (work in progress)

   [Openflow] Openflow Switch Specification,
   http://www.opennetworking.org

   [ONF] Open Networking Foundation https://www.opennetworking.org/

   [Openstack] Openstack cloud software, http://www.openstack.org

   [Cloudstack] Cloudstack, http://www.cloudstack.org

   [OVS] Open vSwitch, http://www.openvswitch.org

   [BGP-EVPN] Aggarwal, R. et al, "BGP MPLS Based Ethernet VPN", draft-
             ietf-l2vpn-evpn (work in progress)

11. Acknowledgments

   In addition to the authors the following people have contributed to
   this document: Thomas Morin, Rotem Salomonovitch.

   This document was prepared using 2-Word-v2.0.template.dot.

























Balus, Stiliadis et al.  Expires April 22, 2013               [Page 15]


Internet-Draft        Federated SDN controllers            October 2012


Authors' Addresses

   Florin Balus
   Nuage Networks
   805 E. Middlefield Road
   Mountain View, CA, USA 94043
   Email: florin@nuagenetworks.net

   Dimitri Stiliadis
   Nuage Networks
   805 E. Middlefield Road
   Mountain View, CA, USA 94043
   Email: dimitri@nuagenetworks.net

   Nabil Bitar
   Verizon
   40 Sylvan Road
   Waltham, MA 02145
   Email: nabil.bitar@verizon.com

   Kenichi Ogaki
   KDDI
   3-10-10 Iidabashi,
   Chiyoda-ku Tokyo, 102-8460 JAPAN
   Email: ke-oogaki@kddi.com

   Marc Lasserre
   Alcatel-Lucent
   Email: marc.lasserre@alcatel-lucent.com

   Wim Henderickx
   Alcatel-Lucent
   Email: wim.henderickx@alcatel-lucent.com
















Balus, Stiliadis et al.  Expires April 22, 2013               [Page 16]