[Search] [txt|pdf|bibtex] [Tracker] [Email] [Nits]

Versions: 00 01 02 03 04 05 06 07                                       
Network Working Group                                         L. Dunbar
Internet Draft                                                   Huawei
Intended status: Informational                               Mehmet Toy
Expires: January 2019                                           Verizon
                                                           May 31, 2018



           Segment routing for SD-WAN paths over hybrid networks
               draft-dunbar-sr-sdwan-over-hybrid-networks-00

Abstract

   This document describes a method for end-to-end (E2E) SD-WAN paths
   (most likely encrypted) to traverse specific list of network
   segments, some of which may be legacy networks that do not support
   SR, to achieve the desired optimal E2E quality.

   The method described in this draft uses the principle of segment
   routing to enforce the SD-WAN paths' head ends selecting paths
   traversing through a list of specific nodes of multiple network
   segments without requiring the nodes in each network segments to
   have the intelligence (or maintaining states) of selecting next hop
   or next domain. Those networks over which the SD-WAN path traverse
   may have different data planes (some maybe IPv4, others maybe IPv6
   or MPLS) and can be owned & managed by different network service
   providers (SP).

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79. This document may not be modified,
   and derivative works of it may not be created, except to publish it
   as an RFC and to translate it into languages other than English.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that




xxx, et al.           Expires November 31, 2018                [Page 1]


Internet-Draft       SD-WAN over multiple domains              May 2018


   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other documents
   at any time.  It is inappropriate to use Internet-Drafts as
   reference material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html

   This Internet-Draft will expire on November 1, 2018.

Copyright Notice

   Copyright (c) 2018 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with
   respect to this document. Code Components extracted from this
   document must include Simplified BSD License text as described in
   Section 4.e of the Trust Legal Provisions and are provided without
   warranty as described in the Simplified BSD License.

Table of Contents


   1. Introduction...................................................3
   2. Definition of terms............................................4
   3. Key Use Cases..................................................5
      3.1. SD-WAN Path over LTE network and SR Domain................5
      3.2. SD-WAN path over public Internet and SR domain............6
      3.3. How SR is useful for those Use Cases......................7


Dunbar, et al.          Expires June 31, 2018                  [Page 2]


Internet-Draft       SD-WAN over multiple domains              May 2018


   4. Mechanism for SD-WAN path over one SR Domain and public Internet
   ..................................................................8
      4.1. Controller Delivers SID Stack to SD-WAN end-points........9
      4.2. Using GRE Key to Differentiate Flows.....................11
      4.3. Using UDP Source Port Number to Differentiate Flows......12
      4.4. GRE Header Extension.....................................15
   5. SD-WAN path over multiple SP managed domains..................15
      5.1. When Both SP domains support SR..........................17
      5.2. When SP-2 does not support SR............................17
      5.3. When SP-1 and SP-2 don't want to share network information18
      5.4. TLV to pass Metadata through SRv6 Domain.................18
   6. Security Considerations.......................................19
   7. IANA Considerations...........................................20
   8. References....................................................20
      8.1. Normative References.....................................20
      8.2. Informative References...................................21
   9. Acknowledgments...............................................22

1. Introduction

   This document describes a method of using the principle of segment
   routing,  i.e. App driven Head End paths selection, to optimize SD-
   WAN services across multiple network segments, some of which are
   legacy networks and don't support SR, to achieve desired E2E path
   quality.

   SD-WAN, as described by ONUG (Open Network User Group), is about
   pooling WAN bandwidth from n service providers to get better WAN
   bandwidth management, visibility & control.

   Throughout this document, the term "Classic SD-WAN" refers to a pair
   of CPEs in two locations aggregating n Service Providers' paths,
   such as MPLS Paths and public internet paths. [SR-SD-WAN] describes
   using explicit routes within the SRv6 or SR-MPLS enabled networks to
   reach the desired quality for SD-WAN paths over the SRv6 or SR-MPLS
   enabled networks respectively.

   Another way of using SD-WAN is for network service providers to
   extend its existing VPN to reach sites to which they do not have
   physical connections, with detailed use cases described in Section 3
   of this document. Under this scenario, the SD-WAN is a path laid


Dunbar, et al.          Expires June 31, 2018                  [Page 3]


Internet-Draft       SD-WAN over multiple domains              May 2018


   over multiple hybrid networks. This document focuses on this type of
   SD-WAN where a portion of SD-WAN path is over SR enabled networks
   and the other portion of the SD-WAN path is over other networks,
   such as legacy IPv4, LTE, etc. Under this scenario, the one or two
   endpoints of the SD-WAN path (e.g. the CPE devices, one or both) are
   not directly attached to PEs of a SR domain.

   The Goal is to place a large portion of the SD-WAN path over a
   provider VPN to reach desired transport quality or making the SD-WAN
   path traversing specific ingress/egress PEs for the desired cost,
   quality or other reasons.

2. Definition of terms



   Cloud DC:   Off-Premise Data Centers that usually host applications
               and workload owned by different organizations or
               tenants.



   Controller: Used interchangeably with SD-WAN controller to manage
               SD-WAN overlay path creation/deletion and monitoring the
               path conditions between two or more sites.

   DMVPN:      Dynamic Multipoint Virtual Private Network. DMVPN is a
               secure network that exchanges data between sites without
               needing to pass traffic through an organization's
               headquarter virtual private network (VPN) server or
               router.

   Heterogeneous Cloud: applications & workloads split among Cloud DCs
               owned & managed by different operators.

   Hybrid Cloud: applications & workloads split between on-premises
               Data centers and Cloud DCs. In this document Hybrid
               Cloud also include heterogeneous cloud as well.



Dunbar, et al.          Expires June 31, 2018                  [Page 4]


Internet-Draft       SD-WAN over multiple domains              May 2018


   SD-WAN:     Software Defined Wide Area Network, which can mean many
               different things. In this document, "SD-WAN" refers to
               the solutions specified by ONUG (Open Network User
               Group), https://www.onug.net/software-defined-wide-area-
               network-sd-wan/, which is about pooling WAN bandwidth
               from n service providers to get better WAN bandwidth
               management, visibility & control.

   SP:         Network Service Provider

   SR:         Segment Routing

   SR Domain:  A domain that supports Segment Routing

   VPC:        Virtual Private Cloud. A service offered by many Cloud
               DC operators to allocate a logically isolated cloud
               resources, including compute, networking and storage.


3. Key Use Cases

3.1. SD-WAN Path over LTE network and SR Domain

   MEF Cloud Service Architecture [MEF-Cloud] describes a use case of
   network operators needing to use SD-WAN over LTE for the last
   segments that they do not have physical infrastructure, as shown
   below:

















Dunbar, et al.          Expires June 31, 2018                  [Page 5]


Internet-Draft       SD-WAN over multiple domains              May 2018


                                SD-WAN Overlay
                 /-------------------------------------------/
                /    [A]-----[E1]***********[E2]--------[Z] /
               /               *              :            /
              /                *******        :           /
             /-----------------------*--------:----------/
                                LTE  *        :
                              Access *        :
                                 *****        ..*****
   +------------+            /---*---*---------:----*--------/
   |     SP     |        :  /    *   *         :    *       /
   |  Underlay  |        : /   [C1]-[C3]------[C4]-[C6]    /
   +------------+        :/       \           /    /      /
                         :         \      /--/----/      /
                        /:          \    /              /
                       / :...........[C2]..........................:
                      /-------------------------------/
          Figure 1: SD-WAN end points are attached to VPN via LTE



3.2. SD-WAN path over public Internet and SR domain

   More and more enterprises are moving their workloads/Apps to cloud
   DCs that are geographically close to their end users to improve end-
   to-end latency & overall user experience, or to comply with local
   data protection regulations. Conversely, workloads/Apps in those
   Cloud DCs can be easily shutdown when their end users' geographic
   base changes.

   Because of the ephemeral property of the selected Cloud DCs, an
   enterprise or its network service provider may not have the direct
   links to the Cloud DCs that are optimal for hosting the enterprise's
   specific workloads/Apps. Under those circumstances, SD-WAN is a very
   flexible choice to interconnect the enterprise on-premises data
   centers or branch offices to its desired Cloud DCs.

   However, SD-WAN paths over public internet can have unpredictable
   performance, especially over long distances and cross state/country
   boundaries. Therefore, it is highly desirable to place as much as


Dunbar, et al.          Expires June 31, 2018                  [Page 6]


Internet-Draft       SD-WAN over multiple domains              May 2018


   possible the portion of SD-WAN paths over service provider VPN (e.g.
   enterprise's existing VPN) that have guaranteed SLA to minimize the
   distance/segments over public internet.

   Under this scenario, one or both of the SD-WAN end points may not
   directly attached to the PEs of a SR Domain.


3.3. How SR is useful for those Use Cases

   First, it is assumed that the SD-WAN Controller is capable of
   computing optimal paths between two end-points (e.g. E1<->E2 in the
   Figure 2), either by communicating with the SR Domain
   controller/management-system, or by other methods which is out of
   the scope of this document.

   The SR domain must have a set of PEs that have at least one port
   facing the external networks (such as the public internet).

   Under this circumstance, SD-WAN end-points usually can reach
   multiple PEs.

   In the diagram below, E1 <-> E2 SD-WAN (most likely IPsec tunnel)
   path can traverse C1 <-> C4, C1<->C6, C3<->C6, or C3<->C4 within the
   VPN. There are many flows (by different Apps) between E1 <-> E2.
   Some flows may need to traverse C1<->C4, others may need to traverse
   C3<->C6 or other segments within the VPN, which are determined by
   the SD-WAN controller based on the characteristics & need of the
   Apps, such as cost, available bandwidth, latency, or special
   functions only available at specific locations, etc.

   Even with the same ingress/egress, some flows may need different
   segments across the SR Domain. It is not practical, or even
   possible, for PEs (e.g. C1, C2, C3 in this example) to differentiate
   which flows should egress C4 or C6 where both C4&C6 can reach E2.

   Segment Routing can easily force the path to traverse the explicit
   egress node (C4 or C6), or explicit segments through the SR Domain
   based on the SLA requested by the SD-WAN head end nodes.







Dunbar, et al.          Expires June 31, 2018                  [Page 7]


Internet-Draft       SD-WAN over multiple domains              May 2018


   +------------+                                       /-----------/
   |   SDWAN    |                                      /           /
   |  Control   |                                     / [SDWAN-C]...
   +---+--------+                                    /           / :
       |                                            /-----------/  :
       |                        SD-WAN Overlay                     :
       |          /-------------------------------------------/    :
       |         /                                           /     :
       |        /    [A]-----[E1]***********[E2]--------[Z] /      :
       |       /               *              :            /       :
       |      /                *******        :           /        :
       |     /-----------------------*--------:----------/         :
       |                             *        :                    :
       |                             *        :                    :
       |              /----------/   *        :                    :
   +---+--------+    /          /    *        :                    :
   |     SP     |   /  [SR-C]  /     *        :                    :
   |  Control   |  /     :    /      *        :                    :
   +------------+ /------:---/   *****        ..*****              :
                         :       *   *         :    *              :
                         :       *   *         :    *              :
   +------------+        :   /---*---*---------:----*--------/     :
   |     SP     |        :  /    *   *         :    *       /      :
   |  Underlay  |        : /   [C1]-[C3]------[C4]-[C6]    /       :
   +------------+        :/       \           /    /      /        :
                         :         \      /--/----/      /         :
                        /:          \    /              /          :
                       / :...........[C2]..........................:
                      /                               /
                     /-------------------------------/

   Figure 2: SDWAN end points not directly attached to PEs of SR Domain


4. Mechanism for SD-WAN path over one SR Domain and public Internet

   There may be two approaches here:
   1) Controller installs the entire SID stack at E1.



Dunbar, et al.          Expires June 31, 2018                  [Page 8]


Internet-Draft       SD-WAN over multiple domains              May 2018


   2) Controller delivers to E1 a "Key" that the SR ingress PE can use
   to map to the SID stack for the packets arriving at the SR Ingress
   PE. Section 4.2 & 4.3 will describe how the "Key" is carried by the
   packets.

   The Approach 1) requires less processing at the SR Ingress PE nodes,
   but only works if the remote CPEs are in the same Administrative
   domain as the SR domain. SR domain usually is not willing to expose
   its internal binding SIDs to devices in different administration
   domains. This approach also requires more changes to SD-WAN end
   nodes and need more header bytes added to the packets when                            rd        traversing through 3  party internet. Some SD-WAN nodes might not be
   capable of supporting encapsulating packets with the SID stack.

   The Approach 2) above requires SR Ingress PE nodes to map the "Key"
   to the SID Stack and prepend the SID stack to the packets (Same
   processing for other traffic except the mapping is from the received
   "Key" carried in the payload).



   4.1. Controller Delivers SID Stack to SD-WAN end-points

   This approach is straightforward.

         E1  -------------------------- > SD-WAN controller
            request for a SD-WAN path E1<->E2 with a specific SLA

         E1  <--------------------------  SD-WAN controller
            Reply with the Ingress PE Node ID or address
            & the Binding SID.

   Here is the packet header for SD-WAN Source Node to prepend to the
   payload:

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

      IPv4 Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Version|  IHL  |Type of Service|          Total Length         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


Dunbar, et al.          Expires June 31, 2018                  [Page 9]


Internet-Draft       SD-WAN over multiple domains              May 2018


      |         Identification        |Flags|      Fragment Offset    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Time to Live | Prot.=17(UDP) |          Header Checksum      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |               SD-WAN Source IPv4 Address                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |              SR Ingress PE IPv4 Address                       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      UDP Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Source Port =              |  Dest. Port = 4754/4755       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           UDP Length          |        UDP Checksum           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      GRE Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |C| |K|S| Reserved0       | Ver |         Protocol Type         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |      Checksum (optional)      |       Reserved1 (Optional)    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                         Key (optional)                        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                 Sequence Number (optional)                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


   To traverse SRv6 domain, SRv6 Header is appended after the GRE
   header [SRv6-SRH]:



   To traverse MPLS-SR domain, a stack of MPLS labels is appended after
   GRE Header [MPLS-SR].








Dunbar, et al.          Expires June 31, 2018                 [Page 10]


Internet-Draft       SD-WAN over multiple domains              May 2018


   4.2. Using GRE Key to Differentiate Flows

   Different flows between SD-WAN end-points (E1<->E2 in the figure
   above) may have different desired route through the SR Domain, or
   need different egress PEs based on cost, performance or other
   factors. The SR Domain ingress will use different SID through the SR
   Domain. One way for the SD-WAN end-points not directly attached to
   SR ingress PE is via the GRE Key.

   We assume that the SD-WAN Controller can determine which ingress PE
   can lead to the optimal path between E1<->E2. It is beyond the scope
   of this document on how SD-WAN controller computes the paths and how
   & what SD-WAN controller communicates with the SR Domain controller.



   Here is the sequence of the flow:



         E1  -------------------------- > SD-WAN controller
            request for a SD-WAN path E1<->E2 with a specific SLA

         E1  <--------------------------  SD-WAN controller
            Reply with the Ingress PE Node ID or address
            & the GRE Key.


   Note: the GRE key from the SD-WAN controller is for the ingress PE
   to correlate desired Path with the list of SIDs to prepend the
   packet across the SR domain.

   When SD-WAN Controller get the E1<->E2 path request, it will
   communicate with the VPN Controller to get the optimal Ingress PE
   Node ID (or IP address) and the GRE key to encapsulate the original
   packets between E1 <-> E2 (assuming IPsec Tunnel mode is used).

   Upon receiving the GRE encapsulated packets, the provider ingress
   Edge C1/C3 decapsulates the outer GRE tunnel header, use the GRE key
   to map to the pre-defined (by the network controller) Binding SIDs,
   prepend the Binding SIDs to the packets, and forward its desired
   paths across the provider VPN.




Dunbar, et al.          Expires June 31, 2018                 [Page 11]


Internet-Draft       SD-WAN over multiple domains              May 2018


   Depending on how the SD-WAN path destination can be reached by the
   egress PE, the egress PE has different processing procedure:

      - If the destination of the SD-WAN path is directly attached to the
        egress VPN PE node, the egress VPN PE decapsulates SR header and
        forward the packets to SD-WAN path destination node, such as the E2
        in the figure above.
      - If the destination of the SD-WAN path is IP reachable via IPv4
        network from the egress VPN PE node, the egress VPN PE node
        decapsulates SR header and forward the packets to SD-WAN path
        destination node via its internet facing port to the SD-WAN path
        destination (i.e. the E2 node in the figure above).
      - If the SD-WAN path is traversing multiple domains owned by different
        network operators, the egress PE processing is described in the next
        session.


   4.3. Using UDP Source Port Number to Differentiate Flows

   [RFC8086] describes how to use GRE-in-UDP source port number as
   entropy for better ECMP performance. When the remotely attached CPEs
   is within very close proximity to the PEs, e.g. only one or two
   hopes away like in LTE access, there is less issue if ECMP put all
   flows with same traffic classifier into one path.  Then, those UDP
   numbers can also be used as a key to SR PE nodes to map to the
   appropriate SID to the packets.

   Same as RFC8086, UDP source port values used as a key for SR PEs to
   map to appropriate SIDs SHOULD be chosen from the ephemeral port
   range (49152-65535) [RFC8085].

   The GRE-in-UDP encapsulation format contains a UDP header [RFC768]
   and a GRE header [RFC2890].  The format is shown as follow
   (presented in bit order):

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

      IPv4 Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Version|  IHL  |Type of Service|          Total Length         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+



Dunbar, et al.          Expires June 31, 2018                 [Page 12]


Internet-Draft       SD-WAN over multiple domains              May 2018


      |         Identification        |Flags|      Fragment Offset    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Time to Live | Prot.=17(UDP) |          Header Checksum      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |               SD-WAN Source IPv4 Address                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |              SR Ingress PE IPv4 Address                       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      UDP Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Source Port = SIDs key Value |  Dest. Port = 4754/4755       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           UDP Length          |        UDP Checksum           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      GRE Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |C| |K|S| Reserved0       | Ver |         Protocol Type         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |      Checksum (optional)      |       Reserved1 (Optional)    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                         Key (optional)                        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                 Sequence Number (optional)                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                    Figure 3: UDP + GRE Headers in IPv4




   Here is the GRE Header for IPv6 network, i.e. the SD-WAN Source SD-
   WAN Destination, and SR PEs are all in IPv6 domain:

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

      IPv6 Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


Dunbar, et al.          Expires June 31, 2018                 [Page 13]


Internet-Draft       SD-WAN over multiple domains              May 2018


      |Version| Traffic Class |           Flow Label                  |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |         Payload Length        | NxtHdr=17(UDP)|   Hop Limit   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      +                                                               +
      |                                                               |
      +                     SD-WAN Source IPv6 Address                +
      |                                                               |
      +                                                               +
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      +                                                               +
      |                                                               |
      +               SR Domain Ingress PE IPv6 Address               +
      |                                                               |
      +                                                               +
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      UDP Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Source Port = SIDs key value |  Dest. Port = 4754/4755       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           UDP Length          |        UDP Checksum           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      GRE Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |C| |K|S| Reserved0       | Ver |         Protocol Type         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |      Checksum (optional)      |       Reserved1 (Optional)    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                         Key (optional)                        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                 Sequence Number (optional)                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+




Dunbar, et al.          Expires June 31, 2018                 [Page 14]


Internet-Draft       SD-WAN over multiple domains              May 2018


                    Figure 4: GRE+UDP for IPv6




   4.4. GRE Header Extension

   A new protocol type can be added to the GRE header [RFC2890] to make
   it easier for the SR PE to do the proper actions:

        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |C|       Reserved0       | Ver |         Protocol Type         |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |      Checksum (optional)      |       Reserved1 (Optional)    |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      The proposed GRE header will have the following format:

       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |C| |K|S| Reserved0       | Ver |         Protocol Type         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |      Checksum (optional)      |       Reserved1 (Optional)    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                         Key (optional)                        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                 Sequence Number (Optional)                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


   New protocol type (value to be assigned by IANA):
     UDP-Key: Using UDP source port value as a Key for SR Ingress PE to
   map to the appropriate SIDs.

     GRE-KEY: Using GRE Key value as a key for SR ingress PE to map to
   the appropriate SIDs


5. SD-WAN path over multiple SP managed domains
   The following figure shows a SD-WAN Path E1<->E2 over two SP domains
   which are interconnected by public internet.



Dunbar, et al.          Expires June 31, 2018                 [Page 15]


Internet-Draft       SD-WAN over multiple domains              May 2018


   +------------+
   |   SDWAN    |                                      /-----------/
   |  Control   |                                     / [SDWAN-C]...
   +---+--------+                                    /           / :
       |                                            /-----------/  :
       |                        SD-WAN Overlay                     :
       |          /-------------------------------------------/    :
       |         /                                           /     :
       |        /    [A]-----[E1]***********[E2]--------[Z] /      :
   +---+       /               *              :            /       :
   |   |      /                *******        :           /        :
   |   |     /-----------------------*--------:----------/         :
   |   |              /----------/   *        .. .. .. .. .. :.    :
   | +-+--------+    /          /    *                        :    :
   | |  SP-1    |   /  [SR-C]  /     *                         :   :
   | |Control   |  /     :    /      *                         :   :
   | +----------+ /------:---/   *****                         :   :
   | +----------+        :   /---*---*-----------------------/ :   :
   | |   SP-1   |        :  /    *   *                      /  :   :
   | |Underlay  |        : /   [C1]-[C3]------[C4]-[C6]    /   :   :
   | +----------+        :/       \           /*   /      /    :   :
   |                     :         \      /--/-*--/      /     :   :
   | +----------+       /:          \    /     *        /      :   :
   +-+  SP-2    |      / :...........[C2]    *         /       :   :
     |Control   |     /-------------**-----*----------/        :   :
     +----------+                  * *   *       .. .. .. .. ..:   :
                                  *    *         :                 :
   +------------+            /---*---*---------:-------------/     :
   |     SP-2   |           /    *   *         :            /      :
   |  Underlay  |          /   [D1]-[D4]------[D2]-[D5]    /       :
   +------------+         /       \           /    /      /        :
                         /         \      /--/----/      /         :
                        /           \    /              /          :
                       /             [D3]..........................:
                      /                               /
                     /-------------------------------/
            Figure 5: SD-WAN path over two different SP domains





Dunbar, et al.          Expires June 31, 2018                 [Page 16]


Internet-Draft       SD-WAN over multiple domains              May 2018


   Let's assume that the SP-1 domain's egress node for the SD-WAN path
   E1<->E2 is C2, which can reach D1 or D4 of SP-2 via public IP
   network (say IPv4 network).

   Let's also assume that the optimal route for some flows over SD-WAN
   path E1<->E2 are C1->C2->D1 and other flows are over C1->C2->D4 (out
   of the scope of this document on how the path is calculated).

   If SP-1 is SR enabled, the mechanism described in Section 4 is
   applicable to the SD-WAN path source node E1 and the SP-1's ingress
   PE (e.g. C1 or C3 in the figure).
   However, the processing at egress node might be different depending
   on how the SP-1's egress edges are connected to the SP-2's ingress
   edge nodes.


5.1. When Both SP domains support SR

   There may be three approaches here:

   1) Controller installs the entire SID stack at E1, and the SID list
   contains SID entries belong to both domains.

   2) Controller delivers to E1 the SID stack that only for the first
   domain, but delivers to C6 (egress node of first domain) the binding
   SID of the second domain.

   3) Controller delivers a "Key" to E1, which can be encoded as GRE
   KEY or represented by the Source UDP port of the GRE encapsulation,
   for Ingress PE of the first SR Domain to map to its own SID stack as
   described in Section 4. The first SR Domain will reserve the "Key"
   through its domain and pass the "Key" to the second SR domain. The
   second SR Domain Ingress node will use the same method to map the
   "Key" to its SID stack.



5.2. When SP-2 does not support SR

   Under this circumstance (which can be caused by SP-2 not supporting
   SR or not willing to share Binding SIDs to SP-1), if the packets
   arriving at SP-1 egress node C6 do not have any metadata indicating
   the types of encrypted payload, C6 does not really have much choice


Dunbar, et al.          Expires June 31, 2018                 [Page 17]


Internet-Draft       SD-WAN over multiple domains              May 2018


   other than simply forwarding the packets to E2 via public IP
   network. This way, the packets may or may not traverse through the
   SP-2 domain. If the distance between C6 and E2 is far, the quality
   of service can be unpredictable.

5.3. When SP-1 and SP-2 don't want to share network information

   If SP-1's ingress node C1 can include the GRE KEY it receives from
   E1 in the data packets' SR header, the SP-1's egress node can map
   the Key to the SP-2's Ingress node and encapsulate the data packet
   in a new GRE header destined towards the SP-2's Ingress node. Then
   the SP-2's Ingress node can follow the procedure described in the
   Section 4 to forward the data packets across its domain.

   If the first SR Domain does not support adding metadata to carry the
   "key" through its domain, the controller can deliver the "key" to
   SP-1's egress node the same time as it delivers the key to E1,
   knowing the SD-WAN path will need to traverse two domains with the
   second one does support SR but the two SPs don't want to exchange
   network information.

5.4. TLV to pass Metadata through SRv6 Domain

   If SP-1 is SRv6 based, the ingress node C1 can append a TLV to the
   end of the SR Header [SRv6-SRH] to carry the KEY it receives from
   E1[Dc1].

   The SP-1 egress node C6 can get the mapping between the KEYs and the
   Node-IDs (or Addresses) of the next domain's ingress edge node (i.e.
   D1 or D4 in the figure 3 above) from its network controller ahead of
   time.

     0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |      Type     |     Length    |          RESERVED             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Key ID (4 octets) from the GRE tunnel remote ingress node     |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Optional                                                     //
      | Node ID or address for the ingress node Next domain          //
      | Variable length (0~32 octets)                                //
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+




Dunbar, et al.          Expires June 31, 2018                 [Page 18]


Internet-Draft       SD-WAN over multiple domains              May 2018


   TYPE: (to be assigned by IANA) is to indicate the TLV is for
   carrying the flow identifier of the packet encoded by the SD-WAN
   source node.



   Upon receiving the packet, the egress node (C6) can

      - find the Node-ID (or the address) for the next domain's ingress
        node,
      - construct a GRE header with the Key received from the TLV above
        and the destination address from the mapping given by the
        controller,
      - encapsulate the GRE header to the data packet (which has
        decapsulated SR header),
      - and forward the packet to the public internet.










   6. Security Considerations

   Remotely attached CPEs might brought the following security risks:

   1) Potential DDoS attack to the PEs with ports facing internet.
     I.e. the PE resourced being attacked by unwanted traffic.
   2) Potential risk of provider VPN network bandwidth being stolen
     by the entities who spoofed the addresses of SD-WAN end nodes.

   To mitigate security risk of 1) above, it is absolutely necessary
   for PEs which accept remotely attached CPEs or simply have ports
   facing internet to enable Anti-DDoS feature to prevent major DDoS
   attack to those PEs.

   To mitigate the security risk of 2) above, RFC7510 defines the use
   of DTLS to authenticate and encrypt the RFC7510 encapsulation.


Dunbar, et al.          Expires June 31, 2018                 [Page 19]


Internet-Draft       SD-WAN over multiple domains              May 2018


   However, for the scenario of SD-WAN source node being remotely
   attached to PEs, using the method recommended by RFC7510  means the
   source node has to perform DTLS on top of the IPSec encryption
   between SD-WAN end points E1<->E2. This can be too processing heavy
   for the SD-WAN end nodes. In addition, if there are many SD-WAN
   flows to traverse through the ingress PE (e.g. C1, C2, C4 in the
   figure 1 above), heavy processing is required on the ingress PEs.

   Since the payload between E2<->E2 is already encrypted, the
   confidentiality of the payload is already ensured.  The network
   operators need to balance between how much they can tolerant some
   percentage of bandwidth being stolen and how much extra cost they
   are willing to pay for completely prevent any unpaid traffic
   traversing through its VPN networks. For operators who opt for lower
   cost ingress PEs and CPEs, but can tolerant some percentage of
   bandwidth being used by unpaid subscribers, a simple approach can be
   considered:

   - Embed a key in the packets, which can be changed periodically,
     like the digital signature used by a certificate authority or
     certification authority (CA).
   - The key can be encoded in the GRE Key field between SD-WAN end
     node and Ingress PE. Since GRE has 24 bits, some fixed bits
     can be used to represent the signature of paid subscribers.



    7. IANA Considerations

   This document requires new protocol type:

   Protocol type to be added to GRE header: SR_Route

8. References


8.1. Normative References
   [RFC2890]   G. Dommety  "Key and Sequence Number Extensions to GRE".
   Sep. 2000.





Dunbar, et al.          Expires June 31, 2018                 [Page 20]


Internet-Draft       SD-WAN over multiple domains              May 2018


    8.2. Informative References

   [RFC2735]   B. Fox, et al "NHRP Support for Virtual Private
   networks". Dec. 1999.

   [RFC8192] S. Hares, et al "Interface to Network Security Functions
             (I2NSF) Problem Statement and Use Cases", July 2017

    [ITU-T-X1036] ITU-T Recommendation X.1036, "Framework for creation,
             storage, distribution and enforcement of policies for
             network security", Nov 2007.

    [RFC6071] S. Frankel and S. Krishnan, "IP Security (IPsec) and
             Internet Key Exchange (IKE) Document Roadmap", Feb 2011.

   [RFC4364] E. Rosen and Y. Rekhter, "BGP/MPLS IP Virtual Private
             Networks (VPNs)", Feb 2006

   [RFC4664] L. Andersson and E. Rosen, "Framework for Layer 2 Virtual
             Private Networks (L2VPNs)", Sept 2006.

   [SR-SD-WAN] D. Dukes, et al, "SR for SDWAN: VPN with Underlay SLA",
             draft-dukes-sr-for-sdwan-00, in progress, Oct 2017

   [SRv6-SRH] S. Previdi, et al, "IPv6 Segment Routing Header (SRH)",
             draft-ietf-6man-segment-routing-header-13, in progress,
             April 2018.

   [MPLS-SR] A. Bashandy, et al, "Segment Routing with MPLS data
             plane", draft-ietf-spring-segment-routing-mpls-13, in
             progress, April 2018.

   [RFC7510] X. Xu, et al, "Encapsulating MPLS in UDP", April 2015.

   [RFC8086] L. Yong, et al, "GRE-in-UDP Encapsulation", March 2017.

   [MEF-Cloud] "Cloud Services Architecture Technical Specification",
             Work in progress, April 2018



Dunbar, et al.          Expires June 31, 2018                 [Page 21]


Internet-Draft       SD-WAN over multiple domains              May 2018


   9. Acknowledgments

   Many thanks to Dean Cheng and Jim Guichard for the discussion and
   contributions.










































Dunbar, et al.          Expires June 31, 2018                 [Page 22]


Internet-Draft       SD-WAN over multiple domains              May 2018


Authors' Addresses


   Linda Dunbar
   Huawei
   Email: Linda.Dunbar@huawei.com

   Mehmet Toy
   Verizon
   One Verizon Way
   Basking Ridge, NJ 07920
   Email: mehmet.toy@verizon.com

































Dunbar, et al.          Expires June 31, 2018                 [Page 23]