Network Working Group                                       K. Majumdar
Internet Draft                                                Microsoft
Intended status: Standard                                     L. Dunbar
Expires: November 31, 2023                                    Futurewei
                                                      V.Kasiviswanathan
                                                                  Arista
                                                           A. Ramchandra
                                                             Microsoft
                                                            May 31, 2023


                    Multi-segment SD-WAN via Cloud DCs
                   draft-dmk-rtgwg-multisegment-sdwan-00

Abstract
   The document describes the methods to optimize the stitching of
   multiple SD-WAN segments on transit nodes across Cloud DCs.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other documents
   at any time.  It is inappropriate to use Internet-Drafts as
   reference material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html

   This Internet-Draft will expire on Dec 31, 2023.







xxx, et al.           Expires November 31, 2023                [Page 1]


Internet-Draft           Multi-segment SD-WAN


Copyright Notice

   Copyright (c) 2023 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with
   respect to this document. Code Components extracted from this
   document must include Simplified BSD License text as described in
   Section 4.e of the Trust Legal Provisions and are provided without
   warranty as described in the Simplified BSD License.

Table of Contents

   1. Introduction...................................................3
   2. Conventions used in this document..............................3
   3. Use Cases......................................................4
      3.1. Multi-segment SD-WAN via Single Cloud GW..................4
      3.2. Multi-segment SD-WAN via Cloud Backbone...................5
      3.3. Why not use policy-based traffic steering.................6
   4. Data Plane encoding for SD-WAN Transit.........................7
      4.1. GENEVE header encoding....................................7
      4.2. Multi-Segment SD-WAN Option Class.........................7
      4.3. SD-WAN Endpoint Sub-TLV...................................8
      4.4. SD-WAN Tunnel Origin Endpoint Sub-TLV.....................9
      4.5. Egress GW Sub-TLV........................................10
      4.6. Include-Transit Sub-TLV..................................10
      4.7. Exclude-Transit Sub-TLV..................................10
   5. IPsec Flow through Cloud GWs Illustration.....................10
      5.1. Single Hop Cloud GW......................................11
      5.2. Multi-hop Transit GWs....................................12
      5.3. Data Authentication and Integrity Check by Cloud GW......14
   6. Illustration of Traffic from Private VPN to IPsec Tunnel......14
   7. Control Plane consideration...................................16
      7.1. Control Plane for CPEs...................................16
      7.2. Control Plane between CPEs and Cloud GWs.................16
   8. Observability Consideration...................................17
   9. Manageability Considerations..................................17
   10. Security Considerations......................................17
   11. IANA Considerations..........................................17
   12. References...................................................17


Dunbar, et al.           Expires Dec 31, 2023                  [Page 2]


Internet-Draft           Multi-segment SD-WAN


      12.1. Normative References....................................17
      12.2. Informative References..................................18
   13. Acknowledgments..............................................20

1. Introduction

   Cloud Providers support SD-WAN interconnecting enterprises' on-prem
                                                              rd        CPEs with the enterprises-owned applications/services and 3  party
   SASE services in cloud DCs. As described in [Net2Cloud], there are
   multiple options for enterprises to connect to Cloud DCs:

     - Direct Interconnect model,
     - Direct Interconnect model with enterprise's own virtual
        appliances in the Cloud,
     - Indirect Interconnect model via SD-WAN paths, and
     - Managed Hybrid WAN model using Enterprise's existing VPN
        connections.

   For the enterprise branches that have private VPN circuits
   interconnecting with a Cloud GW via IXP (Internet eXchange Point),
   the enterprise can extend into Cloud DC without having to set up
   IPsec paths between their on-prem CPEs and the Cloud GWs.

   This document describes a method for a Cloud DCs' gateway (GW) to
   connect multiple SD-WAN segments between the Cloud GW and the
   enterprise's CPEs without the Cloud GW decrypting and encrypting the
   payloads. By integration with Cloud Operators' gateways, enterprises
   can have advanced visibility through the Cloud Providers' global
   network topology, attachment level performance metrics, and
   telemetry data.

2. Conventions used in this document
   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

   The following acronyms and terms are used in this document:





Dunbar, et al.           Expires Dec 31, 2023                  [Page 3]


Internet-Draft           Multi-segment SD-WAN

                                                     rd        Cloud DC:   Off-Premises Data Center, managed by 3  party, that
               hosts applications, services, and workload for different
               organizations or tenants.

   CPE:        Customer (Edge) Premises Equipment.

   OnPrem:     On Premises data centers and branch offices.

   RR          Route Reflector.

   SD-WAN:     Software Defined Wide Area Network. In this document,
               "SD-WAN" refers to a policy-driven transporting of IP
               packets over multiple underlay networks for better WAN
               bandwidth management, visibility, and control.

   VPN         Virtual Private Network.


3. Use Cases

   3.1. Multi-segment SD-WAN via Single Cloud GW

   For enterprise branches that have established SD-WAN paths to a
   Cloud GW for accessing Cloud services, the Cloud GW can be utilized
   to connect those branches, as shown in Figure 1. Here are some
   reasons for connecting those branches via Cloud GW:
  - The public internet among those branches might have limited
     bandwidth, unpredictable connection, or be prone to cyber-attacks.
     In comparison, the network paths from CPEs to the Cloud GW have
     more reliable connections and are constantly monitored by
     sophisticated network functions.
  - It is easier to utilize Cloud based security functions, such as
     Firewall, DDoS, etc., to apply consistent policy enforcement for
     workloads/services to the Cloud and across the branches.
  - Cloud-based tools and SaaS can be easily utilized to collect and
     analyze the threat of traffic.








Dunbar, et al.           Expires Dec 31, 2023                  [Page 4]


Internet-Draft           Multi-segment SD-WAN


                          (^^^^^^^^^^^^)
                        (     Cloud     )
                        ( +----+  +----+  )
                 + -----(-|Edge|  + GW |  )
         Direct  |      ( +----+  +/--\+  )
        Connect  |        (^^^^^^^/^^^^\^)
               {-+---}           /      \  SD-WAN Path CPE<->GW
               { VPN }          /        \
               {-+---}         /          IPsec Tunnel
                 +-------+----/------+    \
                         |   /       |     \
                        ++--/+       |    +-\--+
                        |CPE1|       +----+CPE2|
                        +----+            +----+
       Client Route: 11.1.1.x             10.1.1.x
                     21.1.1.x             20.1.1.x
                                          30.1.1.x
   Figure 1 Multi-Segment SD-WAN stitching via a Cloud GW


3.2. Multi-segment SD-WAN via Cloud Backbone

   For geographic faraway enterprise branches that have established SD-
   WAN paths to their corresponding Cloud GWs to access Cloud services
   in different geographic locations, the Cloud backbone can connect
   those branches, as shown in Figure 2. The reasons to utilize the
   Cloud Backbone to interconnect those branches are similar to
   interconnecting multiple branches via a single Cloud GW described in
   the previous section.















Dunbar, et al.           Expires Dec 31, 2023                  [Page 5]


Internet-Draft           Multi-segment SD-WAN


                       (^^^^^^^^^^^^^^^)
                      (      Cloud      )
                      ( +----+  +----+  )               +-----+
                 + ---(-|Edge|==| GW1|=================== GW2 |
         Direct  |    ( +----+  +/--\+  )               +--|--+
        Connect  |      (^^^^^^^/^^^^\^)                   |
               {-+---}         /      \                    |
               { VPN }        /        \                 +-----+
               {-+---}       /          IPsec Tunnel     |CPE10|
                 +-------+--/--------+   \               +-----+
                         | /         |    \              10.2.1.x
                        ++/--+       |    +\---+         20.2.1.x
                        |CPE1|       +----+CPE2|         30.2.1.x
                        +----+            +----+
       Client Route: 11.1.1.x             10.1.1.x
                     21.1.1.x             20.1.1.x
                                          30.1.1.x

        Figure 2 Multi-Segment SD-WAN Stitching via Cloud Backbone



3.3. Why not use policy-based traffic steering

     There are many well-developed methods, such as SRv6 or MPLS-TE, to
     steer traffic through specific nodes. Those traffic steering
     methods are effective when the entire network domain is under one
     administrative control.

     However, the traffic from on-prem CPEs to Cloud GWs via the public
     internet can only be forwarded based on the packets' destination
     addresses.

     SD-WAN allows for the setup of multiple links (paths) from the
     same SD-WAN branch CPE to a Cloud GW; each link represents a dual
     tunnel connection from a unique public IP of the SD-WAN CPE to two
     different instances of Cloud GW. Using Cloud GW to interconnect
     those on-prem CPEs eliminates the need to manage the multiple
     ISPs' links/paths between the CPEs.










Dunbar, et al.           Expires Dec 31, 2023                  [Page 6]


Internet-Draft           Multi-segment SD-WAN


4. Data Plane encoding for SD-WAN Transit

   As the GENEVE Encapsulation [RFC8926] is supported by most Cloud
   Service Providers, GENEVE is chosen as the encapsulation header to
   steer SD-WAN flows among CPEs through the specified Cloud GWs.

4.1. GENEVE header encoding

   Geneve header shown below is specified by RFC8926:

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Ver|  Opt Len  |O|C|    Rsvd.  |          Protocol Type        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |        Virtual Network Identifier (VNI)       |    Reserved   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~                    Variable-Length Options                    ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


   VNI (virtual network Identifier) is used to represent the Customer
   Identifier.

   The Protocol Type (16 bits) = 50 (ESP) [RFC4303] indicates that
   IPsec ESP encapsulated data are appended at the end of the GENEVE
   header.

4.2. Multi-Segment SD-WAN Option Class

   Need IANA to assign a new GENEVE Option Class dedicated for Multi-
   segment SD-WAN.
















Dunbar, et al.           Expires Dec 31, 2023                  [Page 7]


Internet-Draft           Multi-segment SD-WAN


    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | multi-seg-SD-WAN Option Class |      Type     |R|R|R| Length  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~                  SD-WAN Tunnel Endpoint Sub-TLV               ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~          Optional SD-WAN Tunnel Originator Sub-TLV            ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~          Optional Egress GW Sub-TLV                           ~
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   //                                                             //
   //         Optional Type Length Value objects (variable)       //
   //                                                             //
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


   Type indicates the various types of multi-segment SD-WAN.

     Type = 1: Single Hop Transit SD-WAN

     Type = 2: Multi-Hop Transit SD-WAN with explicitly specified
     egress Cloud GW.

     Type = 3: Multi-Hop Transit SD-WAN without specified egress Cloud
     GW.



4.3. SD-WAN Endpoint Sub-TLV

   The SD-WAN Endpoint sub-TLV indicates the destination CPE of the
   IPsec Tunnel.

   As shown in Figure 1, for the SD-WAN IPsec SA from CPE1 to CPE2, the
   Tunnel Endpoint Sub-TLV of the Geneve Header has the CPE2's IP
   address.






Dunbar, et al.           Expires Dec 31, 2023                  [Page 8]


Internet-Draft           Multi-segment SD-WAN


    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |SD-WAN Endpoint| length        |   Reserved    | TTL          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | SD-WAN Dst Addr Family        | Address                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ (variable)                    +
   ~                                                               ~
   |    SD-WAN end point Address
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   TTL is set by the SD-WAN Tunnel Originator, e.g., CPE1. Each transit node
   or transit region/zone (visible to the CPEs) can decrement the TTL so that
   the destination CPE can know the number of logical transit nodes (cloud
   regions or zones) the packet has traversed. Enterprises can also use TTL to
   set the maximum transit nodes/regions the packets traverse.


4.4. SD-WAN Tunnel Origin Endpoint Sub-TLV

   The SD-WAN Tunnel Origin Endpoint sub-TLV indicates the originating
   CPE of the IPsec Tunnel.

   As shown in Figure 1, for the SD-WAN IPsec SA from CPE1 to CPE2, the
   Tunnel Origin Endpoint Sub-TLV of the Geneve Header indicates CPE1's
   address.

    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |SDWAN OriginEnd| length        |   reserved    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | SD-WAN Org Addr Family        | Address                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ (variable)                    +
   ~                                                               ~
   |    SD-WAN Tunnel Origin Endpoint Address                     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


   The Tunnel Origin Endpoint Sub-TLV in the GENEVE header can assist
   Cloud transit nodes in applying appropriate policies when forwarding
   the packet. However, including the Tunnel Origin Endpoint Sub-TLV in
   the GENEVE header is optional.







Dunbar, et al.           Expires Dec 31, 2023                  [Page 9]


Internet-Draft           Multi-segment SD-WAN


4.5. Egress GW Sub-TLV

   For the multi-segment SD-WAN via Cloud Backbone scenario, the
   originator CPE can use the Egress GW Sub-TLV to specify the Egress
   Cloud GW for reaching the destination CPE.

    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |SDWAN EgressGW | length        |   reserved    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | Egress GW Addr Family         | Address                       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ (variable)                    +
   ~                                                               ~
   |           Egress GW Address                                  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


4.6. Include-Transit Sub-TLV

   Include-Transit Sub-TLV is an optional Sub-TLV for explicitly
   including a list of Cloud Availability Regions or Zones for reasons
   like:

  - Those regions have certain OAM and security functions for the
     improved visibility.
  -  To comply with regulations, etc.

 4.7. Exclude-Transit Sub-TLV

   Exclude-Transit Sub-TLV is an optional Sub-TLV for explicitly
   excluding a list of Cloud Availability Regions or Zones for reasons
   like

       o To comply with regulations
       o To avoid regions that impose certain risks.


5. IPsec Flow through Cloud GWs Illustration

   This section illustrates Cloud GWs connecting traffic flow carried
   by the IPsec tunnels.






Dunbar, et al.           Expires Dec 31, 2023                 [Page 10]


Internet-Draft           Multi-segment SD-WAN


5.1. Single Hop Cloud GW

     Assuming that all CPEs are under one administrative control (e.g.,
     iBGP).

     Using Figure 1 as an example:

       - There is a bidirectional IPsec tunnel between CPE1 and Cloud
          GW; with IPsec SA1 for the traffic from the CPE1 to the
          Cloud-GW; and IPsec SA2 for the traffic from the Cloud-GW to
          the CPE1.
       - There is a bidirectional IPsec tunnel between CPE2 and Cloud
          GW; with IPsec SA3 for the traffic from the CPE2 to the
          Cloud-GW; and IPsec SA4 for the traffic from the Cloud-GW to
          the CPE2.
       - All the CPEs are under one iBGP administrative domain, with a
          Route Reflector (RR) as their controller. The CPEs notify
          their peers of their corresponding Cloud GW addresses (which
          is out of the scope of this document).

     When 11.1.1.x and 10.1.1.x need to communicate with each other,
     CPE1 and CPE2 establish a bidirectional IPsec Tunnel, with SA5 for
     the traffic from CPE1 to CPE2 and SA6 for the traffic from CPE2 to
     CPE1. Assume the IPsec ESP Tunnel Mode is used. A packet from
     11.1.1.1 to 10.1.1.2 has the following outer header:






















Dunbar, et al.           Expires Dec 31, 2023                 [Page 11]


Internet-Draft           Multi-segment SD-WAN


     Outer IP header:
         +---------------------------+
         |    protocol = 17(UDP)     |
         |    src = CPE1             |
         |    dst = Cloud GW         |
         +---------------------------+
         |  Source Port =xxxx        |
         |  Dst Port = 6081 (GENEVE) |
         +===========================+
         | GENEVE Header             |
         | multi-seg-SD-WAN Option   |
         |GENEVE Proto = 50 (ESP)    |
         +- - --  -- - - --      - --+
         |SD-WAN EndPt SubTLV (CPE2) |
         +---------------------------+  < ----------+
         |SPI(Security Parameter Idx)|        Authenticated
         +---------------------------+              |
         |    sequence number        |              |
         +---------------------------+   <-+        |
         | payload IP header:        |     |        |
         |  src =  11.1.1.1          |     |        |
         |  dst =  10.1.1.2          |     |        |
         +---------------------------+  Encrypted   |
         |   TCP header +            |     |        |
         ~    payload (variable)     ~     |        |
         |                           |     |        |
         +===========================+   <-+ -------+
         |   Authentication Data     |
         +---------------------------+



5.2. Multi-hop Transit GWs

     Traffic to/from geographic apart CPEs can cross multiple Cloud DCs
     via Cloud backbone.

     The on-prem CPEs are under one administrative control (e.g.,
     iBGP).

     Using Figure 2 as an example:

       - There is a bidirectional IPsec tunnel between CPE1 and the
          Cloud GW1; with IPsec SA1 for the traffic from the CPE1 to
          the Cloud-GW1; and IPsec SA2 for the traffic from the Cloud-
          GW1 to the CPE1.


Dunbar, et al.           Expires Dec 31, 2023                 [Page 12]


Internet-Draft           Multi-segment SD-WAN


       - There is a bidirectional IPsec tunnel between CPE10 and the
          Cloud GW2; with IPsec SA3 for the traffic from the CPE10 to
          the Cloud-GW2; and IPsec SA4 for the traffic from the Cloud-
          GW2 to the CPE10.
       - All the CPEs are under one iBGP administrative domain, with a
          Route Reflector (RR) as their controller. CPEs notify their
          peers of their corresponding Cloud GW addresses.

     When 11.1.1.x and 10.2.1.x need to communicate with each other,
     CPE1 and CPE10 establish a bidirectional IPsec Tunnel, with SA5
     for the traffic from CPE1 to CPE10 and SA6 for the traffic from
     CPE10 to CPE1. Assume the IPsec ESP Tunnel Mode is used, a packet
     from 11.1.1.1 to 10.2.1.2 has the following outer header:

     Outer IP header:
         +---------------------------+
         |    proto = 17 (UDP)       |
         |    src = CPE1             |
         |    dst = Cloud GW1        |
         +===========================+
         | GENEVE Header             |
         | multi-seg-SD-WAN Option   |
         |GENEVE Proto = 50 (ESP)    |
         +- - --  -- - - --      - --+
         |SD-WAN EndPt SubTLV (CPE10)|
         +---------------------------+
         |   EgressGW-SubTLV         |
         +---------------------------+  < ----------+
         |SPI(Security Parameter Idx)|        Authenticated
         +---------------------------+              |
         |    sequence number        |              |
         +---------------------------+   <-+        |
         | payload IP header:        |     |        |
         |  src =  11.1.1.1          |     |        |
         |  dst =  10.2.1.2          |     |        |
         +---------------------------+  Encrypted   |
         |   TCP header +            |     |        |
         ~    payload (variable)     ~     |        |
         |                           |     |        |
         +===========================+   <-+ -------+
         |   Authentication Data     |
         +---------------------------+






Dunbar, et al.           Expires Dec 31, 2023                 [Page 13]


Internet-Draft           Multi-segment SD-WAN


5.3. Data Authentication and Integrity Check by Cloud GW

     The IPsec SA already encrypts the client payload between the CPEs,
     the Cloud GW doesn't need to decrypt and re-encrypt the payload
     when relaying it to the destination CPE. However, data
     authentication and integrity check are needed as the traffic
     traverse an untrusted network.

     RFC2403 and RFC2404 define authentication algorithms used in AH
     and ESP. MD5, SHA-1, SHA-256 are some of the cryptographic hashes,
     and they are part of a Hashed Message Authentication Code.

5.4. Packet Header Processing

     In Figure 1, upon receiving a GENEVE encapsulated packet with the
     GENEVE Protocol Type = 50 (ESP), the Cloud GW do the following:

      - Authenticate the packet using a preconfigured authentication
         method.
      - Extract the destination CPE address from the SD-WAN Endpoint
         Sub-TLV inside the GENEVE header. Replace the outer IP
         destination address with the destination CPE address.
      - Optionally replace the outer IP source address with the Cloud
         GW address.
      - GENEVE header is unchanged.
      - Forward the packet to the destination CPE.

6. Illustration of Traffic from Private VPN to IPsec Tunnel

   This section illustrates a Cloud GW connecting client traffic from a
   branch CPE via a Private VPN to another CPE via an IPsec tunnel.

   Using Figure 1 as an example:

       - CPE1 send traffic via a Private VPN (Direct Connect to the
          Cloud Edge) to the Cloud GW. The traffic is not encrypted.
       - There is a bidirectional IPsec tunnel between CPE2 and the
          Cloud GW; with IPsec SA1 for the traffic from the CPE2 to the
          Cloud-GW; and IPsec SA2 for the traffic from the Cloud-GW to
          the CPE2.
       - All the CPEs are under one iBGP administrative domain, with a
          Route Reflector (RR) as their controller. CPEs notify their
          peers of their corresponding Cloud GW addresses.



Dunbar, et al.           Expires Dec 31, 2023                 [Page 14]


Internet-Draft           Multi-segment SD-WAN


     Assume the IPsec ESP Tunnel Mode is used for the IPsec SA between
     Cloud GW and CPE2. For a packet from 11.1.1.1 to 10.2.1.2, the
     following header is added by CPE1 sending over the Private VPN:

     Outer IP header:
         +---------------------------+
         |    proto = 17 (UDP)       |
         |    src = CPE1             |
         |    dst = Cloud GW        |
         +===========================+
         | GENEVE Header             |
         | multi-seg-SD-WAN Option   |
         |GENEVE Proto =TCP/UDP/etc. |
         +- - --  -- - - --      - --+
         |SD-WAN EndPt SubTLV (CPE2) |
         +---------------------------+  < -+
         | payload IP header:        |     |
         |  src =  11.1.1.1          |     |
         |  dst =  10.2.1.2          |     |
         +---------------------------+  Not Encrypted
         |   TCP header +            |     |
         ~    payload (variable)     ~     |
         |                           |     |
         +===========================+   <-+

   Upon receiving the GENEVE encapsulated packet with the "Multi-
   Segment-SD-WAN" option, the Cloud GW extracts the destination CPE
   from the GENEVE header and encrypts the packet with the IPsec SA2 to
   forward to the destination (i.e., CPE2). The GENEVE Header is
   carried to the CPE2.



















Dunbar, et al.           Expires Dec 31, 2023                 [Page 15]


Internet-Draft           Multi-segment SD-WAN


      Outer IP header:
         +---------------------------+
         |    proto = 17 (UDP)       |
         |    src = Cloud GW         |
         |    dst = CPE2             |
         +===========================+
         | GENEVE Header             |
         | multi-seg-SD-WAN Option   |
         |GENEVE Proto =50 (ESP)     |
         +- - --  -- - - --      - --+
         |SD-WAN EndPt SubTLV (CPE2) |
         +---------------------------+  < ----------+
         |SPI(Security Parameter Idx)|        Authenticated
         +---------------------------+              |
         |    sequence number        |              |
         +---------------------------+   <-+        |
         | payload IP header:        |     |        |
         |  src =  11.1.1.1          |     |        |
         |  dst =  10.2.1.2          |     |        |
         +---------------------------+  Encrypted   |
         |   TCP header +            |     |        |
         ~    payload (variable)     ~     |        |
         |                           |     |        |
         +===========================+   <-+ -------+
         |   Authentication Data     |
         +---------------------------+


7. Control Plane consideration

7.1. Control Plane for CPEs

   The control plane enables SD-WAN edges to discover their properties
   and attached routes. The on-prem CPEs and their vCPEs (or Virtual
   Appliances in Cloud DC) can be controlled by one iBGP instance.
   [SDWAN-Edge-Discover] describes the mechanism for SD-WAN edges to
   discover each other's properties. The IPsec Key Exchange between on-
   prem CPEs and the vCPE is via the iBGP Update through RR. [SD-WAN-
   Edge-Discovery].

7.2. Control Plane between CPEs and Cloud GWs

   It is common to have eBGP sessions between enterprises CPEs and the
   Cloud GWs. An enterprise-owned vCPE can establish an eBGP session
   with the Cloud VPN GW for accessing the workloads hosted in the
   Cloud DCs. If an IPsec tunnel is required between the Cloud DC GW


Dunbar, et al.           Expires Dec 31, 2023                 [Page 16]


Internet-Draft           Multi-segment SD-WAN


   and the vCPE, the full suite of IPSec IKEv2 must be exchanged
   between the vCPE and the Cloud GW.

8. Observability Consideration

   This section is intended for describing some metrics that
   enterprises can get from Cloud providers for the traffic transited.
   To be added.




9. Manageability Considerations

     To be added.

10. Security Considerations

     To be added.


11. IANA Considerations


   Need IANA to assign a new GENEVE Option Class dedicated for Multi-
   segment SD-WAN.

   In addition, need the following IANA assignment:

      - SD-WAN Endpoint Sub-TLV Type
      - SD-WAN Origin Endpoint Sub-TLV Type
      - SDWAN Egress GW Sub-TLV Type

12. References


12.1. Normative References

   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
             Requirement Levels", BCP 14, RFC 2119, March 1997.






Dunbar, et al.           Expires Dec 31, 2023                 [Page 17]


Internet-Draft           Multi-segment SD-WAN


   [RFC4271] Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
             Border Gateway Protocol 4 (BGP-4)", RFC 4271, DOI
             10.17487/RFC4271, January 2006, <https://www.rfc-
             editor.org/info/rfc4271>.

   [RFC4760] Bates, T., Chandra, R., Katz, D., and Y. Rekhter,
             "Multiprotocol Extensions for BGP-4", RFC 4760, DOI
             10.17487/RFC4760, January 2007, <https://www.rfc-
             editor.org/info/rfc4760>.

   [RFC7296] C. Kaufman, et al, "Internet Key Exchange Protocol Version
             2 (IKEv2)", RFC7296, Oct. 2014.

   [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
             2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
             May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC9012] Patel, K., Van de Velde, G., Sangli, S., and J. Scudder,
             "The BGP Tunnel Encapsulation Attribute", RFC 9012, DOI
             10.17487/RFC9012, April 2021, <https://www.rfc-
             editor.org/info/rfc9012>.


12.2. Informative References

   [RFC8192] S. Hares, et al, "Interface to Network Security Functions
             (I2NSF) Problem Statement and Use Cases", July 2017

   [RFC5521] P. Mohapatra, E. Rosen, "The BGP Encapsulation Subsequent
             Address Family Identifier (SAFI) and the BGP Tunnel
             Encapsulation Attribute", April 2009.

   [RFC9061] Marin-Lopez, R., Lopez-Millan, G., and F. Pereniguez-
             Garcia, "A YANG Data Model for IPsec Flow Protection Based
             on Software-Defined Networking (SDN)", RFC 9061, DOI
             10.17487/RFC9061, July 2021, <https://www.rfc-
             editor.org/info/rfc9061>.

   [CONTROLLER-IKE] D. Carrel, et al, "IPsec Key Exchange using a
             Controller", draft-carrel-ipsecme-controller-ike-01, work-
             in-progress.


Dunbar, et al.           Expires Dec 31, 2023                 [Page 18]


Internet-Draft           Multi-segment SD-WAN


   [LISP-GEOLOC] D. Farinacci, "LISP Geo-Coordinate Use-Case", draft-
             farinacci-lisp-geo-09, April 2020.

   [SDN-IPSEC] R. Lopez, G. Millan, "SDN-based IPsec Flow Protection",
             draft-ietf-i2nsf-sdn-ipsec-flow-protection-07, Aug 2019.

   [SECURE-EVPN] A. Sajassi, et al, "Secure EVPN", draft-sajassi-bess-
             secure-evpn-05, Oct 2021.

   [VPN-over-Internet] E. Rosen, "Provide Secure Layer L3VPNs over
             Public Infrastructure", draft-rosen-bess-secure-l3vpn-00,
             work-in-progress, July 2018

   [DMVPN] Dynamic Multi-point VPN:
             https://www.cisco.com/c/en/us/products/security/dynamic-
             multipoint-vpn-dmvpn/index.html

   [DSVPN] Dynamic Smart VPN:
             http://forum.huawei.com/enterprise/en/thread-390771-1-
             1.html



   [ITU-T-X1036] ITU-T Recommendation X.1036, "Framework for creation,
             storage, distribution and enforcement of policies for
             network security", Nov 2007.

   [Net2Cloud-Problem] L. Dunbar and A. Malis, "Dynamic Networks to
             Hybrid Cloud DCs Problem Statement", draft-ietf-rtgwg-
             net2cloud-problem-statement-22, March, 2023.

   [Net2Cloud-gap] L. Dunbar, A. Malis, and C. Jacquenet, "Networks
             Connecting to Hybrid Cloud DCs: Gap Analysis", draft-ietf-
             rtgwg-net2cloud-gap-analysis-07, July, 2020.

   [RFC9012] K. Patel, et al "The BGP Tunnel Encapsulation Attribute",
             RFC9012, April 2021.






Dunbar, et al.           Expires Dec 31, 2023                 [Page 19]


Internet-Draft           Multi-segment SD-WAN


13. Acknowledgments

   Acknowledgements to XXX for their review and suggestions.

   This document was prepared using 2-Word-v2.0.template.dot.










































Dunbar, et al.           Expires Dec 31, 2023                 [Page 20]


Internet-Draft           Multi-segment SD-WAN


Authors' Addresses


   Linda Dunbar
   Futurewei
   Email: ldunbar@futurewei.com

   Kausik Majumdar
   Microsoft
   Email: kmajumdar@microsoft.com


   Venkit Kasiviswanathan
   Arista
   Email: venkit@arista.com

   Ashok Ramchandra
   Microsoft
   Email: aramchandra@microsoft.com

Contributors' Addresses

























Dunbar, et al.           Expires Dec 31, 2023                 [Page 21]