Skip to main content

BGP UPDATE for SD-WAN Edge Discovery
draft-ietf-idr-sdwan-edge-discovery-12

Document Type Active Internet-Draft (idr WG)
Authors Linda Dunbar , Kausik Majumdar , Susan Hares , Robert Raszuk , Venkit Kasiviswanathan
Last updated 2023-10-14
Replaces draft-dunbar-idr-sdwan-edge-discovery
RFC stream Internet Engineering Task Force (IETF)
Intended RFC status (None)
Formats
Additional resources Mailing list discussion
Stream WG state WG Consensus: Waiting for Write-Up
Document shepherd Keyur Patel
Shepherd write-up Show Last changed 2023-09-08
IESG IESG state I-D Exists
Consensus boilerplate Unknown
Telechat date (None)
Responsible AD (None)
Send notices to keyur@arrcus.com
draft-ietf-idr-sdwan-edge-discovery-12
Network Working Group                                          L. Dunbar
Internet-Draft                                                 Futurewei
Intended status: Standards Track                             K. Majumdar
Expires: 16 April 2024                                   Microsoft Azure
                                                                S. Hares
                                                 Hickory Hill Consulting
                                                               R. Raszuk
                                                                  Arrcus
                                                      V. Kasiviswanathan
                                                                  Arista
                                                         14 October 2023

                  BGP UPDATE for SD-WAN Edge Discovery
                 draft-ietf-idr-sdwan-edge-discovery-12

Abstract

   The document describes the encoding of BGP UPDATE messages for the
   SD-WAN edge node property discovery.

   In the context of this document, BGP Route Reflector (RR) is the
   component of the SD-WAN Controller that receives the BGP UPDATE from
   SD-WAN edges and in turns propagates the information to the intended
   peers that are authorized to communicate via the SD-WAN overlay
   network.

Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119] [RFC8174]
   when, and only when, they appear in all capitals, as shown here.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

Dunbar, et al.            Expires 16 April 2024                 [Page 1]
Internet-Draft            SDWAN Edge Discovery              October 2023

   This Internet-Draft will expire on 16 April 2024.

Copyright Notice

   Copyright (c) 2023 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Conventions used in this document . . . . . . . . . . . . . .   3
   3.  Framework of SD-WAN Edge Discovery  . . . . . . . . . . . . .   4
     3.1.  The Objectives of SD-WAN Edge Discovery . . . . . . . . .   4
     3.2.  Comparing with Pure IPsec VPN . . . . . . . . . . . . . .   5
     3.3.  Client Route UPDATE and SD-WAN Tunnel UPDATE  . . . . . .   6
     3.4.  Edge Node Discovery . . . . . . . . . . . . . . . . . . .   8
   4.  Constrained propagation of BGP UPDATE . . . . . . . . . . . .   9
     4.1.  SD-WAN Segmentation, Virtual Topology and Client VPN  . .   9
     4.2.  4.2.  Constrained Propagation of Edge Capability  . . . .  10
   5.  Client Route UPDATE . . . . . . . . . . . . . . . . . . . . .  11
     5.1.  SD-WAN VPN ID in Client Route Update  . . . . . . . . . .  12
     5.2.  SD-WAN VPN ID in Data Plane . . . . . . . . . . . . . . .  12
   6.  SD-WAN Underlay UPDATE  . . . . . . . . . . . . . . . . . . .  12
     6.1.  NLRI for SD-WAN Underlay Tunnel Update  . . . . . . . . .  12
     6.2.  SD-WAN-Hybrid Tunnel Encoding . . . . . . . . . . . . . .  14
     6.3.  IPsec-SA-ID Sub-TLV . . . . . . . . . . . . . . . . . . .  14
     6.4.  Extended Port Attribute Sub-TLV . . . . . . . . . . . . .  15
     6.5.  Extended SubSub-TLV . . . . . . . . . . . . . . . . . . .  18
       6.5.1.  Underlay Network Transport SubSub-TLV . . . . . . . .  18
   7.  IPsec SA Property Sub-TLVs  . . . . . . . . . . . . . . . . .  19
     7.1.  IPsec SA Nonce Sub-TLV  . . . . . . . . . . . . . . . . .  19
     7.2.  IPsec Public Key Sub-TLV  . . . . . . . . . . . . . . . .  20
     7.3.  IPsec SA Proposal Sub-TLV . . . . . . . . . . . . . . . .  21
     7.4.  Simplified IPsec SA sub-TLV . . . . . . . . . . . . . . .  21
   8.  Error and Mismatch Handling . . . . . . . . . . . . . . . . .  23
     8.1.  Color Mismatch  . . . . . . . . . . . . . . . . . . . . .  23
     8.2.  IPsec Attributes Mismatch . . . . . . . . . . . . . . . .  24
   9.  SD-WAN BGP UPDATE Encoding Examples . . . . . . . . . . . . .  25
     9.1.  Encoding example of WAN Port properties . . . . . . . . .  25

Dunbar, et al.            Expires 16 April 2024                 [Page 2]
Internet-Draft            SDWAN Edge Discovery              October 2023

     9.2.  Encoding example of IPsec SA terminated at the C-PE2  . .  26
     9.3.  Encoding example of using IPsec-SA-ID Sub-TLV . . . . . .  26
   10. Manageability Considerations  . . . . . . . . . . . . . . . .  27
   11. Security Considerations . . . . . . . . . . . . . . . . . . .  27
   12. IANA Considerations . . . . . . . . . . . . . . . . . . . . .  28
     12.1.  Hybrid (SD-WAN) Overlay SAFI . . . . . . . . . . . . . .  28
     12.2.  Tunnel Encapsulation Attribute Type  . . . . . . . . . .  28
     12.3.  Tunnel Encapsulation Attribute Sub-TLV Types . . . . . .  28
   13. References  . . . . . . . . . . . . . . . . . . . . . . . . .  28
     13.1.  Normative References . . . . . . . . . . . . . . . . . .  28
     13.2.  Informative References . . . . . . . . . . . . . . . . .  30
   Appendix A.  Acknowledgments  . . . . . . . . . . . . . . . . . .  31
   Contributors  . . . . . . . . . . . . . . . . . . . . . . . . . .  31
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  31

1.  Introduction

   [SD-WAN-BGP-USAGE] illustrates how BGP [RFC4271] is used as a control
   plane for a SD-WAN network.  SD-WAN network refers to a policy-driven
   network over multiple heterogeneous underlay networks to get better
   WAN bandwidth management, visibility, and control.

   The document describes BGP UPDATE messages for an SD-WAN edge node to
   advertise its properties to its RR which then propagates that
   information to the authorized peers.

2.  Conventions used in this document

   The following acronyms and terms are used in this document:

   Cloud DC:  Off-Premises Data Centers that usually host applications
      and workload owned by different organizations or tenants.

   Controller:  Used interchangeably with SD-WAN controller to manage
      SD-WAN overlay path creation/deletion and monitor the path
      conditions between sites

   CPE:  Customer (Edge) Premises Equipment

   CPE-Based VPN:  Virtual Private Secure network formed among CPEs.
      This is to differentiate such VPNs from most commonly used PE-
      based VPNs discussed in [RFC4364].

   MP-NLRI:  Multi-Protocol Network Layer Reachability Information
      [MP_REACH_NLRI] Path Attribute defined in [RFC4760]

   RR:  Route Reflector

Dunbar, et al.            Expires 16 April 2024                 [Page 3]
Internet-Draft            SDWAN Edge Discovery              October 2023

   SD-WAN:  An overlay connectivity service that optimizes transport of
      IP Packets over one or more Underlay Connectivity Services by
      recognizing applications (Application Flows) and determining
      forwarding behavior by applying Policies to them.  [MEF-70.1]

   SD-WAN Endpoint:  can be the SD-WAN edge node address, a WAN port
      address (logical or physical) of a SD-WAN edge node, or a client
      port address.

   VPN:  Virtual Private Network

   VRF:  VPN Routing and Forwarding instance

   WAN:  Wide Area Network

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC8174] when, and only when, they appear in all capitals, as
   shown here.

3.  Framework of SD-WAN Edge Discovery

3.1.  The Objectives of SD-WAN Edge Discovery

   The objectives of SD-WAN edge discovery for an SD-WAN edge node are:
   to discover its authorized peers and their associated properties in
   order to establish secure overlay tunnels [Net2Cloud].  The
   attributes to be propagated includes:

   *  the SD-WAN (client) VPNs information,

   *  the attached routes under the SD-WAN VPNs, and

   *  the properties of the underlay networks over which the client
      routes can be carried, and potentially more.

   Some SD-WAN peers are connected by both trusted VPNs and untrusted
   public networks.  Some SD-WAN peers are connected only by untrusted
   public networks.  For the traffic over untrusted networks, IPsec
   Security Associations (IPsec SA) must be established and maintained.
   If an edge node has network ports behind a NAT, the NAT properties
   need to be discovered by the authorized SD-WAN peers.

   Like any VPN networks, the attached client routes belonging to
   specific SD-WAN VPNs can only be exchanged with the SD-WAN peer nodes
   authorized to communicate.

Dunbar, et al.            Expires 16 April 2024                 [Page 4]
Internet-Draft            SDWAN Edge Discovery              October 2023

3.2.  Comparing with Pure IPsec VPN

   A pure IPsec VPN has IPsec tunnels connecting all edge nodes over
   public networks.  Therefore, it requires stringent authentication and
   authorization (i.e., IKE Phase 1) before other properties of IPsec SA
   can be exchanged.  The IPsec Security Association (SA) between two
   untrusted nodes typically requires the following configurations and
   message exchanges:

      IPsec IKEV2:  messages sent to authenticate with each other.

      Establish IPsec SA  : requires the following set-up

         -  Local key configuration,

         -  Setting the Remote Peer address (such as 192.10.0.10 -
            172.0.01

         -  Information from IKEv2 Proposal directly sent to peer:
            (Encryption method, Integrity sha512, etc.)

         -  Transform set.

      Attached client prefixes discovery:  which is achieved by:

         -  Running routing protocol within each IPsec SA.

         -  If multiple IPsec SAs between two peer nodes are established
            to achieve load sharing, each IPsec tunnel needs to run its
            own routing protocol to exchange client routes attached to
            the edges.

      Set-up Access List or Traffic Selector:  such as Permit Local-IP1,
         Remote-IP2, and other parameters.

   In a BGP-controlled SD-WAN network over hybrid MPLS VPN and public
   internet underlay networks, all edge nodes and RRs are already
   connected by private secure paths.  The RRs have the policies to
   manage the authentication of all peer nodes.  More importantly, when
   an edge node needs to establish multiple IPsec tunnels to many edge
   nodes, all the management information can be multiplexed into the
   secure management tunnel between RR and the edge node.  Therefore,
   the amount of authentication in a BGP-Controlled SD-WAN network can
   be significantly reduced.

   Client VPNs are configured via VRFs, just like the configuration of
   the existing MPLS VPN.  The IPsec equivalent traffic selectors for
   local and remote routes are achieved by importing/exporting VPN Route

Dunbar, et al.            Expires 16 April 2024                 [Page 5]
Internet-Draft            SDWAN Edge Discovery              October 2023

   Targets.  The binding of client routes to IPsec SA is dictated by
   policies.  As a result, the IPsec configuration for a BGP controlled
   SD-WAN (with mixed MPLS VPN) can be simplified in the following
   manner:

   *  The SD-WAN controller has the authority to authenticate edges and
      peers so the Remote Peer association is controlled by the SD-WAN
      Controller (RR).

   *  The IKEv2 proposals (including the IPsec Transform set) can be
      sent directly to peers, or incorporated in a BGP UPDATE.

   *  The BGP UPDATE announces the client route reachability for all
      permitted parallel tunnels/paths.

   *  Importing/exporting Route Targets under each client VPN (VRF)
      achieves the traffic selection (or permission) among clients'
      routes attached to multiple edge nodes.

   Note that with this method there is no need to run multiple routing
   protocols in each IPsec tunnel.

3.3.  Client Route UPDATE and SD-WAN Tunnel UPDATE

   As described in [SD-WAN-BGP-USAGE], two separate BGP UPDATE messages
   are used for SD-WAN Edge Discovery:

   -  Client routes BGP UPDATE:

      This UPDATE is precisely the same as the BGP VPN client route
      UPDATE.  It uses the Encapsulation Extended Community and the
      Color Extended Community to link with the SD-WAN Tunnels UPDATE
      Message as specified in section 8 of [RFC9012].

      A new Tunnel Type (SD-WAN-Hybrid) is added and used by the
      Encapsulation Extended Community or the Tunnel-Encap Path
      Attribute [RFC9012] to indicate mixed underlay networks.

   -  SD-WAN UPDATE:

      This UPDATE is for an edge node to advertise the properties of
      directly attached underlay networks, including the NAT
      information, pre-configured IPsec SA identifiers, and/or the
      underlay network specific information.  This UPDATE can also
      include the detailed IPsec SA attributes, such as keys, nonce,
      encryption algorithms, etc.

Dunbar, et al.            Expires 16 April 2024                 [Page 6]
Internet-Draft            SDWAN Edge Discovery              October 2023

   In the following figure, four overlay paths between C-PE1 and C-PE2
   are established for illustration purpose.  More overlay paths are
   possible.  One physical port on C-PE2 can terminate multiple overlay
   paths from different ports on C-PE1.

   a)  MPLS-in-GRE path;

   b)  node-based IPsec tunnel [2.2.2.2 - 1.1.1.1].  As C-PE2 has two
      public internet facing WAN ports, either of those two WAN port IP
      addresses can be the outer destination address of the IPsec
      encapsulated data packets;

   c)  port-based IPsec tunnel [192.0.0.1 - 192.10.0.10]; and

   d)  port-based IPsec tunnel [172.0.0.1 - 160.0.0.1].

                                     +---+
                      +--------------|RR |----------+
                     /  Untrusted    +-+-+           \
                    /                                 \
                   /                                   \
           +---------+  MPLS Path                      +-------+
   11.1.1.x| C-PE1   A1-------------------------------B1 C-PE2 |10.1.1.x
           |         |                                 |       |
   21.1.1.x|         A2(192.10.0.10)------( 192.0.0.1)B2       |20.1.1.x
           |         |                                 |       |
           | Addr    A3(160.0.0.1) --------(170.0.0.1)B3 Addr  |
           | 1.1.1.1 |                                 |2.2.2.2|
           +---------+                                 +-------+

                          Figure 1: Hybrid SD-WAN

   C-PE2 advertises the attached client routes as below:

   *  Client Route UPDATE:

      -  Extended community: RT for SD-WAN VPN 1

      -  NLRI: AFI=IPv4/IPv6 and SAFI = VPN

      -  Prefix: 10.1.1.x; 20.1.1.x

      -  NextHop: 2.2.2.2 (C-PE2)

      -  Encapsulation Extended Community: tunnel-type=SD-WAN-Hybrid

      -  Color Extended Community: Site-identifier

Dunbar, et al.            Expires 16 April 2024                 [Page 7]
Internet-Draft            SDWAN Edge Discovery              October 2023

   *  SD-WAN UPDATE:

      -  C-PE2 can use the following Update messages to advertise the
         properties of Internet facing ports 192.0.0.1 and 170.0.0.1,
         and their associated IPsec SA related parameters.

      -  Update #1 for the properties associated with the WAN port
         192.0.0.1, such as the NAT properties, the underlay network
         properties, etc.  (Details in Section 9.1).

      -  Update #2 for the properties associated with the WAN port
         170.0.0.1 associated properties.  (Details in Section 9.1).

      -  Update #3 for IPsec parameters associated with IPsec tunnel
         terminated at the Node level (2.2.2.2), such as the supported
         encryption methods, public keys, etc.  (Details in
         Section 9.2).

3.4.  Edge Node Discovery

   The basic scheme of SD-WAN edge node discovery using BGP consists of
   the following:

   *  Secure connection to a SD-WAN controller (in this context the RR
      ):

      -  For an SD-WAN edge with both MPLS and IPsec paths, the edge
         node should already have a secure connection to its controller
         (the RR in this context).  For an SD-WAN edge that is only
         accessible via Internet, the SD-WAN edge, upon power-up,
         establishes a secure tunnel (such as TLS or SSL) with the SD-
         WAN central controller whose address is preconfigured on the
         edge node.  The central controller informs the edge node of its
         local RR.  The edge node then establishes a transport layer
         secure session with the RR (such as TLS or SSL).

   *  The Edge node will advertise its own properties to its designated
      RR via the secure connection.

   *  The RR propagates the received information to the authorized
      peers.

   *  The authorized peers can establish the secure data channels
      (IPsec) and exchange more information among each other.

Dunbar, et al.            Expires 16 April 2024                 [Page 8]
Internet-Draft            SDWAN Edge Discovery              October 2023

   For an SD-WAN deployment with multiple RRs, it is assumed that there
   are secure connections among those RRs.  How secure connections are
   established among those RRs is out of the scope of this document.
   The existing BGP UPDATE propagation mechanisms control the edge
   properties propagation among the RRs.

   For some environments where the communication to RR is highly
   secured, [RFC9016] IKE-less can be deployed to simplify IPsec SA
   establishment among edge nodes.

4.  Constrained propagation of BGP UPDATE

4.1.  SD-WAN Segmentation, Virtual Topology and Client VPN

   In SD-WAN deployment, SD-WAN Segmentation is a frequently used term,
   referring to partitioning a network into multiple subnetworks, just
   like MPLS VPNs.  SD-WAN Segmentation is achieved by creating SD-WAN
   virtual topologies and SD-WAN VPNs.  An SD-WAN virtual topology
   consists of a set of edge nodes and the tunnels (a.k.a. underlay
   paths), including both IPsec tunnels and/or MPLS VPN tunnels,
   interconnecting those edge nodes.

   An SD-WAN VPN is configured in the same way as the VRFs of an MPLS
   VPN.  One SD-WAN client VPN can be mapped to multiple SD-WAN virtual
   topologies.  SD-WAN Controller governs the policies of mapping a
   client VPN to SD-WAN virtual topologies.

   Each SD-WAN edge node may need to support multiple VPNs.  Route
   Target is used to differentiate the SD-WAN VPNs.  For example, in the
   picture below, the Payment-Flow on C-PE2 is only mapped to the
   virtual topology of C-PEs to/from Payment Gateway, whereas other
   flows can be mapped to a multipoint-to-multipoint virtual topology.

Dunbar, et al.            Expires 16 April 2024                 [Page 9]
Internet-Draft            SDWAN Edge Discovery              October 2023

                                     +---+
                      +--------------|RR |----------+
                     /  Untrusted    +-+-+           \
                    /                                 \
                   /                                   \
           +---------+  MPLS Path                      +-------+
   11.1.1.x| C-PE1   A1-------------------------------B1 C-PE2 |10.1.1.x
           |         |                                 |       |
   21.1.1.x|         A2(192.10.0.10)------( 192.0.0.1)B2       |20.1.1.x
           |         |                                 |       |
           | Addr    A3(160.0.0.1) --------(170.0.0.1)B3 Addr  |11.2.2.x
           | 1.1.1.1 |                              /  |2.2.2.2|
           +---------+                             /   +-------+
                      \                           /
                       \                         /PaymentFlow
                        \                       /
                         \                +----+----+
                          +---------------| payment |
                                          | Gateway |
                                          +---------+

                 Figure 2: SD-WAN Virtual Topology and VPN

4.2.  4.2.  Constrained Propagation of Edge Capability

   BGP has a built-in mechanism [RFC4684] to dynamically achieve the
   constrained distribution of edge information.  In a nutshell, an SD-
   WAN edge sends RT Constraint (RTC) NLRI to the RR for the RR to
   install an outbound route filter, as shown in the figure below:

Dunbar, et al.            Expires 16 April 2024                [Page 10]
Internet-Draft            SDWAN Edge Discovery              October 2023

       RT Constraint                   RT constraint
       NLRI={SD-WAN#1, SD-WAN#2}         NLRI={SD-WAN#1, SD-WAN#3}
               ----->                 +---+      <-----------
                 +--------------------|RR1|------------------+
                 | Outbound Filter    +---+  Outbound Filter |
                 | Permit SD-WAN#1,#2      Permit SD-WAN#1,#3|
                 | Deny all                 Deny all         |
                 |   <-------                --------->      |
                 |                                           |
           +-----+---+  MPLS Path                      +-----+-+
   11.1.1.x| C-PE1   A1-------------------------------B1 C-PE2 |10.1.1.x
           |         |                                 |       |
   21.1.1.x|         A2(192.10.0.10)------( 192.0.0.1)B2       |20.1.1.x
           |         |                                 |       |
           | Addr    A3(160.0.0.1) --------(170.0.0.1)B3 Addr  |
           | 1.1.1.1 |                                 |2.2.2.2|
           +---------+                                 +-------+
   SD-WAN VPN #1                                          SD-WAN VPN #1
   SD-WAN VPN #2                                          SD-WAN VPN #3

             Figure 3: Constraint propagation of Edge Property

   However, a SD-WAN overlay network can span across untrusted networks,
   RR cannot trust the RT Constraint (RTC) NLRI BGP UPDATE from any
   nodes.  RR can only process the RTC NLRI from authorized peers for a
   SD-WAN VPN.

   It is out of the scope of this document on how RR is configured with
   the policies to filter out unauthorized nodes for specific SD-WAN
   VPNs.

   When the RR receives BGP UPDATE from an edge node, it propagates the
   received UPDATE message to the nodes that are in the Outbound Route
   filter for the specific SD-WAN VPN.

5.  Client Route UPDATE

   The SD-WAN network's Client Route UPDATE message is the same as the
   L3 VPN or EVPN client route UDPATE message.  The SD-WAN Client Route
   UPDATE message uses the Encapsulation Extended Community and the
   Color Extended Community to link with the SD-WAN Underlay UPDATE
   Message.

Dunbar, et al.            Expires 16 April 2024                [Page 11]
Internet-Draft            SDWAN Edge Discovery              October 2023

5.1.  SD-WAN VPN ID in Client Route Update

   An SD-WAN VPN is same as a client VPN in a BGP controlled SD-WAN
   network.  The Route Target Extended Community should be included in a
   Client Route UPDATE message to differentiate the client routes from
   routes belonging to other VPNs.

5.2.  SD-WAN VPN ID in Data Plane

   For an SD-WAN edge node which can be reached by both MPLS and IPsec
   paths, the client packets reached by MPLS network will be encoded
   with the MPLS Labels based on the scheme specified by [RFC8277].

   For GRE Encapsulation within an IPsec tunnel, the GRE key field can
   be used to carry the SD-WAN VPN ID.  For network virtual overlay
   (VxLAN, GENEVE, etc.) encapsulation within the IPsec tunnel, the
   Virtual Network Identifier (VNI) field is used to carry the SD-WAN
   VPN ID.

6.  SD-WAN Underlay UPDATE

   The hybrid underlay tunnel UPDATE is to advertise the detailed
   properties associated with the public facing WAN ports and IPsec
   tunnels.  The Edge node will advertise its own properties to its
   designated RR via the secure connection.

6.1.  NLRI for SD-WAN Underlay Tunnel Update

   A new NLRI SAFI[RFC5521](SD-WAN SAFI=74)is introduced within the
   MP_REACH_NLRI Path Attribute of RFC4760 for advertising the detailed
   properties of the SD-WAN tunnels terminated at the edge node:

   +------------------+
   |    Route Type    | 2 octet
   +------------------+
   |     Length       | 2 Octet
   +------------------+
   |   Type Specific  |
   ~ Value (Variable) ~
   |                  |
   +------------------+

                           Figure 4: SD-WAN NLRI

   Where

   Route (NLRI) Type:  2 octet value to define the encoding of the rest

Dunbar, et al.            Expires 16 April 2024                [Page 12]
Internet-Draft            SDWAN Edge Discovery              October 2023

      of the SD-WAN the NLRI.

   Length:  2 octets of length expressed in bits as defined in
      [RFC4760].

   This document defines the following SD-WAN Route type:

   NLRI Route-Type= 1:  For advertising the detailed properties of the
      SD-WAN tunnels terminated at the edge, where the transport network
      port can be uniquely identified by a tuple of three values (Port-
      Local-ID, SD-WAN-Color, SD-WAN-Node-ID).  The SD-WAN NLRI Route-
      Type =1 has the following encoding:

           +------------------+
           |  Route Type = 1  | 2 octet
           +------------------+
           |     Length       | 2 Octet
           +------------------+
           |   Port-Local-ID  | 4 octets
           +------------------+
           |   SD-WAN-Color   | 4 octets
           +------------------+
           |  SD-WAN-Node-ID  | 4 or 16 octets
           +------------------+

                      Figure 5: SD-WAN NLRI Route Type 1

   Port local ID:  SD-WAN edge node Port identifier, which is locally
      significant.  If the SD-WAN NLRI applies to multiple WAN ports,
      this field is NULL.

   SD-WAN-Color:  to represent a group of tunnels, which correlate with
      the Color-Extended-community included in the client routes UPDATE.
      When a client route can be reached by multiple SD-WAN edges co-
      located at one site, the SD-WAN- Color can represent a group of
      tunnels terminated at those SD-WAN edges co-located at the site,
      which effectively represent the site.

   SD-WAN Node ID:  The node's IPv4 or IPv6 address.

   Route Type values outside of 1 are out of scope for this document.
   The Route Type allows for other existing SD-WAN applications to use
   this basic format.

Dunbar, et al.            Expires 16 April 2024                [Page 13]
Internet-Draft            SDWAN Edge Discovery              October 2023

6.2.  SD-WAN-Hybrid Tunnel Encoding

   A new BGP Tunnel-Type=SD-WAN-Hybrid (code point 25) is to indicate
   hybrid underlay tunnels.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | Tunnel-Type=25(SD-WAN-Hybrid )| Length (2 Octets)             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                           Sub-TLVs                            |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                    Figure 6: SD-WAN Hybrid Value Field

6.3.  IPsec-SA-ID Sub-TLV

   IPsec-SA-ID Sub-TLV within the Hybrid Underlay Tunnel UPDATE
   indicates one or more pre-established IPsec SAs by using their
   identifiers, instead of listing all the detailed attributes of the
   IPsec SAs.

   Using an IPsec-SA-ID Sub-TLV not only greatly reduces the size of BGP
   UPDATE messages, but also allows the pairwise IPsec rekeying process
   to be performed independently.

   The following is the structure of the IPsec-SA-ID sub-TLV:

    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Type=64 (IPsec-SA-ID subTLV)   |  Length (2 Octets)            |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      IPsec SA Identifier #1                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      IPsec SA Identifier #2                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                       Figure 7: IPsec-SA-ID Sub-TLV

Dunbar, et al.            Expires 16 April 2024                [Page 14]
Internet-Draft            SDWAN Edge Discovery              October 2023

6.4.  Extended Port Attribute Sub-TLV

   Extended Port Attribute Sub-TLV is to advertise the properties
   associated with a public Internet-facing WAN port that might be
   behind NAT.  An SD-WAN edge node can query a STUN Server (Session
   Traversal of UDP through Network address translation [RFC8489]) to
   get the NAT properties, including the public IP address and the
   Public Port number, to pass to its peers.

   The location of a NAT device can be:

   *  Only the initiator is behind a NAT device.  Multiple initiators
      can be behind separate NAT devices.  Initiators can also connect
      to the responder through multiple NAT devices.

   *  Only the responder is behind a NAT device.

   *  Both the initiator and the responder are behind a NAT device.

   The initiator's address and/or responder's address can be dynamically
   assigned by an ISP or when their connection crosses a dynamic NAT
   device that allocates addresses from a dynamic address pool.

   As one SD-WAN edge can connect to multiple peers, the pair-wise NAT
   exchange as IPsec's IKE[RFC7296] is not efficient.  In the BGP
   Controlled SD-WAN, NAT properties for a WAN port are encoded in the
   Extended Port Attribute sub-TLV, which the following format:

Dunbar, et al.            Expires 16 April 2024                [Page 15]
Internet-Draft            SDWAN Edge Discovery              October 2023

        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |Type=65(extPort| ExtPort Length| Reserved      |I|O|R|R|R|R|R|R|
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | NAT Type      |  Encap-Type   |Trans networkID|     RD ID     |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                  Local  IP Address                            |
               32-bits for IPv4, 128-bits for Ipv6
                       ~~~~~~~~~~~~
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                  Local  Port                                  |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                Public IP                                      |
               32-bits for IPv4, 128-bits for Ipv6
                       ~~~~~~~~~~~~
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                Public Port                                    |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |               Extended SubSub-TLV                             |
       ~                                                               ~
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                 Figure 8: Extended Port Attribute Sub-TLV

   Where:

   *  Extended Port Attribute Type (=65): indicating it is the Extended
      Port Attribute SubTLV

   *  ExtPort Length: the length of the subTLV.

   *  Flags:

      -  I bit (CPE port address or Inner address scheme):

         o  If set to 0, indicate the inner (private) address is IPv4.

         o  If set to 1, indicates the inner address is IPv6.

      -  O bit (Outer address scheme):

         o  If set to 0, indicate the inner (private) address is IPv4.

         o  If set to 1, indicates the inner address is IPv6.

      -  R bits: reserved for future use.  Must be set to 0 now:

   *  NAT Type: the NAT type can be one of the following values:

Dunbar, et al.            Expires 16 April 2024                [Page 16]
Internet-Draft            SDWAN Edge Discovery              October 2023

      -  1: without NAT ;

      -  2: 1-to-1 static NAT;

      -  3: Full Cone;

      -  4: Restricted Cone;

      -  5: Port Restricted Cone;

      -  6: Symmetric; or

      -  7: Unknown (i.e. no response from the STUN server).

   *  Encap-Type: the supported encapsulation types for the port.

      -  Encap-Type=1: GRE;

      -  Encap-Type=2: VxLAN;

      Notes:

      -  The Encap-Type inside the Extended Port Attribute Sub-TLV is
         different from the RFC9012's BGP-Tunnel-Encapsulation type
         (https://www.iana.org/assignments/bgp-tunnel-encapsulation/bgp-
         tunnel-encapsulation.xhtml#tunnel-types).  The Extended Port
         Attribute Sub-TLV is a subTLV attached to the Tunnel Type TLV
         (the BGP-Tunnel-Type = 25 for the SD-WAN Hybrid tunnels).  The
         port can indicate the specific encapsulations, such as:

      -  If the IPsec-SA-ID subTLV or the IPsec SA detailed
         subTLVs(Nonce/publicKey/Proposal) are included in the SD-WAN-
         Hybrid tunnel, the Encap-Type indicates the encapsulation type
         within the IPsec payload.

      -  If the IPsec SA subTLVs are not included in the SD-WAN-Hybrid
         Tunnel, the Encap-Type indicates the encapsulation of the
         payload without IPsec encryption.

   *  Transport Network ID: Central Controller assigns a global unique
      ID to each transport network.

   *  RD ID: Routing Domain ID, need to be globally unique.

Dunbar, et al.            Expires 16 April 2024                [Page 17]
Internet-Draft            SDWAN Edge Discovery              October 2023

      -  Some SD-WAN deployment might have multiple levels, zones, or
         regions that are represented as routing domains.  Policies can
         govern if tunnels can be established across domains.  E.g., a
         hub node can establish tunnels with different domains; but the
         spoke nodes cannot establish tunnels with nodes in different
         domains.

   *  Local IP: The local (or private) IP address of the WAN port.

   *  Local Port: used by Remote SD-WAN edge node for establishing IPsec
      to this specific port.

   *  Public IP: The IP address after the NAT.  If NAT is not used, this
      field is set to NULL.

   *  Public Port: The Port after the NAT.  If NAT is not used, this
      field is set to NULL.

   *  Extended SubSub-TLV: for carrying additional information about the
      underlay networks.

6.5.  Extended SubSub-TLV

   Two types of the Extended SubSub-TLVs are specified in this document:
   Underlay Network Transport SubSub-TLV and the underlay Geo Location
   SubSub-TLV.

6.5.1.  Underlay Network Transport SubSub-TLV

   The Underlay Network Transport SubSub-TLV is an optional Sub-TLV to
   carry the WAN port connection types and bandwidth, such as LTE, DSL,
   Ethernet, etc.

   The format of this Sub-TLV is as follows:

        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | UnderlayType  |      Length   |      Flag     |    Reserved   |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |Connection Type|   Port Type   |        Port Speed             |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                   Figure 9: Underlay Network SubSub-TLV

   Where:

   Underlay Network Properties:  sub Type=66

Dunbar, et al.            Expires 16 April 2024                [Page 18]
Internet-Draft            SDWAN Edge Discovery              October 2023

   Length:  always 6 bytes

   Flag:  a 1 octet value.

   Reserved:  1 octet of reserved bits.  It SHOULD be set to zero on
      transmission and MUST be ignored on receipt.

   Connection Type:  are listed below as:

      *  1 = Wired

      *  2 = WIFI

      *  3 = LTE

      *  4 = 5G

   Port Type:  Port type define as follows:

      *  1 = Ethernet

      *  2 = Fiber Cable

      *  3 = Coax Cable

      *  4 = Cellular

   Port Speed:  The port seed is defined as 2 octet value.  The values
      are defined as Gigabit speed.  For example, a value of 1 would
      mean 1 gigabit.

7.  IPsec SA Property Sub-TLVs

   This section describes the detailed IPsec SA properties sub-TLVs.
   When the IPsec SA properties are associated with the node, any of the
   node's WAN ports can be the outer destination address of the IPsec
   encapsulated data packets.

7.1.  IPsec SA Nonce Sub-TLV

   The Nonce Sub-TLV is based on the Base DIM sub-TLV as described the
   Section 10.1 of [SECURE-EVPN].  The following fields are removed
   because:

      - the Originator ID is same as the Node-ID in the SD-WAN NLRI,

      - the Tenant ID and Subnet ID are represented by the SD-WAN VPN ID
      in the Client UPDATE.

Dunbar, et al.            Expires 16 April 2024                [Page 19]
Internet-Draft            SDWAN Edge Discovery              October 2023

   The format of this Sub-TLV is as follows:

        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |   ID Length   |       Nonce Length            |I|   Flags     |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                             Rekey                             |
       |                            Counter                            |
       +---------------------------------------------------------------+
       |                IPsec SA Identifier                            |
       +---------------------------------------------------------------+
       |                                                               |
       ~                          Nonce Data                           ~
       |                                                               |
       +---------------------------------------------------------------+

                  Figure 10: IPsec SA Nouce Sub-TLV

   where:

      IPsec SA ID - The 4 bytes IPSec SA ID is to differentiate multiple
      IPsec SAs terminated at the edge.

      -  The IPsec SA ID can be used in the IPsec-SA-ID subTLV of a
         different BGP UPDATE message to refer to all the values carried
         in the IPsec Public Key SubTLV and the IPsec SA Proposal Sub-
         TLV that are in the same BGP UPDATE message as the IPsec SA
         Nonce sub-TLV.

7.2.  IPsec Public Key Sub-TLV

   The IPsec Public Key Sub-TLV is derived from the Key Exchange Sub-TLV
   described in [SECURE-EVPN] with an addition of Duration filed to
   define the IPSec SA life span.  The edge nodes would pick the
   shortest duration value advertised by the peers.

   The format of this Sub-TLV is as follows:

Dunbar, et al.            Expires 16 April 2024                [Page 20]
Internet-Draft            SDWAN Edge Discovery              October 2023

        0                   1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |   Diffie-Hellman Group Num    |          Reserved             |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                                                               |
       ~                       Key Exchange Data                       ~
       |                                                               |
       +---------------------------------------------------------------+
       |                            Duration                           |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                Figure 11: IPsec SA Public Key Sub-TLV

7.3.  IPsec SA Proposal Sub-TLV

   The IPsec SA Proposal Sub-TLV is to indicate the number of Transform
   Sub-TLVs.  This Sub-TLV aligns with the sub-TLV structure from
   [SECURE-EVPN].

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |    Transform Attr Length      |Transform Type |    Reserved.  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Transform ID              |          Reserved             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   ~                        Transform Attributes                   ~
   |                                                               |
   +---------------------------------------------------------------+

                    Figure 12: IPsec SA Proposal Sub-TLV

   The Transform Type and the Transform Attributes Sub-sub-TLV are taken
   from the section 3.3.2 and 3.3.5 of RFC7296, respectively.

7.4.  Simplified IPsec SA sub-TLV

   For a simple SD-WAN network with edge nodes supporting only a few
   pre-defined encryption algorithms, a simple IPsec sub-TLV can be used
   to encode the pre-defined algorithms, as below:

Dunbar, et al.            Expires 16 April 2024                [Page 21]
Internet-Draft            SDWAN Edge Discovery              October 2023

        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |IPsec-simType  |IPsecSA Length                 | Flag          |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | Transform     | Mode          | AH algorithms |ESP algorithms |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |         ReKey Counter (SPI)                                   |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | key1 length   |         Key 1                                 ~
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | key2 length   |         Key 2                                 ~
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | key-i length  |         Nonce                                 ~
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |        Duration                                               |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                   Figure 13: Siplified IPsec SA Sub-TLV

   Where:

   IPsec-SimType=70:  indicate the simplified IPsec SA attributes.

   IPsec-SA subTLV Length (2 Byte):  25 (or more).

   Flags:  1 octet of flags.  None are defined at this stage.  Flags
      SHOULD be set to zero on transmission and MUST be ignored on
      receipt.

   Transform (1 Byte):
      *  Transform = 1 means AH,

      *  Transform = 2 means ESP, or

      *  Transform = 3 means AH+ESP.

   IPsec Mode (1 byte):
      *  Mode = 1 indicates that the Tunnel Mode is used.

      *  Mode = 2 indicates that the Transport mode is used.

   AH algorithms (1 byte):  AH authentication algorithms supported.  The
      values are specified by [IANA-AH].  Each SD-WAN edge node can
      support multiple authentication algorithms; send to its peers to
      negotiate the strongest one.

   ESP algorithms (1 byte):  the ESP algorithms supported.  Its values

Dunbar, et al.            Expires 16 April 2024                [Page 22]
Internet-Draft            SDWAN Edge Discovery              October 2023

      are specified by [IANA-ESP].  One SD-WAN edge node can support
      multiple ESP algorithms and send them to its peers to negotiate
      the strongest one.  The default algorithm is AES-256.

         When a node supports multiple authentication algorithms, the
         initial UPDATE needs to include the "Transform Sub-TLV"
         described by [SECURE-EVPN] to describe all of the algorithms
         supported by the node.

   Rekey Counter (Security Parameter Index)):  4 bytes

   Public Key:  IPsec public key

   Nonce:  IPsec Nonce

   Duration:  SA life span.

8.  Error and Mismatch Handling

8.1.  Color Mismatch

   When an SD-WAN edge receives a client route BGP UPDATE from a peer
   with a color that does not match with any of the tunnels advertised
   by the peer, the client route UPDATE should be ignored, and an error
   message (e.g., Syslog) should be generated to its management system
   per[RFC7606].

   For example, for two peers, PA and PB, both PA and PB will first
   advertise their SD-WAN properties (i.e., tunnel properties).  Say PA
   advertises two SD-WAN tunnels (Red and Green), and PB advertises two
   SD-WAN tunnels (Yellow and Purple).  PB should report a mismatch
   error message when PB receives a Client Update from PA with a color
   NOT Red or Green.  PA should report a Mismatch Error when PA receives
   a Client Update from PB with a color that is not Yellow and Purple.

   Upon receiving a Tunnel Update that includes the IPsec-SA-ID subTLV
   from a peer, the BGP node should report Mismatch error if the IPsec
   SA has not been established yet.

   Moreover, if the encap-Types, in the Extended Port Attributes Sub-
   TLV, in the received SDWAN update is not supported by the local
   ports, the corresponding ports between the remote edge and local edge
   will not establish an overlay tunnel.  Overlay tunnels would only be
   established between two ports belonging to different edges, if their
   attributes are compatible.  For instance, the encap Types should
   match.  Policies and configurations outside the scope of this
   document could allow for mismatched attributes to be present and
   allow establishing overlay tunnels.

Dunbar, et al.            Expires 16 April 2024                [Page 23]
Internet-Draft            SDWAN Edge Discovery              October 2023

8.2.  IPsec Attributes Mismatch

   Each C-PE device advertises a SD-WAN SAFI Underlay NLRI to the other
   C-PE devices via a BGP Route Reflector to establish pairwise SAs
   between itself and every other remote C-PEs.  During the SAFI NLRI
   advertisement, the BGP originator would include either simple IPSec
   Security Association properties defined in IPSec SA Sub-TLV based on
   IPSec-SA-Type = 1 or full-set of IPSec Sub-TLVs including Nonce,
   Public Key, Proposal and number of Transform Sub-TLVs based on IPSec-
   SA-Type = 2.

   The C-PE devices compare the IPSec SA attributes between the local
   and remote WAN ports.  If there is a match on the SA Attributes
   between the two ports, the IPSec Tunnel is established.

   The C-PE devices would not try to negotiate the base IPSec-SA
   parameters between the local and the remote ports in the case of
   simple IPSec SA exchange or the Transform sets between local and
   remote ports if there is a mismatch on the Transform sets in the case
   of full-set of IPSec SA Sub-TLVs.

   Here is an example of using Figure 1 in Section 3 to establish an
   IPsec Tunnel between C-PE1 and C-PE2 WAN Ports A2 and B2 (A2:
   192.10.0.10 - B2:192.0.0.1).

   C-PE1 needs to advertise the following attributes for establishing
   the IPsec SA:

   *  NextHop: 192.10.0.10

   *  SD-WAN Node ID

   *  SD-WAN-Site-ID

   *  Tunnel Encap Attr (Type=SD-WAN) -

      -  Extended Port Attribute Sub-TLV

         o  Transport SubSubTLV - with information on ISP3.

      -  Transport-Sub-TLV for detailed information about the ISP3

      -  IPsec SA Nonce Sub-TLV,

      -  IPsec SA Public Key Sub-TLV,

      -  Proposal Sub-TLV with Num Transforms = 1

Dunbar, et al.            Expires 16 April 2024                [Page 24]
Internet-Draft            SDWAN Edge Discovery              October 2023

      -  {Transforms Sub-TLV - Trans 1}

   C-PE2 needs to advertise the following attributes for establishing
   IPsec SA:

   NH:  192.0.0.1

   SD-WAN Node ID:  (value)

   SD-WAN-Site-ID:  (value)

   Tunnel Encap Attr (Type=SD-WAN)
      *  Extended Port Attribute Sub-TLV

         -  Transport SubSubTLV - with information on ISP1.

      *  IPsec SA Nonce Sub-TLV,

      *  IPsec SA Public Key Sub-TLV,

      *  Proposal Sub-TLV with Num Transforms = 1

      *  {Transforms Sub-TLV - Trans 2}

   As there is no matching transform between the WAN ports A2 and B2 in
   C-PE1 and C-PE2, respectively, no IPsec Tunnel will be established.

9.  SD-WAN BGP UPDATE Encoding Examples

9.1.  Encoding example of WAN Port properties

   The C-PE2 of Figure 1 can send the following SD-WAN UPDATE messages
   to advertise the properties associated with WAN Port 192.0.0.1 and
   WAN Port 170.0.0.1, respectively:

   SD-WAN NLRI:  AFI=IPv4/IPv6 and SAFI = SD-WAN

      *  Color match with the Client route UPDATE's Color Extended
         Community[RFC4360]

      *  local port id for WAN port 192.0.0.1

      *  Node-ID= 2.2.2.2 (C-PE2)

   Tunnel-Type:  Hybrid-SD-WAN

   Extended-Port-SubTLV:  for 192.0.0.1

Dunbar, et al.            Expires 16 April 2024                [Page 25]
Internet-Draft            SDWAN Edge Discovery              October 2023

   SD-WAN NLRI:  AFI=IPv4/IPv6 and SAFI = SD-WAN

      *  Color match with the Client route UPDATE's Color Extended
         Community

      *  local port id for WAN port 170.0.0.1

      *  Node-ID= 2.2.2.2 (C-PE2)

   Tunnel-Type:  Hybrid-SD-WAN

   Extended-Port-SubTLV  for 170.0.0.1

9.2.  Encoding example of IPsec SA terminated at the C-PE2

   The C-PE2 of the Figure 1 can send the following SD-WAN UPDATE
   messages to advertise node level IPsec SA:

   SD-WAN NLRI:  AFI=IPv4/IPv6 and SAFI = SD-WAN

      *  Color match with the Client route UPDATE's Color Extended
         Community

      *  Port-ID=0

      *  Node-ID= 2.2.2.2 (C-PE2)

   Tunnel-Type:  Hybrid-SD-WAN

   IPsec-SA-ID Sub-TLV or IPsec SA Property Sub-TLVs

9.3.  Encoding example of using IPsec-SA-ID Sub-TLV

   Here is an encoding example of four IPsec SAs terminated at the same
   node.

Dunbar, et al.            Expires 16 April 2024                [Page 26]
Internet-Draft            SDWAN Edge Discovery              October 2023

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | Tunnel-Type =SD-WAN-Hybrid    |       Length =                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      Tunnel-end-point sub-TLV                                 |
   ~                                                               ~
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | subTLV-Type = IPsec-SA-ID     |       Length =                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                     IPsec SA Identifier = 1                   |
   +---------------------------------------------------------------+
   |                     IPsec SA Identifier = 2                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                     IPsec SA Identifier = 3                   |
   +-------------------------------+-------------------------------+
   |                     IPsec SA Identifier = 4                   |
   +---------------------------------------------------------------+

                 Figure 14: Encoding Example of 4 IPsec SA

   The Length of the Tunnel-Type = SDDWAN-Hybrid is the sum of the
   following:

   *  Tunnel-end-point sub-TLV total length:

   *  The IPsec-SA-ID Sub-TLV length

10.  Manageability Considerations

   Unlike MPLS VPN whose PE nodes are all controlled by the network
   operators, SD-WAN edge nodes can be installed anywhere, in shopping
   malls, in 3rd party Cloud DCs, etc.

   It is very important to ensure that client routes advertisement from
   an SD-WAN edge node are legitimate.  The RR needs to drop all the BGP
   Update messages from an SD-WAN edge nodes that have invalid Route
   Targets.

11.  Security Considerations

   The document describes the encoding for SD-WAN edge nodes to
   advertise its properties to their peers to its RR, which propagates
   to the intended peers via untrusted networks.

Dunbar, et al.            Expires 16 April 2024                [Page 27]
Internet-Draft            SDWAN Edge Discovery              October 2023

   The secure propagation is achieved by secure channels, such as TLS,
   SSL, or IPsec, between the SD-WAN edge nodes and the local controller
   RR.

   SD-WAN edge nodes might not have secure channels with the RR.  In
   this case, BGP connection has be established over IPsec or TLS.

12.  IANA Considerations

12.1.  Hybrid (SD-WAN) Overlay SAFI

   IANA has assigned SAFI = 74 as the Hybrid (SD-WAN) SAFI.

12.2.  Tunnel Encapsulation Attribute Type

   IANA is requested to assign a type from the BGP Tunnel Encapsulation
   Attribute Tunnel Types as follows [RFC8126]:

    Value   Description    Reference
   -----   ------------   ---------
    25       SD-WAN-Hybrid   (this document)

12.3.  Tunnel Encapsulation Attribute Sub-TLV Types

   IANA is requested to assign the following sub-Types in the BGP Tunnel
   Encapsulation Attribute Sub-TLVs registry:

      Value   Type Description               Reference
      -----   -----------------------   ---------------
       64  IPSEC-SA-ID Sub-TLV             (Section 6.3)
       65  Extended Port Property Sub-TLV  (Section 6.4)
       66  Underlay Transport Sub-TLV      (Section 6.5)
       67  IPsec SA Nonce Sub-TLV          (Section 7.1)
       68  IPsec Public Key Sub-TLV        (Section 7.2)
       69  IPsec SA Proposal Sub-TLV       (Section 7.3)
       70  Simplified IPsec SA sub-TLV     (Section 7.4)

13.  References

13.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

Dunbar, et al.            Expires 16 April 2024                [Page 28]
Internet-Draft            SDWAN Edge Discovery              October 2023

   [RFC4271]  Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
              Border Gateway Protocol 4 (BGP-4)", RFC 4271,
              DOI 10.17487/RFC4271, January 2006,
              <https://www.rfc-editor.org/info/rfc4271>.

   [RFC4360]  Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended
              Communities Attribute", RFC 4360, DOI 10.17487/RFC4360,
              February 2006, <https://www.rfc-editor.org/info/rfc4360>.

   [RFC4364]  Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
              Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
              2006, <https://www.rfc-editor.org/info/rfc4364>.

   [RFC4760]  Bates, T., Chandra, R., Katz, D., and Y. Rekhter,
              "Multiprotocol Extensions for BGP-4", RFC 4760,
              DOI 10.17487/RFC4760, January 2007,
              <https://www.rfc-editor.org/info/rfc4760>.

   [RFC7296]  Kaufman, C., Hoffman, P., Nir, Y., Eronen, P., and T.
              Kivinen, "Internet Key Exchange Protocol Version 2
              (IKEv2)", STD 79, RFC 7296, DOI 10.17487/RFC7296, October
              2014, <https://www.rfc-editor.org/info/rfc7296>.

   [RFC7606]  Chen, E., Ed., Scudder, J., Ed., Mohapatra, P., and K.
              Patel, "Revised Error Handling for BGP UPDATE Messages",
              RFC 7606, DOI 10.17487/RFC7606, August 2015,
              <https://www.rfc-editor.org/info/rfc7606>.

   [RFC8126]  Cotton, M., Leiba, B., and T. Narten, "Guidelines for
              Writing an IANA Considerations Section in RFCs", BCP 26,
              RFC 8126, DOI 10.17487/RFC8126, June 2017,
              <https://www.rfc-editor.org/info/rfc8126>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC8277]  Rosen, E., "Using BGP to Bind MPLS Labels to Address
              Prefixes", RFC 8277, DOI 10.17487/RFC8277, October 2017,
              <https://www.rfc-editor.org/info/rfc8277>.

   [RFC8489]  Petit-Huguenin, M., Salgueiro, G., Rosenberg, J., Wing,
              D., Mahy, R., and P. Matthews, "Session Traversal
              Utilities for NAT (STUN)", RFC 8489, DOI 10.17487/RFC8489,
              February 2020, <https://www.rfc-editor.org/info/rfc8489>.

Dunbar, et al.            Expires 16 April 2024                [Page 29]
Internet-Draft            SDWAN Edge Discovery              October 2023

   [RFC9012]  Patel, K., Van de Velde, G., Sangli, S., and J. Scudder,
              "The BGP Tunnel Encapsulation Attribute", RFC 9012,
              DOI 10.17487/RFC9012, April 2021,
              <https://www.rfc-editor.org/info/rfc9012>.

13.2.  Informative References

   [IANA-AH]  IANA, "IANA-AH", <https://www.iana.org/assignments/isakmp-
              registry/isakmp-registry.xhtml#isakmp-registry-9>.

   [IANA-ESP] IANA, "IANA-ESP", <https://www.iana.org/assignments/
              isakmp-registry/isakmp-registry.xhtml#isakmp-registry-9>.

   [Net2Cloud]
              L. Dunbar, A Malis, C. Jacquenet, M. Toy and K. Majumdar,
              "Dynamic Networks to Hybrid Cloud DCs: Problem Statement
              and Mitigation Practice", September 2023,
              <https://datatracker.ietf.org/doc/draft-ietf-rtgwg-
              net2cloud-problem-statement/>.

   [RFC4684]  Marques, P., Bonica, R., Fang, L., Martini, L., Raszuk,
              R., Patel, K., and J. Guichard, "Constrained Route
              Distribution for Border Gateway Protocol/MultiProtocol
              Label Switching (BGP/MPLS) Internet Protocol (IP) Virtual
              Private Networks (VPNs)", RFC 4684, DOI 10.17487/RFC4684,
              November 2006, <https://www.rfc-editor.org/info/rfc4684>.

   [RFC5521]  Oki, E., Takeda, T., and A. Farrel, "Extensions to the
              Path Computation Element Communication Protocol (PCEP) for
              Route Exclusions", RFC 5521, DOI 10.17487/RFC5521, April
              2009, <https://www.rfc-editor.org/info/rfc5521>.

   [RFC9016]  Varga, B., Farkas, J., Cummings, R., Jiang, Y., and D.
              Fedyk, "Flow and Service Information Model for
              Deterministic Networking (DetNet)", RFC 9016,
              DOI 10.17487/RFC9016, March 2021,
              <https://www.rfc-editor.org/info/rfc9016>.

   [SD-WAN-BGP-USAGE]
              L. Dunbar, A Sajassi, J Drake, and B. Najem, "BGP Usage
              for SD-WAN Overlay Networks", September 2023,
              <https://datatracker.ietf.org/doc/draft-ietf-bess-bgp-
              sdwan-usage/>.

   [SECURE-EVPN]
              A Sajassi, et al, "Secure EVPN", July 2023,
              <https://datatracker.ietf.org/doc/draft-ietf-bess-secure-
              evpn/>.

Dunbar, et al.            Expires 16 April 2024                [Page 30]
Internet-Draft            SDWAN Edge Discovery              October 2023

Appendix A.  Acknowledgments

   Acknowledgements to Wang Haibo, Shunwan Zhuang, Hao Weiguo, and
   ShengCheng for implementation contribution; Many thanks to Yoav Nir,
   Graham Bartlett, Jim Guichard, John Scudder, and Donald Eastlake for
   their review and suggestions.

Contributors

   Below is a list of other contributing authors:

   Gyan Mishra Shunwan Zhuang Sheng Cheng Donald Eastlake

Authors' Addresses

   Linda Dunbar
   Futurewei
   Dallas, TX,
   United States of America
   Email: ldunbar@futurewei.com

   Kausik Majumdar
   Microsoft Azure
   California,
   United States of America
   Email: kmajumdar@microsoft.com

   Susan Hares
   Hickory Hill Consulting
   United States of America
   Email: shares@ndzh.com

   Robert Raszuk
   Arrcus
   United States of America
   Email: robert@raszuk.net

   Venkit Kasiviswanathan
   Arista
   United States of America
   Email: venkit@arista.com

Dunbar, et al.            Expires 16 April 2024                [Page 31]