Network Working Group L. Dunbar
Internet Draft J. Guichard
Intended status: Informational Futurewei
Expires: October 7, 2023 Ali Sajassi
Cisco
J. Drake
Juniper
B. Najem
Bell Canada
Ayan Barnerjee
D. Carrel
IPsec Research
April 7, 2023
BGP Usage for SD-WAN Overlay Networks
draft-ietf-bess-bgp-sdwan-usage-09
Abstract
The document discusses the usage and applicability of BGP as the
control plane for multiple SD-WAN scenarios. The document aims to
demonstrate how the BGP-based control plane is used for large-scale
SD-WAN overlay networks with little manual intervention.
SD-WAN edge nodes are commonly interconnected by multiple types of
underlay networks owned and managed by different network providers.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. This document may not be modified,
and derivative works of it may not be created, except to publish it
as an RFC and to translate it into languages other than English.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
xxx, et al. Expires October 7, 2023 [Page 1]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
This Internet-Draft will expire on October 7, 2023.
Copyright Notice
Copyright (c) 2023 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
Table of Contents
1. Introduction...................................................3
2. Conventions used in this document..............................4
3. Use Case Scenario Description and Requirements.................6
3.1. Requirements..............................................6
3.1.1. Supporting SD-WAN Segmentation.......................6
3.1.2. Client Service Requirement...........................6
3.1.3. SD-WAN Traffic Segmentation..........................7
3.1.4. Zero Touch Provisioning..............................7
3.1.5. Constrained Propagation of SD-WAN Edge Properties....8
3.2. Scenario #1: Homogeneous Encrypted SD-WAN.................9
3.3. Scenario #2: Differential Encrypted SD-WAN...............10
3.4. Scenario #3: Private VPN PE based SD-WAN.................12
4. Provisioning Model............................................13
Dunbar, et al. Expires October 7, 2023 [Page 2]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
4.1. Client Service Provisioning Model........................13
4.2. Policy Configuration.....................................14
4.3. IPsec related parameters Provisioning....................14
5. BGP Controlled SD-WAN.........................................14
5.1. Why BGP as Control Plane for SD-WAN?.....................14
5.2. BGP Walk Through for Homogeneous Encrypted SD-WAN........15
5.3. BGP Walk Through for Differential Encrypted SD-WAN.......17
5.4. BGP Walk Through for Application Flow-Based Segmentation.18
5.5. Benefit of Using Recursive Next Hop Resolution...........20
6. SD-WAN Forwarding Model.......................................20
6.1. Forwarding Model for Homogeneous Encrypted SD-WAN........21
6.1.1. Network and Service Startup Procedures..............21
6.1.2. Packet Walk-Through.................................21
6.2. Forwarding Model for Hybrid Underlay SD-WAN..............22
6.2.1. Network and Service Startup Procedures..............22
6.2.2. Packet Walk-Through.................................22
6.3. Forwarding Model for PE based SD-WAN.....................24
6.3.1. Network and Service Startup Procedures..............24
6.3.2. Packet Walk-Through.................................24
7. Manageability Considerations..................................25
8. Security Considerations.......................................25
9. IANA Considerations...........................................25
10. References...................................................26
10.1. Normative References....................................26
10.2. Informative References..................................26
11. Acknowledgments..............................................28
1. Introduction
Software Defined Wide Area Network (SD-WAN) optimizes the transport
of IP Packets over multiple underlay connectivity services. Here are
some of the main characteristics of "SD-WAN" networks:
- Transport Augmentation, referring to utilizing paths over
different underlay networks. There are often multiple parallel
overlay paths between any two SD-WAN edges; some are private
networks over which traffic can traverse with or without
encryption; others require encryption, e.g., over untrusted
public networks.
- Direct Internet breakout from remote branch offices is allowed
instead of all traffic hauled to Corporate HQ for centralized
policy control.
- Some traffic can be forwarded based on their application
identifiers instead of based on destination IP addresses by the
Dunbar, et al. Expires October 7, 2023 [Page 3]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
edge nodes placing the traffic onto specific overlay paths based
on the application-specific policies.
- The traffic forwarding can also be based on specific performance
criteria (e.g., packet delay, packet loss, jitter) to provide
better application performance by choosing the underlay that
meets or exceeds the specified policies.
[Net2Cloud-Problem] describes the network-related problems to
connect enterprises' branch offices to dynamic workloads in
different Cloud Data Centers (DC). SD-WAN has been positioned as a
flexible way to reach dynamic workloads in third-party Cloud DCs.
However, scaling becomes a significant issue when hundreds or
thousands of nodes need to be interconnected by SD-WAN overlay
networks.
This document describes using BGP as a control plane for SD-WAN
overlay networks and services. BGP for SD-WAN overlay is a different
layer from the underlay networks' BGP control plane instances.
2. Conventions used in this document
Cloud DC: Third party data centers that host applications and
workloads owned by different organizations or tenants.
Controller: Used interchangeably with SD-WAN controller to manage
SD-WAN overlay path creation/deletion and monitor the
path conditions between sites.
CPE: Customer Premise Equipment
CPE-Based VPN: Virtual Private Secure network formed among CPEs.
This differentiates from more commonly used PE-based
VPNs [RFC 4364].
Homogeneous Encrypted SD-WAN: A SD-WAN network in which all traffic
to/from the SD-WAN edges are carried by IPsec tunnels
regardless of underlay networks. I.e., the client
traffic is carried by IPsec tunnel even over MPLS
private networks.
ISP: Internet Service Provider
Dunbar, et al. Expires October 7, 2023 [Page 4]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
NSP: Network Service Provider. NSP usually provides more
advanced network services, such as MPLS VPN, private
leased lines, or managed Secure WAN connections, often
within a private, trusted domain. In contrast, an ISP
usually provides plain Internet services over public
untrusted domains.
PE: Provider Edge
SD-WAN Edge Node: an edge node, which can be physical or virtual,
maps the attached clients' traffic to the wide area
network (WAN) overlay tunnels.
SD-WAN: SD-WAN: Software Defined Wide Area Network. A
connectivity service offered by a Service Provider that
optimizes the transport of IP Packets over multiple
underlay connectivity services by recognizing
applications at Ingress and determining forwarding
behavior by applying policies to them.
SD-WAN IPsec SA: IPsec Security Association between two SD-WAN ports
or nodes.
SD-WAN over Hybrid Networks: SD-WAN over Hybrid Networks typically
have edge nodes utilizing bandwidth resources from
different types of underlay networks, some being private
networks and others being public Internet.
WAN Port: A Port or Interface facing an ISP or Network Service
Provider (NSP), with address allocated by the ISP or the
NSP.
C-PE: SD-WAN Edge node, which can be Customer Premises
Equipment (CPE) for customer-managed SD-WAN, or Provider
Edge (PE) for provider-managed SD-WAN services.
ZTP: Zero Touch Provisioning
Dunbar, et al. Expires October 7, 2023 [Page 5]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
3. Use Case Scenario Description and Requirements
This section describes some essential requirements for SD-WAN
networks and several SD-WAN scenarios used by the subsequent
sections to explain how the BGP control plane is applied.
3.1. Requirements
3.1.1. Supporting SD-WAN Segmentation
"SD-WAN Segmentation" is a frequently used term in SD-WAN
deployment, referring to policy-driven network partitioning. An SD-
WAN segment is a virtual private network (SD-WAN VPN) consisting of
a set of edge nodes interconnected by tunnels, such as IPsec tunnels
and MPLS VPN tunnels.
This document assumes that an SD-WAN VPN configuration on a PE
follows the same way as MPLS VPN, i.e., via VRFs. One SD-WAN VPN can
be mapped to one or multiple virtual topologies governed by the SD-
WAN controller's policies.
When using BGP for SD-WAN, the Client Route UPDATE is the same as
MPLS VPN. Route Target in the BGP Extended Community can be used to
differentiate the routes belonging to different SD-WAN VPNs.
As SD-WAN is an overlay network arching over multiple types of
networks, MPLS L2VPN/L3VPN or pure L2 underlay can continue using
the VPN ID, VN-ID, or VLAN in the data plane to differentiate
packets belonging to different SD-WAN VPNs. For packets carried by
an IPsec tunnel, the IPsec tunnel's inner encapsulation header can
have the SD-WAN VPN Identifier to distinguish the packets belonging
to different SD-WAN VPNs.
3.1.2. Client Service Requirement
The client interface of SD-WAN edges can be IP or Ethernet-based.
For Ethernet-based client interfaces, SD-WAN edge should support
VLAN-based service interfaces (EVI100), VLAN bundle service
interfaces (EVI200), or VLAN-Aware bundling service interfaces. EVPN
service requirements apply to client traffic, as described in
Section 3.1 of RFC8388.
For IP-based client interfaces, L3VPN service requirements are
applicable.
Dunbar, et al. Expires October 7, 2023 [Page 6]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
3.1.3. SD-WAN Traffic Segmentation
SD-WAN Traffic Segmentation enables the separation of the traffic
based on the business and the security needs of different user
groups and/or application requirements. Each user group and/or
application may need different isolated topologies and/or policies
to fulfill the business requirements.
For example, a retail business requires the point-of-sales (PoS)
application to be on a different topology from other applications.
The PoS application is routed only to the payment processing entity
at a hub site; other applications can be routed to all other sites.
The traffic from the PoS application follows a tree topology in the
figure below, whereas other traffic can follow a multipoint-to-
multipoint topology.
+--------+
Payment traffic |Payment |
+------+----+-+gateway +------+----+-----+
/ / | +----+---+ | \ \
/ / | | | \ \
+-+--+ +-+--+ +-+--+ | +-+--+ +-+--+ +-+--+
|Site| |Site| |Site| | |Site| |Site| |Site|
| 1 | | 2 | | 3 | | |4 | | 5 | | 6 |
+--+-+ +--+-+ +--|-+ | +--|-+ +--|-+ +--|-+
| | | | | | |
==+=======+=======+====+======+=======+=======+===
Multi-point connection for non-payment traffic
Another example is an enterprise that wants to isolate the traffic
from different departments, with each department having its unique
topology and policy. The HR department may need to access specific
applications that are not accessible by the engineering department.
Also, contractors may have limited access to the enterprise
resources.
3.1.4. Zero Touch Provisioning
SD-WAN zero-touch provisioning (ZTP) allows devices to be configured
and provisioned centrally. When an SD-WAN edge is installed at a
remote location, ZTP automates follow-up steps, including updates to
the OS, software version, and configuration, before client traffic
is forwarded. The ZTP can bootstrap a remote SD-WAN edge and
establish a secure connection to the local SD-WAN Controller, making
Dunbar, et al. Expires October 7, 2023 [Page 7]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
it convenient to add or delete an SD-WAN edge node (virtual or
physical). From the network control perspective, ZTP includes the
following:
- Upon power-up, an SD-WAN edge can establish the transport layer
secure connection (such as TLS, SSL, etc.) to its controller,
whose address can be burned or preconfigured on the device.
- The SD-WAN Controller can designate a local network controller
in the proximity of the SD-WAN edge. Like the Route-Reflector (RR)
for BGP-controlled SD-WAN, the local network controller manages
and monitors the communication policies for traffic to/from the
edge node.
3.1.5. Constrained Propagation of SD-WAN Edge Properties
One SD-WAN edge node may only be authorized to communicate with a
small number of other SD-WAN edge nodes. Under this circumstance,
the property of the SD-WAN edge node cannot be propagated to other
nodes that are not authorized to communicate. But a remote SD-WAN
edge node, upon powering up, might not have the right policies to
know which peers are authorized to communicate. Therefore, SD-WAN
deployment needs to have a central point to distribute the
properties of an SD-WAN edge node to its authorized peers.
BGP is well suited for this purpose. RFC4684 has specified the
procedure to constrain the distribution of BGP UPDATE to only a
subset of nodes. Each edge node informs the Route-Reflector (RR)
[RFC4456] on its interested SD-WAN VPNs. The RR only propagates the
BGP UPDATE for the relevant SD-WAN VPNs to the edge.
The connection between an SD-WAN edge and its RR can be over an
insecure network. Therefore, an SD-WAN edge must establish a secure
transport layer connection (TLS, SSL, etc.) to its designated RR
upon power-up. The BGP UPDATE messages need to be sent over the
secure channel (TLS, SSL, etc.) to the RR.
Dunbar, et al. Expires October 7, 2023 [Page 8]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
+---+
Peer Group 1 |RR | Peer Group 2
+======+====+=+ +======+====+=====+
/ / | +---+ | \ \
/ / | | \ \
+-+--+ +-+--+ +-+--+ +-+--+ +-+--+ +-+--+
|C-PE| |C-PE| |C-PE| |C-PE| |C-PE| |C-PE|
| 1 | | 2 | | 3 | |4 | | 5 | | 6 |
+----+ +----+ +----+ +----+ +----+ +----+
Tenant 1 Tenant 2
Figure 1: Peer Groups managed by RR
Tenant separation is achieved by the SD-WAN VPN identifiers
represented in the control plane and data plane, respectively.
3.2. Scenario #1: Homogeneous Encrypted SD-WAN
Homogeneous Encrypted SD-WAN refers to an SD-WAN network with edge
nodes encrypting all traffic over the WAN underlay to other edge
nodes, regardless of whether the underlay is private or public. For
lack of better terminology, we call this Homogeneous Encrypted SD-
WAN throughout this document.
Here are some typical scenarios for using Homogeneous Encryption:
- A small branch office to connect to its HQ offices via the
Internet. All traffic to/from this small branch office must be
encrypted, usually achieved by IPsec Tunnels [RFC6071].
- A store in a shopping mall may need to securely connect to its
applications in one or more Cloud DCs via the Internet. A common way
of achieving this is to establish IPsec SAs to the Cloud DC gateway
to carry the sensitive data to/from the store.
As described in [SECURE-EVPN], the granularity of the IPsec SAs for
Homogeneous Encryption can be per site, per subnet, per tenant, or
per address. Once the IPsec SA is established for a specific
subnet/tenant/site, all traffic to/from the subnet/tenant/site is
encrypted.
Dunbar, et al. Expires October 7, 2023 [Page 9]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
+---+
+--------------|RR |------------+
/ Untrusted +-+-+ \
/ \
/ \
+----+ +---------+ +------+ +----+
| CN3|--| P1-----+ -------------+------ P1 |--| CN3|
+----+ | C-PE1 P2-----+ | | C-PE2| +----+
+----+ | P3-----+ Wide +------ P2 | +----+
| CN2|--| | | area +------ P3 |--| CN1|
+-+--+ +---------+ | network | +------+ +-+--+
\ | | /
\ +---------+ | all packets | +------+ /
+--| P1-----+ encrypted +------ P1 |-+
| C-PE3 P2-----+ by | | C-PE4|
+----+ | P3-----+ IPsec SAs +------ P2 | +----+
| CN1|--| P4-----+--------------+ | |--| CN2|
+----+ +---------+ +------+ +----+
CN: Client Networks, which is same as Tenant Networks used by NVo3
Figure 2: Homogeneous Encrypted SD-WAN
One of the properties of Homogeneous Encryption is that the SD-WAN
Local Network Controller, e.g., RR in BGP-controlled SD-WAN, might
be connected to C-PEs via an untrusted public network, therefore,
requiring a secure connection between RR and C-PEs (TLS, DTLS,
etc.).
Homogeneous Encrypted SD-WAN has some properties similar to the
commonly deployed IPsec VPN, albeit the IPsec VPN is usually point-
to-point among a small number of nodes and with heavy manual
configuration for IPsec between nodes. In contrast, an SD-WAN
network can have many edge nodes and a central controller to manage
the configurations on the edge nodes.
Existing private VPNs (e.g., MPLS based) can use Homogeneous
Encrypted SD-WAN to extend over the public network to remote sites
to which the VPN operator does not own or lease infrastructural
connectivity, as described in [SECURE-EVPN] and [SECURE-L3VPN]
3.3. Scenario #2: Differential Encrypted SD-WAN
The Differential Encrypted SD-WAN refers to an SD-WAN network in
which traffic over the existing VPN is forwarded natively without
encryption, and the traffic over the Public Internet is encrypted.
Differential Encrypted SD-WAN is over hybrid private VPN and public
Internet underlays. Since IPsec requires additional processing power
Dunbar, et al. Expires October 7, 2023 [Page 10]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
and the encrypted traffic over the Internet does not have the
premium SLA commonly offered by Private VPNs, especially over a long
distance, it is more desirable for traffic over a private VPN to be
forwarded without encryption.
One C-PE might have the Internet-facing WAN ports managed by
different ISPs/NSPs with the WAN ports' addresses assigned by the
corresponding ISPs/NSPs. Clients might have policies to specify:
1) Some flows can only be forwarded over private VPNs.
2) Some flows can be forwarded over either private VPNs or the public
Internet. The packets over the public Internet are encrypted.
3) Some flows, especially Internet-bound browsing ones, can be handed
off to the Internet without any encryption.
Suppose a flow traversing multiple segments, such as A<->B<->C<->D,
has Policy 2) above. The flow can cross different underlays in
different segments, such as over Private underlay between A<->B
without encryption or over the public Internet between B<->C
protected by an IPsec SA.
As shown in the figure below, C-PE-1 has two different types of
interfaces (A1 to Internet and A2 & A3 to VPN). The C-PE's loopback
address and the attached client addresses may or may not be visible
to the ISPs/NSPs. The WAN ports' addresses can be allocated by the
service providers or dynamically assigned (e.g., by DHCP).
Dunbar, et al. Expires October 7, 2023 [Page 11]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
+---+
+--------------|RR |----------+
/ Untrusted +-+-+ \
/ \
/ \
+----+ +---------+ packets encrypted over +------+ +----+
| CN3|--| A1-----+ Untrusted +------ B1 |--| CN1|
+----+ | C-PE1 A2-\ | C-PE2| +----+
+----+ | A3--+--+ +---+---B2 | +----+
| CN2|--| | |PE+--------------+PE |---B3 |--| CN3|
+----+ +---------+ +--+ trusted +---+ +------+ +----+
| WAN |
+----+ +---------+ +--+ packets +---+ +------+ +----+
| CN1|--| C1--|PE| go natively |PE |-- D1 |--| CN1|
+----+ | C-PE3 C2--+--+ without encry+---+ | C-PE4| +----+
| | +--------------+ | |
| | | |
+----+ | | without encrypt over | | +----+
| CN2|--| C3--+---- Untrusted --+------D2 |--| CN2|
+----+ +---------+ +------+ +----+
CN: Client Network
Figure 3: SD-WAN with Hybrid Underlays
Also, the connection between C-PEs and their Controller (RR) might
be via the untrusted public network. It is necessary to encrypt
the communication between RR and C-PEs, by TLS, DTLS, etc.
There could be multiple SD-WAN edges (C-PEs) sharing common
property, such as a geographic location. Some applications over
SD-WAN may need to traverse specific geographic areas for various
reasons, such as to comply with regulatory rules, to utilize
specific value-added services, or others.
Services may not be congruent, i.e., the packets from A-> B may
traverse one underlay network, and the packets from B -> A may go
over a different underlay.
3.4. Scenario #3: Private VPN PE based SD-WAN
This scenario refers to the existing VPN (e.g., EVPN or IPVPN) being
expanded by adding extra ports facing the untrusted Internet for PEs
to offload low-priority traffic when the VPN paths are congested.
Dunbar, et al. Expires October 7, 2023 [Page 12]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
Throughout this document, this scenario is also called Internet
Offload for Private VPN, or PE-based SD-WAN.
Here are some differences from the Hybrid Underlay scenario (Section
3.3):
- For MPLS-based VPN, PEs would have MPLS as payload
encapsulated within the IPsec tunnel egressing the Internet
WAN ports, MPLS-in-IP/GRE-in-IPsec.
- The BGP RR is connected to PEs in the same way as VPN, i.e.,
via the trusted network.
The PE-based SD-WAN can be used by VPN service providers to
temporarily increase bandwidth between sites when not sure if the
demand will sustain for an extended period or as a temporary
solution before the permanent infrastructure is built or leased.
+---+
+======>|PE2|
// +---+
// ^
// || VPN
// VPN v
++--+ ++-+ +---+
|PE1| <====> |RR| <===> |PE3|
+-+-+ +--+ +-+-+
| |
+--- Public Internet -- +
Offload
Figure 4: Additional Internet paths added to the VPN
4. Provisioning Model
4.1. Client Service Provisioning Model
Client service provisioning can follow the same approach as MPLS
VRFs. A client VPN can establish the communication policy by
specifying the Route Targets to be imported and exported.
Alternatively, traditional Match and Action ACLs can identify the
specific routes allowed or denied to or from the client VPN.
Dunbar, et al. Expires October 7, 2023 [Page 13]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
When an SD-WAN edge node is dedicated to one client with one virtual
network, all the prefixes attached to the client port(s) of the edge
node can be considered in one VRF, and the RR can manage the
policies for import/export of the VRF.
4.2. Policy Configuration
One of the characteristics of an SD-WAN service is that packets can
be forwarded over multiple types of underlays. Policies are needed
to govern which underlay paths can carry an application flow, as
described by Section 8 of MEF70.1. An Application Flow consists of
packets that match specific criteria. For example, client-prefix-x
can only be mapped to MPLS topology.
4.3. IPsec related parameters Provisioning
SD-WAN edge nodes must negotiate the supported IPsec encryption
algorithms (DES, 3DES, or AES), the hash algorithm (SHA or MD5), and
the DH groups to establish IPsec tunnels between them. Each SD-WAN
edge can have the default supported values for those attributes or
get them from its controller to minimize the configuration. For a
BGP-controlled SD-WAN, BGP UPDATE messages can propagate each node's
IPsec-related attribute values for peers to choose the common values
supported, traditionally done by IPsec IKEv2 [RFC7296].
5. BGP Controlled SD-WAN
5.1. Why BGP as Control Plane for SD-WAN?
For an SD-WAN network with a small number of nodes, the traditional
hub & spoke model utilizing Next Hop Resolution Protocol (NHRP) or
Dynamic Smart VPN (DSVPN)/Dynamic Multipoint VPN (DMVPN) protocol
had worked reasonably well. DSVPN/DMVPN has a hub node (or
controller) managing the edge nodes, including local & public
addresses and tunnel identifiers mapping. However, for a sizeable
SD-WAN network, say more than 100 nodes with different underlays,
the traditional approach becomes very messy, complex, and error-
prone.
Here are some of the compelling reasons for using BGP:
- Simplified peer authentication process:
With a secure management channel established between an edge node
and RR, RR can perform peer authentication on behalf of the edge
node. RR has policies on peer communication and the built-in
Dunbar, et al. Expires October 7, 2023 [Page 14]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
capability to constrain the propagation of the UPDATE messages to
the authorized edge nodes [RFC4684].
- Scalable IPsec tunnel management
When multiple IPsec tunnels are established between two pairwise
edge nodes, BGP Tunnel Attribute Update can associate multiple
IPsec tunnels with the client routes. In traditional IPsec VPN,
separate routing protocols must run in parallel in each IPsec
Tunnel if the client routes can be load shared among the IPsec
tunnels.
- Simplified IPsec tunnel traffic selection configurations
The IPsec tunnel's traffic selector or admission control can be
inherently realized by specifying importing/exporting the Route
Targets representing the SD-WAN VPNs.
5.2. BGP Walk Through for Homogeneous Encrypted SD-WAN
For the BGP-controlled Homogeneous Encrypted SD-WAN, a C-PE can
advertise its attached client routes and the properties of the IPsec
SA in one BGP UPDATE message.
In the Figure below, the BGP UPDATE message from C-PE2 to RR can
have the client routes encoded in the MP-NLRI Path Attribute and the
IPsec Tunnel associated information encoded in the Tunnel-Encap
[RFC9012] Path Attributes as described in the [SECURE-EVPN].
Dunbar, et al. Expires October 7, 2023 [Page 15]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
+---+
+---------|RR |----------+
/ Untrusted+---+ \
/ \
/ \
+---------+ +-------+
--+ |-----------------------| |-10.1.x.x/16
| | |C-PE2 |- VLAN = 15
| C-PE1 | +-|2.2.2.2|
--| 1.1.1.1 | | | |-12.1.1.x/24
+---------+ | +-------+
|
|
|
+---------+ |
--| |---------------------+
| |
| C-PE3 |
--| 3.3.3.3 |
+---------+
Figure 5: Homogeneous Encrypted SD-WAN
Alternatively, the C-PE2 can use two separate BGP UPDATE messages to
reduce the size of the BGP UPDATE messages, as IPsec SA tunnels have
many attributes and IPsec SA keys periodical changes occur at
different frequency than the clients' routes updates.
As described by Section 4 and 8 of [RFC9012], UPDATE U1 has its
Nexthop to the node loopback address and is recursively resolved to
the IPsec SA tunnel detailed attributes advertised by the UPDATE U2
for the Node Loopback address.
Here are the details of the UPDATE messages:
- Suppose that a given packet P destined towards the client
addresses attached to C-PE2 (e.g., prefix 10.1.x.x/16) can be
carried by any IPsec tunnels terminated at C-PE2.
- The path along which P is to be forwarded is determined by BGP
UPDATE U1.
- UPDATE U1 does not have a Tunnel Encapsulation attribute.
- UPDATE U1 can include the Encapsulation Extended Community with
the option to have the Color Extended Community.
- The address of the next-hop of UPDATE U1 is router C-PE2.
- UPDATE U2 has a Tunnel Encapsulation attribute to describe the
IPsec SA detailed attributes.
Dunbar, et al. Expires October 7, 2023 [Page 16]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
UPDATE U1:
- MP-NLRI Path Attribute:
10.1.x.x/16
12.1.1.x/24
- Nexthop: 2.2.2.2 (C-PE2)
- Encapsulation Extended Community: TYPE = IPsec
UPDATE U2:
- MP-NLRI Path Attribute:
2.2.2.2 (C-PE2)
- Tunnel Encapsulation Path Attributes (as described in the
[SECURE-EVPN]) for IPsec SA detailed attributes, including the WAN
address to be used as the IP address of the IPsec encrypted
packets.
If different client routes attached to C-PE2 need to be reached by
separate IPsec tunnels, the Color-Extended-Community [RFC9012] is
used to associate routes with the tunnels. See Section 8 of
[RFC9012].
Suppose C-PE2 does not have a policy on the authorized peers for the
specific client routes. Then, RR needs to check the client routes
policies to constrain the BGP UPDATE messages propagation only to
the remote authorized edge nodes.
5.3. BGP Walk Through for Differential Encrypted SD-WAN
In this scenario, some client routes can be forwarded over any one
of the tunnels terminating at the edge node. Some client routes can
only be forwarded over specific tunnels (such as only MPLS VPN).
An edge node can use the Color Extended Community (Section 4 & 8 of
[RFC9012]) in its BGP UPDATE message to associate the client routes
with the specific tunnels.
For example, in Figure 5 above, suppose that Route 10.1.x.x/16 can
be carried by either MPLS or IPsec and Route 12.1.1.x/24 can only be
carried by MPLS; C-PE2 can use the following UPDATE messages:
Dunbar, et al. Expires October 7, 2023 [Page 17]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
UPDATE #1a for Route Route 10.1.x.x/16:
- MP-NLRI Path Attribute:
10.1.x.x/16
Nexthop: 2.2.2.2 (C-PE2)
- Encapsulation Extended Community: TYPE = SD-WAN-Hybrid
- Color Extended Community: RED
UPDATE #1b for Route Route 12.1.1.x/24:
- MP-NLRI Path Attribute:
12.1.1.x/24
Nexthop: 2.2.2.2 (C-PE2)
- Encapsulation Extended Community: TYPE = MPLS-in-GRE
- Color Extended Community: YELLOW
UPDATE #2a: for IPsec tunnels terminated at the node:
- MP-NLRI Path Attribute:
2.2.2.2 (C-PE2)
- Tunnel Encapsulation Path Attributes: TYPE=SD-WAN-Hybrid
Including the information about the WAN ports for receiving IPsec
encrypted packets, the IPsec properties, etc.
- Color Extended Community: RED
UPDATE #2b: for MPLS-in-GRE terminated at the node:
- MP-NLRI Path Attribute:
2.2.2.2 (C-PE2)
- Tunnel Encapsulation Path Attributes: TYPE=SD-WAN-Hybrid
- Color Extended Community: YELLOW
SD-WAN-Hybrid Tunnel Type is specified by [SD-WAN-EDGE-Discovery].
5.4. BGP Walk Through for Application Flow-Based Segmentation
Suppose an application flow is identified by its source or
destination IP addresses. Then, constraining the BGP UPDATE messages
for the application only to the nodes that meet the criteria of the
application flow can achieve the Application Flow-based Segmentation
described in Section 3.1.2. In the Figure below, the following BGP
Updates can be advertised to ensure that the Payment Application
only communicates with the Payment Gateway:
Dunbar, et al. Expires October 7, 2023 [Page 18]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
BGP UPDATE #1a from C-PE2 to RR for the P2P topology that is only
propagated to Payment GW node:
UPDATE #1a from C-PE2 to RR is only propagated to the Payment GW
node:
- MP-NLRI Path Attribute:
- 30.1.1.x/24
- Nexthop: 2.2.2.2
- Encapsulation extended community: TYPE = IPsec
- Color Extended Community: BLUE
BGP UPDATE #1b from C-PE2 to RR is propagated to C-PE1 & C-PE3 for
the routes to be reached by C-PE1 and C-PE3:
- MP-NLRI Path Attribute:
- 10.1.x.x
- 12.4.x.x
- Nexthop:2.2.2.2
- Encapsulation extended community: TYPE =IPsec
- Color Extended Community: RED
BGP UPDATE #2 for the detailed IPsec attributes for IPsec tunnels
terminated at C-PE2 2.2.2.2 is propagated to C-PE1 & C-PE3.
UPDATE #2a: for all IPsec SAs terminated at the node:
- MP-NLRI Path Attribute:
2.2.2.2 (C-PE2)
- Tunnel Encapsulation Path Attributes: TYPE=IPsec (for all IPsec
SAs)
- Color Extended Community: RED
UPDATE #2b for the IPsec SA to the Payment GW is only propagated to
Payment GW:
- MP-NLRI Path Attribute:
2.2.2.2 (C-PE2)
- Tunnel Encapsulation Path Attributes: TYPE=IPsec (for the IPsec
SA to Payment GW).
Dunbar, et al. Expires October 7, 2023 [Page 19]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
- Color Extended Community: Blue
+-------+
|Payment|
+------->| GW |<----+
/ +-------+ \
/ Blue Tunnel \
/for Payment App:30.1.1.X/24 \
/ \
+------/--+ +--\----+
--|-----+ | | +---| 30.1.1.x/24
| | Red Tunnels | |
--| C-PE1 |--------------------------| |-10.1.x.x/16
|1.1.1.1 | |C-PE2 |
--| |--------------------------|2.2.2.2|- 12.1.1.x/24
| | | |
+---------+ +--| |- VLAN=25;
/ | | 22.1.1.x/24
+---------+ / +-------+
--| |--------------------+
| C-PE3 | /
| 3.3.3.3 | /
--| |-----------------+
+---------+
Figure 6: Application Based SD-WAN Segmentation
5.5. Benefit of Using Recursive Next Hop Resolution
Using the Recursive Next Hop Resolution described in Section 8 of
[RFC9012], the clients' routes UPDATE messages become very compact,
and any changes of the underlay network tunnels attributes can be
advertised without client route update. This method is handy when
the underlay tunnels are IPsec based, which requires periodic
message exchange for the pairwise re-keying process.
6. SD-WAN Forwarding Model
This section describes how client traffic is forwarded in BGP
Controlled SD-WAN for the use cases described in Section 3.
The procedures described in Section 6 of RFC8388 are applicable for
the SD-WAN client traffic. Like the BGP-based VPN/EVPN client routes
UPDATE message, Route Target can distinguish routes from different
clients.
Dunbar, et al. Expires October 7, 2023 [Page 20]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
6.1. Forwarding Model for Homogeneous Encrypted SD-WAN
6.1.1. Network and Service Startup Procedures
A single IPsec security association (SA) protects data in one
direction. Under the Homogeneous Encrypted SD-WAN Scenario, two SAs
must be present to secure traffic in both directions between two C-
PE nodes, two client ports, or two prefixes. Using Figure 2 of
Section 3.2 as an example, for client CN2 attached to C-PE1, C-PE3,
and C-PE4 to have full-mesh connection, six one-directional IPsec
SAs must be established: C-PE1 <-> C-PE3; C-PE1 <-> C-PE4; C-PE3 <->
C-PE4.
SD-WAN services to clients can be IP-based or Ethernet-based. An SD-
WAN edge can learn client routes from the client-facing ports via
OSPF, RIP, BGP, or static configuration for its IP-based services.
For Layer-2 SD-WAN services, the relevant EVPN parameters, such as
the ESI (Ethernet Segment Identifier), EVI, and CE-VID to EVI
mapping, can be configured similarly to EVPN described in RFC8388.
Instead of running IGP within each IPsec tunnel as done by the
traditional IPsec VPN, BGP UPDATE messages propagate the client
routes attached to SD-WAN edge nodes.
In addition, BGP-RR (SD-WAN Controller) facilitates the IPsec SA
establishment and rekey management described in [SECURE-EVPN]. The
Controller manages how clients' routes are associated with
individual IPsec SA. Therefore, it is no longer necessary to
manually configure the IPsec tunnel's endpoint addresses on each SD-
WAN edge node and set up policies for the allowed client prefixes.
6.1.2. Packet Walk-Through
For an IPsec SA terminated at a C-PE node, multiple client routes
can be multiplexed in the IPsec SA (or tunnel). Traffic to the
client prefixes is encapsulated in an inner tunnel, such as GRE or
VxLAN, carried by the IPsec SA ESP tunnel. Different client traffic
can be differentiated by a unique value in the inner encapsulation
key or ID field.
For unicast packets forwarding:
The C-PE node address (or loopback address) acts as the Next Hop
address for the prefixes attached to the C-PE nodes.
C-PE Node-based IPsec tunnel is inherently protected when the C-PE
has multiple WAN ports to different underlay paths. As shown in
Dunbar, et al. Expires October 7, 2023 [Page 21]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
Figure 2, when one of the underlay paths fails, the IPsec traffic
can be forwarded to or received from a different physical port.
When a C-PE receives a packet from its client port, the packet is
encapsulated inside the IPsec SA, whose destination address
matches the Next Hop address of the packet's destination and
forwarded to the target C-PE.
When a C-PE receives an IPsec encrypted packet from its WAN ports,
it decrypts the packet and forwards the inner packet to the client
port based on the inner packet's destination address.
For multicast packets forwarding:
IPsec was created to be a security protocol between two and only
two devices, so multicast service using IPsec is problematic. An
IPsec peer encrypts a packet so that only one other IPsec peer can
successfully perform the de-encryption. A straight way to forward
a multicast packet for the Homogeneous Encrypted SD-WAN is to
encapsulate the multicast packet in separate unicast IPsec SA
tunnels. More optimized forwarding multicast packets for the
Homogeneous Encrypted SD-WAN is out of the scope of this document.
6.2. Forwarding Model for Hybrid Underlay SD-WAN
In this scenario, as shown in Figure 3 of Section 3.3, traffic
forwarded over the trusted VPN paths can be native (i.e.,
unencrypted). The traffic forwarded over untrusted networks need to
be protected by IPsec SA.
6.2.1. Network and Service Startup Procedures
Infrastructure setup: The proper MPLS infrastructure must be set up
among the edge nodes, i.e., the C-PE1/C-PE2/C-PE3/C-PE4 of Figure 3.
The IPsec SA between WAN ports or nodes must be set up as well.
IPsec SA related attributes on edge nodes can be distributed by BGP
UPDATE messages as described in Section 5.
There could be policies governing how flows can be forwarded, as
specified by MEF70.1. For example, "Private-only" indicates that
the flows can only traverse the MPLS VPN underlay paths.
6.2.2. Packet Walk-Through
For unicast packets forwarding:
Dunbar, et al. Expires October 7, 2023 [Page 22]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
Upon receiving a packet from a client port, if the packet belongs
to a flow that can only be forwarded over the MPLS VPN, the
forwarding processing is the same as the MPLS VPN. Otherwise, the
C-PE node can make the local decision in choosing the least cost
path, including the prior established MPLS paths and IPsec
Tunnels, to forward the packet. Packets forwarded over the trusted
MPLS VPN can be native without any additional encryption, while
the packets sent over the untrusted networks need to be encrypted
by IPsec SA.
For a C-PE with multiple WAN ports provided by different ISPs,
separate IPsec SAs can be established for the different WAN ports.
In this case, the C-PE have multiple IPsec tunnels in addition to
the MPLS path to choose from to forward the packets from the
client ports.
If the IPsec SA is chosen, the packet is encapsulated by the IPsec
inner packet header and encrypted by the IPsec SA before
forwarding to the WAN.
For packets received from a MPLS path, processing is the same as
MPLS VPN.
For IPsec SA encrypted packets received from the WAN ports, the
packets are decrypted, and the inner payload is decapsulated and
forward per the forwarding table of the C-PE. For all packets from
the Internet-facing WAN ports, the additional anti-DDoS mechanism
has to be enabled to prevent potential attacks from the Internet-
facing ports. Control Plane should not learn routes from the
Internet-facing WAN ports.
+---+
+--------------|RR |----------+
/ +-+-+ \
/ \
/ \
+----+ +---------+ packets encrypted over +--------+ +----+
| CN3|--| A1-----+ Untrusted +------ B1 |--| CN1|
+----+ | C-PE-a A2-----+ +-------B2 C-PE-b| +----+
|10.1.1.1 | |10.1.2.1|
+----+ | | +--+ +---+ | | +----+
| CN2|--| A3 |PE+--------------+PE |---B3 |--| CN3|
+----+ +---------+ +--+ trusted +---+ +--------+ +----+
| VPN |
+--------------+
Figure 8: SD-WAN with Hybrid Underlays
Dunbar, et al. Expires October 7, 2023 [Page 23]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
For multicast packets forwarding:
For multicast traffic, MPLS multicast [RFC6513, RFC6514, or
RFC7988] can be used to forward multicast traffic.
If IPsec tunnels are chosen for a multicast packet, the packet is
encapsulated and encrypted by multiple separate IPsec tunnels to
the desired destinations.
6.3. Forwarding Model for PE based SD-WAN
6.3.1. Network and Service Startup Procedures
In this scenario, all PEs have secure interfaces facing the clients
and facing the MPLS backbone, with some PEs having extra connections
by untrusted public Internet. The public Internet paths are for
offloading low priority traffic when the MPLS paths get congested.
The PEs are already connected to their RRs, and the configurations
for the clients and policies are already established.
6.3.2. Packet Walk-Through
For PEs to offload some MPLS packets to the Internet path, each MPLS
packet is wrapped by an outer IP header as MPLS-in-IP or MPLS-in-GRE
[RFC4023]. The outer IP address can be an interface address or the
PE's loopback address.
When IPsec Tunnel mode is used to protect an MPLS-in-IP packet, the
entire MPLS-in-IP packet is placed after the IPsec tunnel header.
When IPsec transport mode is used to protect the MPLS packet, the
MPLS-in-IP packet's IP header becomes the outer IP header of the
IPsec packet, followed by an IPsec header, and then followed by the
MPLS label stack. The IPsec header must set the payload type to MPLS
by using the IP protocol number specified in section 3 of RFC4023.
If IPsec transport mode is applied to an MPLS-in-GRE packet, the GRE
header follows the IPsec header.
The IPsec SA's endpoints should not be the client-facing interface
addresses unless the traffic to/from those clients always goes
through the IPsec SA even when the MPLS backbone has enough capacity
to transport the traffic.
When the PEs' Internet-facing ports are behind the NAT [RFC3715], an
outer UDP field can be added outside the encrypted payload
[RFC3948]. Three UDP ports must be open on the PEs: UDP port 4500
Dunbar, et al. Expires October 7, 2023 [Page 24]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
(used for NAT traversal), UDP port 500 (used for IKE), and IP
protocol 50 (ESP). IPsec IKE (Internet Key Exchange) between the two
PEs would be over the NAT [RFC3947] as well.
Upon receiving a packet from a client port, the forwarding
processing is the same as the MPLS VPN. If the MPLS backbone path to
the destination is deemed congested, the IPsec tunnel towards the
target PEs is used to encrypt the MPLS-in-IP packet.
Upon receiving a packet from the Internet-facing WAN port, the
packet is decrypted, and the inner MPLS payload is extracted to be
sent to the MPLS VPN engine.
Same as Scenario #2, the additional anti-DDoS mechanism must be
enabled to prevent potential attacks from the Internet-facing port.
Control Plane should not learn routes from the Internet-facing WAN
ports.
7. Manageability Considerations
BGP-controlled SD-WAN utilizes the BGP RR to facilitate the routes
and underlay properties distribution among the authorized edge
nodes. With RR having the preconfigured policies about the
authorized peers, the peer-wise authentications of the IPsec IKE
(Internet Key Exchange) are significantly simplified.
8. Security Considerations
Adding an Internet-facing WAN port to a C-PE can introduce the
following security risks:
1) Potential DDoS attacks from the Internet-facing ports.
2) Potential risk of provider VPN network being injected with
illegal traffic from the Internet-facing WAN ports.
Therefore, the additional anti-DDoS mechanism must be enabled on all
Internet-facing ports to prevent potential attacks from those ports.
Control Plane should not learn any routes from the Internet-facing
WAN ports.
9. IANA Considerations
No Action is needed.
Dunbar, et al. Expires October 7, 2023 [Page 25]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
10. References
10.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC4364] E. rosen, Y. Rekhter, "BGP/MPLS IP Virtual Private
networks (VPNs)", Feb 2006.
[RFC6071] S. Frankel, S. Krishan, "IP Security (IPsec) and Internet
Key Exchange (IKE) Document Roadmap", Feb 2011.
[RFC7296] C. Kaufman, et al, "Internet Key Exchange Protocol Version
2 (IKEv2)", Oct 2014.
[RFC7432] A. Sajassi, et al, "BGP MPLS-Based Ethernet VPN", Feb
2015.
[RFC8365] A. Sajassi, et al, "A network Virtualization Overlay
Solution Using Ethernet VPN (EVPN)", March 2018.
[RFC9012] K.Patel, et al "The BGP Tunnel Encapsulation Attribute",
RFC9012, April 2021.
10.2. Informative References
[RFC8192] S. Hares, et al, "Interface to Network Security Functions
(I2NSF) Problem Statement and Use Cases", July 2017
[RFC5521] P. Mohapatra, E. Rosen, "The BGP Encapsulation Subsequent
Address Family Identifier (SAFI) and the BGP Tunnel
Encapsulation Attribute", April 2009.
[RFC8388] J. Rabadan, et al, "Usage and Applicability of BGP MPLS-
Based Ethernet VPN", May 2018.
Dunbar, et al. Expires October 7, 2023 [Page 26]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
[Net2Cloud-Gap] L. Dunbar, A. Malis, C. Jacquenet, "Gap Analysis of
Interconnecting Underlay with Cloud Overlay", draft-dm-
net2cloud-gap-analysis-02, work in progress, Oct. 2018.
[SD-WAN-EDGE-Discovery] L. Dunbar, S. Hares, R. Raszuk, K. Majumdar,
"BGP UPDATE for SD-WAN Edge Discovery", draft-ietf-idr-SD-
WAN-edge-discovery-05, Aug 2022.
[VPN-over-Internet] E. Rosen, "Provide Secure Layer L3VPNs over
Public Infrastructure", draft-rosen-bess-secure-l3vpn-00,
work-in-progress, July 2018
[DMVPN] Dynamic Multi-point VPN:
https://www.cisco.com/c/en/us/products/security/dynamic-
multipoint-vpn-dmvpn/index.html
[DSVPN] Dynamic Smart VPN:
http://forum.huawei.com/enterprise/en/thread-390771-1-
1.html
[SECURE-EVPN] A. Sajassi, et al, "Secure EVPN", draft-sajassi-bess-
secure-evpn-01, Work-in-progress, March 2019.
[SECURE-L3VPN] E. Rosen, R. Bonica, "Secure Layer L3VPN over Public
Infrastructure", draft-rosen-bess-secure-l3vpn-00, Work-
in-progress, June 2018.
[ITU-T-X1036] ITU-T Recommendation X.1036, "Framework for creation,
storage, distribution and enforcement of policies for
network security", Nov 2007.
[Net2Cloud-Problem] L. Dunbar and A. Malis, "Seamless Interconnect
Underlay to Cloud Overlay Problem Statement", draft-rtgwg-
net2cloud-problem-statement-12, March 2022
[Net2Cloud-gap] L. Dunbar, A. Malis, and C. Jacquenet, "Gap Analysis
of Interconnecting Underlay with Cloud Overlay", draft-
rtgwg-net2cloud-gap-analysis-07, work-in-progress, July
2020.
Dunbar, et al. Expires October 7, 2023 [Page 27]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
11. Acknowledgments
Acknowledgements to Adrian Farrel, Joel Halpern, John Scudder,
Darren Dukes, Andy Malis, Donald Eastlake, and Victo Sheng for their
review and contributions.
This document was prepared using 2-Word-v2.0.template.dot.
Dunbar, et al. Expires October 7, 2023 [Page 28]
Internet-Draft BGP Usage for SD-WAN April 7, 2023
Authors' Addresses
Linda Dunbar
Futurewei
Email: ldunbar@futurewei.com
James Guichard
Futurewei
Email: james.n.guichard@futurewei.com
Ali Sajassi
Cisco
Email: sajassi@cisco.com
John Drake
Juniper
Email: jdrake@juniper.net
Basil Najem
Bell Canada
Email: basil.najem@bell.ca
David Carrel
IPsec Research
Email: carrel@ipsec.org
Ayan Banerjee
Cisco
Email: ayabaner@cisco.com
Dunbar, et al. Expires October 7, 2023 [Page 29]
