BGP Usage for SDWAN Overlay Networks
draft-ietf-bess-bgp-sdwan-usage-02
The information below is for an old version of the document.
| Document | Type |
This is an older version of an Internet-Draft whose latest revision state is "Active".
|
|
|---|---|---|---|
| Authors | Linda Dunbar , Jim Guichard , Ali Sajassi , John Drake , Basil Najem , David Carrel | ||
| Last updated | 2021-01-22 (Latest revision 2020-11-02) | ||
| Replaces | draft-dunbar-bess-bgp-sdwan-usage | ||
| RFC stream | Internet Engineering Task Force (IETF) | ||
| Formats | |||
| Reviews |
GENART Last Call review
(of
-14)
by Dan Romascanu
Ready w/issues
|
||
| Additional resources | Mailing list discussion | ||
| Stream | WG state | WG Document | |
| Document shepherd | (None) | ||
| IESG | IESG state | I-D Exists | |
| Consensus boilerplate | Unknown | ||
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-ietf-bess-bgp-sdwan-usage-02
Network Working Group L. Dunbar
Internet Draft J. Guichard
Intended status: Informational Futurewei
Expires: July 22, 2021 Ali Sajassi
Cisco
J. Drake
Juniper
B. Najem
Bell Canada
Ayan Barnerjee
D. Carrel
IPsec Research
January 22, 2021
BGP Usage for SDWAN Overlay Networks
draft-ietf-bess-bgp-sdwan-usage-02
Abstract
The document discusses the usage and applicability of BGP as control
plane for multiple SDWAN scenarios. The goal of the document is to
demonstrate how BGP-based control plane is used for large scale
SDWAN overlay networks with little manual intervention.
SDWAN edge nodes are commonly interconnected by multiple underlay
networks which can be owned and managed by different network
providers.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79. This document may not be modified,
and derivative works of it may not be created, except to publish it
as an RFC and to translate it into languages other than English.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
xxx, et al. Expires July 22, 2021 [Page 1]
Internet-Draft BGP Usage for SDWAN January 22, 2021
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet-Drafts as
reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
This Internet-Draft will expire on July 22, 2021.
Copyright Notice
Copyright (c) 2021 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with
respect to this document. Code Components extracted from this
document must include Simplified BSD License text as described in
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
Table of Contents
1. Introduction...................................................3
2. Conventions used in this document..............................4
3. Use Case Scenario Description and Requirements.................5
3.1. Requirements..............................................6
3.1.1. Supporting Multiple SDWAN Segmentations..............6
3.1.2. Client Service Requirement...........................6
3.1.3. Application Flow Based Segmentation..................7
3.1.4. Zero Touch Provisioning..............................8
3.1.5. Constrained Propagation of SDWAN Edge Properties.....9
Dunbar, et al. Expires July 22, 2021 [Page 2]
Internet-Draft BGP Usage for SDWAN January 22, 2021
3.2. Scenario #1: Homogeneous WAN.............................10
3.3. Scenario #2: Hybrid WAN Underlay.........................11
3.4. Scenario #3: Private VPN PE based SDWAN..................13
4. BGP Walk Through..............................................14
4.1. BGP Walk Through for Homogeneous SDWAN...................14
4.2. BGP Walk Through for Hybrid WAN Underlay.................16
4.3. BGP Walk Through for Application Flow Based Segmentation.17
4.4. Client Service Provisioning Model........................18
4.5. Benefit of Using Recursive Next Hop Resolution...........19
4.6. Why BGP as Control Plane for SDWAN?......................19
5. SDWAN Traffic Forwarding Walk Through.........................20
5.1. Packet Walk-Through for Scenario #1......................20
5.2. Packet Walk-Through for Scenario #2......................21
5.3. Packet Walk-Through for Scenario #3......................22
6. Manageability Considerations..................................22
7. Security Considerations.......................................23
8. IANA Considerations...........................................23
9. References....................................................23
9.1. Normative References.....................................23
9.2. Informative References...................................24
10. Acknowledgments..............................................25
1. Introduction
Here are some key characteristics of "SDWAN" networks:
- Augment of transport, which refers to utilizing overlay paths
over different underlay networks. Very often there are multiple
parallel overlay paths between any two SDWAN edges, some of them
are private networks over which traffic can traverse with or
without encryption, others require encryption, e.g. over
untrusted public networks.
- Enable direct Internet access from remote sites, instead hauling
all traffic to Corporate HQ for centralized policy control.
- Some traffic flows can be forwarded based on their application
identifiers instead of based on destination IP addresses, by the
edge nodes placing the traffic flows onto specific overlay paths
based on their application requirement.
- The traffic flows forwarding can also be based on specific
performance criteria (e.g. packets delay, packet loos, jitter)
to provide better application performance by choosing the right
underlay that meets or exceeds the specified criteria.
Dunbar, et al. Expires July 22, 2021 [Page 3]
Internet-Draft BGP Usage for SDWAN January 22, 2021
[Net2Cloud-Problem] describes the network related problems that
enterprises face to connect enterprises' branch offices to dynamic
workloads in different Cloud DCs, including using SDWAN to aggregate
multiple paths provided by different service providers to achieve
better performance and to accomplish application ID based
forwarding.
Even though SDWAN has been positioned as a flexible way to reach
dynamic workloads in third party Cloud data centers over different
underlay networks, scaling becomes a major issue when there are
hundreds or thousands of nodes to be interconnected by an SDWAN
overlay networks.
This document describes using BGP as control plane to scale large
SDWAN overlay networks.
2. Conventions used in this document
Cloud DC: Third party data centers that host applications and
workloads owned by different organizations or tenants.
Controller: Used interchangeably with SDWAN controller to manage
SDWAN overlay path creation/deletion and monitor the
path conditions between sites.
CPE: Customer Premise Equipment
CPE-Based VPN: Virtual Private Secure network formed among CPEs.
This is to differentiate from more commonly used PE-
based VPNs [RFC 4364].
Homogeneous SDWAN: A type of SDWAN network in which all traffic
to/from the SDWAN edge nodes has to be encrypted
regardless of underlay networks. For lack of better
terminology, we call this Homogeneous SDWAN throughout
this document.
ISP: Internet Service Provider
Dunbar, et al. Expires July 22, 2021 [Page 4]
Internet-Draft BGP Usage for SDWAN January 22, 2021
NSP: Network Service Provider. NSP usually provides more
advanced network services, such as MPLS VPN, private
leased lines, or managed Secure WAN connections, many
times within a private trusted domain, whereas an ISP
usually provides plain internet services over public
untrusted domains.
PE: Provider Edge
SDWAN Edge Node: an edge node, which can be physical or virtual,
maps the attached clients' traffic to the wide area
network (WAN) overlay tunnels.
SDWAN: Software Defined Wide Area Network. A connectivity
service, offered by a Service Provider, that optimizes
transport of IP Packets over multiple underlay
connectivity services by recognizing applications at
Ingress and determining forwarding behavior by applying
policies to them.
SDWAN IPsec SA: IPsec Security Association between two SDWAN ports
or nodes.
SDWAN over Hybrid Networks: SDWAN over Hybrid Networks typically
have edge nodes utilizing bandwidth resources from
different types of underlay networks, some being private
networks and others being public Internet.
WAN Port: A Port or Interface facing an ISP or Network Service
Provider (NSP), with address allocated by the ISP or the
NSP.
C-PE: SDWAN Edge node, which can be CPE for customer managed
SDWAN, or PE for provider managed SDWAN services.
ZTP: Zero Touch Provisioning
3. Use Case Scenario Description and Requirements
SDWAN networks can have different topologies and have different
traffic patterns. To make it easier for the focused discussion in
Dunbar, et al. Expires July 22, 2021 [Page 5]
Internet-Draft BGP Usage for SDWAN January 22, 2021
subsequent drafts on SDWAN control plane and data plane, this
section describes several SDWAN scenarios that may have different
impact on their corresponding control planes & data planes.
3.1. Requirements
3.1.1. Supporting Multiple SDWAN Segmentations
The term "network segmentation", a.k.a. SDWAN instances, is
referring to the process of dividing the network into logical sub-
networks using isolation techniques on a forwarding device such as a
switch, router, or firewall. For a homogeneous network, such as MPLS
VPN or Layer 2 network, VRF or VLAN are used to achieve the network
segmentation.
As SDWAN is an overlay network arching over multiple types of
networks, MPLS L2VPN/L3VPN or pure L2 underlay can continue using
the VRF, VN-ID or VLAN to differentiate SDWAN network segmentations.
For public internet, the IPsec inner encapsulation header can carry
the SDWAN Instance Identifier to differentiate the packets belonging
to different SDWAN instances.
BGP already has the capability to differentiate control packets for
different network instances. When using BGP for SDWAN, the SDWAN
segmentations can be differentiated by the SDWAN Target ID in the
BGP Extended Community in the same way as VPN instances being
represented by the Route Target. Even though the SDWAN Target ID is
for the same purpose as the VPN ID in Route Target, it is better to
use a different name to differentiate from VPN if a CPE supports
traditional VPN with multiple VRFs and supports multiple SDWAN
Segmentations (instances). The actual SDWAN Target ID encoding is
specified by [SDWAN-EDGE-Discovery].
3.1.2. Client Service Requirement
Client interface of SDWAN nodes can be IP or Ethernet based.
For Ethernet based client interfaces, SDWAN edge should support
VLAN-based service interfaces (EVI100), VLAN bundle service
interfaces (EVI200), or VLAN-Aware bundling service interfaces. EVPN
service requirements are applicable to the Client traffic, as
described in the Section 3.1 of RFC8388.
For IP based client interfaces, L3VPN service requirements are
applicable.
Dunbar, et al. Expires July 22, 2021 [Page 6]
Internet-Draft BGP Usage for SDWAN January 22, 2021
3.1.3. Application Flow Based Segmentation
Application Flow based Segmentation, also known as SDWAN Traffic
Segmentation, enables the separation of the traffic based on the
business and the security needs for different users' groups and/or
application requirements. Each user group and/or applications may
need different isolated topology and/or policies to fulfill the
business requirements.
The Application Flow based Segmentation concept is analogous to VLAN
(in L2 network) and VRF (in L3 network).
One can think about the Application Flow based Segmentation as a
feature that can be provided or enabled on a single SDWAN service
(or domain) to a single Subscriber. Each SDWAN Service can have one
or more overlay segments to support the business requirement; each
segment has its own policy, topology and application/user groups.
Applications/users' group can belong to more than one segment.
For example, a retail business requires the point-of-sales (PoS)
application in all stores to be isolated from other applications AND
routed only to the payment processing entity at a hub site (i.e. hub
and spoke); however, the same retail business allows other
applications to be routed to all sites (i.e. multipoint-to
multipoint) AND isolated from the PoS application.
In the figure below, the traffic from the PoS application follows a
Tree topology, whereas other traffic can be multipoint-to-multipoint
topology.
Dunbar, et al. Expires July 22, 2021 [Page 7]
Internet-Draft BGP Usage for SDWAN January 22, 2021
+--------+
Payment traffic |Payment |
+------+----+-+gateway +------+----+-----+
/ / | +----+---+ | \ \
/ / | | | \ \
+-+--+ +-+--+ +-+--+ | +-+--+ +-+--+ +-+--+
|Site| |Site| |Site| | |Site| |Site| |Site|
| 1 | | 2 | | 3 | | |4 | | 5 | | 6 |
+--+-+ +--+-+ +--|-+ | +--|-+ +--|-+ +--|-+
| | | | | | |
==+=======+=======+====+======+=======+=======+===
Multi-point connection for Other traffic
Another example is an enterprise who wants to isolate the traffic
for each department with each department has its own topology and
policy; the HR department may need to access certain applications
that are NOT accessible by the engineering department. In addition,
the contractors may have a limited access to the enterprise
resources.
3.1.4. Zero Touch Provisioning
Unlike traditional EVPN or L3VPN whose PEs are deployed for long
term, SDWAN edge nodes (virtual or physical) deployment at a
specific location can be ephemeral. Therefore, Zero Touch
Provisioning (ZTP), or Plug and Play, is a common requirement for
SDWAN. When an SDWAN edge is physically installed at a location or
instantiated on a VM in a Cloud DC, ZTP automates follow-up steps,
including updates to the OS, software version, and configuration
prior to connection. From network control perspective, ZTP includes
the following:
- Upon power up, an SDWAN node can establish transport layer
secure connection (such as TLS, SSL, etc.) to its controller whose
address can be burned or preconfigured on the device.
- The SDWAN Controller can designate a Local Network Controller
in the proximity of the SDWAN node; the Local Network Controller
manages and monitor the communication policies of the edge node.
Dunbar, et al. Expires July 22, 2021 [Page 8]
Internet-Draft BGP Usage for SDWAN January 22, 2021
3.1.5. Constrained Propagation of SDWAN Edge Properties
One SDWAN edge node may only be authorized to communicate with a
small number of other SDWAN edge nodes. Under this circumstance, the
property of the SDWAN edge node cannot be propagated to other nodes
that are not authorized to communicate. But a remote SDWAN edge node
upon powering up might not have the proper policies to know who the
authorized peers are. Therefore, it is very essential for SDWAN
deployment have a central point to distribute the properties of an
SDWAN edge node to its authorized peers.
BGP is well suited for this purpose. RFC 4684 has specified the
procedure to constrain the distribution of BGP UPDATE to only a
subset of nodes. Basically, each edge node informs the Route
Reflector (RR) [RFC4456] on its interested SDWAN instances. The RR
only propagates the BGP UPDATE for the relevant SDWAN instances to
the edge.
The connection between a SDWAN edge node and its RR can be over
insecure network. Therefore, upon power up, a SDWAN node needs to
establish a secure transport layer connection (TLS, SSL, etc.) to
its designated RR. The BGP UPDATE messages need to be sent over the
secure channel (TLS, SSL, etc.) to the RR.
+---+
Peer Group 1 |RR | Peer Group 2
+======+====+=+ +======+====+=====+
/ / | +---+ | \ \
/ / | | \ \
+-+--+ +-+--+ +-+--+ +-+--+ +-+--+ +-+--+
|C-PE| |C-PE| |C-PE| |C-PE| |C-PE| |C-PE|
| 1 | | 2 | | 3 | |4 | | 5 | | 6 |
+----+ +----+ +----+ +----+ +----+ +----+
Tenant 1 Tenant 2
Figure 1: Peer Groups managed by RR
Tenant separation is achieved by the SDWAN instance identification
represented in control plane and data plane, respectively.
Dunbar, et al. Expires July 22, 2021 [Page 9]
Internet-Draft BGP Usage for SDWAN January 22, 2021
3.2. Scenario #1: Homogeneous WAN
This is referring to a type of SDWAN network with edge nodes
encrypting all traffic over WAN to other edge nodes, regardless of
whether the underlay is private or public. For lack of better
terminology, we call this Homogeneous SDWAN throughout this
document.
Some typical scenarios for the use of a Homogeneous SDWAN network
are as follows:
- A small branch office connecting to its HQ offices via the
Internet. All sensitive traffic to/from this small branch office has
to be encrypted, which is usually achieved using IPsec SAs.
- A store in a shopping mall may need to securely connect to its
applications in one or more Cloud DCs via the Internet. A common way
of achieving this is to establish IPsec SAs to the Cloud DC gateway
to carry the sensitive data to/from the store.
As described in [SECURE-EVPN], the granularity of the IPsec SAs for
Homogeneous SDWAN can be per site, per subnet, per tenant, or per
address. Once the IPsec SA is established for a specific
subnet/tenant/site, all traffic to/from the subnets/tenants/site are
encrypted.
+---+
+--------------|RR |------------+
/ Untrusted +-+-+ \
/ \
/ \
+----+ +---------+ +------+ +----+
| CN3|--| P1-----+ -------------+------ P1 |--| CN1|
+----+ | C-PE1 P2-----+ | | C-PE2| +----+
+----+ | P3-----+ Wide +------ P2 | +----+
| CN2|--| | | area +------ P3 |--| CN3|
+----+ +---------+ | network | +------+ +----+
| |
+----+ +---------+ | all packets | +------+ +----+
| CN1|--| P1-----+ encrypted +------ P1 |--| CN1|
+----+ | C-PE3 P2-----+ by | | C-PE4| +----+
+----+ | P3-----+ IPsec SAs +------ P2 | +----+
| CN2|--| P4-----+--------------+ | |--| CN2|
+----+ +---------+ +------+ +----+
CN: Client Networks, which is same as Tenant Networks used by NVo3
Figure 2: Homogeneous SDWAN
Dunbar, et al. Expires July 22, 2021 [Page 10]
Internet-Draft BGP Usage for SDWAN January 22, 2021
One of the key properties of homogeneous SDWAN is that the SDWAN
Local Network Controller (RR)is connected to C-PEs via untrusted
public network, therefore, requiring secure connection between RR
and C-PEs (TLS, DTLS, etc.).
Homogeneous SDWAN has some similarity to commonly deployed IPsec
VPN, albeit the IPsec VPN is usually point-to-point among a small
number of nodes and with heavy manual configuration for IPsec
between nodes, whereas an SDWAN network can have a large number of
edge nodes and require zero touch provisioning upon powering up.
Existing Private VPNs (e.g. MPLS based) can use homogeneous SDWAN to
extend over public network to remote sites to which the VPN operator
does not own or lease infrastructural connectivity, as described in
[SECURE-EVPN] and [SECURE-L3VPN]
3.3. Scenario #2: Hybrid WAN Underlay
In this scenario, SDWAN edge nodes (a.k.a. C-PEs) have some WAN
ports to private networks such as L3VPN and some WAN ports to the
public Internet. Since IPsec requires additional processing power
and encrypted traffic over Internet usually does not have the
premium SLA commonly offered by Private VPNs, especially over long
distance, this scenario is referring to a SDWAN network in which
traffic over IP VPN are forwarded natively without IPsec protection.
Only the traffic sent over the public Internet are protected by
IPsec SAs.
One C-PE might have Internet facing WAN ports managed by different
ISPs/NSPs and the WAN ports' addresses being assigned by the
corresponding ISPs/NSPs. Clients might have policies to specify:
1) some flows only traversing private VPNs,
2) some can traverse either if their packets are encrypted when over
the public Internet, and
3) Some flows, especially internet bound browsing ones can be handed off to
the internet without any encryption.
If a flow traversing multiple segments, such as A<->B<->C<->D, has
the Policy 2) above, the flow can traverse different underlays in
different segments, such as over Private network underlay between
A<->B without encryption, or over the public internet between B<->C
protected by an IPsec SA.
As shown in the figure below, C-PE-1 has two different types of
interfaces (A1 to Internet and A2 & A3 to VPN). The C-PEs' loopback
addresses and addresses attached to C-PEs may or may not be visible
Dunbar, et al. Expires July 22, 2021 [Page 11]
Internet-Draft BGP Usage for SDWAN January 22, 2021
to the ISPs/NSPs. The addresses for the WAN ports can have addresses
allocated by service providers or dynamically assigned (e.g. by
DHCP).
+---+
+--------------|RR |----------+
/ Untrusted +-+-+ \
/ \
/ \
+----+ +---------+ packets encrypted over +------+ +----+
| CN3|--| A1-----+ Untrusted +------ B1 |--| CN1|
+----+ | C-PE1 A2-\ | C-PE2| +----+
+----+ | A3--+--+ +---+---B2 | +----+
| CN2|--| | |PE+--------------+PE |---B3 |--| CN3|
+----+ +---------+ +--+ trusted +---+ +------+ +----+
| WAN |
+----+ +---------+ +--+ packets +---+ +------+ +----+
| CN1|--| C1--|PE| go natively |PE |-- D1 |--| CN1|
+----+ | C-PE3 C2--+--+ without encry+---+ | C-PE4| +----+
| | +--------------+ | |
| | | |
+----+ | | without encrypt over | | +----+
| CN2|--| C3--+---- Untrusted --+------D2 |--| CN2|
+----+ +---------+ +------+ +----+
CN: Client Network
Figure 3: Hybrid SDWAN
In addition, the connection between C-PEs and their Controller
(RR) might be via the untrusted public network. It is necessary to
encrypt the communication between RR and C-PEs, by TLS, DTLS,
etc..
There could be multiple SDWAN edges (C-PEs) sharing a common
property, such as a geographic location. Some applications over
SDWAN may need to traverse specific geographic locations for
various reasons, such as to comply with regulatory rules, to
utilize specific value-added services, or others.
Services may not be congruent, i.e. the packets from A-> B may
traverse one underlay network, and the packets from B -> A may
traverse a different underlay.
Dunbar, et al. Expires July 22, 2021 [Page 12]
Internet-Draft BGP Usage for SDWAN January 22, 2021
3.4. Scenario #3: Private VPN PE based SDWAN
This scenario refers to existing VPN (e.g. MPLS based VPN, such as
EVPN or IPVPN) adding extra ports facing untrusted public networks
to allow PEs to offload some low priority traffic to public networks
when the VPN MPLS paths are congested. Throughout this document,
this scenario is also called Internet Offload for Private VPN, or PE
based SDWAN.
Here are some differences from the Scenario #2 (Section 3.3):
- The packets offloaded to untrusted public network are still
part of the VPN traffic, therefore, must be encrypted.
- For MPLS based VPN, PEs would have MPLS as payload
encapsulated within the IPsec tunnel egressing to the
Internet WAN ports, MPLS-in-IP-in-IPsec.
- The BGP RR is connected to PEs in the same way as VPN, i.e.
via the trusted network.
PE based SDWAN can be used by VPN service providers to temporarily
increase bandwidth between sites when not sure if the demand will
sustain for long period of time or as a temporary solution before
the permanent infrastructure is built or leased.
+---+
+======>|PE2|
// +---+
// ^
// || VPN
// VPN v
++--+ ++-+ +---+
|PE1| <====> |RR| <===> |PE3|
+-+-+ +--+ +-+-+
| |
+--- Public Internet -- +
Offload
Figure 4: Additional Internet paths added to the VPN
Dunbar, et al. Expires July 22, 2021 [Page 13]
Internet-Draft BGP Usage for SDWAN January 22, 2021
4. BGP Walk Through
4.1. BGP Walk Through for Homogeneous SDWAN
In the figure below, packets destined towards multiple routes
attached to the C-PE2 can be carried by one IPsec tunnel. Then one
BGP UPDATE can be announced by C-PE2 to its RR.
+---+
+---------|RR |----------+
/ Untrusted+---+ \
/ \
C-PE1/ \
+---------+ +------+
--+---+---------------------------------> |-10.1.x.x/16
| / | |C-PE2 |- VLAN = 15
| / | +-----> |
--|/1.1.1.1 | | | |-12.1.1.x/24
+---------+ | +------+
| 2.2.2.2
|
C-PE3 |
+---------+ |
--|---+---------------------------+
| / |
| / |
--|/3.3.3.3 |
+---------+
Figure 5: Homogeneous SDWAN
The BGP UPDATE Message from C-PE2 to RR should have the client
routes encoded in the MP-NLRI Path Attribute and the IPsec Tunnel
associated information encoded in the Tunnel-Encap Path Attributes
as described in the [SECURE-EVPN].
Alternatively, two separate BGP UPDATEs can be used to optimize the
BGP UPDATE packet size, as described by Section 4 and 8 of [Tunnel-
encap]. UPDATE U1 has its Nexthop to the node loopback address and
is reclusively resolved to the IPsec tunnel detailed attributes
advertised by the UPDATE U2 for the Node Loopback address:
Suppose that:
- a given packet P destined towards the client addresses attached
to C-PE2 (e.g. prefix 10.1.x.x/16) can be carried by any IPsec
tunnels terminated at C-PE2;
Dunbar, et al. Expires July 22, 2021 [Page 14]
Internet-Draft BGP Usage for SDWAN January 22, 2021
- The path along which P is to be forwarded is determined by BGP
UPDATE U1;
- UPDATE U1 does not have a Tunnel Encapsulation attribute;
- UPDATE U1 can include Encapsulation Extended Community and/or
Color Extended Community;
- The address of the next hop of UPDATE U1 is router C-PE2;
- The best route to router C-PE2 is a BGP route that was
advertised in UPDATE U2;
- UPDATE U2 has a Tunnel Encapsulation attribute to further
describe the IPsec detailed attributes.
UPDATE U1:
- MP-NLRI Path Attribute:
10.1.x.x/16
12.1.1.x/24
- Nexthop: 2.2.2.2 (C-PE2)
- Encapsulation Extended Community: Type = IPsec
UPDATE U2:
- MP-NLRI Path Attribute:
2.2.2.2 (C-PE2)
- Tunnel Encapsulation Path Attributes (as described in the
[SECURE-EVPN])
If different client routes attached to C-PE2 needs to be reached by
separate IPsec tunnels, then The Color-Extended-Community [Tunnel-
encap] is used to associate routes with the tunnels. See Section 8
of [Tunnel-encap].
If C-PE2 doesn't have the policy on authorized peers for the
specific client routes, RR needs to check the client routes policies
to propagate the BGP UPDATE messages to the remote authorized edge
nodes.
Dunbar, et al. Expires July 22, 2021 [Page 15]
Internet-Draft BGP Usage for SDWAN January 22, 2021
4.2. BGP Walk Through for Hybrid WAN Underlay
In this scenario, some client routes can be forwarded by any tunnels
terminated at the edge node and some client routes can be forwarded
by some specific tunnels (such as only MPLS VPN).
Color Extended Community (Section 4 & 8 of [Tunnel-Encap]) can be
used to represent specific tunnels for the client routes.
For example, in the Figure 5 above, suppose that Route 10.1.x.x/16
can be carried by either MPLS or IPsec, and Route 12.1.1.x/24 can
only be carried by MPLS, the following UDPATE messages can be used:
UPDATE #1a for Route Route 10.1.x.x/16:
- MP-NLRI Path Attribute:
10.1.x.x/16
Nexthop: 2.2.2.2 (C-PE2)
- Encapsulation Extended Community: Type = SDWAN-Hybrid
- Color Extended Community: RED
UPDATE #1b for Route Route 12.1.1.x/24:
- MP-NLRI Path Attribute:
12.1.1.x/24
Nexthop: 2.2.2.2 (C-PE2)
- Encapsulation Extended Community: Type= SDWAN-Hybrid
- Color Extended Community: YELLOW
UPDATE #2a: for IPsec tunnels terminated at the node:
- MP-NLRI Path Attribute:
2.2.2.2 (C-PE2)
- Tunnel Encapsulation Path Attributes: TYPE=SDWAN-Hybrid
- Color Extended Community: RED
UPDATE #2b: for MPLS-in-GRE terminated at the node:
- MP-NLRI Path Attribute:
2.2.2.2 (C-PE2)
- Tunnel Encapsulation Path Attributes: TYPE=SDWAN-Hybrid
- Color Extended Community: YELLOW
IANA Action: require a new value for SDWAN-Hybrid Tunnel Type.
Dunbar, et al. Expires July 22, 2021 [Page 16]
Internet-Draft BGP Usage for SDWAN January 22, 2021
4.3. BGP Walk Through for Application Flow Based Segmentation
If the applications are assigned with unique IP addresses, the
Application Flow based Segmentation described in Section 3.1.2 can
be achieved by advertising different BGP UPDATE messages to
different nodes. In the Figure below, the following BGP Updates can
be advertised to ensure that Payment Application only communicates
with the Payment Gateway:
BGP UPDATE #1a from C-PE2 to RR for the P2P topology that is only
propagated to Payment GW node:
UPDATE #1a (only to the Payment GW node):
- MP-NLRI Path Attribute:
- 30.1.1.x/24
- Nexthop: 2.2.2.2
- Encapsulation extended community: Tunneltype=IPSEC
- Color Extended Community: BLUE
BGP UPDATE #1b from C-PE2 to RR for the routes to be reached by C-
PE1 and C-PE2:
- MP-NLRI Path Attribute:
- 10.1.x.x
- 12.4.x.x
- Nexthop:2.2.2.2
- Encapsulation extended community: Tunnel-type=IPSEC
- Color Extended Community: RED
BGP UPDATE #2 describes the IPsec detailed attributes for IPsec
tunnels terminated at C-PE2 2.2.2.2.
UPDATE #2a: for all IPsec SAs terminated at the node:
- MP-NLRI Path Attribute:
2.2.2.2 (C-PE2)
- Tunnel Encapsulation Path Attributes: TYPE=IPsec (for all IPsec
SAs)
- Color Extended Community: RED
Dunbar, et al. Expires July 22, 2021 [Page 17]
Internet-Draft BGP Usage for SDWAN January 22, 2021
UPDATE #2b: for the IPsec SA to the Payment Gateway:
- MP-NLRI Path Attribute:
2.2.2.2 (C-PE2)
- Tunnel Encapsulation Path Attributes: TYPE=IPsec (for the IPsec
SA to Payment GW).
- Color Extended Community: Blue
+-------+
|Payment|
+-------->| GW |<-------+
/Hub-spoke +-------+ \
/for Payment App \
C-PE1 / \ C-PE2
+------/--+ +----\-+
--|-----/ | | -|- 30.1.1.x/24
+ ---------------------------------------> |-10.1.x.x/16
| | | |-
| | +------------> |- 12.1.1.x/24
--|---------------------------+ | |
+---------+ +------> |- VLAN=25;
/ +------+ 22.1.1.x/24
+---------+ /
--| -----------------------------+
| C-PE3 | /
| | /
--| --------------------------+
+---------+
Figure 6: Application Based SDWAN Segmentation
4.4. Client Service Provisioning Model
The provisioning tasks described in Section 4 of RFC8388 are the
same for the SDWAN client traffic. When client traffic is multi-
homed to two (or more) C-PEs, the Non-Service-Specific parameters
need to be provisioned per the Section 4.1.1 of RFC8388.
Since some SDWAN nodes are ephemeral and have small number of IP
subnets or VLANs attached to their client ports, it is recommended
to have default and simplified Service-specific parameters for each
client port, remotely managed by the SDWAN Network Controller via
the secure channel (TLS/DTLS) between the controller and the C-PEs.
Dunbar, et al. Expires July 22, 2021 [Page 18]
Internet-Draft BGP Usage for SDWAN January 22, 2021
4.5. Benefit of Using Recursive Next Hop Resolution
Using the Recursive Next Hop Resolution described in the Section 8
of [Tunnel-Encap], not only the clients' routes UPDATE messages
become very compact, but also they are not impacted by the underlay
network tunnels attributes changes. This method is especially useful
when the underlay tunnels are IPsec based which requires periodic
messages updates for the pairwise re-keying process.
4.6. Why BGP as Control Plane for SDWAN?
For a small sized SDWAN network, traditional hub & spoke model using
NHRP or DSVPN/DMVPN with a hub node (or controller) managing SDWAN
node WAN ports mapping (e.g. local & public addresses and tunnel
identifiers mapping) can work reasonably well. However, for a large
SDWAN network, say more than 100 nodes with different types of
topologies, the traditional approach becomes very messy, complex and
error prone.
Here are some of the compelling reasons of using BGP instead of
extending NHRP/DSVPN/DMVPN. (Same as the reasons quoted by LSVR on
why using BGP):
- BGP has the built-in capability to constrain the propagation of
SDWAN edge node properties to a small number of edge nodes
[RFC4684].
- RR already has the capability to apply policies to communications
among peers.
- BGP is widely deployed as sole protocol (see RFC 7938)
- Robust and simple implementation
- Wide acceptance - minimal learning
- Reliable transport
- Guaranteed in-order delivery
- Incremental updates
- Incremental updates upon session restart
- No flooding and selective filtering
Dunbar, et al. Expires July 22, 2021 [Page 19]
Internet-Draft BGP Usage for SDWAN January 22, 2021
5. SDWAN Traffic Forwarding Walk Through
BGP based EVPN control plane are still applicable to routes attached
to the client ports of SDWAN nodes. Section 5 of RFC8388 describes
the BGP EVPN NLRI Usage for various routes of client traffic. The
procedures described in the Section 6 of RFC8388 are same for the
SDWAN client traffic.
The only additional consideration for SDWAN is to control how
traffic egress the SDWAN edge node to various WAN ports.
5.1. Packet Walk-Through for Scenario #1
A single IPsec security association (SA) protects data in one
direction. Under the Scenario #1, two SAs must be present to secure
traffic in both directions between any two end points, which can be
two C-PE nodes, two client ports, or two prefixes.
Upon power up, a SDWAN node can learn client routes from the Client
facing ports in the same way as EVPN described in RFC8388.
Controller, i.e. BGP-RR, facilitates the IPsec SA establishment and
rekey management as described in [SECURE-EVPN]. Controller manages
how client's routes are associated with individual IPSec SA.
Using Figure 2 of the Section 3.2 as an example. Let's assume that
the IPsec SAs terminate at the C-PE nodes. To enable full mesh
communication within client CN2 that are attached to C-PE1, C-PE3,
and C-PE4, six one directional IPsec SAs must be established: C-PE1
<-> C-PE3; C-PE1 <-> C-PE4; C-PE3 <-> C-PE4. The C-PE node address
(or loopback address) acts as the Next Hop address for the prefixes
attached to the C-PE node.
When a C-PE receives a packet from its client port, the C-PE uses
the IPsec SA whose destination address matches the Next Hop address
of the packet's destination to encrypt the packet and forward the
encrypted packet to the target C-PE via one of the C-PE's WAN ports.
When a C-PE receives an encrypted packet from its WAN port, it
decrypted the packet and forward the inner packet to the client port
based on the inner packet's destination address.
Dunbar, et al. Expires July 22, 2021 [Page 20]
Internet-Draft BGP Usage for SDWAN January 22, 2021
5.2. Packet Walk-Through for Scenario #2
In this scenario as shown in the Figure 3 of Section 3.3, there are
trusted VPN path and IPsec SAs over Public Internet between two C-
PEs. There could be user specified policy governing that some flows
can only be forwarded to the trusted VPN.
Upon receiving a packet from a client port, if the packet belongs to
a flow that can only be forwarded to the trusted VPN, the forwarding
processing is same as the VPN. If the packet belongs to a flow that
can be forwarded to either VPN or IPsec SA, the C-PE node can make
the local decision in choosing the least congested path to forward
the packet. Since it is not necessary to encrypt the packets
forwarded to the trusted VPN, it is more efficient for the IPsec SAs
to be terminated at the Internet facing WAN ports of the C-PE nodes.
For a C-PE with multiple WAN ports provided by different ISPs, the
C-PE can have multiple IPsec SAs terminated at one WAN port of a
remote C-PE.
If the IPsec SA is chosen, the packet is encrypted and encapsulated
in the inner payload of the IPsec SA and egress out the
corresponding WAN port of the IPsec SA.
Upon receiving a packet from a WAN port, especially from the
Internet facing WAN port, additional anti-DDoS mechanism has to be
enabled to prevent potential attacks from the Internet facing port.
Control Plane should not learn routes from the Internet facing WAN
ports.
+---+
+--------------|RR |----------+
/ Untrusted +-+-+ \
/ Network \
/ \
+----+ +---------+ packets encrypted over +--------+ +----+
| CN3|--| A1-----+ Untrusted +------ B1 |--| CN1|
+----+ | C-PE-a A2-----+ +-------B2 C-PE-b| +----+
|10.1.1.1 | |10.1.2.1|
+----+ | | +--+ +---+ | | +----+
| CN2|--| A3 |PE+--------------+PE |---B3 |--| CN3|
+----+ +---------+ +--+ trusted +---+ +--------+ +----+
| VPN |
+--------------+
Figure 8: SDWAN Scenario #2
Dunbar, et al. Expires July 22, 2021 [Page 21]
Internet-Draft BGP Usage for SDWAN January 22, 2021
5.3. Packet Walk-Through for Scenario #3
In this scenario all PEs have secure interfaces facing clients and
facing MPLS backbone with some PEs having additional connection by
unsecure public Internet. For the MPLS based PEs offloading some
MPLS packets to the internet path, the MPLS data frames need to be
encapsulated in IP [RFC4023] and secured by IPsec SA. The IP header
of the MPLS-in-IP packet becomes the outer IP header of the
resulting packet when IPsec transport mode is used to secure the
MPLS packet. This is followed by an IPsec header, followed by the
MPLS label stack. The IPsec header must set the payload type to MPLS
by using the IP protocol number specified in the section 3 of
RFC4023. If IPsec transport mode is applied on a MPLS-in-GRE packet,
the GRE header follows the IPsec header.
The end points of an IPsec SA between two PEs can be PEs' Internet
facing WAN ports addresses or the PEs' loopback addresses. The IPsec
SA end points should not be the client addresses unless the traffic
to/from those client addresses always go through the IPsec SA even
when the MPLS backbone has enough capacity to transport the traffic.
When the PEs' Internet facing ports are behind the NAT [RFC3715], an
outer UDP field can be added outside the encrypted payload
[RFC3948]. Three UDP ports must be open on the PEs: UDP port 4500
(used for NAT traversal), UDP port 500 (used for IKE) and IP
protocol 50 (ESP). IPsec IKE (Internet Key Exchange) between the two
PEs would be over the NAT [RFC3947] as well.
Upon receiving a packet from a client port, the forwarding and MPLS
processing is same as the MPLS VPN. If the MPLS backbone path to the
packets next hop is congested, the IPsec SA towards the target PEs
are used to encrypt the MPLS packet.
Upon receiving a packet from the Internet facing WAN port, the
packet is decrypted and the inner MPLS payload is extracted to be
sent to the MPLS VPN engine.
Same as the Scenario #2, additional anti-DDoS mechanism must be
enabled to prevent potential attacks from the Internet facing port.
Control Plane should not learn routes from the Internet facing WAN
ports.
6. Manageability Considerations
SDWAN overlay networks utilize the SDWAN controller to facilitate
route distribution, central configurations, and others. SDWAN Edge
Dunbar, et al. Expires July 22, 2021 [Page 22]
Internet-Draft BGP Usage for SDWAN January 22, 2021
nodes need to advertise the attached routes to their controller
(i.e. RR in BGP case).
7. Security Considerations
Having WAN ports facing the public Internet introduces the following
security risks:
1) Potential DDoS attack to the C-PEs with ports facing internet.
2) Potential risk of provider VPN network being injected with
illegal traffic coming from the public Internet WAN ports on the C-
PEs.
8. IANA Considerations
Need a new tunnel type: SDWAN-Hybrid
9. References
9.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC4364] E. rosen, Y. Rekhter, "BGP/MPLS IP Virtual Private
networks (VPNs)", Feb 2006.
[RFC7296] C. Kaufman, et al, "Internet Key Exchange Protocol Version
2 (IKEv2)", Oct 2014.
[RFC7432] A. Sajassi, et al, "BGP MPLS-Based Ethernet VPN", Feb
2015.
[RFC8365] A. Sajassi, et al, "A network Virtualization Overlay
Solution Using Ethernet VPN (EVPN)", March 2018.
Dunbar, et al. Expires July 22, 2021 [Page 23]
Internet-Draft BGP Usage for SDWAN January 22, 2021
9.2. Informative References
[RFC8192] S. Hares, et al, "Interface to Network Security Functions
(I2NSF) Problem Statement and Use Cases", July 2017
[RFC5521] P. Mohapatra, E. Rosen, "The BGP Encapsulation Subsequent
Address Family Identifier (SAFI) and the BGP Tunnel
Encapsulation Attribute", April 2009.
[RFC8388] J. Rabadan, et al, "Usage and Applicability of BGP MPLS-
Based Ethernet VPN", May 2018.
[Net2Cloud-Gap] L. Dunbar, A. Malis, C. Jacquenet, "Gap Analysis of
Interconnecting Underlay with Cloud Overlay", draft-dm-
net2cloud-gap-analysis-02, work in progress, Oct. 2018.
[SDWAN-EDGE-Discovery] L. Dunbar, S. Hares, R. Raszuk, K. Majumdar,
"BGP UPDATE for SDWAN Edge Discovery", draft-dunbar-idr-
sdwan-edge-discovery-01, work-in-progress, Nov 2020.
[VPN-over-Internet] E. Rosen, "Provide Secure Layer L3VPNs over
Public Infrastructure", draft-rosen-bess-secure-l3vpn-00,
work-in-progress, July 2018
[DMVPN] Dynamic Multi-point VPN:
https://www.cisco.com/c/en/us/products/security/dynamic-
multipoint-vpn-dmvpn/index.html
[DSVPN] Dynamic Smart VPN:
http://forum.huawei.com/enterprise/en/thread-390771-1-
1.html
[SECURE-EVPN] A. Sajassi, et al, "Secure EVPN", draft-sajassi-bess-
secure-evpn-01, Work-in-progress, March 2019.
[SECURE-L3VPN] E. Rosen, R. Bonica, "Secure Layer L3VPN over Public
Infrastructure", draft-rosen-bess-secure-l3vpn-00, Work-
in-progress, June 2018.
[ITU-T-X1036] ITU-T Recommendation X.1036, "Framework for creation,
storage, distribution and enforcement of policies for
network security", Nov 2007.
Dunbar, et al. Expires July 22, 2021 [Page 24]
Internet-Draft BGP Usage for SDWAN January 22, 2021
[Net2Cloud-Problem] L. Dunbar and A. Malis, "Seamless Interconnect
Underlay to Cloud Overlay Problem Statement", draft-dm-
net2cloud-problem-statement-02, June 2018
[Net2Cloud-gap] L. Dunbar, A. Malis, and C. Jacquenet, "Gap Analysis
of Interconnecting Underlay with Cloud Overlay", draft-dm-
net2cloud-gap-analysis-02, work-in-progress, Aug 2018.
[Tunnel-Encap] E. Rosen, et al "The BGP Tunnel Encapsulation
Attribute", draft-ietf-idr-tunnel-encaps-10, Aug 2018.
10. Acknowledgments
Acknowledgements to Adrian Farrel, Joel Halpern, John Scudder,
Darren Dukes, Andy Malis and Donald Eastlake for their review and
contributions.
This document was prepared using 2-Word-v2.0.template.dot.
Dunbar, et al. Expires July 22, 2021 [Page 25]
Internet-Draft BGP Usage for SDWAN January 22, 2021
Authors' Addresses
Linda Dunbar
Futurewei
Email: ldunbar@futurewei.com
James Guichard
Futurewei
Email: james.n.guichard@futurewei.com
Ali Sajassi
Cisco
Email: sajassi@cisco.com
John Drake
Juniper
Email: jdrake@juniper.net
Basil Najem
Bell Canada
Email: basil.najem@bell.ca
David Carrel
IPsec Research
Email: carrel@ipsec.org
Ayan Banerjee
Cisco
Email: ayabaner@cisco.com
Dunbar, et al. Expires July 22, 2021 [Page 26]