A Framework for Enhanced Virtual Private Network (VPN+) Services

3.1. Performance Guarantees

Performance guarantees are made by network operators to their customers in relation to the services provided to the customers. They are usually expressed in SLAs as a set of SLOs.¶

There are several kinds of performance guarantee, including guaranteed maximum packet loss, guaranteed maximum delay, and guaranteed delay variation. Note that these guarantees apply to conformance traffic, out-of-profile traffic will be handled according to a separate agreement with the customer.¶

Guaranteed maximum packet loss is usually addressed by setting packet priorities, queue size, and discard policy. However this becomes more difficult when the requirement is combined with latency requirements. The limiting case is zero congestion loss, and that is the goal of DetNet [DETNET] and TSN [TSN]. In modern optical networks, loss due to transmission errors already approaches zero, but there is the possibility of failure of the interface or the fiber itself. This type of fault can only be addressed by some form of signal duplication and transmission over diverse paths.¶

Guaranteed maximum latency is required by a number of applications particularly real-time control applications and some types of virtual reality applications. DetNet [DETNET] is relevant, however additional methods of enhancing the underlay to better support the delay guarantees may be needed, and these methods will need to be integrated with the overall service provisioning mechanisms.¶

Guaranteed maximum delay variation is a performance guarantee that may also be needed. [RFC8578] calls up a number of cases that need this guarantee, for example in electrical utilities. Time transfer is an example service that needs a performance guarantee, although it is in the nature of time that the service might be delivered by the underlay as a shared service and not provided through different enhanced VPNs. Alternatively, a dedicated enhanced VPN might be used to provide this as a shared service.¶

This suggests that a spectrum of service guarantees need to be considered when deploying an enhanced VPN. As a guide to understanding the design requirements we can consider four types of service:¶

• Best effort¶
• Assured bandwidth¶
• Guaranteed latency¶
• Enhanced delivery¶

The best effort service is the basic service as provided by current VPNs.¶

An assured bandwidth service is one in which the bandwidth over some period of time is assured. This can be achieved either simply based on a best effort service with over-capacity provisioning, or it can be based on MPLS traffic engineered label switching paths (TE-LSPs) with bandwidth reservations. Depending on the technique used, however, the bandwidth is not necessarily assured at any instant. Providing assured bandwidth to VPNs, for example by using per-VPN TE-LSPs, is not widely deployed at least partially due to scalability concerns. VPN+ aims to provide a more scalable approach for such services.¶

A guaranteed latency service has an upper bound to edge-to-edge latency. Assuring the upper bound is sometimes more important than minimizing latency. There are several new technologies that provide some assistance with this performance guarantee. Firstly, the IEEE TSN project [TSN] introduces the concept of scheduling of delay- and loss-sensitive packets. The DetNet work [DETNET] is also of relevance in assuring an upper bound of end-to-end packet latency. FlexE [FLEXE] is also useful to help provide these guarantees. The use of such underlying technologies to deliver VPN+ services needs to be considered.¶

An enhanced delivery service is one in which the underlay network (at Layer 3) attempts to deliver the packet through multiple paths in the hope of eliminating packet loss due to equipment or media failures. Such a mechanism may need to be used for VPN+ service.¶

3.2. Isolation between Enhanced VPN Services

One element of the SLA demanded for an enhanced VPN may be a guarantee that the service offered to the customer will not be affected by any other traffic flows in the network. This is termed "isolation" and a customer may express the requirement for isolation as an SLE [I-D.ietf-teas-ietf-network-slices].¶

One way for a network operator to meet the requirement for isolation is simply by setting and conforming to all the SLOs. For example, traffic congestion (interference from other services) might impact on the latency experienced by a VPN+ customer. Thus, in this example, conformance to a latency SLO would be the primary requirement for delivery of the VPN+ service, and isolation from other services might be only a means to that end.¶

Another way for a service provider to meet this SLE is to control the degree to which traffic from one service is isolated from other services in the network.¶

There is a fine distinction between how isolation is requested by a customer and how it is delivered by the service provider. In general, the customer is interested in service performance and not how it is delivered. Thus, for example, the customer wants specific quality guarantees and is not concerned about how the service provider delivers them. However, it should be noted that some aspects of isolation might be directly measurable by a customer if they have information about the traffic patterns on a number services supported by the same service provider. Furthermore, a customer may be nervous about disruption caused by other services, contamination by other traffic, or delivery of their traffic to the wrong destinations. In this way, the customer may want to specify (and pay for) the level of isolation provided by the service provider.¶

Isolation is achieved in the realization of a VPN+ through existing technologies that may be supplemented by new mechanisms. The service provider chooses which processes to use to meet this SLE just as they choose how to meet all other SLOs and SLEs. Isolation may be achieved in the network by various forms of resource partitioning ranging from simple separation of service traffic on delivery (ensuring that traffic is not delivered to the wrong customer), through sharing of resources with some form of safeguards, to dedicated allocation of resources for a specific enhanced VPN. For example, interference avoidance may be achieved by network capacity planning, allocating dedicated network resources, traffic policing or shaping, prioritizing in using shared network resources, etc.¶

The terms hard and soft isolation are used to indicate different levels of isolation. A service has soft isolation if the traffic of one service cannot be received by the customers of another service. The existing IP and MPLS VPNs are examples of services with soft isolation: the network delivers the traffic only to the required customer endpoints. However, with soft isolation, as the network resources are shared, traffic from some services may congest the network, resulting in packet loss and delay for other services. The ability for a service or a group of services to be sheltered from this effect is called hard isolation. Hard isolation may be needed so that applications with exacting requirements can function correctly, despite other demands (perhaps a burst of traffic in another service) competing for the underlying resources. A customer may request different degrees of isolation ranging from soft isolation to hard isolation. In practice isolation may be delivered on a spectrum between soft and hard, and in some cases soft and hard isolation may be used in a hierarchical manner with one enhanced VPN being built on another.¶

To provide the required level of isolation, resources may need to be reserved in the data plane of the underlay network and dedicated to traffic from a specific enhanced VPN or a specific group of enhanced VPNs. This may introduce scalability concerns both in the implementation (as each enhanced VPN would need to be tracked in the network) and in how many resources need to be reserved and may be under-used (see Section 4.4). Thus, some trade-off needs to be considered to provide the isolation between enhanced VPNs while still allowing reasonable resource sharing.¶

An optical underlay can offer a high degree of isolation, at the cost of allocating resources on a long-term and end-to-end basis. On the other hand, where adequate isolation can be achieved at the packet layer, this permits the resources to be shared amongst a group of services and only dedicated to a service on a temporary basis.¶

The next section explores a pragmatic approach to isolation in packet networks.¶

     O=================================================O
     |          \---------------v---------------/
 Statistical                Pragmatic             Absolute
 Multiplexing               Isolation            Isolation
(Traditional VPNs)        (Enhanced VPN)     (Dedicated Network)

3.3. Integration

The way to achieve the characteristics demanded by an enhanced VPN (such as guaranteed or predictable performance) is by integrating the overlay VPN with a particular set of resources in the underlay network which are allocated to meet the service requirement. This needs be done in a flexible and scalable way so that it can be widely deployed in operators' networks to support a reasonable number of enhanced VPN customers.¶

Taking mobile networks and in particular 5G into consideration, the integration of the network with service functions is likely a requirement. The IETF's work on service function chaining (SFC) [SFC] provides a foundation for this. Service functions can be considered as part of enhanced VPN services. The detailed mechanisms about the integration between service functions and enhanced VPNs are out of the scope of this document.¶

3.4. Dynamic Changes

Enhanced VPNs need to be created, modified, and removed from the network according to service demands. An enhanced VPN that requires hard isolation (Section 3.2) must not be disrupted by the instantiation or modification of another enhanced VPN. Determining whether modification of an enhanced VPN can be disruptive to that VPN, and whether the traffic in flight will be disrupted can be a difficult problem.¶

The data plane aspects of this problem are discussed further in Section 5.1,Section 5.2, and Section 5.3.¶

The control plane aspects of this problem are discussed further in Section 5.4.¶

The management plane aspects of this problem are discussed further in Section 5.5.¶

Dynamic changes both to the enhanced VPN and to the underlay network need to be managed to avoid disruption to services that are sensitive to changes in network performance.¶

In addition to non-disruptively managing the network during changes such as the inclusion of a new VPN endpoint or a change to a link, VPN traffic might need to be moved because of changes to traffic patterns and volumes.¶

3.5. Customized Control

In many cases the customers are delivered with enhanced VPN services without knowing the information about the underlying VTNs. However, depends on the agreement between the operator and the customer, in some cases the customer may also be provided with some information about the underlying VTNs. Such information can be filtered or aggregated according to the operator's policy. This allows the customer of the enhanced VPN to have some visibility and even control over how the underlying topology and resources of the VTN are used. For example, the customers may be able to specify the service paths within the VTN for specific traffic flows of their enhanced VPNs. Depending on the requirements, an enhanced VPN customer may have his own network controller, which may be provided with an interface to the control or management system run by the network operator. Note that such control is within the scope of the customer's enhanced VPN, any additional changes beyond this would require some intervention by the network operator.¶

A description of the control plane aspects of this problem are discussed further in Section 5.4. A description of the management plane aspects of this feature can be found in Section 5.5.¶

3.6. Applicability to Overlay Technologies

The concept of enhanced VPN can be applied to any existing and future multi-tenancy overlay technologies including but not limited to :¶

• Layer-2 point-to-point services such as pseudowires [RFC3985]¶
• Layer-2 VPNs [RFC4664]¶
• Ethernet VPNs [RFC7209]¶
• Layer-3 VPNs [RFC4364], [RFC2764]¶

Where such VPN service types need enhanced isolation and delivery characteristics, the technologies described in Section 5 can be used to provide an underlay with the required enhanced performance.¶

4.1. Layered Architecture

The layered architecture of VPN+ is shown in Figure 2.¶

Underpinning everything is the physical network infrastructure layer which provide the underlying resources used to provision the separated VTNs. This layer is responsbile for the partitioning of link and/or node resources for different VTNs. Each subset of link or node resource can be considered as a virtual link or virtual node used to build the VTNs.¶

                              /\
                              ||
                    +-------------------+       Centralized
                    | Network Controller|   Control & Management
                    +-------------------+
                              ||
                              \/
                o---------------------------o    VPN Service 1
                              /-------------o
                o____________/______________o    VPN Service 2
                           _________________o
                     _____/
                o___/     \_________________o    VPN Service 3
                    \_______________________o
                           ......                    ...
                o-----------\ /-------------o
                o____________X______________o    VPN Service n

                   __________________________
                  /       o----o-----o      /
                 /       /          /      /        VTN-1
                / o-----o-----o----o----o /
               /_________________________/
                   __________________________
                  /       o----o            /
                 /       /    / \          /        VTN-2
                / o-----o----o---o------o /
               /_________________________/
                         ......                      ...
                  ___________________________
                 /             o----o       /
                /             /    /       /        VTN-m
               /  o-----o----o----o-----o /
              /__________________________/


                 ++++   ++++   ++++
                 +--+===+--+===+--+
                 +--+===+--+===+--+
                 ++++   +++\\  ++++
                  ||     || \\  ||             Physical
                  ||     ||  \\ ||              Network
          ++++   ++++   ++++  \\+++   ++++  Infrastructure
          +--+===+--+===+--+===+--+===+--+
          +--+===+--+===+--+===+--+===+--+
          ++++   ++++   ++++   ++++   ++++

     o    Virtual Node     ++++
                           +--+  Physical Node with resource partition
     --   Virtual Link     +--+
                           ++++
     ==  Physical Link with resource partition

Various components and techniques discussed in Section 5 can be used to enable resource partitioning, such as FlexE, TSN, DetNet, dedicated queues, etc. These partitions may be physical or virtual so long as the SLA required by the higher layers is met.¶

Based on the network resource partitions provided by the physical network infrastructure, multiple VTNs can be created, each with a set of dedicated or shared network resources allocated from the physical underlay network, and is associated with a customized logical network topology, so as to meet the requirements of different VPN+ services or different groups of VPN+ services. According to the associated logical network topology, each VTN needs to be instantiated on a set of network nodes and links which are involved in the logical topology. And on each node or link, each VTN is associated with a set of local resources which are allocated for the processing of traffic in the VTN. The VTN provides the integration between the virtual network topology and the required underlying network resources.¶

According to the service requirements on connectivity, performance and isolation, etc., VPN services can be mapped to the appropriate VTNs in the network. Different VPN services can be mapped to different VTNs, while it is also possible that multiple VPNs are mapped to the same VTN. Thus VTN is an essential scaling technique, as it has the potential of eliminating per-path state from the network. In addition, when a group of VPN+ services are mapped to a single VTN, only the network state of the single VTN needs to be maintained in the network (see Section 4.4 for more information).¶

The centralized network controller is responsible for creating a VTN, instructing the involved network nodes to allocate network resources to the VTN, and provisioning the VPN services on the VTN. A distributed control plane may be used for distributing the VTN resource and topology attributes among nodes in the VTN.¶

The process used to create VTNs and to allocate network resources for use by the VTNs needs to take a holistic view of the needs of all of its customers and to partition the resources accordingly. However, within a VTN these resources can, if required, be managed via a dynamic control plane. This provides the required scalability and isolation with some flexibility.¶

4.2. Multi-Point to Multi-Point (MP2MP) Connectivity

At the VPN service level, the required connectivity for an MP2MP VPN service is usually full or partial mesh. To support such VPN services, the corresponding VTN also needs to provide MP2MP connectivity among the end points.¶

Other service requirements may be expressed at different granularities, some of which can be applicable to the whole service, while some others may only be applicable to some pairs of end points. For example, when a particular level of performance guarantee is required, the point-to-point path through the underlying VTN of the VPN+ service may need to be specifically engineered to meet the required performance guarantee.¶

4.3. Application Specific Data Types

Although a lot of the traffic that will be carried over VPN+ will likely be IP based, the design must be capable of carrying other traffic types, in particular Ethernet traffic. This is easily accomplished through the various pseudowire (PW) techniques [RFC3985]. Where the underlay is MPLS, Ethernet traffic can be carried over VPN+ encapsulated according to the method specified in [RFC4448]. Where the underlay is IP, Layer Two Tunneling Protocol - Version 3 (L2TPv3) [RFC3931] can be used with Ethernet traffic carried according to [RFC4719]. Encapsulations have been defined for most of the common Layer-2 types for both PW over MPLS and for L2TPv3.¶

4.4. Scaling Considerations

VPNs are instantiated as overlays on top of an operator's network and offered as services to the operator's customers. An important feature of overlays is that they can deliver services without placing per-service state in the core of the underlay network.¶

VPN+ may need to install some additional state within the network to achieve the features that they require. Solutions must consider minimizing and controlling the scale of such state, and deployment architectures should constrain the number of VPN+ services so that the additional state introduced to the network is acceptable and under control. It is expected that the number of VPN+ services will be small at the beginning, and even in future the number of VPN+ services will be fewer than traditional VPNs because pre-existing VPN techniques are good enough to meet the needs of most existing VPN-type services.¶

In general, it is not required that the state in the network be maintained in a 1:1 relationship with the VPN+ services. It will usually be possible to aggregate a set or group of VPN+ services so that they share the same VTN and the same set of network resources (much in the same way that current VPNs are aggregated over transport tunnels) so that collections of VPN+ services that require the same behavior from the network in terms of resource reservation, latency bounds, resiliency, etc. can be grouped together. This is an important feature to assist with the scaling characteristics of VPN+ deployments.¶

[I-D.dong-teas-nrp-scalability] provides more details of scalability considerations for the network resource partitions used to instantiate VTNs, and Section 7 includes a greater discussion of scalability considerations.¶

5.1. Packet Forwarding Plane Technologies

Several candidate Layer 2 packet- or frame-based data plane solutions which provide the required isolation and guarantees are described in the following sections.¶

5.2. Layer Three Data Plane

This section considers the problem of VPN+ service differentiation and the representation of underlying network resources in the network layer. More specifically, it describes the possible data plane mechanisms to determine the network resources and the logical network topology or paths associated with a VTN.¶

5.3. Non-Packet Data Plane

Non-packet underlay data plane technologies often have TE properties and behaviors, and meet many of the key requirements in particular for bandwidth guarantees, traffic isolation (with physical isolation often being an integral part of the technology), highly predictable latency and jitter characteristics, measurable loss characteristics, and ease of identification of flows. The cost is that the resources are allocated on a long-term and end-to-end basis. Such an arrangement means that the full cost of the resources has to be borne by the service that is allocated with the resources.¶

5.4. Control Plane

The control plane of VPN+ would likely be based on a hybrid control mechanism that takes advantage of a logically centralized controller for on-demand provisioning and global optimization, whilst still relying on a distributed control plane to provide scalability, high reliability, fast reaction, automatic failure recovery, etc. Extension to and optimization of the centralized and distributed control plane is needed to support the enhanced properties of VPN+.¶

As described in section 4, the VPN+ control plane needs to provide the following functions:¶

• Collects information about the underlying network topology and network resources, and exports this to network nodes and/or a centralized controller as required.¶
• Creates VTNs with the network resource and topology properties needed by the VPN+ services.¶
• Distribute the attributes of VTNs to network nodes which participate in the VTNs and/or the centralized controller.¶
• Compute and set up VTPs in each VTN.¶
• Map VPN+ services to an appropriate VTN.¶

The collection of underlying network topology and resource information can be done using existing the IGP and BGP-LS based mechanisms. The creation of VTN and the distribution of VTN attributes may need further control protocol extensions. The computation of VTPs based on the attributes and constraints of the VTN can be performed either by the headend node of the path or a centralized Path Computation Element (PCE).¶

There are two candidate mechanisms for the setup of VTPs in the VTN: RSVP-TE and Segment Routing (SR).¶

• RSVP-TE [RFC3209] provides the signaling mechanism for establishing a TE-LSP in an MPLS network with end-to-end resource reservation. This can be seen as an approach of providing a Virtual Transport Path (VTP) which could be used to bind the VPN to specific network resources allocated within the underlay, but there remain scalability concerns as mentioned in Section 5.2.2.¶
• The SR control plane [RFC8665] [RFC8667] [RFC9085] does not have the capability of signaling resource reservations along the path. On the other hand, the SR approach provides a potential way of binding the underlay network resource and the VTNs without requiring per-path state to be maintained in the network. A centralized controller can perform resource planning and reservation for VTNs, and it needs to instruct the network nodes to ensure that resources are correctly allocated for the VTN. The controller could provision the SR paths based on the mechanism in [I-D.ietf-spring-segment-routing-policy] to the headend nodes of the paths.¶

According to the service requirements on connectivity, performance and isolation, one VPN+ service may be mapped a dedicated VTN, or a group of VPN+ services may be mapped to the same VTN. The mapping of VPN+ services to VTN can be achieved using existing control mechanisms with possible extensions, and it can be based on either the characteristics of the data packet or the attributes of the VPN service routes.¶

5.5. Management Plane

The management plane provides the interface between the VPN+ service provider and the customers for life-cycle management of the VPN+ service (i.e., creation, modification, assurance/monitoring, and decommissioning). It relies on a set of service data models for the description of the information and operations needed on the interface.¶

As an example, in the context of 5G end-to-end network slicing [TS28530], the management of VPN+ services is considered as the management of the transport network segment of the 5G end-to-end network slice. The 3GPP management system may provide the connectivity and performance related parameters as requirements to the management plane of the transport network. It may also require the transport network to expose the capabilities and status of the network slice. Thus, an interface between the VPN+ management plane and the 5G network slice management system, and relevant service data models are needed for the coordination of 5G end-to-end network slice management.¶

The management plane interface and data models for VPN+ services can be based on the service models described in Section 5.6.¶

It is important that the management life-cycle supports in-place modification of VPN+ services. That is, it should be possible to add and remove end points, as well as to change the requested characteristics of the service that is delivered. The management system needs to be able to assess the revised VPN+ requests and determine whether they can be provided by the existing VTNs or whether changes must be made, and it will additionally need to determine whether those changes to the VTN are possible. If not, then the customer's modification request may be rejected.¶

When the modification of a VPN+ service is possible, the management system should make every effort to make the changes in a non-disruptive way. That is, the modification of the VPN+ service or the underlying VTN should not perturbate traffic on the VPN+ service in a way that causes the service level to drop below the agreed levels. Furthermore, in the spirit of isolation, changes to one VPN+ service should not cause disruption to other VPN+ services.¶

The network operator for the underlay network (i.e., the provider of the VPN+ service) may delegate some operational aspects of the overlay VPN and the underlying VTN to the customer. In this way, the VPN+ is presented to the customer as a virtual network, and the customer can choose how to use that network. The customer cannot exceed the capabilities of the virtual links and nodes, but can decide how to load traffic onto the network, for example, by assigning different metrics to the virtual links so that the customer can control how traffic is routed through the virtual network. This approach requires a management system for the virtual network, but does not necessarily require any coordination between the management systems of the virtual network and the physical network, except that the virtual network management system might notice when the VTN is close to capacity or considerably under-used and automatically request changes in the service provided by the underlay network.¶

5.6. Applicability of Service Data Models to Enhanced VPN

This section describes the applicability of the existing and in-progress service data models to VPN+. [RFC8309] describes the the scope and purpose of service models and shows where a service model might fit into a SDN based network management architecture. New service models may also be introduced for some of the required management functions.¶

Service data models are used to represent, monitor, and manage the virtual networks and services enabled by VPN+. The VPN customer service models (e.g., the layer 3 VPN service model (L3SM) [RFC8299], the layer 2 VPN service model (L2SM) [RFC8466]), or the ACTN Virtual Network (VN) model [I-D.ietf-teas-actn-vn-yang]) are service models which can provide the customer's view of the VPN+ service. The layer 3 VPN network model (L3NM) [I-D.ietf-opsawg-l3sm-l3nm], the layer 2 VPN network model (L2NM) [I-D.ietf-opsawg-l2nm] provide the operator's view of the managed infrastructure as a set of virtual networks and the associated resources. The NRP model [I-D.wd-teas-nrp-yang] further provides the management of the virtual underlay network topology and resources both in the controller and in the network devices to instantiate the VTNs needed for the VPN+ services.¶

The ACTN framework[RFC8453] supports operators in viewing and controlling different domains and presenting virtualized networks to their customers. [I-D.ietf-teas-applicability-actn-slicing] discusses the applicability of the ACTN approach in the context of network slicing. Since there is a strong correlation between network slices and enhanced VPNs, that document also give guidance on how ACTN can be applied to enhanced VPNs.¶

7.1. Maximum Stack Depth of SR

One of the challenges with SR is the stack depth that nodes are able to impose on packets [RFC8491]. This leads to a difficult balance between adding state to the network and minimizing stack depth, or minimizing state and increasing the stack depth.¶

7.2. RSVP-TE Scalability

The traditional method of creating a resource allocated path through an MPLS network is to use the RSVP-TE protocol. However, there have been concerns that this requires significant continuous state maintenance in the network. Work to improve the scalability of RSVP-TE LSPs in the control plane can be found in [RFC8370].¶

There is also concern at the scalability of the forwarder footprint of RSVP-TE as the number of paths through a label switching router (LSR) grows. [RFC8577] addresses this by employing SR within a tunnel established by RSVP-TE.¶