Routing Area Working Group                                     S. Bryant
Internet-Draft                                                   J. Dong
Intended status: Informational                                    Huawei
Expires: May 3, 2018                                    October 30, 2017

                Enhanced Virtual Private Networks (VPN+)


   This draft describes a number of enhancements that need to be made to
   virtual private networks (VPNs) to support the needs of new
   applications, particularly applications that are associated with 5G
   services.  A network enhanced with these properties may form the
   underpin of network slicing, but will also be of use in its own

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on May 3, 2018.

Copyright Notice

   Copyright (c) 2017 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   ( in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of

Bryant & Dong              Expires May 3, 2018                  [Page 1]

Internet-Draft                    VPN+                      October 2017

   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Requirements Language . . . . . . . . . . . . . . . . . . . .   3
   3.  Overview of the Requirements  . . . . . . . . . . . . . . . .   3
     3.1.  Isolation between VPNs  . . . . . . . . . . . . . . . . .   4
     3.2.  Guaranteed Performance  . . . . . . . . . . . . . . . . .   5
     3.3.  Integration . . . . . . . . . . . . . . . . . . . . . . .   6
     3.4.  Dynamic Configuration . . . . . . . . . . . . . . . . . .   6
     3.5.  Customized Control Plane  . . . . . . . . . . . . . . . .   7
   4.  Architecture and Components of VPN+ . . . . . . . . . . . . .   7
     4.1.  Data-Plane  . . . . . . . . . . . . . . . . . . . . . . .   7
       4.1.1.  Data-plane Layering . . . . . . . . . . . . . . . . .   8
       4.1.2.  Use of Segment Routing Constructs . . . . . . . . . .   8
       4.1.3.  Segment Routing and Isolation . . . . . . . . . . . .   8
     4.2.  Stateful and Stateless Virtual Networks . . . . . . . . .   9
     4.3.  Latency Support . . . . . . . . . . . . . . . . . . . . .  10
     4.4.  Integrating Service Function Chains and VPNs  . . . . . .  11
     4.5.  Application Specific Network Types  . . . . . . . . . . .  11
     4.6.  Control Plane Considerations  . . . . . . . . . . . . . .  12
   5.  Scalability Considerations  . . . . . . . . . . . . . . . . .  12
     5.1.  Maximum Stack Depth . . . . . . . . . . . . . . . . . . .  13
     5.2.  RSVP scalability  . . . . . . . . . . . . . . . . . . . .  13
   6.  OAM and Instrumentation . . . . . . . . . . . . . . . . . . .  14
   7.  Service Disruption During Change  . . . . . . . . . . . . . .  14
   8.  Security Considerations . . . . . . . . . . . . . . . . . . .  15
   9.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  15
   10. References  . . . . . . . . . . . . . . . . . . . . . . . . .  15
     10.1.  Normative References . . . . . . . . . . . . . . . . . .  15
     10.2.  Informative References . . . . . . . . . . . . . . . . .  15
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  16

1.  Introduction

   Virtual networks, often referred to as virtual private networks
   (VPNs) have served the industry well as a means of providing
   different groups of users with logically isolated access to a common
   network.  The common or base network that is used to provide the VPNs
   is often referred to as the underlay, and the VPN is often called an

   Driven largely by needs surfacing from 5G, the concept of network
   slicing has gained traction.  There is a need to create a VPN with
   enhanced characteristics.  Specifically there is a need for a
   transport network supporting a set of virtual networks each of which

Bryant & Dong              Expires May 3, 2018                  [Page 2]

Internet-Draft                    VPN+                      October 2017

   provides the client with dedicated (private) networking, computing
   and storage resources drawn from a shared pool.
   The tenant of such a network can require a degree of isolation and
   performance that previously could only be satisfied by dedicated
   networks.  Additionally the tenant may ask for some level of control
   of their virtual network e.g. to customize the service paths in the
   network slice.

   These properties cannot be met with pure overlay networks, as they
   require tighter coordination and integration between the underlay and
   the overlay network.  This document introduces a new network service
   called enhanced VPN (VPN+).  VPN+ refers to a virtual network which
   has dedicated network resources allocated from the underlay network,
   and can achieve a greater isolation and lower latency than
   traditional VPN.

   These new network layer properties, which have general applicability,
   may also be of interest as part of a network slicing solution

   In this draft we identify the new and modified components that need
   to be provided in the network layer and their associated control and
   monitoring of an enhanced VPN (VPN+).  Specifically we are concerned
   with the technology needed to be provided by the enhanced VPN
   underlay, the enhanced VPN data-plane and the necessary protocols in
   both the underlay and the overlay of enhanced VPN.  One use for
   enhanced VPNs is to create network slices with different isolation
   requirements.  Such slices may be used to provide different tenants
   of vertical industrial markets with their own virtual network with
   the explicit characteristics required.  These slices may be "hard"
   slices providing a high degree of confidence that the VPN+
   characteristics will be maintained over the slice life cycle, of they
   may be "soft" slices in which case some degree of interaction may be

2.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "OPTIONAL" in this document are to be interpreted as described in

3.  Overview of the Requirements

   In this section we provide an overview of the requirements of an
   enhanced VPN.

Bryant & Dong              Expires May 3, 2018                  [Page 3]

Internet-Draft                    VPN+                      October 2017

3.1.  Isolation between VPNs

   The requirement is to provide both hard and soft isolation between
   the tenants/applications using one enhanced VPN and the tenants/
   applications using another enhanced VPN.  Hard isolation is needed so
   that applications with exacting requirements can function correctly
   despite a flash demand being created on another VPN competing for the
   underlying resources.  An example might be a network supporting both
   emergency services and public broadband multi-media services.

   During a major incident the VPNs supporting these services would both
   be expected to experience high data volumes, and it is important that
   both make progress in the transmission of their data.  In these
   circumstances the VPNs would require an appropriate degree of
   isolation to be able to continue to operate acceptably.

   We introduce the terms hard (static) and soft (dynamic) isolation to
   cover cases such as the above.  A VPN has soft isolation if the
   traffic of one VPN cannot be inspected by the traffic of another.
   Both IP and MPLS VPNs are examples of soft isolated VPNs because the
   network delivers the traffic only to the required VPN endpoints.
   However the traffic from one or more VPNs and regular network traffic
   may congest the network resulting in delays for other VPNs operating
   normally.  The ability for a VPN to be sheltered from this effect is
   called hard isolation, and this property is required by some critical

   Although these isolation requirements are triggered by the needs of
   5G networks, they have general utility.

   It is of course possible to achieve high degrees of isolation in the
   optical layer.  However this is done at the cost of allocating
   resources on a long term basis and end-to-end basis.  Such an
   arrangement means that the full cost of the resources must be borne
   by the service that is allocated the resources.  On the other hand,
   isolation at the packet layer allows the resources to be shared
   among-st many services and only dedicated to a service on a temporary
   basis.  This allows greater statistical multiplexing of network
   resources and amortizes the cost over many services, leading to
   better economy.  However, the degree of isolation required by network
   slicing cannot easily be met with MPLS-TE packet LSPs.
   Thus some trade-off between the two approaches needs to be considered
   to provide the required isolation between virtual networks while
   still allows reasonable sharing inside each VPN.  The work of the
   IEEE project on Time Sensitive Networking is introducing the concept
   of packet scheduling where a high priority packet stream may be given
   a scheduled time slot thereby guaranteeing that it experiences no
   queuing delay and hence a reduced latency.  However where no

Bryant & Dong              Expires May 3, 2018                  [Page 4]

Internet-Draft                    VPN+                      October 2017

   scheduled packet arrives its reserved time-slot is handed over to
   best effort traffic, thereby improving the economics of the network.

   One of the key areas in which isolation needs to be provided is at
   the interfaces.  If nothing is done the system falls back to the
   router queuing system in which the ingress places it on a selected
   output queue.  Modern routers have quite sophisticated output queuing
   systems, but normally these do not provide the type of scheduling
   system needed to support the levels of isolation needed for the
   applications that are the target of VPN+ networks.  One alternative
   approach is to employ a true time domain multiplexing system with
   fixed time elements allocated to a number of sub-interfaces.  This is
   the approach that FlexE uses.  As noted elsewhere this produces hard
   isolation but at the cost of making the reclamation of unused
   bandwidth harder.

   Another approach, pursued in the Time Sensitive Networking (TSN)
   space, is to time schedule the transmission of one, or a small number
   of packets from a queue dedicated to a time-slot.  Such an approach
   appears to offer greater flexibility for the reclamation of unused
   bandwidth, thereby improving the economics of the system.

   These approaches can usefully be used in tandem.  It is possible to
   use FlexE to provide tenant isolation, and then to use the TSN
   approach over FlexE to provide service performance guarantee inside
   the a slice/tenant VPN.

3.2.  Guaranteed Performance

   There are several aspects to guaranteed performance, guaranteed
   maximum packet loss, guaranteed maximum delay and guaranteed delay

   Guaranteed maximum packet loss is a common parameter, and is usually
   addressed by setting the packet priorities, queue size and discard
   policy.  However this becomes more difficult when the requirement is
   combine with the latency requirement.  The limiting case is zero
   congestion loss, and than is the goal of the Deterministic Networking
   work that the IETF and IEEE are pursuing.  In modern optical networks
   loss due to transmission errors is already asymptotic to zero due,
   but there is always the possibility of failure of the interface and
   the fiber itself.  This can only be addressed by some form of packet
   duplication and transmission over diverse paths.

   Guaranteed maximum latency is required in a number of applications
   particularly real-time control applications and some types of virtual
   reality applications.  The work of the IETF Deterministic Networking
   (DetNet) Working Group is relevant, however the scope needs to be

Bryant & Dong              Expires May 3, 2018                  [Page 5]

Internet-Draft                    VPN+                      October 2017

   extended to methods of enhancing the underlay to better support the
   delay guarantee, and to integrate these enhancements with the overall
   service provision.

   Guaranteed maximum delay variation is a service that may also be
   needed.  Time transfer is one example of a service that needs this,
   although the fungible nature of time means that it might be delivered
   by the underlay as a shared service and not provided through
   different virtual networks.  Alternatively a dedicated virtual
   network may be used to provide this as a shared service.  The need
   for guaranteed maximum delay variation as a general requirement is
   for further study.

   A useful mechanism to provide these guarantees is to use Flex
   Ethernet [FLEXE] as the underlay.  This is a method of bonding
   Ethernets together and of providing time-slot based channelization
   over an Ethernet bearer.  Such channels are fully isolated from other
   channels running over the same Ethernet bearer.

3.3.  Integration

   A solution to the enhanced VPN problem will need to provide seamless
   integration of both physical and virtual network.  Given the
   targeting of both this technology and service function chaining at
   mobile networks and in particular 5G the co-integration of service
   functions is a likely requirement.

3.4.  Dynamic Configuration

   It is necessary that new enhanced VPNs can be introduced to the
   network, modified, and removed from the network.  In doing so due
   regard must be given to the impact of other enhanced VPNs that are
   operational.  An enhanced VPN that requires hard isolation must not
   be disrupted by the installation or modification of another enhanced

   Whether modification of an enhanced VPN can be disruptive to that
   VPN, and in particular the traffic in flight is to be determined, but
   is likely to be a difficult problem to address.

   The data-plane aspect of this are discussed further in Section 7.

   The control-plane aspects of this, particularly the garbage
   collection are likely to be challenging and are for further study.

Bryant & Dong              Expires May 3, 2018                  [Page 6]

Internet-Draft                    VPN+                      October 2017

3.5.  Customized Control Plane

   In some cases it is desirable that an enhanced VPN has a custom
   control plane, so that the tenant of the enhanced VPN can have some
   control to the resources and functions partitioned for this VPN.
   Each enhanced VPN may have its own dedicated controller, or be
   provided with an interface to the control plane of the underlay.

   Further detail on this requirement will be provided in a future
   version of the draft.

4.  Architecture and Components of VPN+

   VPN+ runs over a substrate or underlay that it draws on to provide
   the resources and features needed to provide enhanced VPN services to
   its tenants.  The assumption is that a number of such enhanced VPNs
   are layered on this underlay, each drawing the resources that they
   need to satisfy the needs of their tenants.  Thus each enhanced VPN
   is bound to a specific set of resources allocated from the underlay,
   with different subsets of the underlay resources dedicated to
   different enhanced VPNs.  The consequence of this is that any VPN+
   solution needs tighter coupling to underlay that is the case with
   classical VPNs.

   An enhanced VPN needs to be designed with consideration given to:

   o  The layering of the VPN+ data-plane onto the substrate.

   o  The amount of static and dynamic state.

   o  The ammount of state in the packet vs the am-mount of state in the
      control plane.

   o  How sufficient isolation is achieved between VPN+ instances and
      between VPN+ instances and the best effort traffic.

   o  How the required latency demands are achieved.

   o  Support of the required integration between network functions and
      service functions.

   o  The design of the control plane.

4.1.  Data-Plane

   The data-plane is required to provide each VPN+ with paths through
   the specific resources allocated to it.  This requires a finer
   granularity of packet steering than is normally provided in networks.

Bryant & Dong              Expires May 3, 2018                  [Page 7]

Internet-Draft                    VPN+                      October 2017

   One of the candidate approaches is to use segment routing.  This is
   discussed Section 4.1.2.

4.1.1.  Data-plane Layering

   An enhanced VPN needs to run on a substrate or underlay that can draw
   on to provide the resources and features it needs.  In order to meet
   the isolation requirement, network resources in the underlay need to
   split into different subsets, which are dedicated to different VPNs.
   For VPNs which require hard isolation, at least the underlay tunnels
   cannot be shared.  In a scalable solution we need to layer a number
   of VPNs on the underlay, and for each enhanced VPN to draws on the
   resources provided by the underlay to deliver a service with the
   required properties.

   Different subsets of the underlay resources are dedicated to
   different VPNs.  The VPN+ solution needs tighter coupling with
   underlay.  We cannot for example share the tunnel between enhanced
   VPNs which require hard isolation.

4.1.2.  Use of Segment Routing Constructs

   Clearly we can use traditional constructs to create a VPN, but there
   are advantages to the use of other constructs such as Segment Routing
   (SR) in the creation of virtual networks with enhanced properties.

   Segment Routing [I-D.ietf-spring-segment-routing] is a method that
   prepends instructions to packets at entry and sometimes at various
   points as it passes though the network.  These instructions allow
   packets to be routed on paths other than the shortest path for
   various traffic engineering reasons.  These paths can be strict or
   loose paths, depending on the compactness required of the instruction
   list and the degree of autonomy granted to the network (for example
   to support ECMP).

   With SR, a path needs to be dynamically created through a set of
   resources by simply specifying the Segment IDs (SIDs), i.e.
   instructions rooted at a particular point in the network.  Thus if a
   path is to be provisioned from some ingress point A to some egress
   point B in the underlay, A is provided with the A..B SID list and
   instructions on how to identify the packets to which the SID list is
   to be prepended.

4.1.3.  Segment Routing and Isolation

   With current segment routing, the instructions are used to specify
   the nodes and links to be traversed.  However, in order to achieve
   the required isolation between different services, new instructions

Bryant & Dong              Expires May 3, 2018                  [Page 8]

Internet-Draft                    VPN+                      October 2017

   can be created which can be prepended to a packet to steer it through
   specific dedicated network resources and functions, e.g. links,
   queues, processors, services etc.

4.2.  Stateful and Stateless Virtual Networks

   A VPN is a network created by applying a multiplexing technique to
   the underlying network (the underlay) in order to distinguish the
   traffic of one VPN from that of another.  A VPN path that travels by
   other than the shortest path through the underlay normally requires
   state in the underlay to specify that path.  State is normally
   applied to the underlay through the use of the RSVP Signaling
   protocol, or directly through the use of an SDN controller, although
   other techniques may emerge as this problem is studied.  This state
   gets harder to manage as the number of VPN paths increases.
   Furthermore, as we increase the coupling between the underlay and the
   overlay to support the VPN which requires enhanced VPN service, this
   state will increase further.

   By encoding the state in the packet, as is done in Segment Routing,
   state is transitioned out of the network.

   |       |     |
   |       |     |

                     Figure 1: An SR Network Fragment

   Consider the network fragment shown in Figure 1.  To send a packet
   from A to E via B, D & E: Node A prepends the ordered list of SIDs:D,
   E to the packet and pushes the packet to B.  SID list {B, D, E} can
   be used as a VPN path.  Thus, to create a VPN, a set of SID Lists is
   created and provided to each ingress node of the VPN together with
   packet selection criteria.  In this way it is possible to create a
   VPN with no state in the core.  However this is at the expense of
   creating a larger packet with possible MTU and hardware restriction
   limits that need to be overcome.

   Note in the above if A and E support multiple VPN an additional VPN
   identifier will need to be added to the packet, but this is omitted
   from this text for simplicity.

Bryant & Dong              Expires May 3, 2018                  [Page 9]

Internet-Draft                    VPN+                      October 2017

   |       |       |
   |       Q       |
   |       |       |

                   Figure 2: Another SR Network Fragment

   Consider a further network fragment shown in Figure 2, and further
   consider VPN A+D+E.

   A has lists: {P, B, Q, D}, {P, B, S, E}
   D has lists: {Q, B, P, A}, {E}
   E has lists: {S, B, P, A}, {D}

   To create a new VPN C+D+B the following list are introduced:

   C lists: {R, D}, {A, P, B}
   D lists: {R, C}, {Q, B}
   B lists: {Q, D}, {P, A, C}

   Thus VPN C+D+B was created without touching the settings of the core
   routers, indeed it is possible to add endpoints to the VPNs, and move
   the paths around simply by providing new lists to the affected

   When SR is extended to support isolation finer granularity state
   needs to be added to the core in anticipation of its use.  We
   therefore need to evaluate the balance between this additional state
   and the performance delivered by the network.

4.3.  Latency Support

   The IETF has ongoing work on support for a latency ceiling
   [I-D.ietf-detnet-architecture].  The provision of a latency ceiling
   is a requirement of the application seeking the use of enhanced
   virtual networks.  The current design of DetNet assumes the design of
   the underlay network is unchanged.  In this document we look at some
   changes that could be used to assist in achieving low latency ceiling
   across the wide area.

   Traditionally a traffic engineered path operates with a granularity
   of a link with hints about priority provided through the use of the
   traffic class field in the header.  However to achieve the latency
   and isolation characteristics that are sought by VPN+ users, steering
   packets through specific queues resources will likely be required.
   The extent to which these needs can be satisfied through existing QoS
   mechanisms is to be determined.  What is clear is that a fine control

Bryant & Dong              Expires May 3, 2018                 [Page 10]

Internet-Draft                    VPN+                      October 2017

   of which services wait for which, with a fine granularity of queue
   management policy is needed.  Note that the concept of a queue is a
   useful abstraction for many types of underlay mechanism that may be
   used to provide enhanced latency support.  From the perspective of
   the control plane and from the perspective of the segment routing the
   method of steering a packet to a queue that provides the required
   properties is a universal construct.  How the queue satisfies the
   requirement is outside the scope of these aspect of the enhanced VPN
   system.  Thus for example a FlexE channel, or time sensitive
   networking packet scheduling slot are abstracted to the same concept
   and bound to the data plane in a common manner.

   We can introduce the specification of finer, deterministic,
   granularity to path selection through extensions to traditional path
   construction techniques such as RSVP-TE and MPLS-TP.

   We can also introduce it by specifying the queue through an SR
   instruction list.  Thus new SR instructions may be created to specify
   not only which resources are traversed, but in some cases how they
   are traversed.  For example, it may be possible to specify not only
   the queue to be used but the policy to be applied when enqueuing and

   This concept can be further generalized, since as well as queuing to
   the output port of a router, it is possible to queue to any resource,
   for example:

   o  A network processor unit (NPU)

   o  A Central Processing Unit (CPU) Core

   o  A Look-up engine such as TCAM

4.4.  Integrating Service Function Chains and VPNs

   There is a significant overlap between the problem of routing a
   packet though a set of network resources and the problem of routing a
   packet through a set of compute resources.  Service Function Chain
   technology is designed to forward a packet through a set of compute

   A future version of this document will discuss this further.

4.5.  Application Specific Network Types

   Although the transport service that underpins the extended VPN is
   likely MPLS/IP based, it needs to be able to carry application

Bryant & Dong              Expires May 3, 2018                 [Page 11]

Internet-Draft                    VPN+                      October 2017

   specific non-MPLS/IP traffic.  This can be accommodated through the
   use of pseudowires (PWs).

4.6.  Control Plane Considerations

   It is expected that VPN+ would be based on a hybrid control
   mechanism, which takes advantage of the logically centralized
   controller for on-demand provisioning and global optimization, whilst
   still relies on distributed control plane to provide scalability,
   high reliability, fast reaction, automatic failure recovery etc.
   Extension and optimization to the distributed control plane is needed
   to support the enhanced properties of VPN+.

   Where SR is used as a the data-plane construct it needs to be noted
   that it does not have the capability of reserving resources along the
   path nor do its currently specified distributed control plane (the
   link state routing protocols).  An SDN controller can clearly do
   this, from the controllers point of view, and no resource reservation
   is done on the device.  Thus if a distributed control plane is needed
   either in place of an SDN controller or as an assistant to it there
   is a risk of resource conflict.  This needs further study.

   On the other hand an advantage of using an SR approach is that it
   provides a way of efficiently binding the network underlay and the
   enhanced VPN overlay.  With a technology such as RSVP-TE LSPs, each
   virtual path in the VPN is bound to the underlay with a dedicated TE-
   RSVP-TE could be enhanced to bind the VPN to specific resources
   within the underlay, but as noted elsewhere in this document there
   are concerns as to the scalability of this approach.  With an SR-
   based approach to resource reservation (per-slice reservation), it is
   straightforward to create dedicated SR network slices, and the VPN
   can be bound to a particular SR network slice.

5.  Scalability Considerations

   For a packet to transit a network, other than on a best effort,
   shortest path basis, it is necessary to introduce additional state,
   either in the packet, or in the network of some combination of both.

   There are at least three ways of doing this:

   o  Introduce the complete state into the packet.  That is how SR does
      this, and this allows the controller to specify the precise series
      of forwarding and processing instructions that will happen to the
      packet as it transits the network.  The cost of this is an
      increase in the packet header size.  The cost is also that systems
      will have capabilities enabled in case they are called upon by a

Bryant & Dong              Expires May 3, 2018                 [Page 12]

Internet-Draft                    VPN+                      October 2017

      service.  This is a type of latent state, and increases as we more
      precisely specify the path and resources that need to be
      exclusively available to a VPN.

   o  Introduce the state to the network.  This is normally done by
      creating a path using RSVP-TE, which can be extended to introduce
      any element that needs to be specified along the path, for example
      explicitly specifying queuing policy.  It is of course possible to
      use other methods to introduce path state, such as via a Software
      Defined Network (SDN) controller, or possibly by modifying a
      routing protocol.  With this approach there is state per path per
      path characteristic that needs to be maintained over its life-
      cycle.  This is more state than is needed using SR, but the packet
      are shorter.

   o  Provide a hybrid approach based on using binding SIDs to create
      path fragments, and bind them together with SR.

   Dynamic creation of a VPN path using SR requires less state
   maintenance in the network core at the expense of larger VPN headers
   on the packet.  The scaling properties will reduce roughly from a
   function of (N/2)^2 to a function of N, where N is the VPN path
   length in intervention points (hops plus network functions).
   Reducing the state in the network is important to VPN+, as VPN+
   requires the overlay to be more closely integrated with the underlay
   than with traditional VPNs.  This tighter coupling would normally
   mean that significant state needed to be created and maintained in
   the core.  However, a segment routed approach allows much of this
   state to be spread amongst the network ingress nodes, and transiently
   carried in the packets as SIDs.

   These approaches are for further study.

5.1.  Maximum Stack Depth

   One of the challenges with SR is the stack depth that nodes are able
   to impose on packets.  This leads to a difficult balance between
   adding state to the network and minimizing stack depth, or minimizing
   state and increasing the stack depth.

5.2.  RSVP scalability

   The traditional method of creating a resource allocated path through
   an MPLS network is to use the RSVP protocol.  However there have been
   concerns that this requires significant continuous state maintenance
   in the network.  There are ongoing works to improve the scalability
   of RSVP-TE LSPs in the control plane

Bryant & Dong              Expires May 3, 2018                 [Page 13]

Internet-Draft                    VPN+                      October 2017

   [I-D.ietf-teas-rsvp-te-scaling-rec].  This will be considered further
   in a future version of this document.

   There is also concern at the scalability of the forwarder footprint
   of RSVP as the number of paths through an LSR grows
   [I-D.sitaraman-mpls-rsvp-shared-labels] proposes to address this by
   employing SR within a tunnel established by RSVP-TE.  This work will
   be considered in a future version of this document.

6.  OAM and Instrumentation

   This will be discussed in a future version of this draft.  A
   discussion should include

   o  Instrumentation of the underlay.

   o  Instrumentation of the overlay by both customer and provider.

   o  Verification of the conformity of the path to the service

7.  Service Disruption During Change

   Each enhanced VPN, of necessity, has a life-cycle, and needs
   modification during deployment as the needs of its user change.
   Additionally as the network as a whole evolves there will need to be
   garbage collection performed to consolidate resources into usable

   Systems in which the path is imposed such as SR, or some form of
   explicit routing tend to do well in these applications because it is
   possible to perform an atomic transition from one path to another.
   However implementations and the monitoring protocols need to make
   sure that the new path is up before traffic is transitioned to it.

   There are however two manifestations of the latency problem that are
   for further study in any of these approaches:

   o  The problem of packets overtaking one and other if a path latency
      reduces during a transition.

   o  The problem of the latency transient in either direction as a path

   There is also the matter of what happens during failure in the
   underlay infrastructure.  Fast reroute is one approach, but that
   still produces a transient loss with a normal goal of rectifying this
   within 50ms.  An alternative is some form of N+1 delivery such as has

Bryant & Dong              Expires May 3, 2018                 [Page 14]

Internet-Draft                    VPN+                      October 2017

   been used for many years to support protection from service
   disruption.  This may be taken to a different level using the
   techniques proposed by the IETF deterministic network work with
   multiple in-network replication and the culling of later packets.

   In addition to the approach used to protect high priority packets,
   consideration has to be given to the impact of best effort traffic on
   the high priority packets during a transient.  Specifically if a
   conventional re-convergence process is used there will inevitably be
   micro-loops and whilst some form of explicit routing will protect the
   high priority traffic, lower priority traffic on best effort shortest
   paths will micro-loop without the use of a loop prevention
   technology.  To provide the highest quality of service to high
   priority traffic, either this traffic must be shielded from the
   micro-loops, or micro-loops must be prevented.

8.  Security Considerations

   All types of virtual network require special consideration to be
   given to the isolation between the tenants.  However in an enhanced
   virtual network service hard isolation needs to be considered.  If a
   service requires a specific latency then it can be damaged by simply
   delaying the packet through the activities of another tenant.  In a
   network with virtual functions, depriving a function used by another
   tenant of compute resources can be just as damaging as delaying
   transmission of a packet in the network.

9.  IANA Considerations

   There are no requested IANA actions.

10.  References

10.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997, <https://www.rfc-

10.2.  Informative References

   [FLEXE]    "Flex Ethernet Implementation Agreement", March 2016,

Bryant & Dong              Expires May 3, 2018                 [Page 15]

Internet-Draft                    VPN+                      October 2017

              67, 4., Dong, J., Bryant, S.,,
              k., Galis, A., Foy, X., and S. Kuklinski, "Network Slicing
              Architecture", draft-geng-netslices-architecture-02 (work
              in progress), July 2017.

              Finn, N., Thubert, P., Varga, B., and J. Farkas,
              "Deterministic Networking Architecture", draft-ietf-
              detnet-architecture-03 (work in progress), August 2017.

              Filsfils, C., Previdi, S., Ginsberg, L., Decraene, B.,
              Litkowski, S., and R. Shakir, "Segment Routing
              Architecture", draft-ietf-spring-segment-routing-13 (work
              in progress), October 2017.

              Beeram, V., Minei, I., Shakir, R., Pacella, D., and T.
              Saad, "Techniques to Improve the Scalability of RSVP
              Traffic Engineering Deployments", draft-ietf-teas-rsvp-te-
              scaling-rec-07 (work in progress), September 2017.

              Sitaraman, H., Beeram, V., Parikh, T., and T. Saad,
              "Signaling RSVP-TE tunnels on a shared MPLS forwarding
              plane", draft-sitaraman-mpls-rsvp-shared-labels-02 (work
              in progress), September 2017.

Authors' Addresses

   Stewart Bryant


   Jie Dong


Bryant & Dong              Expires May 3, 2018                 [Page 16]