Skip to main content

DetNet Data Plane: MPLS
draft-ietf-detnet-mpls-00

The information below is for an old version of the document.
Document Type
This is an older version of an Internet-Draft that was ultimately published as RFC 8964.
Authors Balazs Varga , János Farkas , Lou Berger , Don Fedyk , Andrew G. Malis , Stewart Bryant , Jouni Korhonen
Last updated 2019-06-08 (Latest revision 2019-05-06)
Replaces draft-ietf-detnet-dp-sol-mpls
RFC stream Internet Engineering Task Force (IETF)
Formats
Reviews
Additional resources Mailing list discussion
Stream WG state WG Document
Document shepherd (None)
IESG IESG state Became RFC 8964 (Proposed Standard)
Consensus boilerplate Unknown
Telechat date (None)
Responsible AD (None)
Send notices to (None)
draft-ietf-detnet-mpls-00
DetNet                                                     B. Varga, Ed.
Internet-Draft                                                 J. Farkas
Intended status: Standards Track                                Ericsson
Expires: November 6, 2019                                      L. Berger
                                                                D. Fedyk
                                                 LabN Consulting, L.L.C.
                                                                A. Malis
                                                               S. Bryant
                                                     Huawei Technologies
                                                             J. Korhonen
                                                             May 5, 2019

                        DetNet Data Plane: MPLS
                       draft-ietf-detnet-mpls-00

Abstract

   This document specifies the Deterministic Networking data plane when
   operating over an MPLS Packet Switched Networks.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on November 6, 2019.

Copyright Notice

   Copyright (c) 2019 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect

Varga, et al.           Expires November 6, 2019                [Page 1]
Internet-Draft                 DetNet MPLS                      May 2019

   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
     2.1.  Terms Used in This Document . . . . . . . . . . . . . . .   3
     2.2.  Abbreviations . . . . . . . . . . . . . . . . . . . . . .   4
   3.  Requirements Language . . . . . . . . . . . . . . . . . . . .   5
   4.  DetNet MPLS Data Plane Overview . . . . . . . . . . . . . . .   5
     4.1.  Layers of DetNet Data Plane . . . . . . . . . . . . . . .   5
     4.2.  DetNet MPLS Data Plane Scenarios  . . . . . . . . . . . .   6
   5.  MPLS-Based DetNet Data Plane Solution . . . . . . . . . . . .   8
     5.1.  DetNet Over MPLS Encapsulation Components . . . . . . . .   8
     5.2.  MPLS Data Plane Encapsulation . . . . . . . . . . . . . .   9
       5.2.1.  DetNet Control Word and the DetNet Sequence Number  .  10
       5.2.2.  S-Labels  . . . . . . . . . . . . . . . . . . . . . .  12
       5.2.3.  F-Labels  . . . . . . . . . . . . . . . . . . . . . .  14
     5.3.  OAM Indication  . . . . . . . . . . . . . . . . . . . . .  17
     5.4.  Flow Aggregation  . . . . . . . . . . . . . . . . . . . .  17
       5.4.1.  Aggregation at the LSP  . . . . . . . . . . . . . . .  17
       5.4.2.  Aggregating DetNet Flows as a new DetNet flow . . . .  18
       5.4.3.  Simple Aggregation at the DetNet Layer  . . . . . . .  19
     5.5.  Service Sub-Layer Considerations  . . . . . . . . . . . .  19
       5.5.1.  Edge Node Processing  . . . . . . . . . . . . . . . .  20
       5.5.2.  Relay Node Processing . . . . . . . . . . . . . . . .  20
     5.6.  Forwarding Sub-Layer Considerations . . . . . . . . . . .  20
       5.6.1.  Class of Service  . . . . . . . . . . . . . . . . . .  20
       5.6.2.  Quality of Service  . . . . . . . . . . . . . . . . .  21
   6.  Flow Identification Management and Control Information  . . .  22
   7.  Security Considerations . . . . . . . . . . . . . . . . . . .  22
   8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  22
   9.  Contributors  . . . . . . . . . . . . . . . . . . . . . . . .  22
   10. Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  23
   11. References  . . . . . . . . . . . . . . . . . . . . . . . . .  24
     11.1.  Normative References . . . . . . . . . . . . . . . . . .  24
     11.2.  Informative References . . . . . . . . . . . . . . . . .  26
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  28

1.  Introduction

   Deterministic Networking (DetNet) is a service that can be offered by
   a network to DetNet flows.  DetNet provides these flows extremely low
   packet loss rates and assured maximum end-to-end delivery latency.

Varga, et al.           Expires November 6, 2019                [Page 2]
Internet-Draft                 DetNet MPLS                      May 2019

   General background and concepts of DetNet can be found in
   [I-D.ietf-detnet-architecture].

   The DetNet Architecture models the DetNet related data plane
   functions decomposed into two sub-layers: a service sub-layer and a
   forwarding sub-layer.  The service sub-layer is used to provide
   DetNet service functions such as protection and reordering.  The
   forwarding sub-layer is used to provide forwarding assurance (low
   loss, assured latency, and limited reordering).

   This document specifies the DetNet data plane operation and the on-
   wire encapsulation of DetNet flows over an MPLS-based Packet Switched
   Network (PSN) using the service sub-layer reference model.  MPLS
   encapsulation already provides a solid foundation of building blocks
   to enable the DetNet service and forwarding sub-layer functions.
   MPLS encapsulated DetNet can be carried over a variety of different
   network technologies that can provide the DetNet required level of
   service.  However, the specific details of how DetNet MPLS is carried
   over different network technologies is out of scope of this document.

   MPLS encapsulated DetNet flows can carry different types of traffic.
   The details of the types of traffic that are carried in DetNet are
   also out of scope of this document.  An example of IP using DetNet
   MPLS sub-networks can be found in [I-D.ietf-detnet-ip].  DetNet MPLS
   may use an associated controller and Operations, Administration, and
   Maintenance (OAM) functions that are defined outside of this
   document.

   Important background information common to all data planes for DetNet
   can be found in the DetNet Data Plane Framework
   [I-D.ietf-detnet-data-plane-framework].

2.  Terminology

2.1.  Terms Used in This Document

   This document uses the terminology established in the DetNet
   architecture [I-D.ietf-detnet-architecture] and the the DetNet Data
   Plane Framework [I-D.ietf-detnet-data-plane-framework].  The reader
   is assumed to be familiar with these documents and any terminology
   defined therein.

   The following terminology is introduced in this document:

   F-Label       A Detnet "forwarding" label that identifies the LSP
                 used to forward a DetNet flow across an MPLS PSN, e.g.,
                 a hop-by-hop label used between label switching routers
                 (LSR).

Varga, et al.           Expires November 6, 2019                [Page 3]
Internet-Draft                 DetNet MPLS                      May 2019

   S-Label       A DetNet "service" label that is used between DetNet
                 nodes that implement also the DetNet service sub-layer
                 functions.  An S-Label is also used to identify a
                 DetNet flow at DetNet service sub-layer.

   d-CW          A DetNet Control Word (d-CW) is used for sequencing
                 information of a DetNet flow at the DetNet service sub-
                 layer.

2.2.  Abbreviations

   The following abbreviations are used in this document:

   AC            Attachment Circuit.

   CE            Customer Edge equipment.

   CoS           Class of Service.

   CW            Control Word.

   DetNet        Deterministic Networking.

   DF            DetNet Flow.

   DN-IWF        DetNet Inter-Working Function.

   L2            Layer 2.

   L2VPN         Layer 2 Virtual Private Network.

   L3            Layer 3.

   LSR           Label Switching Router.

   MPLS          Multiprotocol Label Switching.

   MPLS-TE       Multiprotocol Label Switching - Traffic Engineering.

   MPLS-TP       Multiprotocol Label Switching - Transport Profile.

   MS-PW         Multi-Segment PseudoWire (MS-PW).

   NSP           Native Service Processing.

   OAM           Operations, Administration, and Maintenance.

   PE            Provider Edge.

Varga, et al.           Expires November 6, 2019                [Page 4]
Internet-Draft                 DetNet MPLS                      May 2019

   PEF           Packet Elimination Function.

   PRF           Packet Replication Function.

   PREOF         Packet Replication, Elimination and Ordering Functions.

   POF           Packet Ordering Function.

   PSN           Packet Switched Network.

   PW            PseudoWire.

   QoS           Quality of Service.

   S-PE          Switching Provider Edge.

   T-PE          Terminating Provider Edge.

   TSN           Time-Sensitive Network.

3.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

4.  DetNet MPLS Data Plane Overview

4.1.  Layers of DetNet Data Plane

   MPLS provides a wide range of capabilities that can be utilised by
   DetNet.  A straight forward approach utilizing MPLS for a DetNet
   service sub-layer is based on the existing pseudowire (PW)
   encapsulations and by utilizing existing MPLS Traffic Engineering
   encapsulations and mechanisms.  Background on PWs can be found in
   [RFC3985] and [RFC3031].  Background on MPLS Traffic Engineering can
   be found in [RFC3272] and [RFC3209].

Varga, et al.           Expires November 6, 2019                [Page 5]
Internet-Draft                 DetNet MPLS                      May 2019

                                  DetNet        MPLS
                                    .
               Bottom of Stack      .
               (inner label)    +------------+
                                |  Service   | d-CW, S-Label
                                +------------+
                                | Forwarding | F-Label(s)
                                +------------+
               Top of Stack         .
               (outer label)        .

              Figure 1: DetNet Adaptation to MPLS Data Plane

   The DetNet MPLS data plane representation is illustrated in Figure 1.
   The service sub-layer includes a DetNet control word (d-CW) and a
   identifying service label (S-Label).  The DetNet control word (d-CW)
   conforms to the Generic PW MPLS Control Word (PWMCW) defined in
   [RFC4385].

   A node operating on a DetNet flow in the Detnet service sub-
   layer,uses the local context associated with that S-Label, provided
   by a received F-Label, to determine what local DetNet operation(s)
   are applied to that packet.  An S-Label may be taken from the
   platform label space [RFC3031], making it unique, enabling DetNet
   flow identification regardless of which input interface or LSP the
   packet arrives on.

   The DetNet forwarding sub-layer is supported by one or more
   forwarding labels (F-Labels).  MPLS Traffic Engineering
   encapsulations and mechanisms can be utilized to provide a forwarding
   sub-layer that is responsible for providing resource allocation and
   explicit routes.

4.2.  DetNet MPLS Data Plane Scenarios

Varga, et al.           Expires November 6, 2019                [Page 6]
Internet-Draft                 DetNet MPLS                      May 2019

   DetNet MPLS       Relay       Transit         Relay       DetNet MPLS
   End System        Node         Node           Node        End System
      (T-PE)        (S-PE)       (LSR)          (S-PE)         (T-PE)
   +----------+                                             +----------+
   |   Appl.  |<------------ End to End Service ----------->|   Appl.  |
   +----------+   +---------+                 +---------+   +----------+
   | Service  |<--| Service |-- DetNet flow --| Service |-->| Service  |
   +----------+   +---------+  +----------+   +---------+   +----------+
   |Forwarding|   |Fwd| |Fwd|  |Forwarding|   |Fwd| |Fwd|   |Forwarding|
   +-------.--+   +-.-+ +-.-+  +----.---.-+   +-.-+ +-.-+   +---.------+
           :  Link  :    /  ,-----.  \   : Link :    /  ,-----.  \
           +........+    +-[  Sub  ]-+   +......+    +-[  Sub  ]-+
                           [Network]                   [Network]
                            `-----'                     `-----'
           |<- LSP -->| |<-------- LSP -----------| |<--- LSP -->|

           |<----------------- DetNet MPLS --------------------->|

                      Figure 2: A DetNet MPLS Network

   Figure 2 illustrates a hypothetical DetNet MPLS-only network composed
   of DetNet aware MPLS enabled end systems, operating over a DetNet
   aware MPLS network.  In this figure, the relay nodes are PE devices
   that define the MPLS LSP boundaries and the transit nodes are LSRs.

   DetNet end system and relay nodes understand the particular needs of
   DetNet flows and provide both DetNet service and forwarding sub-layer
   functions.  In the case of MPLS, DetNet nodes add, remove and process
   d-CWs, S-Labels and F-labels as needed.  DetNet MPLS nodes provide
   functionality analogous to T-PEs when they sit at the edge of an MPLS
   domain, and S-PEs when they are in the middle of an MPLS domain, see
   [RFC6073].

   In a DetNet MPLS network, transit nodes may be DetNet service aware
   or may be DetNet unaware MPLS Label Switching Routers (LSRs).  In
   this latter case, such LSRs would be unaware of the special
   requirements of the DetNet service sub-layer, but would still provide
   traffic engineering functions and the QoS capabilities needed to
   ensure that the (TE) LSPs meet the service requirements of the
   carried DetNet flows.

   The application of DetNet using MPLS supports a number of control
   plane/management plane types.  These types support certain MPLS data
   plane deployments.  For example RSVP-TE, MPLS-TP, or MPLS Segment
   Routing (when extended to support resource allocation) are all valid
   MPLS deployment cases.

Varga, et al.           Expires November 6, 2019                [Page 7]
Internet-Draft                 DetNet MPLS                      May 2019

   Figure 3 illustrates how an end-to-end MPLS-based DetNet service is
   provided in a more detail.  In this figure, the customer end systems,
   CE1 and CE2, are able to send and receive MPLS encapsulated DetNet
   flows, and R1, R2 and R3 are relay nodes in the middle of a DetNet
   network.  The 'X' in the end systems, and relay nodes represents
   potential DetNet compound flow packet replication and elimination
   points.  In this example, service protection is supported utilizing
   least two DetNet member flows and TE LSPs.  For a unidirectional
   flow, R1 supports PRF and R3 supports PEF and POF.  Note that the
   relay nodes may change the underlying forwarding sub-layer, for
   example tunneling MPLS over IEEE 802.1 TSN
   [I-D.ietf-detnet-mpls-over-tsn], or simply over interconnect network
   links.

   DetNet                                           DetNet
   MPLS  Service          Transit          Transit       Service MPLS
   DetNet  |             |<-Tnl->|        |<-Tnl->|            | DetNet
   End     |             V   1   V        V   2   V            | End
   System  |    +--------+       +--------+       +--------+   | System
   +---+   |    |   R1   |=======|   R2   |=======|   R3   |   |  +---+
   |  X...DFa...|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|.DFa..|.X |
   |CE1|========|    \   |       |   X    |       |   /    |======|CE2|
   |   |   |    |     \_.|..DF2..|._/ \__.|..DF4..|._/     |   |  |   |
   +---+        |        |=======|        |=======|        |      +---+
       ^        +--------+       +--------+       +--------+      ^
       |        Relay Node       Relay Node       Relay Node      |
       |          (S-PE)           (S-PE)           (S-PE)        |
       |                                                          |
       |<---------------------- DetNet MPLS --------------------->|
       |                                                          |
       |<--------------- End to End DetNet Service -------------->|

       -------------------------- Data Flow ------------------------->

   X   = Optional service protection (none, PRF, PREOF, PEF/POF)
   DFx = DetNet member flow x over a TE LSP

                    Figure 3: MPLS-Based Native DetNet

5.  MPLS-Based DetNet Data Plane Solution

5.1.  DetNet Over MPLS Encapsulation Components

   To carry DetNet over MPLS the following is required:

   1.  A method of identifying the MPLS payload type.

Varga, et al.           Expires November 6, 2019                [Page 8]
Internet-Draft                 DetNet MPLS                      May 2019

   2.  A method of identifying the DetNet flow group to the processing
       element.

   3.  A method of distinguishing DetNet OAM packets from DetNet data
       packets.

   4.  A method of carrying the DetNet sequence number.

   5.  A suitable LSP to deliver the packet to the egress PE.

   6.  A method of carrying queuing and forwarding indication.

   In this design an MPLS service label (the S-Label), similar to a
   pseudowire (PW) label [RFC3985], is used to identify both the DetNet
   flow identity and the payload MPLS payload type satisfying (1) and
   (2) in the list above.  OAM traffic discrimination happens through
   the use of the Associated Channel method described in [RFC4385].  The
   DetNet sequence number is carried in the DetNet Control word which
   carries the Data/OAM discriminator.  To simplify implementation and
   to maximize interoperability two sequence number sizes are supported:
   a 16 bit sequence number and a 28 bit sequence number.  The 16 bit
   sequence number is needed to support some types of legacy clients.
   The 28 bit sequence number is used in situations where it is
   necessary ensure that in high speed networks the sequence number
   space does not wrap whilst packets are in flight.

   The LSP used to forward the DetNet packet may be of any type (MPLS-
   LDP, MPLS-TE, MPLS-TP [RFC5921], or MPLS-SR
   [I-D.ietf-spring-segment-routing-mpls]).  The LSP (F-Label) label
   and/or the S-Label may be used to indicate the queue processing as
   well as the forwarding parameters.  Note that the possible use of
   Penultimate Hop Popping (PHP) means that the S-Label may be the only
   label received at the terminating DetNet service.

5.2.  MPLS Data Plane Encapsulation

   Figure 4 illustrates a DetNet data plane MPLS encapsulation.  The
   MPLS-based encapsulation of the DetNet flows is well suited for the
   scenarios described in [I-D.ietf-detnet-data-plane-framework].
   Furthermore, an end to end DetNet service i.e., native DetNet
   deployment (see Section 4.2) is also possible if DetNet end systems
   are capable of initiating and termination MPLS encapsulated packets.

   The MPLS-based DetNet data plane encapsulation consists of:

   o  DetNet control word (d-CW) containing sequencing information for
      packet replication and duplicate elimination purposes, and the OAM
      indicator.

Varga, et al.           Expires November 6, 2019                [Page 9]
Internet-Draft                 DetNet MPLS                      May 2019

   o  DetNet service Label (S-Label) that identifies a DetNet flow at
      the receiving DetNet service sub-layer processing node.

   o  Zero or more Detnet MPLS Forwarding label(s) (F-Label) used to
      direct the packet along the label switched path (LSP) to the next
      service sub-layer processing node along the path.  When
      Penultimate Hop Popping is in use there may be no label F-Label in
      the protocol stack on the final hop.

   o  The necessary data-link encapsulation is then applied prior to
      transmission over the physical media.

        DetNet MPLS-based encapsulation

      +---------------------------------+
      |                                 |
      |         DetNet App-Flow         |
      |         Payload  Packet         |
      |                                 |
      +---------------------------------+ <--\
      |       DetNet Control Word       |    |
      +---------------------------------+    +--> DetNet data plane
      |           S-Label               |    |    MPLS encapsulation
      +---------------------------------+    |
      |           F-Label(s)            |    |
      +---------------------------------+ <--/
      |           Data-Link             |
      +---------------------------------+
      |           Physical              |
      +---------------------------------+

     Figure 4: Encapsulation of a DetNet App-Flow in an MPLS(-TP) PSN

5.2.1.  DetNet Control Word and the DetNet Sequence Number

   A DetNet control word (d-CW) conforms to the Generic PW MPLS Control
   Word (PWMCW) defined in [RFC4385].  The d-CW formatted as shown in
   Figure 5 MUST be present in all DetNet packets containing app-flow
   data.

Varga, et al.           Expires November 6, 2019               [Page 10]
Internet-Draft                 DetNet MPLS                      May 2019

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |0 0 0 0|                Sequence Number                        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                       Figure 5: DetNet Control Word

   (bits 0 to 3)

      Per [RFC4385], MUST be set to zero (0).

   Sequence Number (bits 4 to 31)

      An unsigned value implementing the DetNet sequence number.

   A separate sequence number space MUST be maintained by the node that
   adds the d-CW for each DetNet app-flow.  The following sequence
   number field lengths MUST be supported:

      0 bits

      16 bits

      28 bits

   The sequence number length MUST be provisioned (configured) on a per
   app-flow basis via configuration, e.g., the controller plane
   described in [I-D.ietf-detnet-data-plane-framework].

   A 0 bit sequence number field length indicates that there is no
   DetNet sequence number used for the flow.  When the length is zero,
   the sequence number field MUST be set to zero (0) on all packets sent
   for the flow.

   When the sequence number field length is 16 or 28 bits for a flow,
   the sequence number MUST be incremented by one for each new app-flow
   packet sent.  When the field length is 16 bits, d-CW bits 4 to 15
   MUST be set to zero (0).  This values carried in this field can wrap
   and it is important to note that zero (0) is a valid field value.
   For example, were the sequence number size is 16 bits, the sequence
   will contain: 65535, 0, 1.  In this case, zero (0) is an ordinary
   sequence number.  This differs from [RFC4448] where a sequence number
   of zero (0) does not indicate that no sequence number field value is
   in use.

Varga, et al.           Expires November 6, 2019               [Page 11]
Internet-Draft                 DetNet MPLS                      May 2019

   The sequence number is optionally used during receive processing as
   described below in Section 5.2.2.1 and Section 5.2.2.2.

5.2.2.  S-Labels

   App-flow identification at a DetNet service sub-layer is realized by
   an S-Label.  Each app-flow MUST be sent by the node that adds a d-CW
   with a single specific S-Label value.  MPLS-aware DetNet end systems
   and edge nodes, which are by definition MPLS ingress and egress
   nodes, MUST add and remove the d-CW and S-Label.  Relay nodes MAY
   swap S-Label values when processing an app-flow.

   The S-Label value MUST be provisioned per app-flow via configuration,
   e.g., via the controller plane described in
   [I-D.ietf-detnet-data-plane-framework].  Note that S-Labels provide
   app-flow identification at the downstream DetNet service sub-layer
   receiver, not the sender.  As such, S-Labels MUST be allocated by the
   entity that controls the service sub-layer receiving node's label
   space, and MAY be allocated from the platform label space [RFC3031].

   The S-Label will normally be at the bottom of the label stack once
   the last F-Label is removed, immediately preceding the d-CW.  To
   support service sub-layer level OAM, an OAM Associated Channel Header
   (ACH) [RFC4385] together with a Generic Associated Channel Label
   (GAL) [RFC5586] MAY be used in place of a d-CW.

   Similarly, an Entropy Label Indicator/Entropy Label (ELI/EL)
   [RFC6790] MAY be carried below the S-Label in the label stack in
   networks where DetNet flows would otherwise received ECMP treatment.
   When ELs are used, the same EL value SHOULD be used for all of the
   packets sent using a specific S-Label to force the flow to follow the
   same path.  However, as outlines in
   [I-D.ietf-detnet-data-plane-framework] the use of ECMP for DetNet
   flows is NOT RECOMMENDED.  ECMP MAY be used for non-DetNet flows
   within a DetNet domain.

   When receiving a DetNet MPLS flow, an implementation MUST identify
   the app-flow associated with the incoming packet based on the
   S-Label.  When a node is using platform labels for S-Labels, no
   additional information is needed as the S-label uniquely identifies
   the app-flow.  In the case where platform labels are not used, one or
   more F-Labels proceeding the S-Label MUST be used together with the
   S-Label to uniquely identify the incoming app-flows.  When PHP is
   used, the incoming interface MAY also be used to together with any
   present F-Label(s) and the S-Label to uniquely identify an incoming
   app-flows.  Note that choice to use platform label space for S-Label
   or S-Label plus one or more F-Labels to identify app flows is a local
   implementation choice, with one caveat.  When one or more F-labels,

Varga, et al.           Expires November 6, 2019               [Page 12]
Internet-Draft                 DetNet MPLS                      May 2019

   or incoming interface, is needed together with an S-Label to uniquely
   identify, the controller plane MUST ensure that incoming DetNet MPLS
   packets arrive with the needed information (F-label(s) and/or
   incoming interface); the details of such are outside the scope of
   this document.

   The use of platform labels for S-Labels matches other pseudowire
   encapsulations for consistency but there is no hard requirement in
   this regard.

5.2.2.1.  Packet Elimination Function Processing

   Implementations MAY support the Packet Elimination Function (PEF) for
   received DetNet MPLS flows.  When supported, use of the PEF for a
   particular app-flow MUST be provisioned via configuration, e.g., via
   the controller plane described in
   [I-D.ietf-detnet-data-plane-framework].

   After an app-flow is identified for a received DetNet MPLS packet, as
   described above, an implementation MUST check if PEF is configured
   for that app-flow.  When configured the implementation MUST track the
   sequence number contained in received d-CWs and MUST ensure that
   duplicate (replicated) instances of a particular sequence number are
   discarded.  The specific mechanisms used for an implementation to
   identify which received packets are duplicates and which are new is
   an implementation choice.  Note that per Section 5.2.1 the sequence
   number field length may be 16 or 28 bits, and the field value can
   wrap.

   Note that an implementation MAY wish to constrain the maximum number
   sequence numbers that are tracked, on platform-wide or per flow
   basis.  Some implementations MAY support the provisioning of the
   maximum number sequence numbers that are tracked number on either a
   platform-wide or per flow basis.

5.2.2.2.  Packet Ordering Function Processing

   A function that is related to PEF is the Packet Ordering Function
   (POF).  Implementations MAY support POF.  When supported, use of the
   POF for a particular app-flow MUST be provisioned via configuration,
   e.g., via the controller plane described by
   [I-D.ietf-detnet-data-plane-framework].  Implementations MAY required
   that PEF and POF be used in combination.  There is no requirement
   related to the order of execution of the Packet Elimination and
   Ordering Functions in an implementation.

   After an app-flow is identified for a received DetNet MPLS packet, as
   described above, an implementation MUST check if POF is configured

Varga, et al.           Expires November 6, 2019               [Page 13]
Internet-Draft                 DetNet MPLS                      May 2019

   for that app-flow.  When configured the implementation MUST track the
   sequence number contained in received d-CWs and MUST ensure that
   packets are processed in the order indicated in the received d-CW
   sequence number field, which may not be in the order the packets are
   received.  As defined in Section 5.2.1 the sequence number field
   length may be 16 or 28 bits, is incremented by one (1) for each new
   app-flow packet sent, and the field value can wrap.  The specific
   mechanisms used for an implementation to identify the order of
   received packets is an implementation choice.

   Note that an implementation MAY wish to constrain the maximum number
   of out of order packets that can be processed, on platform-wide or
   per flow basis.  Some implementations MAY support the provisioning of
   this number on either a platform-wide or per flow basis.  The number
   of out of order packets that can be processed also impacts the
   latency of a flow.

5.2.3.  F-Labels

   F-Labels are supported the DetNet forwarding sub-layer.  F-Labels are
   used to provide LSP-based connectivity between DetNet service sub-
   layer processing nodes.

5.2.3.1.  Service Sub-Layer and Packet Replication Function Processing

   DetNet MPLS end systems, edge nodes and relay nodes may operate at
   the DetNet service sub-layer with understand of app-flows and their
   requirements.  As mentioned earlier, when operating at this layer
   such nodes can push, pop or swap (pop then push) S-Labels.  In all
   cases, the F-Labels used for the app-flow are always replaced and the
   following procedures apply.

   When sending a DetNet flow, Zero or more F-Labels MAY be added on top
   of an S-Label by the node pushing an S-Label.  The F-Labels to be
   pushed when sending a particular app-flow MUST be provisioned per
   app-flow via configuration, e.g., via the controller plane discussed
   in [I-D.ietf-detnet-data-plane-framework].  To allow for the omission
   of F-Labels, an implementation SHOULD also allow an outgoing
   interface to be provisioned.

   The Packet Replication Function (PRF) function MAY be supported by an
   implementation for outgoing DetNet flows.  When replication is
   supported, the same app-flow data will be sent over multiple outgoing
   forwarding sub-layer LSPs.  To support PRF an implementation MUST
   support the setting of different sets of F-Labels.  Therefore, to
   allow for the omission of F-Labels, an implementation SHOULD also
   allow multiple outgoing interfaces to be provisioned.  PRF MUST NOT

Varga, et al.           Expires November 6, 2019               [Page 14]
Internet-Draft                 DetNet MPLS                      May 2019

   be used with app-flows configured with a d-CW sequence number field
   length of 0 bits.

   When a single set of F-Labels is provisioned for a particular
   outgoing app-flow, that set of F-labels MUST be pushed after the
   S-Label is pushed.  The outgoing packet is then forwarded as
   described below in Section 5.2.3.2.  When a single outgoing interface
   is provisioned, the outgoing packet is then forwarded as described
   below in Section 5.2.3.2.

   When multiple sets of F-Labels or interfaces are provisioned for a
   particular outgoing app-flow, a copy of the outgoing packet,
   including the pushed S-Label, MUST be made per F-label set and
   outgoing interface.  Each set of provisioned F-Labels are then pushed
   onto a copy of the packet.  Each copy is then forwarded as described
   below in Section 5.2.3.2.

   As described in the previous section, when receiving a DetNet MPLS
   flow, an implementation identifies the app-flow associated with the
   incoming packet based on the S-Label.  When a node is using platform
   labels for S-Labels, any F-Labels can be popped and the S-label
   uniquely identifies the app-flow.  In the case where platform labels
   are not used, F-Label(s) MUST also be provisioned for incoming app-
   flows.  When PHP is used, incoming interface MUST be provisioned.
   The provisioned information MUST then be used to identify incoming
   app-flows based on the combination of S-Label and F-Label(s) or
   incoming interface.

5.2.3.2.  Common F-Label Processing

   All DetNet aware MPLS nodes process F-Labels as needed to meet the
   service requirements of the DetNet flow or flows carried in the LSPs
   represented by the F-Labels.  This includes normal push, pop and swap
   operations.  Such processing is essentially the same type of
   processing enabled for TE LSPs, although the specific service
   parameters, or traffic specification, can differ.  When the DetNet
   service parameters of the app-flow or flows carried in an LSP
   represented by an F-Label can be met by an exiting TE mechanism, the
   forwarding sub-layer processing node MAY be a DetNet unaware, i.e.,
   standard, MPLS LSR.  Such TE LSPs may provide LSP forwarding service
   as defined in, but not limited to, [RFC3209], [RFC3270], [RFC3272],
   [RFC3473], [RFC4875], [RFC5440], and [RFC6006].

   More specifically, as mentioned above, the DetNet forwarding sub-
   layer provides explicit routes and allocated resources, and F-Labels
   are used to map to each.  Explicit routes are supported based on the
   topmost (outermost) F-Label that is pushed or swapped and the LSP
   that corresponds to this label.  This topmost (outgoing) label MUST

Varga, et al.           Expires November 6, 2019               [Page 15]
Internet-Draft                 DetNet MPLS                      May 2019

   be associated with a provisioned outgoing interface and, for non-
   point-to-point outgoing interfaces, a next hop LSR.  Note that this
   information MUST be provisioned via configuration or the controller
   plane.  In the previously mentioned special case where there is no
   added F-labels and the outgoing interface is not a point-to-point
   interface, the outgoing interface MUST also be associated with a next
   hop LSR.

   Resources may be allocated in a hierarchical fashion per LSP that is
   represented by each F-Label.  Each LSP MAY be provisioned with a
   service parameters that dictates the specific traffic treatment to be
   received by the traffic carried over that LSP.  Implementations of
   this document MUST ensure that traffic carried over each LSP
   represented by an F-Label receives the traffic treatment provisioned
   for that LSP.  Typical mechanisms used to provide different treatment
   to different flows includes the allocation of system resources (such
   as queues and buffers) and provisioning or related parameters (such
   as shaping, and policing).  Support can also be provided via an
   underlying network technology such IEEE802.1 TSN
   [I-D.ietf-detnet-mpls-over-tsn].  Other than in the TSN case, the
   specific mechanisms used by a DetNet node to ensure DetNet service
   delivery requirements are met for supported DetNet flows is outside
   the scope of this document.

   Packets that are marked in a way that does not correspond to
   allocated resources, e.g., lack a provisioned F-Label, can disrupt
   the QoS offered to properly reserved DetNet flows by using resources
   allocated to the reserved flows.  Therefore, the network nodes of a
   DetNet network:

   o  MUST defend the DetNet QoS by discarding or remarking (to an
      allocated DetNet flow or non-competing non-DetNet flow) packets
      received that are not associated with a completed resource
      allocation.

   o  MUST NOT use a DetNet allocated resource, e.g. a queue or shaper
      reserved for DetNet flows, for any packet that does match the
      corresponding DetNet flow.

   o  MUST ensure a QoS flow does not exceed its allocated resources or
      provisioned service level,

   o  MUST ensure a CoS flow or service class does not impact the
      service delivered to other flows.  This requirement is similar to
      requirement for MPLS LSRs to that CoS LSPs do not impact the
      resources allocated to TE LSPs, e.g., via [RFC3473].

Varga, et al.           Expires November 6, 2019               [Page 16]
Internet-Draft                 DetNet MPLS                      May 2019

   Subsequent sections provide additional considerations related to CoS
   (Section 5.6.1), QoS (Section 5.6.2) and aggregation (Section 5.4).

5.3.  OAM Indication

   OAM follows the procedures set out in [RFC5085] with the restriction
   that only Virtual Circuit Connectivity Verification (VCCV) type 1 is
   supported.

   As shown in Figure 3 of [RFC5085] when the first nibble of the d-CW
   is 0x0 the payload following the d-CW is normal user data.  However,
   when the first nibble of the d-CW is 0X1, the payload that follows
   the d-DW is an OAM payload with the OAM type indicated by the value
   in the d-CW Channel Type field.

   The reader is referred to [RFC5085] for a more detailed description
   of the Associated Channel mechanism, and to the DetNet work on OAM
   for more information DetNet OAM.

5.4.  Flow Aggregation

   [Author's note: need to revisit this section and ensure that we cover
   (and fully specify) desired types of aggregation.]

   1.  Aggregate at the LSP (Forwarding)

   2.  Aggregating DetNet flows as a new DetNet flow

   3.  Simple Aggregation at the DetNet layer

   The resource control and management aspects of aggregation (including
   the queuing/shaping/ policing implications) will be covered in other
   documents.

   The ability to aggregate individual flows, and their associated
   resource control, into a larger aggregate is an important technique
   for improving scaling of control in the data, management and control
   planes.  The DetNet data plane allows for the aggregation of DetNet
   flows, to improved scaling.  There are three methods of introducing
   flow aggregation:

5.4.1.  Aggregation at the LSP

   DetNet flows forwarded via MPLS can leverage MPLS-TE's existing
   support for hierarchical LSPs (H-LSPs), see [RFC4206].  H-LSPs are
   typically used to aggregate control and resources, they may also be
   used to provide OAM or protection for the aggregated LSPs.  Arbitrary
   levels of aggregation naturally falls out of the definition for

Varga, et al.           Expires November 6, 2019               [Page 17]
Internet-Draft                 DetNet MPLS                      May 2019

   hierarchy and the MPLS label stack [RFC3032].  DetNet nodes which
   support aggregation (LSP hierarchy) map one or more LSPs (labels)
   into and from an H-LSP.  Both carried LSPs and H-LSPs may or may not
   use the TC field, i.e., L-LSPs or E-LSPs.  Such nodes will need to
   ensure that traffic from aggregated LSPs are placed (shaped/policed/
   enqueued) onto the H-LSPs in a fashion that ensures the required
   DetNet service is preserved.

   Additional details of the traffic control capabilities needed at a
   DetNet-aware node may be covered in the new service descriptions
   mentioned above or in separate future documents.  Controller plane
   mechanisms will also need to ensure that the service required on the
   aggregate flow (H-LSP or DSCP) are provided, which may include the
   discarding or remarking mentioned in the previous sections.

5.4.2.  Aggregating DetNet Flows as a new DetNet flow

   An aggregate can be built by layering DetNet flows as shown below:

   +---------------------------------+
   |                                 |
   |           DetNet Flow           |
   |         Payload  Packet         |
   |                                 |
   +---------------------------------+ <--\
   |       DetNet Control Word       |    |
   +=================================+    |
   |            S-Label              |    |
   +---------------------------------+    +----DetNet data plane
   |       DetNet Control Word       |    |    MPLS encapsulation
   +=================================+    |
   |            A-Label              |    |
   +---------------------------------+    |
   |           F-Label(s)            | <--/
   +---------------------------------+
   |           Data-Link             |
   +---------------------------------+
   |           Physical              |
   +---------------------------------+

             Figure 6: DetNet Aggregation as a new DetNet Flow

   Both the Aggregation (A) label and the S-Label have their MPLS S bit
   set indicating bottom of stack, and the d-CW allows the PREOF to
   work.

   It is a property of the A-label that what follows is d-CW followed by
   an S-Label.  A relay node processing the A-label would not know the

Varga, et al.           Expires November 6, 2019               [Page 18]
Internet-Draft                 DetNet MPLS                      May 2019

   underlying payload type.  This would only be known to a node that was
   a peer of the node imposing the S-Label.  However there is no real
   need for it to know the payload type during aggregation processing.

5.4.3.  Simple Aggregation at the DetNet Layer

   Another approach would be not to include a d-CW for the aggregated
   flow.  This would be functionally similar to aggregation at the
   forwarding sub-layer using H-LSPs, but would confine knowledge of the
   aggregation to the DetNet layer.  Such an approach shares the
   disadvantage that PREOF operations would not be possible.  OAM
   operation in this mode is for further study.

    +---------------------------------+
    |                                 |
    |           DetNet Flow           |
    |         Payload  Packet         |
    |                                 |
    +---------------------------------+ <--\
    |       DetNet Control Word       |    |
    +=================================+    |
    |            S-Label              |    +----DetNet data plane
    +---------------------------------+    |    MPLS encapsulation
    |            A-Label              |    |
    +---------------------------------+    |
    |           F-Label(s)            | <--/
    +---------------------------------+
    |           Data-Link             |
    +---------------------------------+
    |           Physical              |
    +---------------------------------+

                    Figure 7: Simple DetNet Aggregation

5.5.  Service Sub-Layer Considerations

   The edge and relay node internal procedures related to PREOF are
   implementation specific.  The order of a packet elimination or
   replication is out of scope in this specification.

   It is important that the DetNet layer is configured such that a
   DetNet node never receives its own replicated packets.  If it were to
   receive such packets the replication function would make the loop
   more destructive of bandwidth than a conventional unicast loop.
   Ultimately the TTL in the S-Label will cause the packet to die during
   a transient loop, but given the sensitivity of applications to packet
   latency the impact on the DetNet application would be severe.

Varga, et al.           Expires November 6, 2019               [Page 19]
Internet-Draft                 DetNet MPLS                      May 2019

5.5.1.  Edge Node Processing

   An edge node is responsible for matching ingress packets to the
   service they require and encapsulating them accordingly.  An edge
   node may participate in the packet replication and duplicate packet
   elimination.

   The DetNet-aware forwarder selects the egress DetNet member flow
   segment based on the flow identification.  The mapping of ingress
   DetNet member flow segment to egress DetNet member flow segment may
   be statically or dynamically configured.  Additionally the DetNet-
   aware forwarder does duplicate frame elimination based on the flow
   identification and the sequence number combination.  The packet
   replication is also done within the DetNet-aware forwarder.  During
   elimination and the replication process the sequence number of the
   DetNet member flow MUST be preserved and copied to the egress DetNet
   member flow.

   The internal design of a relay node is out of scope of this document.
   However the reader's attention is drawn to the need to make any PREOF
   state available to the packet processor(s) dealing with packets to
   which the PREOF functions must be applied, and to maintain that state
   is such as way that it is available to the packet processor operation
   on the next packet in the DetNet flow (which may be a duplicate, a
   late packet, or the next packet in sequence.

5.5.2.  Relay Node Processing

   A DetNet Relay node operates in the DetNet forwarding sub-layer .
   For DetNet using MPLS this processing is performed on the F-Label.
   This processing is done within an extended forwarder function.
   Whether an ingress DetNet member flow receives DetNet specific
   processing depends on how the forwarding is programmed.  Some relay
   nodes may be DetNet service aware, while others may be unmodified
   LSRs that only understand how to swicth MPLS-TE LSPs.

   It is also possible to treat the relay node as a transit node, see
   Section 5.4.  Again, this is entirely up to how the forwarding has
   been programmed.

5.6.  Forwarding Sub-Layer Considerations

5.6.1.  Class of Service

   Class and quality of service, i.e., CoS and QoS, are terms that are
   often used interchangeably and confused with each other.  In the
   context of DetNet, CoS is used to refer to mechanisms that provide
   traffic forwarding treatment based on aggregate group basis and QoS

Varga, et al.           Expires November 6, 2019               [Page 20]
Internet-Draft                 DetNet MPLS                      May 2019

   is used to refer to mechanisms that provide traffic forwarding
   treatment based on a specific DetNet flow basis.  Examples of
   existing network level CoS mechanisms include DiffServ which is
   enabled by IP header differentiated services code point (DSCP) field
   [RFC2474] and MPLS label traffic class field [RFC5462], and at Layer-
   2, by IEEE 802.1p priority code point (PCP).

   CoS for DetNet flows carried in PWs and MPLS is provided using the
   existing MPLS Differentiated Services (DiffServ) architecture
   [RFC3270].  Both E-LSP and L-LSP MPLS DiffServ modes MAY be used to
   support DetNet flows.  The Traffic Class field (formerly the EXP
   field) of an MPLS label follows the definition of [RFC5462] and
   [RFC3270].  The Uniform, Pipe, and Short Pipe DiffServ tunneling and
   TTL processing models are described in [RFC3270] and [RFC3443] and
   MAY be used for MPLS LSPs supporting DetNet flows.  MPLS ECN MAY also
   be used as defined in ECN [RFC5129] and updated by [RFC5462].

5.6.2.  Quality of Service

   In addition to explicit routes, and packet replication and
   elimination, described in Section 5 above, DetNet provides zero
   congestion loss and bounded latency and jitter.  As described in
   [I-D.ietf-detnet-architecture], there are different mechanisms that
   maybe used separately or in combination to deliver a zero congestion
   loss service.  This includes Quality of Service (QoS) mechanisms at
   the MPLS layer, that may be combined with the mechanisms defined by
   the underlying network layer such as 802.1TSN.

   Quality of Service (QoS) mechanisms for flow specific traffic
   treatment typically includes a guarantee/agreement for the service,
   and allocation of resources to support the service.  Example QoS
   mechanisms include discrete resource allocation, admission control,
   flow identification and isolation, and sometimes path control,
   traffic protection, shaping, policing and remarking.  Example
   protocols that support QoS control include Resource ReSerVation
   Protocol (RSVP) [RFC2205] (RSVP) and RSVP-TE [RFC3209] and [RFC3473].
   The existing MPLS mechanisms defined to support CoS [RFC3270] can
   also be used to reserve resources for specific traffic classes.

   A baseline set of QoS capabilities for DetNet flows carried in PWs
   and MPLS can provided by MPLS with Traffic Engineering (MPLS-TE)
   [RFC3209] and [RFC3473].  TE LSPs can also support explicit routes
   (path pinning).  Current service definitions for packet TE LSPs can
   be found in "Specification of the Controlled Load Quality of
   Service", [RFC2211], "Specification of Guaranteed Quality of
   Service", [RFC2212], and "Ethernet Traffic Parameters", [RFC6003].
   Additional service definitions are expected in future documents to
   support the full range of DetNet services.  In all cases, the

Varga, et al.           Expires November 6, 2019               [Page 21]
Internet-Draft                 DetNet MPLS                      May 2019

   existing label-based marking mechanisms defined for TE-LSPs and even
   E-LSPs are use to support the identification of flows requiring
   DetNet QoS.

6.  Flow Identification Management and Control Information

   [Editor's Note] The following will summarize the set of information
   that is needed to support the MPLS DetNet data plane.

7.  Security Considerations

   The security considerations of DetNet in general are discussed in
   [I-D.ietf-detnet-architecture] and [I-D.sdt-detnet-security].  Other
   security considerations will be added in a future version of this
   draft.

8.  IANA Considerations

   This document makes no IANA requests.

9.  Contributors

   RFC7322 limits the number of authors listed on the front page of a
   draft to a maximum of 5, far fewer than the many individuals below
   who made important contributions to this draft.  The editor wishes to
   thank and acknowledge each of the following authors for contributing
   text to this draft.  See also Section 10.

      Loa Andersson
      Huawei
      Email: loa@pi.nu

      Yuanlong Jiang
      Huawei
      Email: jiangyuanlong@huawei.com

      Norman Finn
      Huawei
      3101 Rio Way
      Spring Valley, CA  91977
      USA
      Email: norman.finn@mail01.huawei.com

      Janos Farkas
      Ericsson
      Magyar Tudosok krt. 11.
      Budapest  1117
      Hungary

Varga, et al.           Expires November 6, 2019               [Page 22]
Internet-Draft                 DetNet MPLS                      May 2019

      Email: janos.farkas@ericsson.com

      Carlos J. Bernardos
      Universidad Carlos III de Madrid
      Av. Universidad, 30
      Leganes, Madrid  28911
      Spain
      Email: cjbc@it.uc3m.es

      Tal Mizrahi
      Marvell
      6 Hamada st.
      Yokneam
      Israel
      Email: talmi@marvell.com

      Lou Berger
      LabN Consulting, L.L.C.
      Email: lberger@labn.net

      Stewart Bryant
      Huawei Technologies
      Email: stewart.bryant@gmail.com

      Mach Chen
      Huawei Technologies
      Email: mach.chen@huawei.com

      Andrew G. Malis
      Huawei Technologies
      Email: agmalis@gmail.com

      Don Fedyk
      LabN Consulting, L.L.C.
      Email: dfedyk@labn.net

10.  Acknowledgements

   The author(s) ACK and NACK.

   The following people were part of the DetNet Data Plane Solution
   Design Team:

      Jouni Korhonen

      Janos Farkas

      Norman Finn

Varga, et al.           Expires November 6, 2019               [Page 23]
Internet-Draft                 DetNet MPLS                      May 2019

      Balazs Varga

      Loa Andersson

      Tal Mizrahi

      David Mozes

      Yuanlong Jiang

      Andrew Malis

      Carlos J.  Bernardos

   The DetNet chairs serving during the DetNet Data Plane Solution
   Design Team:

      Lou Berger

      Pat Thaler

   Thanks for Stewart Bryant for his extensive review of the previous
   versions of the document.

11.  References

11.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC2211]  Wroclawski, J., "Specification of the Controlled-Load
              Network Element Service", RFC 2211, DOI 10.17487/RFC2211,
              September 1997, <https://www.rfc-editor.org/info/rfc2211>.

   [RFC2212]  Shenker, S., Partridge, C., and R. Guerin, "Specification
              of Guaranteed Quality of Service", RFC 2212,
              DOI 10.17487/RFC2212, September 1997,
              <https://www.rfc-editor.org/info/rfc2212>.

   [RFC3031]  Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol
              Label Switching Architecture", RFC 3031,
              DOI 10.17487/RFC3031, January 2001,
              <https://www.rfc-editor.org/info/rfc3031>.

Varga, et al.           Expires November 6, 2019               [Page 24]
Internet-Draft                 DetNet MPLS                      May 2019

   [RFC3032]  Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y.,
              Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack
              Encoding", RFC 3032, DOI 10.17487/RFC3032, January 2001,
              <https://www.rfc-editor.org/info/rfc3032>.

   [RFC3209]  Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V.,
              and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP
              Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001,
              <https://www.rfc-editor.org/info/rfc3209>.

   [RFC3270]  Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen,
              P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi-
              Protocol Label Switching (MPLS) Support of Differentiated
              Services", RFC 3270, DOI 10.17487/RFC3270, May 2002,
              <https://www.rfc-editor.org/info/rfc3270>.

   [RFC3443]  Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing
              in Multi-Protocol Label Switching (MPLS) Networks",
              RFC 3443, DOI 10.17487/RFC3443, January 2003,
              <https://www.rfc-editor.org/info/rfc3443>.

   [RFC3473]  Berger, L., Ed., "Generalized Multi-Protocol Label
              Switching (GMPLS) Signaling Resource ReserVation Protocol-
              Traffic Engineering (RSVP-TE) Extensions", RFC 3473,
              DOI 10.17487/RFC3473, January 2003,
              <https://www.rfc-editor.org/info/rfc3473>.

   [RFC4206]  Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP)
              Hierarchy with Generalized Multi-Protocol Label Switching
              (GMPLS) Traffic Engineering (TE)", RFC 4206,
              DOI 10.17487/RFC4206, October 2005,
              <https://www.rfc-editor.org/info/rfc4206>.

   [RFC4385]  Bryant, S., Swallow, G., Martini, L., and D. McPherson,
              "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for
              Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385,
              February 2006, <https://www.rfc-editor.org/info/rfc4385>.

   [RFC5085]  Nadeau, T., Ed. and C. Pignataro, Ed., "Pseudowire Virtual
              Circuit Connectivity Verification (VCCV): A Control
              Channel for Pseudowires", RFC 5085, DOI 10.17487/RFC5085,
              December 2007, <https://www.rfc-editor.org/info/rfc5085>.

   [RFC5129]  Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion
              Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January
              2008, <https://www.rfc-editor.org/info/rfc5129>.

Varga, et al.           Expires November 6, 2019               [Page 25]
Internet-Draft                 DetNet MPLS                      May 2019

   [RFC5462]  Andersson, L. and R. Asati, "Multiprotocol Label Switching
              (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic
              Class" Field", RFC 5462, DOI 10.17487/RFC5462, February
              2009, <https://www.rfc-editor.org/info/rfc5462>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

11.2.  Informative References

   [I-D.ietf-detnet-architecture]
              Finn, N., Thubert, P., Varga, B., and J. Farkas,
              "Deterministic Networking Architecture", draft-ietf-
              detnet-architecture-12 (work in progress), March 2019.

   [I-D.ietf-detnet-data-plane-framework]
              Korhonen, J., Varga, B., "DetNet Data Plane Framework",
              2019.

   [I-D.ietf-detnet-mpls-over-tsn]
              Korhonen, J., Varga, B., "DetNet MPLS over IEEE 802.1 TSN
              Sub-Networks", 2019.

   [I-D.ietf-spring-segment-routing-mpls]
              Bashandy, A., Filsfils, C., Previdi, S., Decraene, B.,
              Litkowski, S., and R. Shakir, "Segment Routing with MPLS
              data plane", draft-ietf-spring-segment-routing-mpls-22
              (work in progress), May 2019.

   [I-D.sdt-detnet-security]
              Mizrahi, T., Grossman, E., Hacker, A., Das, S.,
              "Deterministic Networking (DetNet) Security
              Considerations, draft-sdt-detnet-security, work in
              progress", 2017.

   [RFC2205]  Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S.
              Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1
              Functional Specification", RFC 2205, DOI 10.17487/RFC2205,
              September 1997, <https://www.rfc-editor.org/info/rfc2205>.

   [RFC2474]  Nichols, K., Blake, S., Baker, F., and D. Black,
              "Definition of the Differentiated Services Field (DS
              Field) in the IPv4 and IPv6 Headers", RFC 2474,
              DOI 10.17487/RFC2474, December 1998,
              <https://www.rfc-editor.org/info/rfc2474>.

Varga, et al.           Expires November 6, 2019               [Page 26]
Internet-Draft                 DetNet MPLS                      May 2019

   [RFC3272]  Awduche, D., Chiu, A., Elwalid, A., Widjaja, I., and X.
              Xiao, "Overview and Principles of Internet Traffic
              Engineering", RFC 3272, DOI 10.17487/RFC3272, May 2002,
              <https://www.rfc-editor.org/info/rfc3272>.

   [RFC3985]  Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation
              Edge-to-Edge (PWE3) Architecture", RFC 3985,
              DOI 10.17487/RFC3985, March 2005,
              <https://www.rfc-editor.org/info/rfc3985>.

   [RFC4448]  Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron,
              "Encapsulation Methods for Transport of Ethernet over MPLS
              Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006,
              <https://www.rfc-editor.org/info/rfc4448>.

   [RFC4875]  Aggarwal, R., Ed., Papadimitriou, D., Ed., and S.
              Yasukawa, Ed., "Extensions to Resource Reservation
              Protocol - Traffic Engineering (RSVP-TE) for Point-to-
              Multipoint TE Label Switched Paths (LSPs)", RFC 4875,
              DOI 10.17487/RFC4875, May 2007,
              <https://www.rfc-editor.org/info/rfc4875>.

   [RFC5440]  Vasseur, JP., Ed. and JL. Le Roux, Ed., "Path Computation
              Element (PCE) Communication Protocol (PCEP)", RFC 5440,
              DOI 10.17487/RFC5440, March 2009,
              <https://www.rfc-editor.org/info/rfc5440>.

   [RFC5586]  Bocci, M., Ed., Vigoureux, M., Ed., and S. Bryant, Ed.,
              "MPLS Generic Associated Channel", RFC 5586,
              DOI 10.17487/RFC5586, June 2009,
              <https://www.rfc-editor.org/info/rfc5586>.

   [RFC5921]  Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau,
              L., and L. Berger, "A Framework for MPLS in Transport
              Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010,
              <https://www.rfc-editor.org/info/rfc5921>.

   [RFC6003]  Papadimitriou, D., "Ethernet Traffic Parameters",
              RFC 6003, DOI 10.17487/RFC6003, October 2010,
              <https://www.rfc-editor.org/info/rfc6003>.

   [RFC6006]  Zhao, Q., Ed., King, D., Ed., Verhaeghe, F., Takeda, T.,
              Ali, Z., and J. Meuric, "Extensions to the Path
              Computation Element Communication Protocol (PCEP) for
              Point-to-Multipoint Traffic Engineering Label Switched
              Paths", RFC 6006, DOI 10.17487/RFC6006, September 2010,
              <https://www.rfc-editor.org/info/rfc6006>.

Varga, et al.           Expires November 6, 2019               [Page 27]
Internet-Draft                 DetNet MPLS                      May 2019

   [RFC6073]  Martini, L., Metz, C., Nadeau, T., Bocci, M., and M.
              Aissaoui, "Segmented Pseudowire", RFC 6073,
              DOI 10.17487/RFC6073, January 2011,
              <https://www.rfc-editor.org/info/rfc6073>.

   [RFC6790]  Kompella, K., Drake, J., Amante, S., Henderickx, W., and
              L. Yong, "The Use of Entropy Labels in MPLS Forwarding",
              RFC 6790, DOI 10.17487/RFC6790, November 2012,
              <https://www.rfc-editor.org/info/rfc6790>.

Authors' Addresses

   Balazs Varga (editor)
   Ericsson
   Magyar Tudosok krt. 11.
   Budapest  1117
   Hungary

   Email: balazs.a.varga@ericsson.com

   Janos Farkas
   Ericsson
   Magyar Tudosok krt. 11.
   Budapest  1117
   Hungary

   Email: janos.farkas@ericsson.com

   Lou Berger
   LabN Consulting, L.L.C.

   Email: lberger@labn.net

   Don Fedyk
   LabN Consulting, L.L.C.

   Email: dfedyk@labn.net

   Andrew G. Malis
   Huawei Technologies

   Email: agmalis@gmail.com

Varga, et al.           Expires November 6, 2019               [Page 28]
Internet-Draft                 DetNet MPLS                      May 2019

   Stewart Bryant
   Huawei Technologies

   Email: stewart.bryant@gmail.com

   Jouni Korhonen

   Email: jouni.nospam@gmail.com

Varga, et al.           Expires November 6, 2019               [Page 29]