DETNET                                                          Q. Xiong
Internet-Draft                                           ZTE Corporation
Intended status: Standards Track                            October 2021
Expires: 27 April 2022


       The Requirements for Wide-area IP Deterministic Networking
            draft-xiong-detnet-wide-area-ip-requirements-00

Abstract

   In wide-area IP networks, more requirements need to be taken into
   considerations such as differentiated DetNet QoS of multiple
   services, high-efficiency of resources utilization and routes
   steering, integration of large-scale heterogeneous network and
   guarantees of multiple dynamic deterministic flows.  This document
   describes the requirements in wide-area applications and proposes the
   solution with deterministic resources, routes and QoS.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 4 April 2022.

Copyright Notice

   Copyright (c) 2021 IETF Trust and the persons identified as the
   document authors.  All rights reserved.











Xiong                     Expires 27 April 2022                 [Page 1]


Internet-Draft  The Requirements for Wide-area IP Determ    October 2021


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Simplified BSD License text
   as described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Conventions used in this document . . . . . . . . . . . . . .   4
     2.1.  Terminology . . . . . . . . . . . . . . . . . . . . . . .   4
     2.2.  Requirements Language . . . . . . . . . . . . . . . . . .   4
   3.  Requirements for Wide-area IP Deterministic Networking  . . .   4
     3.1.  Differentiated DetNet QoS of Multiple Services  . . . . .   4
     3.2.  Integration of Large-scale Heterogeneous Network  . . . .   5
     3.3.  Efficiency of Resources Utilization and Routes
           Steering  . . . . . . . . . . . . . . . . . . . . . . . .   6
     3.4.  Guarantees of Multiple Dynamic Deterministic Flows  . . .   6
   4.  Solutions Considerations of Wide-area IP Deterministic
           Networking  . . . . . . . . . . . . . . . . . . . . . . .   7
     4.1.  The Deterministic Resources . . . . . . . . . . . . . . .   7
     4.2.  The Deterministic Routes  . . . . . . . . . . . . . . . .   7
     4.3.  The Deterministic QoS . . . . . . . . . . . . . . . . . .   8
   5.  Security Considerations . . . . . . . . . . . . . . . . . . .   8
   6.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   8
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   8
   8.  Normative References  . . . . . . . . . . . . . . . . . . . .   8
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .   9

1.  Introduction

   5G network is oriented to the internet of everything.  In addition to
   the Enhanced Mobile Broadband (eMBB) and Massive Machine Type
   Communications(mMTC) services, it also supports the Ultra-reliable
   Low Latency Communications (uRLLC) services.  The uRLLC services
   cover the industries such as intelligent electrical network,
   intelligent factory, internet of vehicles, industry automation and
   other industrial internet scenarios, which is the key demand of
   digital transformation of vertical domains.  These uRLLC services
   demand SLA guarantees such as low latency and high reliability and
   other deterministic and precise properties.

   For the intelligent electrical network, there are deterministic
   requirements for communication delay, jitter and packet loss rate.
   For example, in the electrical current difference model, a delay of



Xiong                     Expires 27 April 2022                 [Page 2]


Internet-Draft  The Requirements for Wide-area IP Determ    October 2021


   3~10ms and a jitter variation is no more than 100us are required.
   The isolation requirement is also important.  For example, the
   automatic operation, control of a process, isochronous data and low
   priority service need to meet the requirements of hard isolation.  In
   addition to the requirements of delay and jitter, the differential
   protection (DP) service needs to be isolated from other services.

   The industrial internet is the key infrastructure that coordinate
   various units of work over various system components, e.g. people,
   machines and things in the industrial environment including big data,
   cloud computing, Internet of Things (IOT), Augment Reality (AR),
   industrial robots, Artificial Intelligence (AI) and other basic
   technologies.  For example, automation control is one of the basic
   application and the the core is closed-loop control system.  The
   control process cycle is as low as millisecond level, so the system
   communication delay needs to reach millisecond level or even lower to
   ensure the realization of precise control.  There are three levels of
   real-time requirements for industrial interconnection: factory level
   is about 1s, and process level is 10~100ms, and the highest real-time
   requirement is motion control, which requires less than 1ms.

   The applications in 5G networks demand much more deterministic and
   precise properties.  But traditional Ethernet, IP and MPLS networks
   which is based on statistical multiplexing provides best-effort
   packet service and offers no delivery and SLA guarantee.  The
   deterministic forwarding can only apply to flows with such well-
   defined characteristics as periodicity and burstiness.  Technologies
   to provide deterministic service has been proposed to provide bounded
   latency and jitter based on a best-effort packet network.  IEEE 802.1
   Time-Sensitive Networking (TSN) has been proposed to provide bounded
   latency and jitter in L2 LAN networks.  According to [RFC8655],
   Deterministic Networking (DetNet) operates at the IP layer and
   delivers service which provides extremely low data loss rates and
   bounded latency within a network domain.

   The deterministic networks not only need to offer the Service Level
   Agreements (SLA) guarantees such as low latency and jitter, low
   packet loss and high reliability, but also need to support the
   precise services such as flexible resource allocation and service
   isolation.  However, under the existing IP network architecture with
   statistical multiplexing characteristics, the existing deterministic
   technologies are facing large scale number of nodes and long-distance
   transmission, traffic scheduling, dynamic flows, and other
   controversial issues especially in Wide Area Network (WAN)
   applications.






Xiong                     Expires 27 April 2022                 [Page 3]


Internet-Draft  The Requirements for Wide-area IP Determ    October 2021


   In wide-area IP networks, more requirements need to be taken into
   considerations such as differentiated DetNet QoS of multiple
   services, high-efficiency of resources utilization and routes
   steering, integration of large-scale heterogeneous network and
   guarantees of multiple dynamic deterministic flows.  This document
   describes the requirements in wide-area applications and proposes the
   solution with deterministic resources, routes and QoS.

2.  Conventions used in this document

2.1.  Terminology

   The terminology is defined as [RFC8655].

2.2.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

3.  Requirements for Wide-area IP Deterministic Networking

3.1.  Differentiated DetNet QoS of Multiple Services

   As defined in [RFC8655], the DetNet QoS can be expressed in terms of
   : Minimum and maximum end-to-end latency, bounded jitter (packet
   delay variation), packet loss ratio and an upper bound on out-of-
   order packet delivery.  As described in [RFC8578], DetNet
   applications differ in their network topologies and specific desired
   behavior and different services requires differentiated DetNet QoS.
   In the WAN scenarios, multiple services with differentiated DetNet
   QoS is co-existed in the same DetNet network.  The classification of
   the deterministic flows is should be taken into considerations.  It
   is required to provide Latency, bounded jitter and packet loss
   dynamically and flexibly in all scenarios for each characterizd flow.

   As the Figure 1 shown, the services is divided into 4 levels and
   level 1~3 is the DetNet flows and level-4 is non-DetNet flow.  DetNet
   applications and DetNet QoS is differentiated within each level.










Xiong                     Expires 27 April 2022                 [Page 4]


Internet-Draft  The Requirements for Wide-area IP Determ    October 2021


   +--------------+---------------+------------+-----------+-----------+
   | Item         | Level-1       | Level-2    | Level-3   | Level-4   |
   +--------------+---------------+------------+-----------+-----------+
   | Applications | Industrial    | VR/AR      | Audio and | Broadcast |
   | Examples     |               |            | Video     |           |
   +--------------+---------------+------------+-----------+-----------+
   | DetNet QoS   | Ultra-low     | Low        | Low       | Best      |
   |              | latency and   | latency    | latency   | Effort    |
   |              | jitter        | and jitter |           |           |
   +--------------+---------------+------------+-----------+-----------+


     Figure 1: Figure 1: The classification of multiple services


   Moreover, different DetNet services is required to tolerate different
   percentage of packet loss ratio such as 99.9%, 99.99%, 99.999%, and
   so on.  It is also required to provide service isolation.  In some
   scenarios, such as intelligent electrical network, the isolation
   requirements are very important.  For example, the automatic
   operation or control of a process or isochronous data and service
   with different priorities need to meet the requirements of hard
   isolation.  In addition to the requirements of delay and jitter, the
   differential protection (DP) service needs to be isolated from other
   services and hard isolated tunnel is required.

3.2.  Integration of Large-scale Heterogeneous Network

   In WAN application, large-scale number of nodes and long-distance
   transmission in the network will lead to latency and jitter, such as
   increasing transmission latency, jitter and packet loss.  It is to
   required reduce the scale of the network topology by establishing
   cutthrough channels.  The existing technologies such as FlexE and SR
   tunnels should be taken into consideration.  And multiple
   capabilities is also provided by the nodes and links within the
   network topology such as FlexE tunnels, TSN sub-network and IP/MPLS/
   SRv6 tunnels.  It is required to integrate the multi-capability
   resources to achieve the optimal DetNet QoS.

   Another option is to divide the network into several domains and
   segments.  And the deadline of latency and jitter of each domain and
   segment should be determined and controlled.  It is required to
   control the DetNet QoS at the inter-domain boundary nodes and achieve
   the end-to-end latency, bounded jitter and packet loss ratio across.







Xiong                     Expires 27 April 2022                 [Page 5]


Internet-Draft  The Requirements for Wide-area IP Determ    October 2021


3.3.  Efficiency of Resources Utilization and Routes Steering

   Traditional Ethernet, IP and MPLS networks which is based on
   statistical multiplexing provides best-effort packet service and
   offers no delivery and SLA guarantee.  As described in [RFC8655], the
   primary technique by which DetNet achieves its QoS is to allocate
   sufficient resources.  But it can not be achieved by not sufficient
   resource which can be allocated due to practical and cost reason.  So
   it is required to achieve the high-efficiency of resources
   utilization when provide the DetNet service.

   Network resources include nodes, links, ports, bandwidth, queues,
   etc.  The congestion control, shaping and queue scheduling and other
   traffic mechanisms which have been proposed in IEEE 802.1 TSN such as
   IEEE802.1Qbv, IEEE802.1Qch, IEEE802.1Qav, IEEE802.1Qcr and so on.
   Heterogeneous resource should be used and unified and simplified
   resources mechanism under the selection of existing multiple
   technical methods to realize the elastic of deterministic capability.

   Resource classification and modeling is required along with the
   explicit path with more SLA guarantee parameters like bandwidth,
   latency, jitter, packet loss and so on.  On the basic of the
   resources, the steering path and routes for deterministic flows
   should be programmed before the flows coming and able to provide SLA
   capability.  And the routes should be considered to be established in
   distributed and centralized control Plane.

3.4.  Guarantees of Multiple Dynamic Deterministic Flows

   As described in [RFC8557], deterministic forwarding can only apply to
   flows with such well-defined characteristics as periodicity and
   burstiness.  As defined in DetNet architecture [RFC8655], the traffic
   characteristics of an App-flow can be CBR (constant bit rate) or VBR
   (variable bit rate) of L1, L2 and L3 layers (VBR takes the maximum
   value when reserving resources).  But the current scenarios and
   technical solutions only consider CBR flow, without considering the
   coexistence of VBR and CBR, the burst and aperiodicity of flows.  The
   operations such as shaping or scheduling have not been specified.
   Even TSN mechanisms are based on a constant and forecastable traffic
   characteristics.

   It will be more complicated in WAN applications where much more flows
   coexist and the traffic characteristics is more dynamic.  It is
   required to offer reliable delivery and SLA guarantee for dynamic
   flows.  For example, periodic flow and aperiodic flow (including
   micro burst flow, etc.), CBR and VBR flow, flow with different
   periods or phases, etc.  When the network needs to forward these
   deterministic flows at the same time, it must solve the problems of



Xiong                     Expires 27 April 2022                 [Page 6]


Internet-Draft  The Requirements for Wide-area IP Determ    October 2021


   time window selection, queue processing and aggregation of multiple
   flows.  It is required to classify the dynamic deterministic flows
   and map them into different virtual topologies to limit the number of
   the concurrent flows and reduce the micro bursts.

4.  Solutions Considerations of Wide-area IP Deterministic Networking

4.1.  The Deterministic Resources

   As defined in RFC8655, the resource allocation is one of the
   techniques to achieve the DetNet QoS.  Network resources include
   nodes, links, ports, bandwidth, queues, etc.  The deterministic
   resources require planning and arrangement of network resources,
   resources modeling, resource allocation and reservation, resource
   isolation and resource scheduling, etc.  In order to meet the
   requirements of deterministic service, resources need to be
   classified, including ultra-low delay resources, low delay resources,
   low jitter resources, etc.

   Deterministic resources guarantee the delay, jitter and other
   requirements of deterministic services by reserving resources for
   flows.  If the network resources are sufficient, congestion and
   packet loss can be eliminated to meet the requirements of low delay
   jitter.  If the network resources are insufficient, congestion
   control, queue mechanisms of deterministic flows need to be carried
   out.  The nodes with different queue mechanisms provide different
   latency and bounded jitter.  Moreover, network resources could to be
   reconstructed to provide ultra-low latency, for example, L1 layer
   resources could be used to provide cutthrough channels, FlexE pipes,
   etc.

4.2.  The Deterministic Routes

   The deterministic routes is based on the provision of deterministic
   resources.  The deterministic routes refers to the requirements to
   select the network routes for the deterministic flows to guarantee
   the stability of the routing at least during the packets
   transmission, and the path will not change within the real-time
   change of network topology.  Moreover, the deterministic routes
   should provide the capability including the latency, jitter and
   packet loss ratio.

   Routes generally perform forwarding function including receiving the
   incoming packets and forwarding the packets to a Router based on the
   header information and a forwarding information base.  It is
   necessary to provide pre-routes with SLA capability and generate
   endogenous deterministic routing with deterministic capability.  The
   deterministic routes perform the functions of forwarding and QoS



Xiong                     Expires 27 April 2022                 [Page 7]


Internet-Draft  The Requirements for Wide-area IP Determ    October 2021


   guarantee at the same time.  The types of deterministic routes can be
   classified into ultra-low delay routes, low delay routes, low jitter
   routes, and so on.  There can also has replication routes and
   aggregation routes.

   The mechanisms of path establishment include traffic engineering
   technology (MPLS-TE, SR-TE, static configuration, etc.), IGP
   technology, etc.  Explicit strict routing can guarantee the delay
   jitter and other requirements of services.  Loose routing only
   generates some endogenous deterministic routes, and other routes
   still need forwarding and scheduling, such as dynamic resource-aware
   routing and queue scheduling.

4.3.  The Deterministic QoS

   The deterministic QoS is to arrange and schedule the deterministic
   flows on the basis of providing deterministic resources and routes,
   so as to control of each flow and meet the DetNet QoS goals.

   The scheduling and control include the classification of the
   deterministic flows, queue scheduling mechanism for each class of
   deterministic flow, deterministic shaping at boundary nodes, limiting
   the number of concurrent flows and reducing micro bursts, mapping the
   dynamic concurrent flows into different virtual topologies.
   Moreover, flow aggregation is performed at the aggregation node to
   reduce flow state maintenance and replication or elimination is
   performed at the relay node to achieve reliability.

   If the deterministic flows crosses multiple domains, the end-to-end
   latency is the sum of delay from all domains.  It is required to
   control the deadline delay of each domain.  Moreover, bounded jitter
   (packet delay variation) should be adjusted and scheduled at the
   inter-domain boundary nodes.

5.  Security Considerations

   TBA

6.  Acknowledgements

   TBA

7.  IANA Considerations

   TBA

8.  Normative References




Xiong                     Expires 27 April 2022                 [Page 8]


Internet-Draft  The Requirements for Wide-area IP Determ    October 2021


   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC8557]  Finn, N. and P. Thubert, "Deterministic Networking Problem
              Statement", RFC 8557, DOI 10.17487/RFC8557, May 2019,
              <https://www.rfc-editor.org/info/rfc8557>.

   [RFC8578]  Grossman, E., Ed., "Deterministic Networking Use Cases",
              RFC 8578, DOI 10.17487/RFC8578, May 2019,
              <https://www.rfc-editor.org/info/rfc8578>.

   [RFC8655]  Finn, N., Thubert, P., Varga, B., and J. Farkas,
              "Deterministic Networking Architecture", RFC 8655,
              DOI 10.17487/RFC8655, October 2019,
              <https://www.rfc-editor.org/info/rfc8655>.

Author's Address

   Quan Xiong
   ZTE Corporation
   No.6 Huashi Park Rd
   Wuhan
   Hubei, 430223
   China

   Email: xiong.quan@zte.com.cn



















Xiong                     Expires 27 April 2022                 [Page 9]