DETNET Q. Xiong
Internet-Draft ZTE Corporation
Intended status: Standards Track Z. Du
Expires: 28 August 2022 China Mobile
February 2022
DetNet Enhancements for Large-Scale Deterministic Networks
draft-xiong-detnet-large-scale-enhancements-00
Abstract
This document describes enhancements to DetNet to achieve the
differentiated DetNet QoS in large-scale deterministic networks
including the overall requirements and solutions with deterministic
resources, routes and services.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 5 August 2022.
Copyright Notice
Copyright (c) 2022 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Xiong & Du Expires 28 August 2022 [Page 1]
Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Conventions used in this document . . . . . . . . . . . . . . 3
2.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3
2.2. Requirements Language . . . . . . . . . . . . . . . . . . 3
3. DetNet Applicability for Large-Scale Deterministic
Networks . . . . . . . . . . . . . . . . . . . . . . . . 4
4. Overall Requirements of Large-Scale Deterministic Networks . 5
4.1. Service Requirements . . . . . . . . . . . . . . . . . . 5
4.1.1. Support the Differentiated DetNet QoS of Multiple
Services . . . . . . . . . . . . . . . . . . . . . . 5
4.1.2. Guarantees of Multiple Dynamic Deterministic Flows . 7
4.2. Route Requirements . . . . . . . . . . . . . . . . . . . 7
4.2.1. Support the Distributed Deterministic Routes . . . . 8
4.2.2. Support the Inter-domain Deterministic Routes . . . . 8
4.2.3. Support the Replication and Elimination Deterministic
Routes . . . . . . . . . . . . . . . . . . . . . . . 8
4.3. Resource Requirements . . . . . . . . . . . . . . . . . . 8
4.3.1. Management and Scheduling of the Network Resources . 8
4.3.2. Support the Utilization of Heterogeneous Resources . 9
5. Solutions of Large-Scale Deterministic Networks . . . . . . . 9
5.1. Enhanced Layering Model . . . . . . . . . . . . . . . . . 9
5.2. Mechanisms to Achieve Differentiated DetNet QoS . . . . . 10
5.2.1. Deterministic Resources . . . . . . . . . . . . . . . 10
5.2.2. Deterministic Routes . . . . . . . . . . . . . . . . 11
5.2.3. Deterministic Services . . . . . . . . . . . . . . . 11
6. Security Considerations . . . . . . . . . . . . . . . . . . . 12
7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12
9. Normative References . . . . . . . . . . . . . . . . . . . . 12
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 12
1. Introduction
5G network is oriented to the internet of everything. In addition to
the Enhanced Mobile Broadband (eMBB) and Massive Machine Type
Communications(mMTC) services, it also supports the Ultra-reliable
Low Latency Communications (uRLLC) services. The uRLLC services
demand SLA guarantees such as low latency and high reliability and
other deterministic and precise properties especially in Wide Area
Network (WAN) applications.
The uRLLC services should be provided in large-scale networks which
cover the industries such as intelligent electrical network,
intelligent factory, internet of vehicles, industry automation and
other industrial internet scenarios. The industrial internet is the
key infrastructure that coordinate various units of work over various
Xiong & Du Expires 28 August 2022 [Page 2]
Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022
system components, e.g. people, machines and things in the industrial
environment including big data, cloud computing, Internet of Things
(IOT), Augment Reality (AR), industrial robots, Artificial
Intelligence (AI) and other basic technologies. For the intelligent
electrical network, there are deterministic requirements for
communication delay, jitter and packet loss rate. For example, in
the electrical current difference model, a delay of 3~10ms and a
jitter variation is no more than 100us are required. For the
automation control, it is one of the basic application and the the
core is closed-loop control system. The control process cycle is as
low as millisecond level, so the system communication delay needs to
reach millisecond level or even lower to ensure the realization of
precise control. There are three levels of real-time requirements
for industrial interconnection: factory level is about 1s, and
process level is 10~100ms, and the highest real-time requirement is
motion control, which requires less than 1ms.
According to [RFC8655], Deterministic Networking (DetNet) operates at
the IP layer and delivers service which provides extremely low data
loss rates and bounded latency within a network domain. The
applications in 5G networks demand much more deterministic and
precise properties in WAN. The existing deterministic technologies
are facing large-scale number of nodes and long-distance
transmission, traffic scheduling, dynamic flows, and other
controversial issues in large-scale networks.
This document describes enhancements to DetNet to achieve the
differentiated DetNet QoS in large-scale deterministic networks
including the overall requirements and solutions with deterministic
resources, routes and services.
2. Conventions used in this document
2.1. Terminology
The terminology is defined as [RFC8655].
2.2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
Xiong & Du Expires 28 August 2022 [Page 3]
Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022
3. DetNet Applicability for Large-Scale Deterministic Networks
As per [RFC8655], it defined the overall architecture for DetNet,
which provides a capability for real-time applications with extremely
low data loss rates and bounded latency within a network domain. It
has three goals: minimum and maximum end-to-end latency from source
to destination, bounded jitter (packet delay variation), packet loss
ratio and upper bound on out-of-order packet delivery. To achieve
the above objectives, multiple techniques need to be used in
combination, including explicit routes, service protection and
resource allocation defined by DetNet. And the DetNet functionality
is implemented at DetNet service sub-layer and DetNet forwarding sub-
layer. It is required to analyse the applicability in DetNet for
large-scale deterministic networks.
From the perspective of services requirements discussed in section
4.1, a large-scale network needs to provide the deterministic service
for various applications. And the deterministic service may demand
different deterministic QoS requirements according to different
application scenarios. The service protection in service sub-layer
is not sufficient to meet the services requirements of large-scale
networks, it should provide unified planning and scheduling
mechanisms for service flows to perform end-to-end delay and jitter
control and achieve differentiated DetNet QoS of multiple services.
The large-scale deterministic networks have a large number of hops
and high link delay, which makes it difficult to achieve network-wide
precise time synchronization. It may across multiple IP domains, or
there may be different heterogeneous forwarding plane transport
technologies. It is required to consider the efficiency of resources
utilization and routes steering.
From the perspective of routes requirements discussed in section 4.3,
a large-scale network should provide the deterministic paths for the
services in large-scale networks. The deterministic routes should be
calculated based on the deterministic metrics such as the end-to-end
bounded latency and jitter. The forwarding sub-layer should
establish the deterministic routes with SLA guarantees based on the
deterministic resources. Moreover, other than explicit routes in
centralized control scenarios, the distributed routes when the DetNet
deployed with no controller may be more important for large-scale
networks.
From the perspective of resources requirements discussed in section
4.3, a large-scale network should utilize the bandwidth, nodes,
links, jitter resource, and queue scheduling resource and the other
heterogeneous resources to establish the deterministic links which
could provide SLA guarantees for the deterministic forwarding
Xiong & Du Expires 28 August 2022 [Page 4]
Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022
capabilities at different levels. Other than resource allocation,
the forwarding sub-layer should support the unified and simplified
scheduling and management mechanism for resources. For example,
resource modeling, isolation and reservation should be considered to
guarantee the deterministic transmission.
It is required to provide mechanisms within DetNet service and
forwarding sub-layers to meet the requirements of large-scale
deterministic networks. This document describes enhancements to
DetNet to achieve the differentiated DetNet QoS in large-scale
deterministic networks including the overall requirements and
solutions with deterministic resources, routes and services.
4. Overall Requirements of Large-Scale Deterministic Networks
As per [draft-liu-detnet-large-scale-requirements], the technical and
operational requirements have been specified for large-scale
deterministic networks. For DetNet architecture to support
deterministic service in a large-scale network, the requirements from
services, routes and resources also need to be considered.
4.1. Service Requirements
4.1.1. Support the Differentiated DetNet QoS of Multiple Services
As defined in [RFC8655], the DetNet QoS can be expressed in terms of
: Minimum and maximum end-to-end latency, bounded jitter (packet
delay variation), packet loss ratio and an upper bound on out-of-
order packet delivery. As described in [RFC8578], DetNet
applications differ in their network topologies and specific desired
behavior and different services requires differentiated DetNet QoS.
In the large-scale networks, multiple services with differentiated
DetNet QoS is co-existed in the same DetNet network. The
classification of the deterministic flows within different levels is
should be taken into considerations. It is required to provide
Latency, bounded jitter and packet loss dynamically and flexibly in
all scenarios for each characterized flow.
As the Figure 1 shows, the services can be divided into 5 levels and
level 2~5 is the DetNet flows and level-1 is non-DetNet flow. DetNet
applications and DetNet QoS is differentiated within each level.
Xiong & Du Expires 28 August 2022 [Page 5]
Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022
+-------------+-----------+----------+----------+----------+-----------+
| Item | Level-1 | Level-2 | Level-3 | Level-4 | Level-5 |
+-------------+-----------+----------+----------+----------+-----------+
| Applications| Broadcast | Voice | Audio and| AR/VR | Industrial|
| Examples | | | Video | | |
+-------------+-----------+----------+----------+----------+-----------+
| DetNet QoS | Bandwidth | Jitter | Latency | Low | Ultra-low |
| | Guarantee | Guarantee| Guarantee| latency |latency and|
| | | | |and jitter| jitter |
+-------------+-----------+----------+----------+----------+-----------+
Figure 1: Figure 1: The classification of multiple services
From the perspective of deterministic service requirements,
deterministic Quality of Service (QoS) in the network can be divided
into five types or levels:
Level-1: bandwidth guarantee. The indicator requirements include
basic bandwidth guarantee and certain packet loss tolerance. There
is no requirement for the upper bound of the latency, and no
requirement for the jitter. Typical services include download and
FTP services.
Level-2: jitter guarantee. The indicator requirements include:
jitter 50ms, delay 300ms. Typical services include synchronous voice
services, such as voice call.
Level-3: Latency guarantee. The indicator requirements include:
delay 50ms, jitter 50ms. Typical services include real-time
communication services, such as video, production monitoring, and
communication services.
Level-4: low delay and low jitter guarantee. The indicator
requirements include: delay 20ms, jitter 5ms. Typical services
include video interaction services, such as AR/VR, holographic
communication, cloud video and cloud games.
Level-5: ultra-low delay and jitter guarantee. The indicator
requirements include: delay 10ms, jitter 100us. Typical services
include production control services, such as power protection and
remote control.
Moreover, different DetNet services is required to tolerate different
percentage of packet loss ratio such as 99.9%, 99.99%, 99.999%, and
so on. It is also required to provide service isolation. In some
scenarios, such as intelligent electrical network, the isolation
Xiong & Du Expires 28 August 2022 [Page 6]
Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022
requirements are very important. For example, the automatic
operation or control of a process or isochronous data and service
with different priorities need to meet the requirements of hard
isolation. In addition to the requirements of delay and jitter, the
differential protection (DP) service needs to be isolated from other
services and hard isolated tunnel is required.
4.1.2. Guarantees of Multiple Dynamic Deterministic Flows
As described in [RFC8557], deterministic forwarding can only apply to
flows with such well-defined characteristics as periodicity and
burstiness. As defined in DetNet architecture [RFC8655], the traffic
characteristics of an App-flow can be CBR (constant bit rate) or VBR
(variable bit rate) of L1, L2 and L3 layers (VBR takes the maximum
value when reserving resources). But the current scenarios and
technical solutions only consider CBR flow, without considering the
coexistence of VBR and CBR, the burst and aperiodicity of flows. The
operations such as shaping or scheduling have not been specified.
Even TSN mechanisms are based on a constant and forecastable traffic
characteristics.
It will be more complicated in WAN applications where much more flows
coexist and the traffic characteristics is more dynamic. A huge
number of flows with different DetNet QoS requirements is dynamically
concurrent and the state of each flow cannot be maintained. It is
required to offer reliable delivery and SLA guarantee for dynamic
flows. For example, periodic flow and aperiodic flow (including
micro burst flow, etc.), CBR and VBR flow, flow with different
periods or phases, etc. When the network needs to forward these
deterministic flows at the same time, it must solve the problems of
time micro bursts, queue processing and aggregation of multiple
flows. It is required to guarantee the deterministic QoS of multiple
dynamic flows. Flow shaping and concurrent and micro-burst control
should be provided.
4.2. Route Requirements
Traditional routes only have reachability. Deterministic
requirements such as delay and jitter are only used as path
computation constraints. The paths vary with the real-time change of
the network topology. They do not have Service Level Agreement (SLA)
capability, and cannot meet the deterministic requirements at
different levels. On the basic of the resources, the steering path
and routes for deterministic flows should be programmed before the
flows coming and able to provide SLA capability. And the routes
should be considered to be established in distributed and centralized
control Plane.
Xiong & Du Expires 28 August 2022 [Page 7]
Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022
4.2.1. Support the Distributed Deterministic Routes
In large-scale deterministic networks, the distributed scenario with
no controller should be taken into consideration. It is required to
support the distributed deterministic routes which are established by
distributed protocols such as IGP.
4.2.2. Support the Inter-domain Deterministic Routes
In large-scale deterministic networks, it may across multiple network
domains, it is required to support the inter-domain deterministic
routes to achieve the end-to-end latency, bounded jitter. And the
deadline of latency and jitter of each domain and segment should be
determined and controlled. The inter-domain mechanism MUST be
considered at the boundary nodes such as BGP configurations.
4.2.3. Support the Replication and Elimination Deterministic Routes
As described in [RFC8557], the packet replication and elimination
service protection should be provided to achieve the low packet loss
ratio. It will copy the flows and spread the data over multiple
disjoint forwarding paths. The bounded latency and jitter of each
path should be meet service deterministic requirement. And the
difference of latency within these paths should be limited. So the
replication and elimination deterministic routes with configured
latency and jitter policy should be supported.
4.3. Resource Requirements
4.3.1. Management and Scheduling of the Network Resources
Traditional Ethernet, IP and MPLS networks which is based on
statistical multiplexing provides best-effort packet service and
offers no delivery and SLA guarantee. As described in [RFC8655], the
primary technique by which DetNet achieves its QoS is to allocate
sufficient resources. But it can not be achieved by not sufficient
resource which can be allocated due to practical and cost reason. So
it is required to achieve the high-efficiency of resources
utilization when provide the DetNet service.
Network resources include nodes, links, ports, bandwidth, queues,
etc. The congestion control, shaping and queue scheduling and other
traffic mechanisms which have been proposed in IEEE 802.1 TSN such as
IEEE802.1Qbv, IEEE802.1Qch, IEEE802.1Qav, IEEE802.1Qcr and so on.
Resource classification and modeling is required along with the
explicit path with more SLA guarantee parameters like bandwidth,
latency, jitter, packet loss and so on.
Xiong & Du Expires 28 August 2022 [Page 8]
Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022
4.3.2. Support the Utilization of Heterogeneous Resources
In large-scale application, a large-scale number of nodes and long-
distance transmission in the network will lead to latency and jitter,
such as increasing transmission latency, jitter and packet loss. It
is required to reduce the scale of the network topology by
establishing cut-through channels. The existing technologies such as
FlexE and SR tunnels should be taken into consideration. And
multiple capabilities is also provided by the nodes and links within
the network topology such as FlexE tunnels, TSN sub-network and
IP/MPLS/SRv6 tunnels. It is required to integrate the multi-
capability resources to achieve the optimal DetNet QoS.
Heterogeneous resource should be used and unified and simplified
resources mechanism under the selection of existing multiple
technical methods to realize the elastic of deterministic capability.
5. Solutions of Large-Scale Deterministic Networks
5.1. Enhanced Layering Model
The large-scale IP network can provide three levels of determinism,
deterministic resources, deterministic routes and deterministic
services, to establish a unified large-scale deterministic IP network
architecture. The deterministic resources maintains the resources of
the entire network, and performs unified modeling for deterministic
resources to form deterministic links to shield the differences in
heterogeneous resource capabilities. The deterministic routes
computes routes based on the deterministic links modeled at the
resource layer to provide deterministic transport capabilities. The
deterministic services performs traffic monitoring on ingress nodes
by planning the traffic characteristics of service flows, and maps
them to deterministic routes to meet the time requirements of
different types and levels of services.
Xiong & Du Expires 28 August 2022 [Page 9]
Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022
+-----------------------------------------+
| Service sub-layer |
+-----------------------------------------+
| Differentiated DetNet QoS for Services |
+-----------------------------------------+
| Forwarding sub-layer |
+-----------------------------------------+
| Routes with Deterministic metrics |
| Distributed Deterministic Routes |
| Inter-domain Deterministic Routes |
| Replication and Elimination Routes |
+-----------------------------------------+
| Resource Modeling |
| Resource Reservation |
| Resource Isolation |
+-----------------------------------------+
Figure 2: Figure 2: The Enhanced Layering Model of Large-Scale
Deterministic Networks
5.2. Mechanisms to Achieve Differentiated DetNet QoS
5.2.1. Deterministic Resources
Differentiated deterministic service requirements require the
networks to provide different deterministic capabilities. The
resources related to deterministic capabilities are also
differentiated. The networks need to shield the differences between
network capabilities. Deterministic resource is the basis for
providing deterministic network services. It refers to the resources
that meet the deterministic indicators of a node and link processing
as well as the corresponding resource processing mechanisms (such as
link bandwidth, queues, and scheduling algorithms). It is necessary
to make overall resource planning for the network and make unified
modeling for heterogeneous deterministic resources to form unified
deterministic links to provide guarantee for the deterministic
forwarding capabilities at different levels. A deterministic link
can be a sub-network that provides deterministic transmission or a
Point-to-Point (P2P) link. When the existing resources in the
network are insufficient to meet the SLA requirements, virtual
networks need to be reconstructed.
Xiong & Du Expires 28 August 2022 [Page 10]
Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022
5.2.2. Deterministic Routes
To meet the requirements of different types and levels of
deterministic services, deterministic route is to create
deterministic routes with different SLA levels based on the
deterministic link resources after unified modeling.
Deterministic routes can be based on strict explicit paths or loose
routes. The former is applicable to centralized scenarios with
controllers, and the latter is applicable to distributed scenarios
without controllers. In the centralized scenario, when the source
and sink PEs of a deterministic service are located at the two ends
of a WAN with a limited physical range, one controller (single
domain) or multiple controllers (cross domain) compute one or more
paths with deterministic SLA in advance according to the typical
Traffic Specification (T-SPEC) based on the collected deterministic
resources, or compute dynamically according to the service T-SPEC as
required by the services. It is suggested to generate two non-
intersecting paths with very close delay to form 1+1 protection and
perform concurrent transmission and dual reception, and make
replication and elimination on the egress PE. In the distributed
scenario, intrinsic deterministic loose routes are computed on the
device side through routing protocols. Interior Gateway Protocol
(IGP) is used to compute deterministic routes based on deterministic-
delay inside a domain, and Border Gateway Protocol (BGP) is used to
compute deterministic routes based on accurate delay/jitter across
domains.
5.2.3. Deterministic Services
Deterministic services provide unified planning and scheduling
mechanisms for service flows and perform end-to-end delay and jitter
control. It is necessary to implement admission control and traffic
policing at the ingress PE node based on the SLA of deterministic
service flows, and map the service flows to deterministic routes to
achieve the final goal of deterministic QoS.
Deterministic services support that the end-to-end delay/jitter of
the traffic with a specific T-SPEC in the network will be strictly
limited within a bounded range on the basis of deterministic resource
and route . As different service levels have different requirements
for delay and jitter, the resources and routing mechanisms used for
mapping services to deterministic routes are also different. For
example, the extremely low delay and jitter can be guaranteed by
multiplexing the rigid pipes at L1, so as to avoid the excessive
intra-node delay contributed by too many hops of intermediate nodes
at L3. Or in the customized virtual network, the bounded delay and
jitter can be guaranteed by forwarding along the paths composed of
Xiong & Du Expires 28 August 2022 [Page 11]
Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022
links based on the ATS or CQF scheduling algorithm. Traffic policing
on the ingress PE ensures that the service traffic does not exceed
the reserved bandwidth, and then performs traffic shaping on the
egress node. Different scheduling algorithms have different shaping
effects.
6. Security Considerations
TBA
7. Acknowledgements
The authors would like to thank Peng Liu, Bin Tan, Aihua Liu Shaofu
Peng for their review, suggestions and comments to this document.
8. IANA Considerations
TBA
9. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8557] Finn, N. and P. Thubert, "Deterministic Networking Problem
Statement", RFC 8557, DOI 10.17487/RFC8557, May 2019,
<https://www.rfc-editor.org/info/rfc8557>.
[RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases",
RFC 8578, DOI 10.17487/RFC8578, May 2019,
<https://www.rfc-editor.org/info/rfc8578>.
[RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas,
"Deterministic Networking Architecture", RFC 8655,
DOI 10.17487/RFC8655, October 2019,
<https://www.rfc-editor.org/info/rfc8655>.
Authors' Addresses
Quan Xiong
ZTE Corporation
No.6 Huashi Park Rd
Xiong & Du Expires 28 August 2022 [Page 12]
Internet-Draft DetNet Enhancements for Large-Scale Dete February 2022
Wuhan
Hubei, 430223
China
Email: xiong.quan@zte.com.cn
ZongPeng Du
China Mobile
Beijing
China
Email: duzongpeng@chinamobile.com
Xiong & Du Expires 28 August 2022 [Page 13]