Requirements for Large-Scale Deterministic Networks
draft-liu-detnet-large-scale-requirements-04
The information below is for an old version of the document.
| Document | Type |
This is an older version of an Internet-Draft whose latest revision state is "Replaced".
|
|
|---|---|---|---|
| Authors | Peng Liu , Yizhou Li , Toerless Eckert , Quan Xiong , Jeong-dong Ryoo , zhushiyin , Xuesong Geng | ||
| Last updated | 2022-09-06 | ||
| Replaced by | draft-ietf-detnet-scaling-requirements, draft-ietf-detnet-scaling-requirements | ||
| RFC stream | (None) | ||
| Formats | |||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | I-D Exists | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-liu-detnet-large-scale-requirements-04
Deterministic Networking Working Group P. Liu
Internet-Draft China Mobile
Intended status: Informational Y. Li
Expires: 11 March 2023 Huawei
T. Eckert
Futurewei Technologies USA
Q. Xiong
ZTE Corporation
J. Ryoo
ETRI
S. Zhu
New H3C Technologies
X. Geng
Huawei
7 September 2022
Requirements for Large-Scale Deterministic Networks
draft-liu-detnet-large-scale-requirements-04
Abstract
Aiming at the large-scale deterministic network, this document
describes the technical and operational requirements when the
different deterministic levels of applications co-exist and are
transported over a wide area. This document also describes the
corresponding Deterministic Networking (DetNet) data plane
enhancement requirements.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 11 March 2023.
Liu, et al. Expires 11 March 2023 [Page 1]
Internet-Draft Deterministic Networking September 2022
Copyright Notice
Copyright (c) 2022 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Conventions Used in This Document . . . . . . . . . . . . . . 4
3. The Overall Characteristics of Large-Scale Deterministic
Networks . . . . . . . . . . . . . . . . . . . . . . . . 4
4. Technical Requirements in Large-Scale Deterministic
Networks . . . . . . . . . . . . . . . . . . . . . . . . 6
4.1. Tolerate Time Asynchrony . . . . . . . . . . . . . . . . 6
4.1.1. Support Asynchronous Clocks Across Domains . . . . . 6
4.1.2. Tolerate Clock Jitter & Wander within a Clock
Synchronous Domain . . . . . . . . . . . . . . . . . 7
4.1.3. Provide Mechanisms not Requiring Full Time
Synchronization . . . . . . . . . . . . . . . . . . . 7
4.1.4. Support Asynchronization based Methods . . . . . . . 7
4.2. Support Large Single-hop Propagation Latency . . . . . . 8
4.3. Accommodate the Higher Link Speed . . . . . . . . . . . . 9
4.4. Be Scalable to Numerous Network Devices and Massive Traffic
Flows . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.5. Tolerate Failures of Links or Nodes and Topology
Changes . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.6. Support Configuration of Multiple Queuing Mechanisms . . 11
4.7. Support Queuing Mechanisms Switchover Crossing
Multi-domains . . . . . . . . . . . . . . . . . . . . . . 12
5. Data Plane Enhancement Requirements . . . . . . . . . . . . . 12
5.1. Support Aggregated Flow Identification . . . . . . . . . 13
5.2. Support Information used by Functions ensuring
Deterministic Latency . . . . . . . . . . . . . . . . . . 13
5.3. Support Redundancy Related Fields . . . . . . . . . . . . 14
5.4. Support Explicit Path Selection . . . . . . . . . . . . . 14
6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 14
7. Security Considerations . . . . . . . . . . . . . . . . . . . 14
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 14
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 14
Liu, et al. Expires 11 March 2023 [Page 2]
Internet-Draft Deterministic Networking September 2022
10. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 15
11. Normative References . . . . . . . . . . . . . . . . . . . . 15
Appendix A. Examples of Large-Scale Deterministic Network
Trials . . . . . . . . . . . . . . . . . . . . . . . . . 17
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 18
1. Introduction
Packet networks are evolving from bandwidth-guaranteed Quality of
Service (QoS) to latency-guaranteed QoS that guarantees bounded
latency and definite latency. Bounded latency and definite latency
can be further understood as in-time delivery, in which a packet
arrives without exceeding a predetermined time, and on-time delivery,
in which a packet arrives at a predetermined time, respectively. In
addition, network survivability, which typically guarantees traffic
recovery within 50 ms in the event of a network failure, is evolving
to a level that guarantees lossless recovery. In order to realize
the evolution of QoS and network survivability of these networks,
Time-Sensitive Networking (TSN) technology and Deterministic
Networking (DetNet) technology are considered to be essential.
TSN is a set of standards developed by the IEEE 802.1 TSN Task Group
(TG) [IEEE802.1TSN] and specifies mechanisms and protocols necessary
to realize highly available IEEE 802.1 networks with bounded latency
to carry time-sensitive, real-time application traffic.
DetNet, of which architecture is defined in RFC 8655 [RFC8655],
provides a capability to carry specified unicast or multicast data
flows for real-time applications with extremely low data loss rates
and bounded latency within a network domain. The overall framework
for DetNet data plane is provided in [RFC8938], and various documents
on different data plane technologies and their interworking
technologies to extend the service range of data that TSN intends to
deliver to the IP (Internet Protocol) and MPLS (Multi-Protocol Label
Switching) networks have been standardized.
Liu, et al. Expires 11 March 2023 [Page 3]
Internet-Draft Deterministic Networking September 2022
Since TSN and DetNet were proposed, application use cases have always
been one of the hottest topics. As documented in RFC 8578 [RFC8578],
the scope of networks addressed by the current DetNet is limited to
networks that can be centrally controlled, i.e., an "enterprise" (aka
"corporate") network, excluding "the open Internet," explicitly.
After years of development, TSN has been used in several industries,
and has enough public awareness of the industry for its scope.
DetNet also has done a lot of work and the standards are mature, and
people become concerned about how to meet deterministic service
demand in large-scale networks. The current DetNet is limited to a
single administrative domain network, and there are technical
elements necessary for application to a large-scale network spanning
multiple domains.
This document describes requirements for large-scale deterministic
networks where different deterministic levels of applications co-
exist and large-scale deterministic networking across multiple
administrative domains is possible. This document also describes the
requirements for enhancing the DetNet data plane defined prior to
this document.
2. Conventions Used in This Document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119][RFC8174] when, and only when, they appear in all
capitals, as shown here.
While [RFC2119] and [RFC8174] describe interpretations of these key
words in terms of protocol specifications and implementations, they
are used in this document to describe technical and operational
requirements to realize large-scale deterministic networks.
3. The Overall Characteristics of Large-Scale Deterministic Networks
When deterministic network services are introduced, network providers
always face the problem of how to match application needs to the
technology, so more works are needed for network service providers to
successfully sell DetNet type services to customers. The providers
are in need of the following:
Service level objective definitions, considering absolute or relative
latency and jitter bounds, flows types and physical network scale
Suitable queuing mechanisms, considering more options for queuing
mechanisms for different service level, and
Liu, et al. Expires 11 March 2023 [Page 4]
Internet-Draft Deterministic Networking September 2022
Deployment strategies, considering how to integrate into existing
networks, service, and control plane.
[RFC8578] provides various use cases and their requirements in the
areas of industry, electricity, buildings, etc. Some of them clearly
specify the requirements for latency and jitter, while some others do
not for the jitter. Different types of users have different demands,
just as a network provider provides different network services for
personal business or enterprise business.
One kind has critical SLA requirement, such as remote control or
cloud Programmable Logic Controller (PLC) of manufacturing and
differential protection of electricity. If these services exceed the
boundaries of latency and jitter, it will bring property losses and
security risks, so they cannot tolerate with any non-deterministic
situation and can pay more on the network service.
Another kind has relatively loose levels of SLA requirement, such as
cloud gaming, cloud VR and online meeting for "consumer" networks.
The users of these applications hope to have a better network
experience, but they can tolerate it to a certain extent. If the
network quality is not good sometime, they might be willing to spend
more money for high-quality network services. In some aspects,
because such services have no industry barriers and can tolerate
exceeding the upper boundary of latency within a small probability,
they have relatively lower requirements for the network and may be
easier to deploy.
Different application demands are actually related to cost. For
strict deterministic services, strict technologies need to be used,
and all network devices may need to be upgraded. For non-strict
deterministic services, it may only be necessary to upgrade some
network devices (maybe edge nodes) or share corresponding network
resources. From the perspective of deployment, it is helpful if
there is a clear classification of application demands, including
latency, jitter, reliability, etc. In this way, the corresponding
technology to implement could be chosen, taking into account both
performance and cost, but how to make choice is not within the scope
of this document.
Liu, et al. Expires 11 March 2023 [Page 5]
Internet-Draft Deterministic Networking September 2022
Critical latency requirements:
| <->| Industrial, tight jitter, hard latency limit
|<------->| Industrial, hard latency limit
|
|<-------------.....> Relatively lower latency requirements
|
|<-------------........................> Best effort
|
+---------------------------------------------------------->
latency
Figure 1: Different levels of application requirements
4. Technical Requirements in Large-Scale Deterministic Networks
Due to the different kinds of application requirements in large-scale
networks, the corresponding technical requirements should be
considered.
4.1. Tolerate Time Asynchrony
4.1.1. Support Asynchronous Clocks Across Domains
A large-scale network may span over multiple networks with one or
more administrative domains. One of DetNet's objectives is to stitch
TSN islands together. All devices inside a TSN domain are time-
synchronized, and most of TSN technologies rely on precise time
synchronization [IEEE802.1Qbv][IEEE802.1Qch][IEEE802.1Qav]. However,
different TSN islands may have different clocks which are not
synchronized as shown in Figure 2, where the time difference of two
TSN domains is D. DetNet needs to connect these two TSN domains
together and provide end-to-end deterministic latency service. The
mechanism adopted by a large-scale deterministic network MUST be
prepared to cope with non-synced TSN domains. This can be done, for
example, by putting extra buffer space at the ingress of a new
domain, increasing the dead time as a guard band, or using some
timing compensation mechanism. This document does not intend to list
all the potential ways.
Liu, et al. Expires 11 March 2023 [Page 6]
Internet-Draft Deterministic Networking September 2022
+--------------+ +--------------+
| | DetNet Connection | |
| TSN Domain I +-----------------------------+ TSN Domain II|
| | | |
+--------------+ +--------------+
| | | | |
Clock of TSN +--------+--------+--------+--------+
Domain I =
=
= | | | | |
Clock of TSN = +--------+--------+--------+--------+
Domain II = =
=<==D==>=
= =
Figure 2: Clock asynchrony between two TSN islands
4.1.2. Tolerate Clock Jitter & Wander within a Clock Synchronous Domain
Within a single time synchronization domain, different clock accuracy
is expected, for example the crystal oscillator in Ethernet is
specified at 100 ppm [Fast-Ethernet-MII-clock], Synchronous Ethernet
(SyncE) can achieve 50 ppb [G.8262], and more precise time
synchronization [G.8273] is expected in 5G mobile backhaul. The
clocks experience different jitter and wander. It may cause
different level of asymmetry of the path. The large-scale networks
SHOULD be able to recover or absorb such time variance within a
domain and across multiple domains.
4.1.3. Provide Mechanisms not Requiring Full Time Synchronization
Some networks like mobile backhaul use frequency synchronization,
such as SyncE, instead of the strict time synchronization. It is
usually hard to achieve the full time synchronization in large-scale
networks when considering the size of the network topology. It is
desired that the same deterministic performance in term of the
bounded latency and jitter SHOULD be achieved when full time
synchronization is not available, that is to say, when only partial
synchronization (SyncE is one of the examples) is in use.
4.1.4. Support Asynchronization based Methods
There are a large number of traffic flows in a large-scale network
and some of them are acyclic. Asynchronization based methods can
meet the requirements of those traffic flows. Moreover, The
mechanisms not requiring the time and/or frequency synchronization
eliminate the hardware cost and difficulty at the network nodes.
[IEEE802.1Qcr] conceptually uses per-flow based asynchronous shaper
Liu, et al. Expires 11 March 2023 [Page 7]
Internet-Draft Deterministic Networking September 2022
to achieve bounded latency. The effectiveness of the per-flow based
asynchronous shaper can be proven through mathematical analysis. It
can naturally tolerate the time variance, but it exhibits the
concerns of per-flow state buffer management as shown in
[I-D.eckert-detnet-bounded-latency-problems]. When it is in use, the
requirement in Section 4.3 SHOULD be carefully met.
4.2. Support Large Single-hop Propagation Latency
In a large-scale network, a single hop distance is enough to generate
large latency. The speed of optical transmission in fiber is 200 km/
ms. Thus, the propagation delay of a single hop can be in the order
of a few milliseconds. It is much greater than that of a LAN, and
introduces impacts on queuing mechanisms, such as cyclic or time
aware scheduling method. So the queuing mechanism for LAN networks
needs to be extended, such as considering the propagation latency
when setting the period in both time synchronization or frequency
synchronization based methods, or setting the extra buffer in the
asynchronization based methods, to meet the requirements of
deterministic forwarding between the network nodes.
Here, we use an example to describe the influence of Large Single-hop
Propagation Latency on cycle based methods, but not to specify any
solution. For a cyclic based method, suppose a large-scale network
wants to keep using the simple cycle mapping relationship, however
the link distance between two nodes is longer. Moreover, a
downstream node may have many upstream nodes each with different link
propagation delays (e.g., 9 us, 10 us, 11 us, 15 us and 20 us). In
order to absorb the longest link propagation delay, the length of
cycle must be set to at least 20 us. However, since packet's arrival
time varies within the receiving cycle, larger cycle length means
larger delay variance.
Liu, et al. Expires 11 March 2023 [Page 8]
Internet-Draft Deterministic Networking September 2022
Upstream Node X |sending cycle | |
+--"------------+------------+
= "\ = =
= " \ = =
= " \ = =
= " \ = =
= " V = =
Downstream Node Y |receiving cycle| |
+--"----"-------+----\-------+
= " " = \ =
= " " = V resent out
= " " = =
Time Line -=--"----"-------=------------=----->
(in us) 0 | | 10 20
v v
Transmission Latency
Figure 3: The influence of transmission latency on a cyclic method
4.3. Accommodate the Higher Link Speed
A large-scale network normally uses higher speed links, especially
for its backbone. Current deterministic mechanisms used in a local
network is usually deployed in link speed of 10 Mbps or 1 Gbps, or
possibly 10 Gbps. The data rates of 10G, 100G, 400G and even higher
are commonly used in wide area networks. With the increasing of the
data rate, the network scheduling cycle can be reduced if the same
amount of the data is required to be sent each cycle for each
application. Or more data can be sent if the network cycle time
remains the same. For the former, it requires the more precise time
control (e.g. cycle in the order of a few microseconds or sub-
microseconds) for the input stream gate and the timed output buffer.
For the latter, more buffer space is required which imposes more
complex buffer or queue management and larger memory consumption.
Another aspect to consider is the aggregation of the flows. In the
large-scale network, the number of flows can be hundreds or tens of
thousands. They can be aggregated into a small number of
deterministic path or tunnels. It is practical to have a few flow-
based or aggregated-flow based status in the local network. But in
higher speed and larger scale networks, it is hardly feasible. If
[IEEE802.1Qcr] is in use, it requires more buffers comparing to the
other full/partial time synchronized mechanisms. Therefore, it
requires optimizations to support higher link speeds.
Liu, et al. Expires 11 March 2023 [Page 9]
Internet-Draft Deterministic Networking September 2022
4.4. Be Scalable to Numerous Network Devices and Massive Traffic Flows
Comparing to a LAN, a large-scale network may have more network
devices and traffic flows, and there is a greater possibility of
adding or removing network devices and traffic flows. The
deterministic latency forwarding mechanisms MUST scale to networks of
significant size with numerous network devices and a massive traffic
flows.
The increase or decrease of network devices in large-scale networks
is more frequent than that in LANs. The change of the number of
devices may affect the implementation and adjustment of deterministic
network mechanism, such as the topology discovery, queuing mechanism
and packet replication and elimination. A simple use case to
understand is ultra-low-latency (public) 5G transport networks, which
would require DetNet extend to every 5G base station. For some
network operators, their networks may need to connect to ~100 K base
stations (serving multiple mobile networks operators), and this
number will only increase with 5G.
It is almost impossible to identify individual IP flows at the DetNet
data plane because of the large overhead and resource reservation for
a massive number of flows. DetNet allows the leverage of the flow
aggregation. With the large scaling of the network, proper provision
at the control plane to accommodate such higher aggregation is
required. Individual flows may join and exit the aggregated flow
rapidly which causes the dynamic in identification of the aggregated
DetNet flow. The wildcards and value ranges used in the
identification may have to change in order to ensure the aggregated
flows have compatible deterministic characteristics.
The micro-burst will happen more often due to the massive traffic
flows, so some methods to decrease it are needed.
[I-D.du-detnet-layer3-low-latency] introduces a reference method
requiring a scalable buffer to adjust the speed of sending the
packets, so as to keep a uniform transmission rate, and it also
support the flow aggregation. Moreover, the edge shaping based
solution to reduce the micro-burst may also work to some extent.
4.5. Tolerate Failures of Links or Nodes and Topology Changes
Network link failures are more common in large-scale networks. Path
switching or re-convergence of routing will cause high latency of
packet loss and retransmission, which is usually in seconds before
the network becomes stable again. It is necessary to support certain
mechanisms to adapt to failures of links or nodes and topology
changes.
Liu, et al. Expires 11 March 2023 [Page 10]
Internet-Draft Deterministic Networking September 2022
The change of path or topology poses a higher challenge to packet
replication and elimination. The full disjoint paths when
implementing the Packet Replication, Elimination, and Ordering
Functions (PREOF) gives a better chance of survival when one of the
nodes or links in the path fails. At the same time, it brings the
challenges of finding paths with similar distance and/or number of
hops so that there is enough buffer space to absorb the latency
difference caused by different paths when the scale is large. So, a
method is expected to support flexible planning of multiple paths and
provide a solid foundation for the implementation of PREOF.
4.6. Support Configuration of Multiple Queuing Mechanisms
It is required to provide diversified deterministic service for
various applications in a large-scale network and to support the
corresponding diversified queuing mechanisms (possibly at multiple
DetNet QoS levels). Different queuing mechanisms can provide
different levels of latency, jitter and other guarantees, and there
may be situations where a network device provides multiple queuing
mechanisms at the same time. For example, a network aggregation
device may use the mechanisms specified in [IEEE802.1Qbv] and
[IEEE802.1Qcr], and other mechanisms to forward traffic to different
paths at the same time. By providing a variety of queuing mechanisms
to meet diversified deterministic service requirements, compared with
LAN environment, this demand is particularly prominent in large-scale
networks. There are usually eight traffic classes in TSN enabled
networks. The different queuing mechanisms can be employed to the
queues of one or more of those traffic class. In practice, there may
be more than eight queues or sub-queues to support more complicated
queuing mechanisms.
Accordingly, the configuration for multiple queuing mechanisms is
complicated in large-scale deterministic networks and MUST support
the unified or simplified scheduling and management of multiple
queuing mechanisms. For example, in the distributed scenario with no
controller, the related information of the queuing mechanisms could
be advertised among the domain , including the types and related
algorithms, queue forwarding capability with different levels of
latency and jitter guarantees , etc. In the centralized scenario,
the queuing mechanisms and other information could be reported to the
controller to build a deterministic network resource topology pool
for path calculation.
In addition, for refined management of forward resources and
providing resource assurance for deterministic forwarding when
establishing/ deleting sessions, it is required to establish unified
mechanisms on quantification and measurement of resources associated
with interfaces and queues.
Liu, et al. Expires 11 March 2023 [Page 11]
Internet-Draft Deterministic Networking September 2022
4.7. Support Queuing Mechanisms Switchover Crossing Multi-domains
In large-scale deterministic networks, it may across multiple network
domains and adopt a variety of different queuing mechanisms within
each domain. It is required to support the inter-domain
deterministic mechanism at the inter-domain boundary nodes such as
the priority redefinition and rescheduling of queues to achieve the
end-to-end latency, bounded jitter and packet loss ratio.
Moreover, changing from one queuing mechanism to another may generate
additional end-to-end latency and/or jitter which should be taken
into consideration,because the different scheduling granularities or
phase differences between the two domains requires flexible flow
aggregation and queue stitching function. For example, when a flow
is forwarded across multiple network domains based on different
queuing mechanisms, such as a time synchronous Qbv mechanism
[IEEE802.1Qbv] and an asynchronous Qcr mechanism [IEEE802.1Qcr], a
collaboration mechanism crossing multi-domains MUST be considered,
such as increasing the buffer of inter-domain devices to provide
enough adjustment space for the flow to cross different queuing
mechanisms, the expected method of jitter compression to reduce the
coupling between two domains' queuing mechanisms, or the additional
traffic shaping solutions to make the traffic smooth, so as to
provide end-to-end deterministic services across multiple network
domains.
5. Data Plane Enhancement Requirements
According to [RFC8938], the DetNet data plane can provide or carry
two metadata in MPLS and IP data planes: Flow-ID and sequence number.
The Flow-ID could be used for identification of the DetNet flow or
aggregate flow, and the sequence number could be used for PREOF for
each DetNet flow. The Flow-ID is used by both the service and
forwarding sub-layers, but the sequence number is only used by the
service layer. Metadata can also be used for OAM indications and
instrumentation of DetNet data plane operation.
Generally speaking, more data plane metadata and related processing
SHOULD be supported in the large-scale deterministic networks. By
introducing IPv6 Extension Headers [RFC8200] and Segment Routing over
IPv6 [RFC8986], native IPv6 data plane should be supported with the
IPv6-sepcific enhancement. This section lists the data plane
enhancement requirements based on but not limited to the technical
requirements in Section 4, describing how to use IP and/or MPLS, and
related OAM, to support a data plane method of flow identification
and packet treatment over Layer 3.
Liu, et al. Expires 11 March 2023 [Page 12]
Internet-Draft Deterministic Networking September 2022
5.1. Support Aggregated Flow Identification
Current IPv6 aggregated flow identification is generally based on 5
or 6 tuples, IP prefixes, or wildcards as indicated in [RFC8938].
However, in large-scale deterministic networks the number of
individual flows is huge, and they may randomly join and leave the
aggregated flow at each hop. Such behaviours lead to the difficulty
in identifying aggregated flows by relying on the prefixes or
wildcards.
In addition, the deterministic services may demand different
deterministic QoS requirements according to different levels of
application requirements. The flow identification with service-level
aggregation should be supported. Flow identification is also used to
quickly push a packet to a suitable queue. In a large-scale network,
there are large amount of flows requiring deterministic latency
service and normal forwarding service. Explicit flow identification
makes it easier to quickly distinguish the DetNet flows without
requiring the longest match rule on multiple tuples in IP data plane.
Therefore, explicit aggregated flow identification SHOULD be
supported.
5.2. Support Information used by Functions ensuring Deterministic
Latency
According to Section 4.1, a large-scale network should support
synchronized or asynchronized queuing mechanisms. Different queuing
mechanisms require different information to be defined as the DetNet-
specific metadata to help the functions of ensuring deterministic
latency, including regulation, queue management.etc. For instance,
the data plane needs to support the identification of cycle for
cyclic queuing and forwarding or the latency related information for
time based queuing in order to enable the applicability of the
respective queuing and/or scheduling mechanisms in the large scale
network.
When different queuing mechanisms are deployed at a network node,
metadata used for each queuing mechanism should be provided at the
same time. When multiple metadata carried in one packet, metadata
should be self-described and extensible to tolerate variable number
of metadata. Meanwhile, extra data will cause extra processing,
referring to fixed or variable length lookups, total number of read/
write access to the packet header.etc. So the data plane processing
efficiency also needs to be considered when ensuring deterministic
latency, but the specific method or solution is out of scope of this
document.
Liu, et al. Expires 11 March 2023 [Page 13]
Internet-Draft Deterministic Networking September 2022
This document does not specify what metadata and what format to be
carried in data plane. The solution document should be specific
enough on why and how the information carried as data plane meta data
works in conjunction with the queuing or other functions and how it
helps the large scale network deployment.
5.3. Support Redundancy Related Fields
Sequence number is the only metadata currently defined for redundancy
feature of Detnet. MPLS data plane uses Detnet-over-MPLS label stack
to carry it. At the same time, native IPv6 data plane should be able
to carry this information too. If specific IP encapsulation or
tunnel is in use, this meta data should be defined explicitly for
that data plane.
5.4. Support Explicit Path Selection
Explicit route at the control plane and/or management is required so
that the "best" path can be selected to meet the latency requirement
for DetNet flows. At the data planes, MPLS label stack can be used
for this purpose. IP data plane enhancement is required to support
the explicit path selection based on IP source routing or SRv6.
6. Conclusion
This document specifies the technical requirements when ensuring the
deterministic features in the large-scale networks, and the
corresponding data plane enhancement requirements to support the
them. Some of the proposed queuing mechanisms and trials are cited
and the authors of the document think those proposals give reasonably
sound insights to enhancement the current queuing mechanisms to meet
the deterministic requirements of the large-scale networks.
7. Security Considerations
There are no IANA actions required by this document.
8. IANA Considerations
This section will be described later.
9. Acknowledgements
The authors would like to thank Bala'zs Varga, Fan Yang, Tianran
Zhou,Yaakov Stein for helpful suggestions. The authors also would
like to thank Liang Geng, Peter Willis, Shunsuke Homma and Li Qiang
for their previous works.
Liu, et al. Expires 11 March 2023 [Page 14]
Internet-Draft Deterministic Networking September 2022
10. Contributors
The following people have substantially contributed to this document:
Zongpeng Du
China Mobile
EMail: duzongpeng@chinamobile.com
Lei Zhou
New H3C Technologies
Email: zhou.leih@h3c.com
11. Normative References
[Fast-Ethernet-MII-clock]
"Fast Ethernet MII clock".
[G.8262] International Telecommunication Union, "Timing
characteristics of a synchronous equipment slave clock",
ITU-T Recommendation G.8262, November 2018.
[G.8273] International Telecommunication Union, "Framework of phase
and time clocks", ITU-T Recommendation G.8273, March 2018.
[I-D.dang-queuing-with-multiple-cyclic-buffers]
Liu, B. and J. Dang, "A Queuing Mechanism with Multiple
Cyclic Buffers", Work in Progress, Internet-Draft, draft-
dang-queuing-with-multiple-cyclic-buffers-00, 22 February
2021, <https://www.ietf.org/archive/id/draft-dang-queuing-
with-multiple-cyclic-buffers-00.txt>.
[I-D.du-detnet-layer3-low-latency]
Du, Z. and P. Liu, "Micro-burst Decreasing in Layer3
Network for Low-Latency Traffic", Work in Progress,
Internet-Draft, draft-du-detnet-layer3-low-latency-05, 7
July 2022, <https://www.ietf.org/archive/id/draft-du-
detnet-layer3-low-latency-05.txt>.
[I-D.eckert-detnet-bounded-latency-problems]
Eckert, T. and S. Bryant, "Problems with existing DetNet
bounded latency queuing mechanisms", Work in Progress,
Internet-Draft, draft-eckert-detnet-bounded-latency-
problems-00, 12 July 2021,
<https://www.ietf.org/archive/id/draft-eckert-detnet-
bounded-latency-problems-00.txt>.
Liu, et al. Expires 11 March 2023 [Page 15]
Internet-Draft Deterministic Networking September 2022
[I-D.geng-detnet-requirements-bounded-latency]
Geng, L., Willis, P., Homma, S., and L. Qiang,
"Requirements of Layer 3 Deterministic Latency Service",
Work in Progress, Internet-Draft, draft-geng-detnet-
requirements-bounded-latency-03, 7 July 2019,
<https://www.ietf.org/archive/id/draft-geng-detnet-
requirements-bounded-latency-03.txt>.
[I-D.qiang-detnet-large-scale-detnet]
Qiang, L., Geng, X., Liu, B., Eckert, T., Geng, L., and G.
Li, "Large-Scale Deterministic IP Network", Work in
Progress, Internet-Draft, draft-qiang-detnet-large-scale-
detnet-05, 2 September 2019,
<https://www.ietf.org/archive/id/draft-qiang-detnet-large-
scale-detnet-05.txt>.
[IEEE802.1Qav]
IEEE, "IEEE Standard for Local and metropolitan area
networks -- Virtual Bridged Local Area Networks -
Amendment 12: Forwarding and Queuing Enhancements for
Time-Sensitive Streams", IEEE 802.1Qav-2009,
DOI 10.1109/IEEESTD.2010.8684664, 5 January 2010,
<https://doi.org/10.1109/IEEESTD.2010.8684664>.
[IEEE802.1Qbv]
IEEE, "IEEE Standard for Local and metropolitan area
networks -- Bridges and Bridged Networks - Amendment 25:
Enhancements for Scheduled Traffic", IEEE 802.1Qbv-2015,
DOI 10.1109/IEEESTD.2016.8613095, 18 March 2016,
<https://doi.org/10.1109/IEEESTD.2016.8613095>.
[IEEE802.1Qch]
IEEE, "IEEE Standard for Local and metropolitan area
networks -- Bridges and Bridged Networks - Amendment 29:
Cyclic Queuing and Forwarding", IEEE 802.1Qch-2017,
DOI 10.1109/IEEESTD.2017.7961303, 28 June 2017,
<https://doi.org/10.1109/IEEESTD.2017.7961303>.
[IEEE802.1Qcr]
IEEE, "IEEE Standard for Local and Metropolitan Area
Networks -- Bridges and Bridged Networks - Amendment 34:
Asynchronous Traffic Shaping", IEEE 802.1Qcr-2020,
DOI 10.1109/IEEESTD.2020.9253013, 6 November 2020,
<https://doi.org/10.1109/IEEESTD.2020.9253013>.
Liu, et al. Expires 11 March 2023 [Page 16]
Internet-Draft Deterministic Networking September 2022
[IEEE802.1TSN]
IEEE Standards Association, "IEEE 802.1 Time-Sensitive
Networking Task Group",
<https://www.ieee802.org/1/pages/tsn.html>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8200] Deering, S. and R. Hinden, "Internet Protocol, Version 6
(IPv6) Specification", STD 86, RFC 8200,
DOI 10.17487/RFC8200, July 2017,
<https://www.rfc-editor.org/info/rfc8200>.
[RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases",
RFC 8578, DOI 10.17487/RFC8578, May 2019,
<https://www.rfc-editor.org/info/rfc8578>.
[RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas,
"Deterministic Networking Architecture", RFC 8655,
DOI 10.17487/RFC8655, October 2019,
<https://www.rfc-editor.org/info/rfc8655>.
[RFC8938] Varga, B., Ed., Farkas, J., Berger, L., Malis, A., and S.
Bryant, "Deterministic Networking (DetNet) Data Plane
Framework", RFC 8938, DOI 10.17487/RFC8938, November 2020,
<https://www.rfc-editor.org/info/rfc8938>.
[RFC8986] Filsfils, C., Ed., Camarillo, P., Ed., Leddy, J., Voyer,
D., Matsushima, S., and Z. Li, "Segment Routing over IPv6
(SRv6) Network Programming", RFC 8986,
DOI 10.17487/RFC8986, February 2021,
<https://www.rfc-editor.org/info/rfc8986>.
Appendix A. Examples of Large-Scale Deterministic Network Trials
Some trials have been carried out to verify the concept of large-
scale deterministic networks.
Liu, et al. Expires 11 March 2023 [Page 17]
Internet-Draft Deterministic Networking September 2022
In order to verify the deterministic technology of large-scale
networks, a trial of Deterministic IP on China Environment for
Network Innovations (CENI), which is a network built for new network
technology trial, was deployed. A network with a distance of 3,000
km over 13 hops was tested, and the jitter was controlled within
100us.
In order to verify the remote control on Deterministic IP, which
required that the latency should be controlled within 4 ms and jitter
should be controlled within 20 us. A trial cooperated with Baosteel
spanned 600 km was deployed. Baosteel is a Chinese steel company and
put forward this demand. Both of the first and second trials are
based on a frequency synchronization solution. The mechanism details
could be found in . [I-D.dang-queuing-with-multiple-cyclic-buffers][I
-D.qiang-detnet-large-scale-detnet].
In order to realize multi flows synchronization on an inter-
provincial network in an exhibition, Emergen proposed the requirement
that two flows of video and virtual reality (VR) were sent from
province A, and arrived at province B together, so people can see the
synchronization of video collected by camera and the VR model. This
requirement was proposed to facilitate the virtual industry product
deployment. Due to time and other problems, it was realized by the
edge network device for a relatively lower levels of service level
agreement (SLA).
Teaming up with a smart factory operator, network operators,
equipment companies, and universities, ETRI demonstrated an ultra-low
latency, high-reliability 5G wired and wireless network-based remote
industrial Internet of Things (IIoT) service by connecting a control
center and a smart factory through three different operators'
networks at a distance of 280 km. In this trail, it was demonstrated
that real-time remote smart manufacturing service is possible by
making round-trip delay below 3 ms within a smart factory and below
10 ms between remote 5G industrial devices. In the future, the team
plans to examine feasibility of large-scale deterministic networking
by connecting smart factories in Gyeongsan, South Korea and Oulu,
Finland.
These trials show that both operators and enterprise users begin to
put forward requirements for the certainty of large-scale networks,
but the implementation technologies are not exactly the same.
Authors' Addresses
Liu, et al. Expires 11 March 2023 [Page 18]
Internet-Draft Deterministic Networking September 2022
Peng Liu
China Mobile
Beijing
100053
China
Email: liupengyjy@chinamobile.com
Yizhou Li
Huawei
Nanjing
210012
China
Email: liyizhou@huawei.com
Toerless Eckert
Futurewei Technologies USA
Santa Clara, 95014
United States of America
Email: tte@cs.fau.de
Quan Xiong
ZTE Corporation
Wuhan
430223
China
Email: xiong.quan@zte.com.cn
Jeong-dong Ryoo
ETRI
Daejeon
34129
South Korea
Email: ryoo@etri.re.kr
Shiyin Zhu
New H3C Technologies
Beijing
100094
China
Email: zhushiyin@h3c.com
Liu, et al. Expires 11 March 2023 [Page 19]
Internet-Draft Deterministic Networking September 2022
Xuesong Geng
Huawei
Beijing
100095
China
Email: gengxuesong@huawei.com
Liu, et al. Expires 11 March 2023 [Page 20]