Skip to main content

Resilient Cycle Queuing and Forwarding

Document Type Active Internet-Draft (individual)
Authors Rubing Liu , zuopin cheng
Last updated 2024-05-24
RFC stream (None)
Intended RFC status (None)
Stream Stream state (No stream defined)
Consensus boilerplate Unknown
RFC Editor Note (None)
IESG IESG state I-D Exists
Telechat date (None)
Responsible AD (None)
Send notices to (None)
DetNet                                                         R. Liu
Internet Draft                                               Z. Cheng
Intended status: Informational                                    H3C
Expires: November 24, 2024
                                                          May 24, 2024

                  Resilient Cycle Queuing and Forwarding


   The document proposes a solution called Resilient Cycle Queuing and
   Forwarding (RCQF) for DetNet extremely low data loss rates, bounded
   latency and minimal jitter. RCQF is a technological advancement built
   upon Cycle Queuing and Forwarding (CQF), designed to provide a
   capability for the delivery of end-to-end DetNet flows. RCQF extends
   the capabilities of CQF and makes it suitable for large scaling
   networks. The elasticity of RCQF includes adaptive bonded latency,
   minimal jitter, high bandwidth, large packet size, interface speed,
   and only requires frequency synchronization. This document aims to
   introduce the concept of RCQF and its potential in providing
   resilient and adaptable solutions for Deterministic Networks.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79. This document may not be modified,
   and derivative works of it may not be created, and it may not be
   published except as an Internet-Draft.

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79. This document may not be modified,
   and derivative works of it may not be created, except to publish it
   as an RFC and to translate it into languages other than English.

   This document may contain material from IETF Documents or IETF
   Contributions published or made publicly available before November 10,
   2008. The person(s) controlling the copyright in some of this
   material may not have granted the IETF Trust the right to allow
   modifications of such material outside the IETF Standards Process.
   Without obtaining an adequate license from the person(s) controlling
   the copyright in such materials, this document may not be modified

Liu, et al            Expires November 24, 2024               [Page 1]
Internet-Draft                  RCQF                         May 2024

   outside the IETF Standards Process, and derivative works of it may
   not be created outside the IETF Standards Process, except to format
   it for publication as an RFC or to translate it into languages other
   than English.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at

   The list of Internet-Draft Shadow Directories can be accessed at

   This Internet-Draft will expire on November 24, 2024.

Copyright Notice

   Copyright (c) 2024 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   ( in effect on the date of
   publication of this document. Please review these documents carefully,
   as they describe your rights and restrictions with respect to this
   document. Code Components extracted from this document must include
   Simplified BSD License text as described in Section 4.e of the Trust
   Legal Provisions and are provided without warranty as described in
   the Simplified BSD License.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   ( in effect on the date of
   publication of this document. Please review these documents carefully,
   as they describe your rights and restrictions with respect to this

Table of Contents

Liu, et al.           Expires November 24, 2024               [Page 2]
Internet-Draft                  RCQF                         May 2024

   1. Introduction...................................................3
      1.1. Requirements Language.....................................4
   2. Problem Statement..............................................4
   3. Network Model..................................................4
      3.1. Explicit Path.............................................4
      3.2. Queue Mapping.............................................5
      3.3. Periodic Forwarding.......................................5
      3.4. Elastic Scheduling........................................5
      3.5. Edge-to-Edge Collaboration................................5
   4. Scheduling cycle...............................................6
   5. Dynamic Queue Sharing..........................................7
   6. OAM............................................................9
   7. Security Considerations.......................................10
   8. IANA Considerations...........................................10
   9. References....................................................10
      9.1. Normative References.....................................10
      9.2. Informative References...................................11
   10. Acknowledgments..............................................11

1. Introduction

   As an extension of CQF technology, RCQF has the following specific

   o Queue Flexibility: RCQF allows for flexible definition of queue
      parameters such as queue quantity, queue length, and cycle width,
      enabling adjustment of queue parameters according to requirements.

   o Dynamic Load Sharing: In the event of uneven packet distribution
      within the queue, dynamic shaping and load-sharing operations
      enable smooth packet transmission to prevent congestion within the

   o Latency Compensation: During the process of data forwarding, the
      latency and jitter of data transmission constantly fluctuate. By
      utilizing compensation mechanism, it is possible to maintain end-
      to-end packet latency and jitter within a baseline value, thereby
      enabling controlled upper limits for latency and jitter.

   o Loose Coupling: RCQF only requires frequency synchronization
      without the need for phase synchronization. As a result, it
      reduces the system's dependency on clock synchronization
      technology and subsequently lowers the difficulty of deterministic

Liu, et al.           Expires November 24, 2024               [Page 3]
Internet-Draft                  RCQF                         May 2024

1.1. Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

2. Problem Statement

   Deterministic Networking presents several challenges in terms of
   planning and scheduling. Firstly, the diversity of traffic models,
   including variations in traffic burstiness, packet sizes, quantities,
   and inter-packet intervals, poses significant challenges for planning
   and scheduling. Secondly, there are challenges associated with
   forwarding multiple services on a single device due to variations in
   device types and performance, such as differences in crystal
   oscillator accuracy, port rates, buffer resources, and system
   performance, which make it challenging to provide timely and accurate
   forwarding for different services. Lastly, achieving efficient
   collaboration across the entire network is also a challenge, given
   the differences in nodes throughout the network, including variations
   in clock synchronization capabilities, interconnection, and
   adaptation, all of which pose significant challenges for achieving a
   unified and efficient end-to-end deterministic transmission system.

   While the periodic forwarding mechanism of CQF can meet the low-
   latency and low-jitter requirements of deterministic services, it is
   too complex in terms of queue control and configuration. CSQF
   requires the prior specification of mapped queues. In the event of
   topological or traffic changes resulting in burst traffic or an
   overload of packets, the pre-specified queues may overflow. There is
   a lack of load balancing between queues, which makes it inflexible
   and unsuitable for upper-layer services and network planning.

3. Network Model

   RCQF guarantees end-to-end deterministic latency and jitter through
   the following mechanisms, while also possessing elasticity and
   dynamic adjustment capabilities to optimize overall system

3.1. Explicit Path

   By collecting network topology and delay information, RCQF computes
   end-to-end data forwarding paths that satisfy different application
   latency requirements.

Liu, et al.           Expires November 24, 2024               [Page 4]
Internet-Draft                  RCQF                         May 2024

3.2. Queue Mapping

   By embedding forwarding node, interface, queue, and other information
   in the SRv6 SID, RCQF establishes a mapping relationship between
   deterministic service flows and queues to guide deterministic traffic
   forwarding through specific queues.

3.3. Periodic Forwarding

   Network nodes perform queue periodic forwarding according to
   specified interfaces, queues, and periods based on relevant
   information in the SRv6 SID.

3.4. Elastic Scheduling

   RCQF achieves elasticity and dynamic adjustment through dynamic load
   balancing between queues, delay compensation, and other operations.

3.5. Edge-to-Edge Collaboration

   Through control plane coordination, the controller facilitates
   client-side and network-side collaboration to dynamically adjust
   forwarding paths, queues, windows, and other resources in the network,
   thereby improving the efficiency of network data transmission.

Liu, et al.           Expires November 24, 2024               [Page 5]
Internet-Draft                  RCQF                         May 2024

4. Scheduling cycle

      +---------------+              Sector number
      |  Quece0       |
      +---------------+         +--+--+--+--+--+--+--+--+
      +---------------+         |  |  |  |  |  |  |  |  |
      |  Quece1       |         |Q0|Q1|Q2|Q3|Q4|Q5|Q6|Q7|
      +---------------+         |  |  |  |  |  |  |  |  |
      +---------------+         +--+--+--+--+--+--+--+--+
      |  Quece2       |
      +---------------+                 A Cycle
      |  Quece3       |
      +---------------+                   ----
      +---------------+               ///-    -\\\
      |  Quece4       |              /            \
      +---------------+             /              \
      +---------------+            |                |
      |  Quece5       |            |        --------|
      +---------------+            |                |
      +---------------+            |                |
      |  Quece6       |             \              /
      +---------------+              \            /
      +---------------+               \\\-    -///
      |  Quece7       |                   ----

                         Figure 1 Scheduling Cycle

   As shown in the above figure, one circle represents one scheduling
   cycle, and one sector represents one time slot which can scheduling
   the queue. The entire scheduling process can be broken down into the
   following parameters:

   1. Plan multiple queues based on the characteristics of the flows,
      such as the quantity and bandwidth of flows, i.e., pre-define the
      number of queues.

   2. Set the queue length based on the attributes of the flows (e.g.,
      flow size) and available resources (e.g., buffer resources).

   3. Set the sectors of a cycle based on the number of queues;
      typically, the number of queues and the sector of a cycle are

Liu, et al.           Expires November 24, 2024               [Page 6]
Internet-Draft                  RCQF                         May 2024

   4. Define the time of the sector based on the business requirements
      (e.g., end-to-end jitter accuracy of less than 10 microseconds)
      and hardware conditions (e.g., crystal oscillator and scheduling
      accuracy). This refers to the time duration occupied by each
      sector (time slot), such as 10 microseconds. If set to 10
      microseconds, it means that deterministic packets can be sent
      within 10 microseconds. For instance, for a 10G interface,
      approximately 12,000 bytes of packets can be sent within 10

   Once these parameters are set, a deterministic flow will be mapped to
   one or multiple queues. These queues are associated with sectors,
   with each sector being a part of the entire scheduling cycle (i.e., a
   time slot). The scheduling cycle is continuously rotating, and the
   change of cycles is alternated. For example, if the scheduling cycle
   is divided into 8 parts, then there will be 8 sectors. If the width
   of a single sector is 10 microseconds, then one complete rotation of
   the scheduling cycle will take 80 microseconds. In other words,
   within 1 second, the scheduling cycle can complete 12.5 rotations.
   From the operational mechanism described above, it can be concluded
   that by adjusting parameters such as "queue length," "queue
   quantity," "sector quantity," and "sector width," the scheduling
   strategy can be changed to accommodate different application

   For instance, if the end-to-end jitter requirement of an application
   is low, the time of a single sector can be longer, enabling more
   packets to be sent within a single sector. This approach has two
   advantages: 1. The system's forwarding efficiency increases because
   the frequency of cycle switching within a unit of time decreases,
   thereby reducing the overhead caused by cycle switching; 2. More
   flows and packets can be aggregated, as a longer sector width allows
   for the transmission of more packets within a single sector, thus
   enabling more flows and packets to be aggregated for forwarding
   within a single sector.

   Furthermore, by increasing the number of queues and sectors,
   different flows can be mapped to different queues to better support a
   greater number of flows and provide dedicated queues for specific

5. Dynamic Queue Sharing

   Packets are received from the incoming port, processed internally,
   and then forwarded from the outgoing port. To achieve deterministic
   flow, the process is going to involve: 1. entering the RCQF Queue, 2.
   mapping the RCQF Queue to the corresponding sector, and 3. the

Liu, et al.           Expires November 24, 2024               [Page 7]
Internet-Draft                  RCQF                         May 2024

   scheduler cyclically schedules (scheduling cycle) according to
   deterministic scheduling. In this process, some queues may be idle
   while others are busy, and this mapping to the cycle schedule results
   in some sectors being idle while others are busy.

   To address the uneven scheduling of cycles, this patent introduces a
   method of mutual sharing as follows:

   1. Set a "Sector threshold," for example, the threshold value can be
      set as 50% of the maximum capacity of a single sector for

   2. Determination of exceeding the threshold: When it is determined
      that the threshold is exceeded, the sector is traced back to the
      queue, and based on the occupancy of the queue, it is determined
      whether the threshold is exceeded, and the corresponding queue is
      marked if the threshold is exceeded.

   3. Mutual sharing of scheduling between sectors: Based on the above
      exceeding criteria, mutual sharing of scheduling between adjacent
      sectors is performed.

   4. For the sharing in the third step, it can be moved forward. While
      forwarding Sector 0, if it does not exceed the threshold, the
      status of Queue 1 is evaluated. If Queue 1 exceeds the threshold,
      and there is remaining time after forwarding the packets from
      Queue 0, the packets are taken from Queue 1 and sent externally.
      This helps alleviate the burden on Sector 1.

   5. In addition to forward and backward shifting, the stepping can
      also be adjusted. If the step size is set to 2, it can be shifted
      two sectors forward or backward. As shown in the diagram, if it is
      set to shift two sectors forward, when Sectors 4 is scheduled, it
      can take into account the threshold mark for Queue 6 (6 = 4 + 2).
      Since Queue 6 exceeds the threshold and has a large number of
      packets, after scheduling the packets from Queue 4, the remaining
      time can be used to schedule the packets from Queue 5, thereby
      reducing the pressure on Sector 5 and allowing more Queues 6
      packets to be scheduled later in Sector 5.

   6. By adjusting and optimizing the above methods such as thresholds,
      sharing, forward and backward shifting, and step size, it is
      possible to achieve queue shaping, sharing between sectors, and
      smoother output of packets, thereby improving overall forwarding
      efficiency and resource utilization.

Liu, et al.           Expires November 24, 2024               [Page 8]
Internet-Draft                  RCQF                         May 2024

   RCQF first proposed the Cycle-level sharing scheme, which has higher
   forwarding efficiency and resource utilization. It is more friendly
   to control plane resource planning, topology, and traffic changes.
   Additionally, RCQF's sharing processing occurs during the cycle
   scheduling period, rather than during the queue entry period.

6. OAM

   The core function of Operations, Administration, and Maintenance (OAM)
   is to detect time slot deviations, where forwarding nodes map time
   slot deviations to the egress interface queues of service packets,
   thereby guiding packet forwarding. Deterministic networking
   technology uses maximum time slot deviations to guide forwarding,
   ultimately achieving deterministic latency jitter.

                 |                                  |
           +-----+         SDN Controller           +-------+
           |     |                                  |       |
           |     +----------+-----------------+-----+       |
           |                |                 |             |
           |                |                 |             |
           |                |                 |             |
      t1=Qm2-Qm1      t2=Qn-Qm2        t3=Qx-Qn|       t4=Qy-Qx
           |                |                 |             |
       +---+----+      +----+---+        +----+---+     +---+----+
   Qm1 |        | Qm2  |        |Qn      |        |Qx   |   |    |Qy
       +   PE1  +------+   P1   +--------+   P2   +-----+  PE2   +
       |        |      |        |        |        |     |        |
       +--------+      +--------+        +--------+     +--------+

                            Figure 2 DetNet OAM

   As shown in Figure, the workflow of OAM detecting the maximum time
   slot deviation between the source node and adjacent nodes is as

   The ingress node of the SRv6 path periodically simulates the
   generation of OAM detection packets for traffic, with 10 detections
   performed per period. The detection packets are forwarded along the
   Segment List.

   The ingress node on the Segment List detects the time slot deviations
   between the egress and ingress interface queues of this device for
   the detection packets (t1) and other nodes calculate the time slot
   deviations for egressing the detection packets from this node to the

Liu, et al.           Expires November 24, 2024               [Page 9]
Internet-Draft                  RCQF                         May 2024

   upstream node (t2, t3, and t4) and upload the maximum, minimum, and
   average time slot deviation during the period to the controller.

   The controller selects the maximum time slot deviation within the
   detection period, generates a list of time slot deviations (t1, t2,
   t3, t4), and sends the list of time slot deviations to the ingress

   The ingress node encapsulates the list of time slot deviations in the
   SRv6 packet header to guide packet forwarding.

   [I-D.guo-detnet-jitter-reduction-mechanism] describes a compensation
   mechanism designed to reduce end-to-end jitter and meet the
   requirements of applications with tightly bounded jitter. The
   compensation mechanism can be a part of RCQF and could achieve on

7. Security Considerations


8. IANA Considerations

   This document has no IANA actions.

9. References

9.1. Normative References

   [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas,
             "Deterministic Networking Architecture", RFC 8655, DOI
             10.17487/RFC8655, October 2019, <https://www.rfc-

   [RFC8938] Varga, B., Ed., Farkas, J., Berger, L., Malis, A., and S.
             Bryant, "Deterministic Networking (DetNet) Data Plane
             Framework", RFC 8938, DOI 10.17487/RFC8938, November 2020,

Liu, et al.           Expires November 24, 2024              [Page 10]
Internet-Draft                  RCQF                         May 2024

9.2. Informative References

   [I-D.ietf-detnet-scaling-requirements] Liu, P., Li, Y., Eckert, T. T.,
             Xiong, Q., Ryoo, J., Yin, Z., and Geng, X., "Requirements
             for Scaling Deterministic Networks", Work in Progress,
             Internet-Draft, draft-ietf-detnet-scaling-requirements-05,
             20 November 2023,

   [I-D.chen-detnet-sr-based-bounded-latency] Chen, M., Geng, X., Li, Z.,
             Joung, J., and J. Ryoo, "Segment Routing (SR) Based Bounded
             Latency", Work in Progress, Internet-Draft, draft-chen-
             detnet-sr-based-bounded-latency-03, 7 July 2023,

   [I-D.guo-detnet-jitter-reduction-mechanism] Guo, D., Xu, S., "Jitter
             Reduction Mechanism for DetNet",Work in Progress, Internet-
             Draft, draft-guo-detnet-jitter-reduction-mechanism-01, 23
             October 2023, <

   [I-D.guo-detnet-vpfc-planning] D. Guo, G. Wen, K. Yao, Q. Xiong, G.
             Peng, "Deterministic Networking (DetNet) Controller Plane -
             VPFC Planning Information Model Based on VPFP in Scaling
             Deterministic Networks", Work in Progress, Internet-Draft,
             draft-guo-detnet-vpfc-planning-03, 5 January 2024,

10. Acknowledgments


Liu, et al.           Expires November 24, 2024              [Page 11]
Internet-Draft                  RCQF                         May 2024

Authors' Addresses

   Rubing Liu
   New H3C Technologies Co., Ltd
   Hangzhou, China

   Zuopin Cheng
   New H3C Technologies Co., Ltd
   Hangzhou, China

Liu, et al.           Expires November 24, 2024              [Page 12]