Deterministic Networking (DetNet) Data Plane - Tagged Cyclic Queuing and Forwarding (TCQF) for bounded latency with low jitter in large scale DetNets

1.1. Terminology

CQF

Cyclic Queuing and Forwarding. A queuing mechanism defined by annex T of [IEEE802.1Q].¶

DT

Dead Time. A term from CQF indicating the time during each cycle in which no frames can be sent because the the receiving node could not receive it into the desired cycle buffer.¶

TCQF

Tagged Cyclic Queuing and Forwarding. The mechanism specified in this memo.¶

2.1. Cyclic Queuing and Forwarding (CQF)

Cyclic Queuing and Forwarding (CQF) is a bounded (guaranteed) per-hop latency forwarding mechanism standardized for use in ethernet switched networks by the IEEE TSN working group originally via [IEEE802.1Qch] (802.1 Qch), which later became annex T of [IEEE802.1Q]. See also [RFC9320], Section 6.6.¶

CQF is not a separate forwarding mechanism, but it is simple a profile of the IEEE Time Aware Shaper (TAS) standard, [IEEE802.1Qbv], which introduce Time-Gated Queues.¶

CQF uses a two-queue based forwarding mechanism on every switch along a path between a sender and receiver. One queue is used to receive and store frames destined toward a particular outgoing interface on the switch, the other queue is used simultaneously to send frames to the same outgoing interface. At every cycle time T_c interval these two queues are swapped, or in terms of Time-Gated Queus, one is closed for sending, the other is opened for sending. This operation is synchronized across all switches in the network by network wide synchronized clocks, so that all queues open and close at the same time.¶

For a path of h hops, the end-to-end latency bound is between (h-1) * T_c + DT and (h+1) * T_c. DT is the so-called dead time at the end of a cycle during which no frames can be transmitted from the sending queue to ensure that the last byte of the last frame will be received earlier than the end of the same cycle on the receiving switch.¶

A core contributor to DT is the (physical) link between the sending and receiving switch. DT needs to be larger than the latency of this link, including physical propagation latency (speed of light), possible error correction latencies, and interface serialization latency.¶

T_c needs to be choosen carefully: The larger it is, the higher the bounded latency. The smaller it is, the fewer bytes (and hence frames) will fit into a cycle.¶

To admit flows into a CQF network, the ingress switch uses per-flow Time-Gated Queues. In the most simple case, such a gate is configured to admit a maximum amount of bytes from the flow into every cycle. More advanced admission control can be performed for bursty flows. For example N bursty flows f_i = 0...(N-1) could share admitted bandwidth by each having their burst admitted in different cycles c_i = c % N + c_i, where c is a continuous increasing cycle number.¶

2.2. Benefits of CQF with higher speed links

The typical CQF deployments in manufacturing networks with 1Gbps links uses no less than hundreds of microseconds as a cycle interval. In a network with a small diameter, say less than 8 hops, it is sufficiently good to provide an end-to-end latency bound in the order of several milliseconds.¶

With the increasing of link speed from 100Mbps to 1Gbps, 10Gbps, 100Gbps or even higher in larger networks, either more bytes can be transmitted within the same cycle interval or the smaller cycle interval is required to transmit the same amount of bytes in a cycle as that in low speed networks. Likewise, the serialization latency reduces with higher speed links and DT reduces. This overall makes CQF for higher speed networks more attractive than for lower speed networks.¶

Figure 1 shows a simple calculation on the number of bytes that can be transmitted in a cycle with different cycle intervals and link speeds. A minimum of 1500 bytes is labeled with * as a baseline because a typical maximum Ethernet frame is 1500 bytes and a selected cycle interval should at least allow one such frame size to be transmitted unless otherwise specified.¶

TBD: These numbers probbly need to be adjusted to reflect reducing DT based on serialization latency.¶

+----------+------------------------------------------------+
|          |           Bytes Transmitted in a Cycle         |
|Cycle Time+------------------------------------------------+
|          |             Link Speed                         |
|  (us)    |   100Mbps  |   1Gbps     |   10Gbps  | 100Gbps |
+----------+------------+-------------+-----------+---------+
|    1     |    12.5    |    125      |    1250   |   12500*|
+----------+------------+-------------+-----------+---------+
|   1.2    |     15     |    150      |    1500*  |   15000 |
+----------+------------+-------------+-----------+---------+
|    2     |     25     |    250      |    2500   |   25000 |
+----------+------------+-------------+-----------+---------+
|    4     |     50     |    500      |    5000   |   50000 |
+----------+------------+-------------+-----------+---------+
|    10    |    125     |   1250      |   12500   |  125000 |
+----------+------------+-------------+-----------+---------+
|    12    |    150     |   1500*     |   15000   |  150000 |
+----------+------------+-------------+-----------+---------+
|   120    |    1500*   |   15000     |   150000  | 1500000 |
+----------+------------+-------------+-----------+---------+

When the link speed is at 10Gbps, the cycle interval could be as small as 1.2 us if a 1500 byte frame needs to be transmitted in one cycle interval, and with 100Gbps links even 1 usec cycle time allows for 8 frames of 1500 byte each. These are not accurate calculations because there are certainly other factors to determine the cycle interval. However, it shows that as the link speed increases, cycle interval can be greatly reduced in practice while satisfying the minimum amount of data transmitted in a single cycle. The end-to-end latency bound when applying CQF is determined by cycle interval and number of hops. That is to say, CQFs with a smaller cycle interval have the potential to meet more strict end-to-end latency requirements in higher link speed networks or meet the same end-to-end latency requirement in networks with much larger network diameter (number of hops).¶

Industry automation has some typical application period requirement, e.g. 100 us to 2 ms for isochronous traffic, 500 us to 1 ms for cyclic-synchronous and 2 to 20 ms for cyclic-asynchronous traffic. The network cycle interval is usually a fraction of the application period. When the cycle interval is in the order of tens of microseconds, CQF can be used to meet the most strict end-to-end latency requirements. For instance, if we assume the number of hops is 24, when cycle interval is set to 10us, the end-to-end latency bound can be around (24+1)*10 = 250 us which has the potential to meet the latency bound requirement for isochronous traffic.¶

In summary a higher speed network makes the shorter cycle interval feasible because sufficiently large traffic volume can be transmitted within one cycle interval. A shorter cycle interval further offers shorter end-to-end latency and jitter bounds which provide CQF with the potentials to meet more strict latency requirements in wider deployments while preserving its simplicity of latency calculation and provisioning. Therefore there is a strong motivation to leverage CQF and at the same time to make cycle interval as short as possible.¶

2.3. Challenges of CQF with higher latency links

Unlike the original targets for IEEE TSN work, DetNet not only targets to support IETF forwarding planes (IP, MPLS,...), but also wide-area networks with therefore longer physcial propagation latencies.¶

As shown in Figure 2 for fundamental (two buffer) CQF, the last byte sent by node A in cycle (i-1) has to be ready for sending at node B before the start of cycle i. To realize it, DT or dead time is imposed. It is a time interval usually at the end of a cycle so that a node should not send the scheduled CQF packets.¶

Dead time is at least the sum of the maximum propagation delay to the next node, the maximum processing delay at the next node and the maximum other time variations. Therefore either the longer propagation or longer processing delay makes dead time larger. Packets from DetNet service is likely to be propagated over long links in the wider area. It takes around 5us per kilometer to propagate, i.e. 0.5ms every hundred kilometers. Hence the dead time can be as large as milliseconds or tens of milliseconds in case of hundred kilometers of longer links and larger processing delays. That would make the dead time eat up most of the cycle interval when cycle interval is short (e.g., at the same order or one order higher of magnitude in time as dead time). Then the useful time in a cycle will be much reduced. In some extreme cases, when the link is long and the cycle interval is set to extremely short, the first packet sent in a cycle by a node will not be possibly received in the same cycle interval at the next node. That makes the useful time in a cycle reaches zero in two buffer CQF. Then two buffer CQF will be no longer suitable.¶

In result of these considerations, reasonable limits for the size of TSN CQF networks are in the order of at most few Km per hop, beyond which DT exceeds common cycle times and possible through of CQF traffic is hence 0.¶

--------------------------------------------------------> Time

          |             |             |             |
Node A    |  cycle i-1  |   cycle i   |  cycle i+1  |
          |             |             |             |
Sending  ---------------+----------------------------------
          |+------+     |+------+     |+------+     |
          ||//////|     ||//////|     ||//////|     |
          |+------+     |+------+     |+------+     |
          |  buf_1      |  buf_2      |  buf_1      |
          |       |     |       |     |       |     |
          |       | DT  |       | DT  |       | DT  |
Node B    |       |<--->|       |<--->|       |<--->|
          |             |             |             |
Receiving--------------------------------------------------
          |     +------+|     +------+|     +------+|
          |     |//////||     |//////||     |//////||
          |     +------+|     +------+|     +------+|
          |       buf_1 |       buf_2 |       buf_1 |
          |             |             |             |
          |             |             |             |
Node B    |             |             |             |
          |             |             |             |
Sending  --------------------------------------------------
          |             |+------+     |+------+     |
          |             ||//////|     ||//////|     |
          |             |+------+     |+------+     |
          |             |  buf_1      |  buf_2

DT=Dead Time

2.4. Review of CQF benefits and challenges for DetNet

In review, CQF has a range of benefits for DetNet.¶

1. It provides bounded latency.¶

2. It provided tightly bounded jitter.¶

3. It has a very simple and easily standardized calculus for its bounded latency and jitter.¶

4. It has very simple per-hop forwarding machinery (cyclic queues) easily supportable in high-speed network equipment.¶

5. Like Diffserv forwarding, it does not use per-hop,per-flow state in the forwarding plane and therefore does not require per-hop,per-flow signaling with the DetNet controller-plane, allowing it to scale to large number of flows.¶

6. The faster the links are, the lower the per-hop latency impact of the cyclic queuing mechanism.¶

The core limitation of CQF, which TCQF intends to solve, lies in its use of arrival time clock to determine the cycle into which the packet is to be placed, see [I-D.eckert-detnet-bounded-latency-problems] for more details.¶

1. Cycles times should be as short as feasible to support lower end to end latency (Section 2.2).¶

2. When networks have longer links, or links with higher propagation jitter as in Metro and WAN, this increases the dead time, and hence reduces the possible utilization or need to increase cycle times.¶

3. When shorter cycle times are feasible because of higher speed links, this would require an increase in clock-synchronization accuracy.¶

2.5. Tagged CQF

Tagging of CQF packets with cycle identifiers can be used to solve the dilemma aforementioned with minor changes to the fundamental two buffer CQF. This section introduces this mechanism with multipl buffers and CQF cycle identification in the packet header. Note that we are also now using the term packet (as used for IP, MPLS and other IETF forwarding planes) and buffers for packets, as opposed to frames as used by IEEE.¶

--------------------------------------------------------> Time

          |             |             |             |
Node A    |  cycle i-1  |   cycle i   |  cycle i+1  |
          |             |             |             |
Sending  ---------------+----------------------------------
          |+----------+ |+----------+ |+----------+ |
          ||//////////| ||//////////| ||//////////| |
          |+----------+ |+----------+ |+----------+ |
          |  buf_1      |  buf_2         buf_3      |
          |           | |           | |           | |
          |         ->| |<-       ->| |<-       ->| |<-
          |            DT            DT            DT
          |
          -------------------------------------------------
Node B    |     +-----------+ +-----------+ +-----------+
          |     |///////////| |///////////| |///////////|
Receiving |     +-----------+ +-----------+ +-----------+
          |       buf_1 |       buf_2 |       buf_3 |
          |             |             |             |
          |             |             |             |
          |             |             |             |
          |             |             |             |
         ---------------|----------------------------------
Node B    |             |             |+----------+ |+----------+
          |             |             ||//////////| ||//////////|
Sending   |             |             |+----------+ |+----------+
          |             |                buf_1         buf_2

DT=Dead Time

--------------------------------------------------------> Time

           |        |        |        |        |
Node A     | cycle  | cycle  | cycle  | cycle  |
           |  i-1   |   i    |  i+1   |  i+2   |
Sending   ----------+--------+--------+--------+
           |+-----+ |+-----+ |+-----+ |+-----+ |
           ||/////| ||/////| ||/////| ||/////| |
           |+-----+ |+-----+ |+-----+ |+-----+ |
           | buf_1  | buf_2  | buf_3  | buf_4  |
           |      | |      | |      | |      | |
           |    ->| |<-  ->| |    ->| |    ->| |
           |      DT       DT       DT       DT
           |
          --------------------------------------
           |      +-----------+receiving window
Node B     |      |///////////|for A's buf_1
           |      +-----------+
Receiving  |    put to B's buf_1
           |
           |             ->|  |<- ambiguity window 1
           |
           |               +-----------+receiving window
           |               |///////////|for A's buf_2
           |        |      +-----------+
           |        |     put to B's buf_2
           |        |
           |        |             ->|  |<- ambiguity window 2
           |        |        |
           |        |        |      +-----------+receiving window
           |        |        |      |///////////|for A's buf_3
           |        |        |      +-----------+
           |        |        |     put to B's buf_3
           |        |        |
           |        |        |             ..........
           |        |        |
          -|--------|--------|--------|---------------
Node B     |        |        |        |        |        |
           |        |        | +-----+|+-----+ |+-----+ |+-----+
Sending    |        |        | |/////|||/////| ||/////| ||/////|
           |        |        | +-----+|+-----+ |+-----+ |+-----+
           |        |        |  buf_4 | buf_1  | buf_2  | buf_3

DT=Dead Time

2.6. Summary of TCQF benefits and goals for DetNet

TCQF inherits the benefits of CQF for DetNet as outlined in Section 2.4, and byusing a configurable number of three or more cycles, and signaling the cycle as part of a packet header, it resolves these problems as follows.¶

1. With three cycles, TCQF can support arbitrary latency links at arbitrary speeds without reduction of utilization because of longer links or higher link speeds (same cycle time, same clock accuracy, only change in lengths and speeds).¶

2. With four or more cycles, TCQF can also eliminate Dead Time caused by variation of clock synchronization inaccuracies (MTIE) as well as jitter caused by link propagation and processing variation. The sum of cycles times needs to be larger than the total jitter to achieve this.¶

Prior documents describing the concept of TCQF (without using that name) include [I-D.qiang-detnet-large-scale-detnet] and [I-D.dang-queuing-with-multiple-cyclic-buffers]. TCQF does not depend on other elements of [RFC8655], so it can also be used stand alone in otherwise non-deterministic IP/IPv6 or MPLS networks to achieve bounded latency and low jitter.¶

TCQF is likely especially beneficial when networks are architected to avoid per-hop, per-flow state even for traffic steering, which is the case for networks using SR-MPLS [RFC8402] for traffic steering of MPLS unicast traffic, SRv6 [RFC8986] for traffic steeering of IPv6 unicast traffic and/or BIER-TE [I-D.ietf-bier-te-arch] for tree engineering of MPLS multicast traffic by using the TC and/or DSCP header fields of BIER packets according to [RFC8296].¶

In these networks, it is specifically undesirable to require per-flow signaling to non-edge forwarders (such as P-LSR in MPLS networks) solely for DetNet QoS because such per-flow state is unnecessary for traffic steering and would only be required for the bounded latency QoS mechanism and require likely even more complex hardware and manageability support than what was previously required for per-hop steering state (such as in RSVP-TE, [RFC4875]). Note that the DetNet architecture [RFC8655] does not include full support for this DiffServ model, which is why this memo describes how to use TCQF with the DetNet architecture per-hop, per-flow processing as well as without it.¶

4.1. Configuration Data model and tag processing for MPLS TC tags

The following data model summarizes the configuration parameters as required for TCQF and discussed in further sections. 'tcqf' includes the parameters independent of the tagging on an interface. 'tcqf_*' describes the parameters for interfaces using MPLS TC and IP DSCP tagging.¶

# Encapsulation agnostic data
tcqf
+-- uint16 cycles
+-- uint16 cycle_time
+-- uint32 cycle_clock_offset
+-- if_config[oif] # Outgoing InterFace
    +-- uint32 cycle_clock_offset
    +-- cycle_map[iif] # Incoming InterFace
        +--uint8 oif_cycle[iif_cycle]

4.2. Packet processing

This section explains the TCQF packet processing and through it, introduces the semantic of the objects in Figure 6¶

tcqf contains the router wide configuration of TCQF parameters, independent of the specific tagging mechanism on any interface. Any interface can have a different tagging method. This document uses the term router when it is irrelevant whether forwarding is for IP or MPLS packet, and the term Label Switched Router (LSR) to indicate MPLS is used, or IP router to indicate IP or IPv6 are used - independent of the specific encapsulation used for IP or MPLS to carry the cycle identification.¶

The model represents a single TQCF domain, which is a set of interfaces acting both as ingress (iif) and egress (oif) interfaces, capable to forward TCQF packets amongst each other. A router may have multiple TCQF domains each with a set of interfaces disjoint from those of any other TCQF domain.¶

tcqf.cycles is the number of cycles used across all interfaces in the TCQF domain. routers MUST support 3 and 4 cycles. The maximum number of supportable cycles depends on the encapsulation. For example, to support interfaces with MPLS TC tagging, 7 or fewer cycles MUST be used across all interfaces in the CQF domain. See Section 4.3.¶

The unit of tcqf.cycle_time is micro-seconds. routers MUST support configuration of cycle-times of 20,50,100,200,500,1000,2000 usec.¶

Cycles start at an offset of tcqf.cycle_clock_offset in units of nsec as follows. Let clock1 be a timestamp of the local reference clock for TCQF, at which cycle 1 starts, then:¶

tcqf.cycle_clock_offset = (clock1 mod (tcqf.cycle_time * tcqf.cycles) )¶

The local reference clock of the router is expected to be synchronized with the neighboring LSR/router in TCQF domain. tcqf.cycle_clock_offset can be configurable by the operator, or it can be read-only. In either case will the operator be able to configure working TCQF forwarding through appropriately calculated cycle mapping.¶

tcqf.if_config[oif] is optional per-interface configuration of TCQF parameters. tcqf.if_config[oif].cycle_clock_offset may be different from tcqf.cycle_clock_offset, for example, when interfaces are on line cards with independently synchronized clocks, or when non-uniform ingress-to-egress propagation latency over a complex router/LSR fabric makes it beneficial to allow per-egress interface or line card configuration of cycle_clock_offset. It may be configurable or read-only.¶

The value of -1 for tcqf.if_config[oif].cycle_clock_offset is used to indicate that the domain wide tcqf.cycle_clock_offset is to be used for oif. This is the only permitted negative number for this parameter.¶

When a packet is received from iif with a cycle value of iif_cycle and the packet is routed towards oif, then the cycle value (and buffer) to use on oif is tcqf.if_config[oif].cycle_map[iif].oif_cycle[iif_cycle]. This is called the cycle mapping and is must be configurable. This cycle mapping always happens when the packet is received with a cycle tag on an interface in a TCQF domain and forwarded to another interface in the same TCQF domain.¶

This encapsulation independent data model only defines how to map from a received packets cycle to a sending interface cycle buffer and hence sent packet cycle. It does not specify how the cycle identifier is encoded in the received or sent packet. This is amended by the specification in the following sections.¶

This data model does therefore also not determine whether interfaces use IP/IPv6, MPLS or any other encapsulation. This is determined by the configuration of the DetNet domain. A mixed use of MPLS and IP/IPv6 interfaces is possible with this data model, but at the time of writing this document not supported by DetNet.¶

4.3. TCQF for MPLS with TC tagging

This section describes operation of TCQF for MPLS packets using the Traffic Class (TC) field of MPLS label to carry the cycle-id. To support this encapsulation, the TCQF Data Model as defined in Figure 6 is expanded as follows.¶

# MPLS TC tagging specific data
tcqf_tc[oif]
+--uint8 tc[oif_cycle]

tcqf_tc[oif].tc[oif_cycle] defines how to map from the internal cycle number oif_cycle to an MPLS TC value on interface oif. tcqf_tc[oif] MUST be configured, when oif uses MPLS. This oif_cycle <=> tc mapping is not only used to map from internal cycle number to MPLS TC tag when sending packets, but also to map from MPLS TC tag to the internal cycle number when receiving packets.¶

In the terminology of [RFC3270], TCQF QoS as defined here, is TC-Inferred-PSC LSP (E-LSP) behavior: Packets are determined to belong to the TCQF PSC solely based on the TC of the received packet.¶

The internal cycle number SHOULD be assigned from the Top of Stack (ToS) MPLS label TC bits before any other label stack operations happens. On the egress side, the TC value of the ToS MPLS label SHOULD be assigned from the internal cycle number after any label stack processing.¶

With this order of processing, TCQF can support forwarding of packets with any label stack operations such as label swap in the case of LDP or RSVP-TE created LSP, Penultimate Hop Popping (PHP), or no label changes from SID hop-by-hop forwarding and/or SID/label pop as in the case of SR-MPLS traffic steering.¶

4.4. TCQF for IP/IPv6 with DSCP tagging

This section describes operation of TCQF for IP/IPv6 packets using the Differentiated Services Code Point (DSCP) field of IP/IPv6 packets to carry the cycle-id. To support this encapsulation, the TCQF Data Model as defined in Figure 6 is expanded as follows.¶

# IP/IPv6 DSCP tagging specific data
tcqf_dscp[oif]
+--uint8 dscp[oif_cycle]

tcqf_dscp[oif].dscp[oif_cycle] defines how to map from the internal cycle number oif_cycle to an IP/IPv6 DSCP value on interface oif. tcqf_dscp[oif] MUST be configured, when oif uses DSCP tagging of IP/IPv6 packets for TCQF. This oif_cycle <=> idscp mapping is not only used to map from internal cycle number to the DSCP tag when sending packets, but also to map from IP/IPv6 DSCP to the internal cycle number when receiving packets.¶

As how DetNet domains are currently assumed to be single administrative network operator domains, this document does not ask for standardization of the DSCP to use with TCQF. Instead, deployments wanting to use TCQF with IP/IPv6 encapsulation and DSCP tagging need to assign within their domain DSCP from the xxxx11 "EXP/LU" Codepoint space according to [RFC2474], Section 6. This allows up to 16 DSCP for intradomain use and hence up to 16 cycle identifiers.¶

4.5. TCQF for IPv6 with IPv6 Option tagging

This section describes operation of TCQF for IPv6 packets without having to rely on DSCP by defining a new IPv6 option for DetNet. This option is to be placed in the IPv6 HbH (Hop-by-Hop) Options or DOH (Destination Option Header) header. To support this encapsulation, the TCQF Data Model as defined in Figure 6 is expanded as follows.¶

# IPv6 TCQF Option tagging specific data
tcqf_ipv6oh[oif]
+--uint8 ipv6oh[oif_cycle]

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Option Type  |  Opt Data Len |E|    Flags    |   Cycle Id    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.                                                               .
~         (64-bit extension if flag E-bit is 1)                 ~
.                                                               .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

+-----------------------------------+
|         DetNet IP Packet          |
+-----------------------------------+
|            other EHs              |
+-----------------------------------+
|        IPv6 Hop-by-Hop Ex Hdr     |
|         (DIP-TCQF Option)         |
+-----------------------------------+
|            IPv6 Header            |
+-----------------------------------+
|             Data-Link             |
+-----------------------------------+
|             Physical              |
+-----------------------------------+

+-----------------------------------+
|         DetNet IP Packet          |
+-----------------------------------+
|         other EHs including RH    |
+-----------------------------------+
|       IPv6 Destination Ex Hdr     |
|         (DIP-TCQF Option)         |
+-----------------------------------+
|            IPv6 Header            |
+-----------------------------------+
|             Data-Link             |
+-----------------------------------+
|             Physical              |
+-----------------------------------+

4.6. TCQF Pseudocode (normative)

The following pseudocode restates the forwarding behavior of Section 4 in an algorithmic fashion as pseudocode. It uses the objects of the TCQF configuration data model defined in Section 4.1.¶

void receive(pak) {
  // Receive side TCQF - retrieve cycle of received packet
  // from packet internal header
  iif = pak.context.iif
  if (tcqf.if_config[iif]) { // TCQF enabled on iif
    if (tcqf_tc[iif]) {      // MPLS TCQF enabled on iif
      tc = pak.mpls_header.lse[tos].tc
      pak.context.tcqf_cycle = map_tc2cycle( tc, tcqf_tc[iif])
    } else
    if (tcqf_ipv6oh[iif]) {    // IPv6 Option Header used on iif
      cycle_id = pak.ipv6_header.tcqf_oh[cycle_id]
      pak.context.tcqf_cycle =
        map_ipv6oh2cycle( cycle_id, tcqf_ipv6oh[iif])
    } else
    if (tcqf_dscp[iif]) {      // IP DSCP TCQF used on iif
      dscp = pak.ip_header.dscp
      pak.context.tcqf_cycle = map_dscp2cycle( dscp, tcqf_dscp[iif])
    } else // ... other encaps
  }
  forward(pak);
}

// ... Forwarding including any label stack operations

void forward(pak) {
  oif = pak.context.oif = forward_process(pak)

  if(ingres_flow_enqueue(pak))
    return // ingress packets are only enqueued here.

  if(pak.context.tcqf_cycle) // non TCQF packets cycle is 0
    if(tcqf.if_config[oif]) {    // TCQF enabled on OIF
      // Map tcqf_cycle iif to oif - encap agnostic
      cycle = pak.context.tcqf_cycle
            = map_cycle(cycle,
              tcqf.if_config[oif].cycle_map[[iif])

      // MPLS TC-TCQF
      if(tcqf.tc[oif]) {
        pak.mpls_header.lse[tos].tc = map_cycle2tc(cycle, tcqf_tc[oif])
      } else
      if (tcqf_ipv6oh[oif]) { // IPv6 Option Header used on iif
        pak.ipv6_header.tcqf_oh[cycle_id] =
          map_cycle2ipv6oh(cycle, tcqf_ipv6oh[oif])
      } else
      // IP DSCP TCQF enabled on iif
      if (tcqf_dscp[oif]) {
        pak.ip_header.dscp = map_cycle2dscp(cycle, tcqf_dscp[oif])
      } // else...  other future encap/tagging options for TCQF

      tcqf_enqueue(pak, oif.cycleq[cycle,iif])  // [3]
      return
    } else {
      // Forwarding of egress TCQF packets [1]
    }
  }
  // ... non TCQF OIF forwarding [2]
}

// Started when TCQF is enabled on an interface
// dequeues packets from oif.cycleq
// independent of encapsulation
void send_tcqf(oif) {
  cycle = 1
  cc =  tcqf.cycle_time *
        tcqf.cycle_time
  o =   tcqf.cycle_clock_offset
  nextcyclestart = floor(tnow / cc) * cc + cc + o

  while(1) {
    ingress_flow_2_tcqf(oif,cycle) // [5]
    wait_until(tnow >= nextcyclestart); // wait until next cycle
    nextcyclestart += tcqf.cycle_time
    forall(iif) {
      forall(pak = tcqf_dequeue(oif.cycleq[cycle,iif]) {
        schedule to send pak on oif before nextcyclestart; // [4]
      }
    }
    cycle = (cycle + 1) mod tcqf.cycles + 1
  }
}

Processing of ingress TCQF packets is performed via ingres_flow_enqueue(pak) and ingress_flow_2_tcqf(oif,cycle) as explained in Section 5.2.¶

Packets in a cycle buffer can be sent almost arbitrarily within the time period of the cycle. They also do not need to be sent as soon as possible, as long as all will be sent within that period. There is no need to send them in the order of their arrival except that packets from the same ingres flow that end up in the same cycle must not be reordered across any number of tcqf hops. The pseudocode describes this by using a queue oif.cycleq[cycle,iif] ([3]) for all packets from the same iif. The pseudocode describes the oterwise arbitrary scheduling of all packets within the cycle time via the statement shown in [4].¶

Ingress packets are passed from their ingress queues to the next cycle queue via [5].¶

Processing of egres TCQF packets is out-of-scope. It can performed by any non-TCQF packet forwarding mechanism such as some strict priority queuing in step [2], and packets could accordingly be marked with an according packet header traffic class indicator for such a traffic class in step [1].¶

5.1. Ingress Flows Configuration Data Model

# Extends above defined tcqf
tcqf
...
| Ingress Flows, see below (TBD:
+-- iflow[flowid]
    +-- uint32 csize # in bits

The data model shown in Figure 14 expands the tcqf data model from Figure 6. For every DetNet flow for which this router is the TCQF ingress, the controller plane has to specify a maximum number of bits called csize (cycle size) that are permitted to go into each individual cycle.¶

Note, that iflow[flowid].csize is not specific to the sending interface because it is a property of the DetNet flow.¶

5.2. Ingress Flows Pseudocode

When a TCQF ingress is received, it first has to be enqueued into a per-flow queue. This is necessary because the permitted burst size for the flow may be larger than what can fit into a single cycle, or even into the number of cycles used in the network.¶

bool ingres_flow_enqueue(pak) {
  if(!pak.context.tcqf_cycle &&
      flowid = match_detnetflow(pak)) {
    police(pak) // according to RFC9016 5.5
    enqueue(pak, flowq[oif][flowid])
    return true
  }
  return false
}

ingres_flow_enqueue(pak) as shown in Figure 15 performs this enqueuing of the packet. Its position in the DetNet/TCQF forwarding code is shown in Figure 13.¶

police(pak): If the router is not only the TCQF ingress router, but also the first-hop router from the source, ingres_flow_enqueue(pak) will also be the place where policing of the flows packet according to the Traffic Specification of the flow would happen - to ensure that packets violating the Traffic Specification will not be forwarded, or be forwarded with lower priority (e.g.: as best effort). This policing and resulting forwarding action is not specific to TCQF and therefore out of scope for this text. See [RFC9016], section 5.5.¶

void ingress_flow_2_tcqf(oif, cycle) {
  foreach flowid in flowq[oif][*] {
    free = tcqf.iflow[flowid].csize
    q = flowq[oif][flowid]
    while(notempty(q) &&
          (l = head(q).size) <= free) {
      pak = dequeue(q)
      free -= l
      tcqf_enqueue(pak, oif.cycleq[cycle,internal])
    }
  }
}

ingress_flow_2_tcqf(oif, cycle) as shown in Figure 16 transfers ingress DetNet flow packets from their per-flow queue into the queue of the cycle that will be sent next. The position of ingress_flow_2_tcqf() in the DetNet/TCQF forwarding code is shown in Figure 13.¶

6.1. High-Speed Implementation

High-speed implementations with programmable forwarding planes of TCQF packet forwarding require Time-Gated Queues for the cycle queues, such as introduced by [IEEE802.1Qbv] and also employed in CQF [IEEE802.1Qch].¶

Compared to CQF, the accuracy of clock synchronization across the nodes is reduced as explained in Section 6.2 below.¶

High-speed forwarding for ingress packets as specified in Section 5 above would require to pass packets first into a per-flow queue and then re-queue them into a cycle queue. This is not ideal for high speed implementations. The pseudocode for ingres_flow_enqueue() and ingress_flow_2_tcqf(), like the rest of the pseudocode in this document is only meant to serve as the most compact and hopefully most easy to read specification of the desired externally observable behavior of TCQF - but not as a guidance for implementation, especially not for high-speed forwarding planes.¶

High-speed forward could be implemented with single-enqueueing into cycle queues as follows:¶

Let B[f] be the maximum amount of data that the router would need to buffer for ingress flow f at any point in time. This can be calculated from the flows Traffic Specification. For example, when using the parameters of [RFC9016], section 5.5.¶

B[f] <= MaxPacketsPerInterval*MaxPayloadSize*8

maxcycles = max( ceil( B[f] / tcqf.iflow[f].csize) | f)

Maxcycles is the maximum number of cycles required so that packets from all ingress flows can be directly enqueued into maxcycles queues. The router would then not cycle across tcqf.cycles number of queues, but across maxcycles number of queues, but still cycling across tcqf.cycles number of cycle tags.¶

Calculation of B[f] and in result maxcycles may further be refined (lowered) by additionally known constraints such as the bitrates of the ingress interface(s) and TCQF output interface(s).¶

6.2. Controller plane computation of cycle mappings

The cycle mapping is computed by the controller plane by taking at minimum the link, interface serialization and node internal forwarding latencies as well as the cycle_clock_offsets into account.¶

Router  . O1
 R1     . | cycle 1 | cycle 2 | cycle 3 | cycle 1 |
        .    .
        .     ............... Delay D
        .                    .
        .                    O1'
        .                     | cycle 1 |
Router  .   | cycle 1 | cycle 2 | cycle 3 | cycle 1 |
  R2    .   O2

CT  = cycle_time
C   = cycles
CC  = CT * C
O1  = cycle_clock_offset router R1, interface towards R2
O2  = cycle_clock_offset router R2, output interface of interest
O1' = O1 + D

Consider in Figure 17 that Router R1 sends packets via C = 3 cycles with a cycle_clock offset of O1 towards Router R2. These packets arrive at R2 with a cycle_clock offset of O1' which includes through D all latencies incurred between releasing a packet on R1 from the cycle buffer until it can be put into a cycle buffer on R2: serialization delay on R1, link delay, non_CQF delays in R1 and R2, especially forwarding in R2, potentially across an internal fabric to the output interface with the sending cycle buffers.¶

A = ( ceil( ( O1' - O2 ) / CT) + C + 1) mod CC
map(i) = (i - 1 + A) mod C + 1

Figure 18 shows a formula to calculate the cycle mapping between R1 and R2, using the first available cycle on R2. In the example of Figure 17 with CT = 1, (O1' - O2) =~ 1.8, A will be 0, resulting in map(1) to be 1, map(2) to be 2 and map(3) to be 3.¶

The offset "C" for the calculation of A is included so that a negative (O1 - O2) will still lead to a positive A.¶

In general, D will be variable [Dmin...Dmax], for example because of differences in serialization latency between min and max size packets, variable link latency because of temperature based length variations, link-layer variability (radio links) or in-router processing variability. In addition, D also needs to account for the drift between the synchronized clocks for R1 and R2. This is called the Maximum Time Interval Error (MTIE).¶

Let A(d) be A where O1' is calculated with D = d. To account for the variability of latency and clock synchronization, map(i) has to be calculated with A(Dmax), and the controller plane needs to ensure that that A(Dmin)...A(Dmax) does cover at most (C - 1) cycles.¶

If it does cover C cycles, then C and/or CT are chosen too small, and the controller plane needs to use larger numbers for either.¶

This (C - 1) limitation is based on the understanding that there is only one buffer for each cycle, so a cycle cannot receive packets when it is sending packets. While this could be changed by using double buffers, this would create additional implementation complexity and not solve the limitation for all cases, because the number of cycles to cover [Dmin...Dmax] could also be (C + 1) or larger, in which case a tag of 1...C would not suffice.¶

6.3. Link speed and bandwidth sharing

TCQF hops along a path do not need to have the same bitrate, they just need to use the same cycle time. The controller plane has to then be able to take the TCQF capacity of each hop into account when admitting flows based on their Traffic Specification and TCQF csize.¶

TCQF does not require to be allocated 100% of the link bitrate. When TCQF has to share a link with other traffic classes, queuing just has to be set up to ensure that all data of a TCQF cycle buffer can be sent within the TCQF cycle time. For example by making the TCQF cycle queues the highest priority queues and then limiting their capacity through admission control to leave time for other queues to be served as well.¶

6.4. Controller-plane considerations

TCQF is applicable to both centralized as well as decentralized/distributed controller-plane models. From the perspective of the controller plane. If the controller-plane is centralized, then it is logically very simple to perform admission control for any additional flow by checking that there is sufficient bandwidth for the amount of bits required for the flow on every cycle along the intended path. Likewise, path computation can be done to determine on which non-shortest path those resources are available.¶

More efficient use of resources can be achieved by considering that flows with low bit rates would not need bits reserved in every cycle, but only in every N'th cyce. This requires different gates on ingres to admit packets from such flows than shown in this document and more complex admission control that attempts for example to interleave multiple flows across different set of cycles to as best as possible utilize all cycles. This is the same complexity as possible in TSN technologies. Beside the admission control and different ingres policing, such enhancements have no impact on the per-hop TCQF forwarding and can thus potentially be added incrementally.¶

Decentralized or distributed controller planes including on-path, per-flow signaling, such as one using the mechanisms of RSVP-TE, [RFC3209] is equally feasible with TCQF. In this case one of the potential benefits of TCQ is not leveraged, which is the complete removal of per-hop,per-flow awarenes on each router. Nevertheless, the controller-plane only introduces the need for this state maintenance into the control-plane of each router, but does not change the TCQF forwarding plane, but maintains its per-hop, per-flow non-stateful nature and resulting performance/cost benefits.¶

6.5. Validation

[LDN] describes an accurate simualtion based validation of TCQF and provides further details on the mathematical models.¶

[CENI] is a report summary of a 100Gbps link speed commercial router validation implementation of TCQF deployed and measured in a research testbed with a range of up to 2000km across China, operated by the China Environment for Network Innovations (CENI). The report also provides a reference to a more detailled version of the report. Note that both reports are in chinese. TCQF is called DIP in these reports.¶