DetNet                                                      Shaofu. Peng
Internet-Draft                                                       ZTE
Intended status: Standards Track                               Peng. Liu
Expires: 23 November 2023                                   China Mobile
                                                         Kashinath. Basu
                                               Oxford Brookes University
                                                              Aihua. Liu
                                                                     ZTE
                                                              Dong. Yang
                                             Beijing Jiaotong University
                                                             22 May 2023


              Generic Packet Timeslot Scheduling Mechanism
             draft-peng-detnet-packet-timeslot-mechanism-02

Abstract

   IP/MPLS networks use packet switching (with the feature store-and-
   forward) and are based on statistical multiplexing.  Statistical
   multiplexing is essentially a variant of time division multiplexing,
   which refers to the asynchronous and dynamic allocation of link
   timeslot resources.  In this case, the service flow does not occupy a
   fixed timeslot, and the length of the timeslot is not fixed, but
   depends on the size of the packet.  Statistical multiplexing has
   certain challenges and complexity in meeting deterministic QoS, and
   its delay performance is dependent on the the used queueing
   mechanism.  This document further describes a generic time division
   multiplexing scheme in IP/MPLS networks, which we call packet
   timeslot scheme.  It aims to make it easier for the control plane to
   calculate the delay performance and more flexible to allocate
   deterministic resources, and also make it easier for the data plane
   to create more flexible timeslot mapping.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."



Peng, et al.            Expires 23 November 2023                [Page 1]


Internet-Draft           Packet Generic Timeslot                May 2023


   This Internet-Draft will expire on 23 November 2023.

Copyright Notice

   Copyright (c) 2023 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   5
   3.  Overview  . . . . . . . . . . . . . . . . . . . . . . . . . .   6
     3.1.  Timeslot Resource Reservation in Control-plane  . . . . .   8
       3.1.1.  Timeslot Mapping Relationship . . . . . . . . . . . .   9
         3.1.1.1.  Deduced by Single Timeslot Mapping Detection  . .  10
         3.1.1.2.  Deduced by Phase Difference of Orchestration
                 Period  . . . . . . . . . . . . . . . . . . . . . .  11
       3.1.2.  Timeslot Resource Definition  . . . . . . . . . . . .  12
       3.1.3.  Arrival Postion in the Orchestration Period . . . . .  14
       3.1.4.  Proccess of Each Reservation Sub-task . . . . . . . .  16
         3.1.4.1.  Resource Reservation on the Ingress Node  . . . .  18
         3.1.4.2.  Resource Reservation on the Transit Node  . . . .  19
         3.1.4.3.  Resource Reservation on the Egress Node . . . . .  20
         3.1.4.4.  End-to-end Delay and Jitter . . . . . . . . . . .  20
     3.2.  Timeslot Resource Access in Data-plane  . . . . . . . . .  21
       3.2.1.  Conversion of Timeslot ID . . . . . . . . . . . . . .  22
   4.  Global Timeslot ID  . . . . . . . . . . . . . . . . . . . . .  24
     4.1.  Fixed Timeslot Mapping Mode . . . . . . . . . . . . . . .  24
     4.2.  Variable Timeslot Mapping Mode  . . . . . . . . . . . . .  27
   5.  Summary of Timeslot Style . . . . . . . . . . . . . . . . . .  28
   6.  Queue Design  . . . . . . . . . . . . . . . . . . . . . . . .  29
     6.1.  Full Queues . . . . . . . . . . . . . . . . . . . . . . .  29
     6.2.  Non-full Queues . . . . . . . . . . . . . . . . . . . . .  30
   7.  Admission Control on the Headend  . . . . . . . . . . . . . .  30
   8.  Frequency Synchronization . . . . . . . . . . . . . . . . . .  31
   9.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  31
   10. Security Considerations . . . . . . . . . . . . . . . . . . .  31
   11. Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  31
   12. References  . . . . . . . . . . . . . . . . . . . . . . . . .  31



Peng, et al.            Expires 23 November 2023                [Page 2]


Internet-Draft           Packet Generic Timeslot                May 2023


     12.1.  Normative References . . . . . . . . . . . . . . . . . .  31
     12.2.  Informative References . . . . . . . . . . . . . . . . .  32
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  32

1.  Introduction

   IP/MPLS networks use packet switching (with the feature store-and-
   forward) and are based on statistical multiplexing.  The discussion
   of supporting multiplexing in the network was first seen in the time
   division multiplexing (TDM), frequency division multiplexing (FDM)
   and other technologies of telephone communication network (using
   circuit switching).  Statistical multiplexing is essentially a
   variant of time division multiplexing, which refers to the
   asynchronous and dynamic allocation of link resources.  In this case,
   the service flow does not occupy a fixed timeslot, and the length of
   the timeslot is not fixed, but depends on the size of the packet.  In
   contrast, synchronous time division multiplexing means that a
   sampling frame (or termed as time frame) includes a fixed number of
   fixed length timeslots, and the timeslot at a specific position is
   allocated to a specific service.  The utilization rate of link
   resources in statistical multiplexing is higher than that in
   synchronous time division multiplexing.  However, if we want to
   provide deterministic end-to-end delay in packet switched networks
   based on statistical multiplexing, the difficulty is greater than
   that in synchronous time division multiplexing.  The main challenge
   is to obtain a deterministic upper bound on the queuing delay, which
   is closely related to the queuing mechanism used in the network.

   In addition to IP/MPLS network, other packet switched network
   technologies, such as ATM, also discusses how to provide
   corresponding transmission quality guarantee for different service
   types.  Before service communication, ATM needs to establish a
   connection to reserve virtual path/channel resources, and use fixed-
   length short cells and timeslots.  The advantage of short cell is
   small interference delay, but the disadvantage is low encoding
   efficiency.  The mapping relationship between ATM cells and timeslots
   is not fixed, so it still depends on a specific cells scheduling
   mechanism (such as [ATM-LATENCY]) to ensure delay performance.
   Although the calculation of delay performance based on short and
   fixed-length cells is more concise than that of IP/MPLS networks
   based on non-fixed-length packets, they all essentially depend on the
   queueing mechanism.

   [CQF] introduce a hybrid of asynchronous and synchronous time-
   division multiplexing method based on fixed-length cycle in Ethernet
   LAN.  [Multi-CQF] is a further enhancement of the classic CQF to be
   applicable to IP/MPLS networks.  Generally, the service flow is not
   mapped to a fixed cycle at the network entrance, but dynamically



Peng, et al.            Expires 23 November 2023                [Page 3]


Internet-Draft           Packet Generic Timeslot                May 2023


   selects an idle cycle, which can be regarded as asynchronous, but at
   the intermediate nodes it is based on a fixed and inherent cycle
   mapping, which can be regarded as synchronous.  CQF with 2-buffer
   mode or Mult-CQF with 3-buffer mode only uses a small number of
   cycles to establish the inherent cycle mapping between a port-pair of
   two adjacent nodes, which can minimize the residence delay in the
   node and limit end-to-end jitter at the same time.  The inherent
   cycle mapping is independent of the service and is also irrelevant to
   the resource reservation of the service.  That is, the cycle of CQF/
   Mult-CQF is not a resource that is open to the service and can be
   reserved.  [Multi-CQF] describes the deterministic behavior for each
   cycle-level based on traditional bandwidth resource allocation, with
   cycle-based admission control.  However, overprovisioning (i.e.,
   burst / cycle is larger than service bandwidth) may affect the
   service scale that it can support.  Alternatively, other literatures
   have also been discussing the cycle-based resource allocation, which
   mainly affects the sending cycle selected for the service flow on the
   entry node, but does not change the inherent cycle mapping in the
   network.  The path composed of inherent cycle mapping relationship is
   like a highway without traffic lights along the way.  Traffic control
   can only be implemented at the entrance of the path, so it may be
   possible to avoid car conflict of multiple paths at the intermediate
   nodes, but not always likely.  In addition, since the number of
   cycles used by CQF/Multi-CQF is small, such as 3 or 4, it is not easy
   to determine whether the cycle conflict of service flows on multiple
   paths is true or false conflict (True conflict is when two flows used
   the same slot in the same scheduling period whereas false conflict
   arise when two flows use the same slot in different scheduling
   period).

   In order to improve the service scale supported by the network, this
   document further discusses a generic time division multiplexing
   scheduling mechanism using timeslot resource in IP/MPLS networks,
   which we call Timeslot Queueing and Forwarding (TQF) mechanism.  The
   timeslot resource is the enhancement of bandwidth resource, has the
   feature of time awareness, is visible and assignable to the service.
   From the perspective of data plane, the timeslot resource is a time
   interval that can continuously send packets, while from the control
   plane, it is the amount of bits in the time interval (e.g, the total
   amount of bits that is reservable, and the amount of unreserved
   bits).  Based on the timeslot resources, it is easier for the control
   plane to reserve strictly non-conflicting resources for different
   paths that can get deterministic delay performance, and the data
   plane can create more flexible (i.e., not inherent, but based on
   reservation) timeslot mapping to avoid conflicts.






Peng, et al.            Expires 23 November 2023                [Page 4]


Internet-Draft           Packet Generic Timeslot                May 2023


2.  Terminology

   The following terminology is introduced in this document:

   Timeslot:  The smallest unit of packet timeslot scheduling.  It needs
       to design a reasonable value, such as 10us, to send at least one
       complete packet.  Different nodes can be configured with
       different length of timeslot.

   Timeslot Scheduling:  The packet is stored in the buffer
       corresponding to a specific timeslot, then sent in that timeslot.

   Service Burst Interval:  The traffic specification of deterministic
       services generally follows the principle of generating a specific
       burst amounts within a specific length of cyclic burst interval.
       For example, a service generates 1000 bits of burst per 1 ms,
       where 1 ms is the service burs interval.

   Orchestration Period:  The orchestration Period is adopted by the
       control plane according to the needs of all deterministic
       services, including a fixed number (termed as N and numbered from
       0 to N-1) of timeslots, for example, the orchestration period
       include 1000 timeslots and each timeslot length is 10 us.  The
       timeslot resources within the orchestration period can be
       allocated for services, i.e., which timeslots are occupied by
       services and how many bits are occupied in timeslots.  The
       orchestration period is the Least Common Multiple of all service
       burst intervals.  It is also a multiple of the scheduling period.
       It is recommended that all nodes of the network be configured
       with the same length of orchestration period (note that timeslot
       length may still be different), because it is service-related and
       also crucial for establishing a stable timeslot mapping
       relationship.

   Ongoing Sending Period:  The orchestration period which the ongoing
       sending timeslot belongs to.

   Scheduling Period:  The period of the packet timeslot scheduling
       mechanism implemented by the forwarding plane of the network
       node, including a fixed number (termed as M and numbered from 0
       to M-1) of timeslots, for example, the scheduling period include
       100 timeslots and each timeslot length is 10 us.  Different nodes
       can be configured with different length of scheduling period.

   Incoming Timeslot:  For an intermediate node in a specific path, the






Peng, et al.            Expires 23 November 2023                [Page 5]


Internet-Draft           Packet Generic Timeslot                May 2023


       timeslot contained in the packet received from the upstream node
       (i.e., the outgoing timeslot of the upstream node) is its
       incoming timeslot.  An incoming timeslot is the timeslot number
       in the orchestration period.

   Outgoing Timeslot:  For an intermediate node in a specific path, when
       it continues to send packets received from the upstream node to
       downstream nodes, according to resource reservation or certain
       rules, it chooses to send packets in the specified timeslot,
       which is the outgoing timeslot.  An outgoing timeslot is the
       timeslot number in the orchestration period.

   Ongoing Sending Timeslot:  For the headend of the path, packets
       received from the client side and sent to the downstream node.
       When the packet reaches the outgoing port, the timeslot currently
       in the sending state is the ongoing sending timeslot; For
       intermediate nodes of the path, packets received from the
       upstream node and sent to the downstream node.  When the end of
       the incoming timeslot to which the packet belongs reaches the
       outgoing port, the timeslot currently in the sending state is the
       ongoing sending timeslot.  Note that the ongoing sending timeslot
       is different with the outgoing timeslot.  An ongoing sending
       timeslot is the timeslot number in the orchestration period.

3.  Overview

   This scheme introduces the time-division multiplexing scheduling
   mechanism based on the fixed length timeslot in the IP/MPLS network.
   Note that the time-division multiplexing here is a L3 packet-level
   scheduling mechanism, rather than the TDM port (such as SONET/SDH)
   implemented in L1.  The latter generally involves the time frame and
   the corresponding framing specification, which is not necessary in
   this document.

   Figure 1 shows the packet timeslot scheduling behavior implemented by
   the intermediate node P through which multiple deterministic paths
   passes on to the outgoing port (P-PE2).














Peng, et al.            Expires 23 November 2023                [Page 6]


Internet-Draft           Packet Generic Timeslot                May 2023


       +---+                 +---+                 +---+
       |PE1| --------------- | P | --------------- |PE2|
       +---+                 +---+                 +---+

                                orchestration period
                            +---+---+-+-+---+---------+---+
                            | 0 | 1 | 2 | 3 | ... ... |N-1|
                            +---+---+---+---+---------+---+
                                   ^  ^
                    reserve slots: |  | reserve slots:
                      a,b,c        |  |   x,y
   path-1 -------------------------o--|---------------->
   path-2 -------------------------|--o---------------->
                                   |  |
                     access slots: |  | access slots:
                       a',b',c'    v  v   x',y'
                        /  +-------------------+    ___
                        |  |  queue-0 @slot 0  |   /   \
                        |  +-------------------+  |     |
                        |  |  queue-1 @slot 1  |  |     |
           Scheduling  <   +-------------------+  |
           Period       |  |  ... ...          |  |     ^
                        |  +-------------------+  |     |
                        |  |  queue-n @slot M-1|   \___/
                        \  +-------------------+


                                  Figure 1

   Where, both the orchestration period and the scheduling period
   consist of multiple timeslots, the total amount of bits that can be
   reserved or send in each timeslot can be preset, generally not
   exceeding the result of the total bandwidth of the link multiplied by
   the timeslot length.  The orchestration period or scheduling period
   of all nodes in the network does not need to be synchronized, and
   phase difference is allowed.  However, for each node, the phase of
   timeslot of orchestration period and the scheduling period are
   strictly aligned.  This is indeed natural because multiple scheduling
   periods forms an orchestration period.












Peng, et al.            Expires 23 November 2023                [Page 7]


Internet-Draft           Packet Generic Timeslot                May 2023


   In the figure, path-1 and path-2 allocate timeslot resource from the
   orchestration period of link P-PE2 respectively.  Path-1 reserves
   timeslot a, b, c from orchestration period, and finally accesses
   timeslot a', b', c' from scheduling period.  Path-2 reserves timeslot
   x, y from orchestration period, and finally accesses timeslot x', y'
   from scheduling period.  There is a mapping relationship function
   between the timeslot i of orchestration period and the timeslot i' of
   scheduling period, i.e., i' = f(i).  There are many mapping options,
   such as a'=a, a'=a+offset, a'=a%M, and a'=random(a), etc.

   In general, the packet timeslot scheduling mechanism implemented on
   all nodes in the network may use the same length of timeslot and
   scheduling period.  However, considering the capability differences
   of each node in the network (for example, the capabilities of the
   edge nodes are weaker than the core nodes), it is feasible for
   different nodes to use different length of timeslot and scheduling
   period.

   The scheme involves two aspects: the path calculation and timeslot
   resource reservation in the control plane, and timeslot resource
   access in the data plane.

3.1.  Timeslot Resource Reservation in Control-plane

   The control plane (centralized controller or distributed protocol)
   can reserve corresponding timeslot resources along the deterministic
   path.  Note that a path may carry multiple services, then the path
   will reserve timeslot resources for the combined services, and may
   reserve the bit resources in multiple timeslots at the same time in
   the orchestration period.  However, it would still be beneficial to
   distinguish between reservation sub-tasks corresponding to different
   service flows in the combined reservation task.  In this document, we
   refer to a reservation sub-task as a timeslot resource reservation
   action related to the service flow.  Note that one or more
   reservation sub-tasks for a specific service flow may be derived
   based on its TSpec, and each reservation sub-task will allocate
   corresponding timeslot resource.  The intermediate nodes do not
   maintain the state of service flow and only reserve timeslot
   resources based on the reservation sub-tasks.












Peng, et al.            Expires 23 November 2023                [Page 8]


Internet-Draft           Packet Generic Timeslot                May 2023


   During resource reservation, it is necessary to distinguish the
   requirements between low latency service and non-low latency service
   . For low latency service requirements, the physical offset between
   the reserved outgoing timeslot and the incoming timeslot is small;
   while for non-low latency service requirements, this physical offset
   can be large.  It is necessary to maintain the end-to-end total
   residence delay budget for each reservation sub-task, used to select
   outgoing timeslot, as long as the sum of residence delays caused by
   all nodes should not exceed the total residence delay budget.

   Multiple reservation sub-tasks may generate different incoming/
   outgoing timeslot mapping relationships on node P.  For example:

   *  The timeslot mapping relationship created by the sub-task-1:

         <(incoming port a, incoming slot number 3), (outgoing port b,
         outgoing slot number 60)>

   *  The timeslot mapping relationship created by the sub-task-2:

         <(incoming port a, incoming slot number 3), (outgoing port b,
         outgoing slot number 61)>

   Special care should be taken not to confuse the use of different
   mapping relationships.  For specific service flows, P need to
   explicitly use specific timeslot mapping relationships.

   It is recommended, but not mandatory, to reserve timeslot resources
   on the outgoing port of each hop from the headend of the path to the
   endpoint, that is, first determine the timeslot reserved on the
   headend, then determine the timeslot reserved on the next hop , and
   so on.  We assume that the service flow has a periodic arrival time,
   and there is a fixed position relationship between the arrival time
   and the orchestration period of the headend, so selecting the
   outgoing timeslot closed to the arrival time or within the expected
   offset range in the orchestration period can minimize the residency
   delay of the packet on the headend.  However, sometimes it is
   necessary to get a larger residence delay on the headend and a
   smaller residence delay on other nodes to ensure successful path
   calculation.

3.1.1.  Timeslot Mapping Relationship

   In order to reserve outgoing timeslot resources for the service flow
   , it is necessary to first determine the ongoing sending timeslot
   that the incoming timeslot falls into, i.e., the mapping relationship
   between the incoming timeslot and the ongoing sending timeslot.




Peng, et al.            Expires 23 November 2023                [Page 9]


Internet-Draft           Packet Generic Timeslot                May 2023


   Suppose a path contains three nodes U, V, and W in turn along the
   path.  All nodes are configured with orchestration period of the same
   length (termed as LOP), which is crucial for establishing a fixed
   timeslot mapping relationship.  Node U config timeslot length L_u,
   and an orchestration period contains N_u timeslots.  Node V config
   timeslot length L_v, and an orchestration period contains N_v
   timeslots.  In general, the link bandwidth of edge nodes is small,
   and they will be configured with a larger timeslot length than the
   aggregated/backbone nodes.

   It has been mathematically proven that if the least common multiple
   of Lu and Lv is LCM, LOP is also a multiple of LCM.

   Two methods are provided in the following sub-sections to determine
   the timeslot mapping relationship.

3.1.1.1.  Deduced by Single Timeslot Mapping Detection

   Figure 2 shows that Node U sends a detection packet from the end (or
   head, the process is similar) of an arbitrary timeslot i on the
   outgoing port connected to node V.  After a certain link propagation
   delay (D_propagation), the packet is received by the incoming port of
   node V, and i is regarded as the incoming timeslot by V.  The packet
   finally arrives at the outgoing port connected to node W after the
   intra-node forwarding delay (D_forwarding) including parsing, table
   lookup, internal fabric exchange, etc.  At this time, the ongoing
   sending timeslot is j, and there is time T_ij left before the end of
   the timeslot j.

       |<------------------------ LOP ---------------------------->|

       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
   U   |   |   |   | i |   |   |   |   |   | x |   |   |   |   |   |
       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
                       |                       |
                       |<--T_ij->|             |<--T_xy->|
                       v                       v
         +-----------+-----------+-----------+-----------+-----------+
   V     |           |     j     |  ... ...  |     y     |           |
         +-----------+-----------+-----------+-----------+-----------+

         |<------------------------ LOP ---------------------------->|


                                  Figure 2






Peng, et al.            Expires 23 November 2023               [Page 10]


Internet-Draft           Packet Generic Timeslot                May 2023


   Then, based one the detection result of the mapping relationship
   between incomming timeslot i and ongoing sending timeslot j, for any
   other outgoing timeslot x of node U, the mapped ongoing sending
   timeslot y of node V is:

   *  y = (j + ((N_u+x-i)*L_u-T_ij)/L_v + 1) % N_v

   And the time Txy left before the end of the timeslot y is:

   *  T_xy = L_v - ((N_u+x-i)*L_u-T_ij)%L_v

   Note that the detection message used to get the mapping relationship
   i->j does not really need to be sent to the outgoing port, i.e., the
   mapping relationship cannot be obtained only on the outgoing port,
   but on the incoming port side.  Assuming that the orchestration
   period of all ports within a node are strictly synchronized, this is
   easy to achieve.  On the incoming port, upon receiving the detection
   message, immediately determine the ongoing sending timeslot j' that
   the incoming timeslot falls into and the corresponding T_ij', and
   then based on a fixed forwarding delay evaluation value (but not less
   than the actual forwarding delay D_forwarding) to estimate the
   timeslot j that the incoming timeslot falls into and the
   corresponding T_ij.

3.1.1.2.  Deduced by Phase Difference of Orchestration Period

   Figure 3 shows that Node U sends a detection packet from the end (or
   head, the process is similar) of the orchestration period on the
   outgoing port connected to node V.  After a certain link propagation
   delay (D_propagation), the packet is received by the incoming port of
   node V and finally arrives at the outgoing port connected to node W
   after the intra-node forwarding delay (D_forwarding).  At this time,
   there is time P_uv left before the end of the ongoing sending period.

       |<------------------- LOP --------------------->|

       +---+---+---+---+---+---+---+---+---+---+---+---+
   U   |   |   |   |   |   |   | x |   |   |   |   |   |
       +---+---+---+---+---+---+---+---+---+---+---+---+
       |                           |
       |<----Puv---->|             |<--T_xy->|
                     |             v
                     +-----------+-----------+-----------+-----------+
   V                 |           |     y     |           |           |
                     +-----------+-----------+-----------+-----------+

                     |<------------------- LOP --------------------->|




Peng, et al.            Expires 23 November 2023               [Page 11]


Internet-Draft           Packet Generic Timeslot                May 2023


                                  Figure 3

   Then, based one the phase difference of orchestration period, for any
   outgoing timeslot x of node U, the mapped ongoing sending timeslot y
   of node V is:

   *  y = ((LOP+(x+1)*L_u-P_uv)/L_v) % N_v

   And the time Txy left before the end of the timeslot y is:

   *  T_xy = Lv - (LOP+(x+1)*L_u-P_uv)%Lv

   Note that the detection message used to get the phase difference of
   orchestration period does not really need to be sent to the outgoing
   port, i.e., the phase difference cannot be obtained only on the
   outgoing port, but on the incoming port side.  Assuming again that
   the orchestration period of all ports within a node are strictly
   synchronized, on the incoming port, upon receiving the detection
   message, immediately determine the phase difference P_uv', and then
   based on a fixed forwarding delay evaluation value (but not less than
   the actual forwarding delay D_forwarding) to estimate the phase
   difference P_uv.

   Note that in Section 3.1.1.1, the phase difference of orchestration
   period may also be derived firstly by the mapping relationship of
   i->j, and then get the mapping relationship for other timeslots
   according to the above formula.

3.1.2.  Timeslot Resource Definition

   The timeslot resources of a link can be represented as the
   corresponding bit amounts of all timeslots included in an
   orchestration period.  Basically, the link capability should contain
   the following information:

   *  Timeslot Length (L_T): Represents the length of the timeslot, in
      units of us.  Generally, the length of each timeslot included in
      the orchestration period is the same.

   *  Length of Orchestration Period (LOP): Indicates the number of
      timeslots (N) included in the orchestration period, numbered
      sequentially from 0 to N-1.

   *  Length of Scheduling Period (LSP): Indicates the number of
      timeslots (M) included in the scheduling period, numbered
      sequentially from 0 to M-1.





Peng, et al.            Expires 23 November 2023               [Page 12]


Internet-Draft           Packet Generic Timeslot                May 2023


   Figure 4 shows the timeslot resource model of the link, with an
   orchestration period consisting of N timeslots numbered from 0 to
   N-1.  The resource information of each timeslot includes the
   following attributes:

   *  Sequence Number: Indicates the number of the timeslot in the
      orchestration period.  The number of the first timeslot is 0, and
      the number of the last timeslot is N-1.

   *  Maximum Reservable Bursts (MRB): Refers to the maximum amount of
      bit quota corresponding to this timeslot, with unit of bits.  It
      is a configurable preset value that is related to the total
      bandwidth of the link (termed as C) and the length of the timeslot
      (termed as L_T), then the Maximum Reservable Bursts should be set
      to a value not exceeding C*L_T, considering the transmission needs
      of other non-deterministic service flows in the network.
      Generally, the Maximum Reservable Bursts of each timeslot included
      in the orchestration period are all the same.

   *  Unreserved Bursts (UB): Refers to the amount of bits reservable
      corresponding to this timeslot, with unit of bits.

       #N-1       +-------------------------------------+
                  | Timeslot Length:           L_T(n-1) |
                  | Maximum Reservable Bursts: MRB(n-1) |
                  | Unreserved Bursts:         UB(n-1)  |
                  +-------------------------------------+
       ...                      ... ...
       ...                      ... ...

       #1         +-------------------------------------+
                  | Timeslot Length:           L_T(1)   |
                  | Maximum Reservable Bursts: MRB(1)   |
                  | Unreserved Bursts:         UB(1)    |
                  +-------------------------------------+

       #0         +-------------------------------------+
                  | Timeslot Length:           L_T(0)   |
                  | Maximum Reservable Bursts: MRB(0)   |
                  | Unreserved Bursts:         UB(0)    |
                  +-------------------------------------+
       ----------------------------------------------------------->
                      Timeslot Resource of the Link


                                  Figure 4





Peng, et al.            Expires 23 November 2023               [Page 13]


Internet-Draft           Packet Generic Timeslot                May 2023


   The IGP/BGP extensions to advertise the link's capability and
   timeslot resource will be specified in seperate documents.

3.1.3.  Arrival Postion in the Orchestration Period

   Generally, a deterministic service flow has its TSpec, such as
   periodically generating traffic of a specific burst size within a
   specific length of burst interval, which regularly reaches the
   network entry.  The headend executes traffic regulation (e.g, setting
   appropriate parameters for leaky bucket shaping), which generally
   make packets evenly distributed within the service burst interval,
   i.e, there are one or more shaped sub-burst in the service burst
   interval.  There is a fixed positional relationship between the
   departure time when each sub-burst leaves the regulator and the
   orchestration period, based on that a specific outgoing time slot is
   reserved for the sub-burst.  Note that there may be jitter when a
   sub-burst departure from the regulator occurs, that is, there may be
   jitter in the positional relationship between it and the
   orchestration period.  Therefore, when reserving the outgoing
   timeslot, this jitter should be included (i.e., selecting a later
   outgoing timeslot).

   Figure 5 shows, for some typical service flows, the relationship
   between the service burst interval (SBI) and the length of
   orchestration period (LOP) of headend, as well as the possible
   timeslot resource reservation results for these service flows.

























Peng, et al.            Expires 23 November 2023               [Page 14]


Internet-Draft           Packet Generic Timeslot                May 2023


               |<--------------------- LOP ---------------------->|
               +----+----+----+----+----+----+----+----------+----+
               | #0 | #1 | #2 | #3 | #4 | #5 | #6 |  ... ... |#N-1|
               +----+----+----+----+----+----+----+----------+----+

                     +--+
    Service 1: |     |b1|                                         |
               +-----+--+-----------------------------------------+
               |<------------------- SBI ------------------------>|


                            +--+                        +--+
    Service 2: |            |b1|                        |b2|
               +------------+--+------------------------+--+------+
               |<------------------- SBI ------------------------>|


                                           +------+
    Service 3: |                           |  b1  |
               +---------------------------+------+---------------+
               |<------------------- SBI ------------------------>|


                    +--+             +--+             +--+
    Service 4: |    |b1|        |    |b1|        |    |b1|        |
               +----+--+--------+----+--+--------+----+--+--------+
               |<----- SBI ---->|<----- SBI ---->|<----- SBI ---->|

                                  Figure 5

   As shown in the figure, the length of service burst intervals for
   services 1, 2, 3 is equal to the length of orchestration period.
   Service 1 generates a very small single burst amounts within its
   burst interval, which may reserve timeslot 2 or other subsequent
   timeslot in the orchestration period; Service 2 generates two small
   discrete sub-bursts within its burst interval and also be shaped,
   which may reserve slots 4 and N-1 in the orchestration period for
   each sub-burst respectively; Service 3 generates a large single burst
   amount within its burst interval but not be shaped (due to purchasing
   a larger bucket depth), which may also be split to multiple sub-
   bursts and reserve multiple timeslots in the orchestration period,
   such as timeslots 8 and 9.  The length of the service burst interval
   for service 4 is only 1/3 of the orchestration cycle, then first
   construct service 4' whose burst interval is equal to the length of
   orchestration period and contains three times of service 4.  So
   service 4' is similar to service 2, generating a small amount of
   three separate sub-bursts within its burst interval.  It may reserve
   timeslots 3, 7, and N-1 in the orchestration period.



Peng, et al.            Expires 23 November 2023               [Page 15]


Internet-Draft           Packet Generic Timeslot                May 2023


   Each sub-burst corresponds to a reservation sub-task.

   For a specific service flow, to determine how many reservation sub-
   tasks are required, can be summarized as:

   *  First, align the service burst interval with the Orchestration
      Period of the headend to ensure that the two are of equal length.
      If the service burst interval is only a fraction of the
      Orchestration Period, then multiply it several times to obtain the
      expanded service burst interval to get a new service'.

   *  Check how many discrete sub-bursts will be generated during the
      orchestration Period, and for each sub-burst:

      -  If the proportion of the sub-burst size to the MRB of a single
         timeslot does not exceed a specific value, then the sub-burst
         corresponds to a reservation sub-task;

      -  Otherwise, continue to split the sub-burst into multiple sub-
         sub-bursts (note that each sub-sub-burst must contain a
         complete packet), so that the proportion of each sub-sub-burst
         size to the MRB of a single timeslot does not exceed the
         specific value, and each sub-sub-burst corresponds to a
         reservation sub-task.

3.1.4.  Proccess of Each Reservation Sub-task

   Each reservation sub-task contains a separate parameter set, which is
   used in the process of timeslot resource reservation.  Note that this
   set may be a local information for the path compuation engine (e.g, a
   controller), or may signal between nodes (e.g, RSVP-TE).

   *  Total Residence Budget: It is the sum of the residence delay
      allowed by the service flow within all nodes in the path, which is
      equal to the end-to-end delay requirement of the service flow
      minus the propagation delay of all links included in the path.

   *  Node Residence Budget: It refers to the resident delay budget of
      the current node traversed during the process of reserving
      timeslot resources on each node along the path in sequence.  A
      simple way is to divide the Total Residence Budget by the number
      of nodes included in the path to obtain the average resident delay
      budget as the Node Residence Budget for each node, or use a
      specified budget list to specify the resident delay budget for
      each node separately.






Peng, et al.            Expires 23 November 2023               [Page 16]


Internet-Draft           Packet Generic Timeslot                May 2023


   *  Accumulated Node Residence Budget: It refers to the cumulative
      residence delay budget of those nodes that have executed resource
      reservation.

   *  Accumulated Node Residence Evaluation: It refers to the cumulative
      evaluation value of the residence delay of nodes that have
      executed timeslot resource reservation.  The residence delay
      evaluation value of a node refers to the residence delay
      evaluation value calculated based on the delay formula (see below)
      when the node actually reserves a certain outgoing timeslot for
      the reservation sub-task.  Generally, if a node is able to reserve
      the expected outgoing timeslot according to its residence delay
      budget, the residence delay evaluation value does not differ from
      the residence delay budget.  However, in some cases, due to
      insufficient resources in the expected timeslot, resources have to
      be reserved in the timeslot adjacent to the expected timeslot,
      which can lead to a difference between the residence delay
      evaluation value and the budget value.

   *  Accumulated Node Residence Deviation: It is equal to the
      Accumulated Node Residence Budget minus the Accumulated Node
      Residence Evaluation.

   *  Node Residence Budget Adjustment: It is equal to the Node
      Residence Budget plus the Accumulated Node Residence Deviation.

   The usage for the above parameter set is:

   *  For specific reservation sub-task, determine the Node Residence
      Budget for each node in the path, which can be taken from the
      average residence delay budget per node or the specified budget
      list.

   *  From the headend to the endpoint, on each node's outgoing port in
      sequence, reserve outgoing timeslot resources based on the Node
      Residence Budget Adjustment, to let the residence delay evaluation
      value of the node obtained from the reserved outgoing timeslot be
      equal to or close to the Node Residence Budget Adjustment.

      -  On the headend, the Accumulated Node Residence Deviation is the
         initial value of 0.  Therefore, the Node Residence Budget
         Adjustment is equal to the Node Residence Budget.









Peng, et al.            Expires 23 November 2023               [Page 17]


Internet-Draft           Packet Generic Timeslot                May 2023


      -  On any other nodes, the Accumulated Node Residence Deviation is
         generally not 0.  If the residence delay evaluation value of
         the node obtained from the reserved outgoing timeslot be equal
         to the Node Residence Budget Adjustment, it will cause the
         Accumulated Node Residence Deviation faced by the downstream
         node in the path to be 0 again.

   Note that the above parameter set is only an implementation choice
   and is not mandatory.  There may be more intelligent path calculation
   methods available.

3.1.4.1.  Resource Reservation on the Ingress Node

   On the headend H, as mentioned above, there is a fixed positional
   relationship (with possible jitter) between the departure time when
   the sub-burst leaves the regulator and the orchestration period.
   From the departure time when the sub-burst leaves the regulator,
   after the intra-node forwarding delay (d_f) including parsing, table
   lookup, internal fabric exchange, etc, the sub-burst finally arrives
   at the ougoing port, and at this time the ongoing sending timeslot is
   j, and there is time T_j left before the end of the timeslot j.

   The outgoing timeslot reserved for the sub-burst by the headend is
   offset by o (>=1) timeslots after timeslot j, which means the
   outgoing timeslot is (j+o)%N_h, where N_h is the number of timeslots
   in the orchestration period for node H.

   Note that o must be less than M.

   Thus, on the headend H the residence delay evaluation value obtained
   from the reserved outgoing timeslot (j+o)%N_h is:

      Best Node Residence Evaluation = d_f + T_j + (o-1)*L_h

      Worst Node Residence Evaluation = d_f + T_j + o*L_h

      Average Node Residence Evaluation = d_f + T_j + (2o-1)*L_h/2

      where, L_h is the length of timeslot for node H.

   The Best Node Residence Evaluation occurs when the sub-burst is sent
   at the head of outgoing timeslot j+o.  The Worst Node Residence
   Evaluation occurs when the sub-burst is sent at the end of outgoing
   timeslot j+o.  The delay jitter within the node is L_h.  However, the
   jitter of the entire path is not the sum of the jitters of all nodes.






Peng, et al.            Expires 23 November 2023               [Page 18]


Internet-Draft           Packet Generic Timeslot                May 2023


   Depending on the implementation, the above Best Node Residence
   Evaluation, Worst Node Residence Evaluation, or Average Node
   Residence Evaluation can be used to compare with the Node Residence
   Budget Adjustment, so that when selecting the appropriate outgoing
   timeslot (j+o)%N_h, the two are equal or nearly equal, and the
   corresponding Unreserved Burst resources of the outgoing timeslot
   (j+o)%N_h meet the burst demand of the sub-burst.  However, this
   document suggests using the Average Node Residence Evaluation to
   compare with the Node Residence Budget Adjustment, because the
   characteristic of the forwarding behavior based on packet timeslot is
   that adjacent nodes on the path will not simultaneously face the best
   or worst residency delay.

3.1.4.2.  Resource Reservation on the Transit Node

   On the transit node V, as described in Section 3.1.1, there is a
   timeslot mapping relationship between the outgoing timeslot of the
   upstream node U and the ongoing sending timeslot of node V.

   For a specific sub-task, assume that an outgoing timeslot i is
   reserved for it on the outgoing port of the upstream node U, and
   after the intra-node forwarding delay (d_f) then mapped to the
   ongoing sending timeslot j of node V, and there is time T_ij left
   before the end of the timeslot j.

   The outgoing timeslot reserved for the sub-task by node V is offset
   by o (>=1) timeslots after timeslot j, which means the outgoing
   timeslot is (j+o)%N_v, where N_v is the number of timeslots in the
   orchestration period of node V.

   Note that o must be less than M.

   Thus, on the transit node V the residence delay evaluation value
   obtained from the reserved outgoing timeslot (j+o)%N_v is:

      Best Node Residence Evaluation = d_f + T_ij + (o-1)*L_v

      Worst Node Residence Evaluation = d_f + T_ij + L_u + o*L_v

      Average Node Residence Evaluation = d_f + T_ij + (L_u+(2o-
      1)*L_v)/2

      where, L_u and L_v is the length of timeslot for node U and V
      respectively.

   The Best Node Residence Evaluation occurs when the packet is received
   at the end of incoming timeslot i and sent at the head of outgoing
   slot j+o; The Worst Node Residence Evaluation occurs when t he packet



Peng, et al.            Expires 23 November 2023               [Page 19]


Internet-Draft           Packet Generic Timeslot                May 2023


   is received at the head of incoming timeslot i and sent at the end of
   outgoing timeslot j+o.  The delay jitter within the node is (L_u +
   L_v).  However, the jitter of the entire path is not the sum of the
   jitters of all nodes.

   Depending on the implementation, the above Best Node Residence
   Evaluation, Worst Node Residence Evaluation, or Average Node
   Residence Evaluation can be used to compare with the Node Residence
   Budget Adjustment, so that when selecting the appropriate outgoing
   timeslot (j+o)%N_v, the two are equal or nearly equal, and the
   corresponding Unreserved Burst resources of the outgoing timeslot
   (j+o)%N_v meet the burst demand of the sub-burst.  However, this
   document suggests using the Average Node Residence Evaluation to
   compare with the Node Residence Budget Adjustment, because the
   characteristic of the forwarding behavior based on packet timeslot is
   that adjacent nodes on the path will not simultaneously face the best
   or worst residency delay.

3.1.4.3.  Resource Reservation on the Egress Node

   Generally, for the deterministic path carrying the service flow, the
   flow needs to continue forwarding from the outgoing port of the
   egress node to the client side, and also faces the issues of queuing.
   However, the outgoing port facing the client side is not part of the
   deterministic path.  If it is necessary to continue supporting packet
   timeslot scheduling mechanism on that port, timeslot resources should
   be reserved on the higher-level service path (an overlay path) using
   the above reservation method.  In this case, the deterministic path
   will serve as a virtual link of the overlay path, providing a
   deterministic delay performance.

   Therefore, for deterministic paths, the residence dalay evaluation
   value on the egress node is only contributed by the forwarding delay
   (d_f) including parsing, table lookup, internal fabric exchange, etc.

3.1.4.4.  End-to-end Delay and Jitter

   Figure 6 shows that a path from headend P1 to endpoint E, for each
   node Pi, the length of timeslot is L_i, the intra-node forwarding
   delay is F_i, the remaining time from the end of the mapped ongoing
   sending timeslot is T_i, the number of timeslots offset by outgoing
   timeslot relative to ongoing sending timeslot is X_i, then the end to
   end delay can be evaluted as follows:

      Best E2E Delay = sum(F_i+T_i+X_i*L_i, for 1<=i<=n) - L_n + F_e

      Worst E2E Delay = sum(F_i+T_i+X_i*L_i, for 1<=i<=n) + F_e




Peng, et al.            Expires 23 November 2023               [Page 20]


Internet-Draft           Packet Generic Timeslot                May 2023


       +---+     +---+     +---+             +---+     +---+
       | P1| --- | P2| --- | P3| --- ... --- | Pn| --- | E |
       +---+     +---+     +---+             +---+     +---+


                                  Figure 6

   The Best E2E Delay occurs when the sub-burst is sent at the head of
   outgoing timeslot of node Pn.  The Worst E2E Delay occurs when the
   sub-burst is sent at the end of outgoing timeslot of node Pn.  The
   delay jitter is L_n.  Note that at the headend P1, regardless of
   whether it has the best or worst residence latency, it will be
   aligned to the worst latency on the downstream node; Every hop is
   like this, except for the last one.

3.2.  Timeslot Resource Access in Data-plane

   The headend of the path needs to maintain the timeslot resource
   information with the granularity of sub-burst, so that each sub-burst
   of the service flow can access the mapped timeslot resources.
   However, the intermediate node does not need to maintain this mapping
   state.  The intermediate node only access the timeslot resources
   based on the timeslot number carried in the packets or indicated by
   FIB entries.

   The entry node determines the appropriate outgoing timeslot and sends
   the packet according to the periodic arrival time of the sub-burst,
   and the maintained mapping relationship between the sub-burst of
   service flow and the outgoing timeslot.

   The relationship between the incoming timeslot and the outgoing
   timeslot can be installed on the intermediate node or carried in the
   packet, so that the packet can access the corresponding outgoing
   timeslot on the intermediate node.

   Note that the incoming and outgoing timeslots mentioned here are both
   timeslot numbers within the orchestration period.

   It should be noted that the forwarding outgoing port for the service
   flow is still determined according to the traditional routing entries
   (e.g, Segment Routing), but the outgoing timeslot used by the packet
   is determined by the timeslot resource reservation information.









Peng, et al.            Expires 23 November 2023               [Page 21]


Internet-Draft           Packet Generic Timeslot                May 2023


3.2.1.  Conversion of Timeslot ID

   Figure 1 shows that the scheduling period implemented on the
   forwarding plane is not completely equivalent to the orchestration
   period of the control plane.  The scheduling period includes timeslot
   numbers from 0 to M-1, while the orchestration period includes
   timeslot numbers from 0 to N-1.  Therefore, it is necessary to
   convert the outgoing timeslot of the orchestration period to the
   target timeslot of the scheduling period, and insert the packet to
   the queue corresponding to the target timeslot for transmission.

   A simple conversion method is:

   *  target timeslot = outgoing timeslot % M

   This is safe because during resource reservation, o < M is always
   followed, and N is an integer multiple of M.

   In the orchestration period, from timeslot 0 to M-1 is the first
   scheduling period, from timeslot M to slot 2M-1 is the second
   scheduling period, and so on.  From timeslot N-M to slot N-1 is the
   N/M scheduling period.  Each timeslot in the scheduling period
   corresponds to an associated queue, which is used to store packets
   for sending in the corresponding timeslot.

   According to the timeslot resource reservation process mentioned
   above, when the sub-burst corresponding to any outgoing timeslot
   (e.g, z) arrived at the outgoing port of any node of the path, the
   ongoing sending timeslot (e.g, j) in the orchestration period of the
   outgoing port must be offset by o before the outgoing timeslot (z),
   and meet o < M, which means that the sub-burst does not randomly
   arrive at this node, but strictly abide by the time so that when it
   reaches the outgoing port, it will definitely fall into the ongoing
   sending timeslot (j).

   Next, we briefly demonstrate that the sub-burst that arrives at the
   outgoing port during the ongoing sending timeslot (j) can be safely
   inserted into the corresponding queue in the scheduling period, and
   that queue will not overflow.

   Assuming that each timeslot in the orchestration period has a virtual
   queue, the length of the virtual queue is the MRB of that timeslot.
   For example, termed the virtual queue corresponding to the outgoing
   timeslot (z) as queue-z, the packets that can be inserted into
   queue-z may only come from the following bursts:






Peng, et al.            Expires 23 November 2023               [Page 22]


Internet-Draft           Packet Generic Timeslot                May 2023


      During the ongoing sending timeslot j = (z-M+1+N)%N, the bursts
      that arrive at the outgoing port, that is, these bursts may
      reserve the outgoing timeslot (z) according to o = M-1.

      During the ongoing sending timeslot j = (z-M+2+N)%N, the bursts
      that arrive at the outgoing port, that is, these bursts may
      reserve the outgoing timeslot (z) according to o = M-2.

      ... ...

      During the ongoing sending timeslot j = (z-1+N)%N, the bursts that
      arrive at the outgoing port, that is, these bursts may reserve the
      outgoing timeslot (z) according to o = 1;

      The total reserved amount of all these bursts does not exceed the
      MRB of the outgoing timeslot (z).

   Then, when the ongoing sending timeslot changes to z, queue-z will be
   sent and cleared.  In the following time, starting from timeslot z+1
   to the last timeslot N-1 in the orchestration period, there are no
   longer any packets inserted into queue-z.  Obviously, this virtual
   queue is a great waste of queue resources.  In fact, queue-z can be
   reused by the subsequent outgoing timeslot (z+M)%N.  Namely:

      During the ongoing sending timeslot j = (z+1)%N, the bursts that
      arrive at the outgoing port, that is, these bursts may reserve the
      outgoing timeslot (z+M)%N according to o = M-1.

      During the ongoing sending timeslot j = (z+2)%N, the bursts that
      arrive at the outgoing port, that is, these bursts may reserve the
      outgoing timeslot (z+M)%N according to o = M-2.

      ... ...

      During the ongoing sending timeslot j = (z+M-1)%N, the bursts that
      arrive at the outgoing port, that is, these bursts may reserve the
      outgoing timeslot (z+M)%N according to o = 1.

      The total reserved amount of all these bursts does not exceed the
      MRB of the outgoing timeslot (z+M)%N.

   It can be seen that queue-z can be used by any outgoing timeslot
   (z+k*M)%N, where k is a non negative integer.  By observing
   (z+k*M)%N, it can be seen that the minimum z satisfies 0<= z< M, that
   is, the entire orchestration period actually only requires M queues
   to store packets, which are the queues corresponding to M timeslots
   in the scheduling period.  That is to say, the minimum z is the
   timeslot id in the scheduling period, while the outgoing timeslot



Peng, et al.            Expires 23 November 2023               [Page 23]


Internet-Draft           Packet Generic Timeslot                May 2023


   (z+k*M)% N is the timeslot id in the orchestration period.  The
   latter obtains the former by moduling M, which can then access the
   queue corresponding to the former.  In short, the reason why a queue
   can store packets from multiple outgoing timeslots without being
   overflowed is that the packets stored in the queue earlier (more than
   M timeslots ago) have already been sent.

4.  Global Timeslot ID

   The outgoing timeslots we discussed in the previous sections are all
   local timeslots for nodes.  This section discusses the situation
   based on global timeslot.

   Global timeslot style refers to that all nodes in the path are
   identified with the same timeslot id, which of course requires all
   nodes to use the same timeslot length.  The advantages are that the
   resource reservation based on global timeslots is simple, always
   reserving a specified outgoing timeslot for the service flow.  There
   is no need to establish a local timeslot mapping relationship on each
   node or carry this mapping relationship in packets.  The packet only
   needs to carry the unique global timeslot id.  However, the
   disadvantage is that the latency performance of the path may be
   large, which depends on the phase difference between the inherent
   orchestration periods between the adjacent nodes.  Another
   disadvantage is that the success rate of finding a path that matches
   the service requirements is not as high as local timeslot style.

   Global timeslot style requires that the orchestration period is equal
   to the scheduling period, mainly considering that arrival packets
   with any global timeslot id can be successfully inserted into the
   corresponding queue.  However, as the scheduling period is less than
   the orchestration period is the ideal design goal, further research
   is needed on other methods(such as basically aligning orchestration
   period between nodes), to ensure that packets with any global
   timeslot id can queue normally when the scheduling period is less
   than the orchestration period.

4.1.  Fixed Timeslot Mapping Mode

   So far, the packet timeslot scheduling scheme presented above is to
   reserve a fixed outgoing timeslot for services in the orchestration
   period, even if global slot-id is used.

   Compared to the local timeslot style, global timeslot style means
   that the incoming timeslot i must map to the outgoing timeslot i too.
   As the example shown in Figure 7, each orchestration period contains
   6 timeslots.  Node V has three connected upstream nodes U1, U2, and
   U3.  During each hop forwarding, the packet accesses the outgoing



Peng, et al.            Expires 23 November 2023               [Page 24]


Internet-Draft           Packet Generic Timeslot                May 2023


   timeslot corresponding to the global timeslot id and forwards to the
   downstream node with the global timeslot id unchanged.  For example,
   U1 sends some packets with global slot-id 0, termed as g0, in the
   outgoing timeslot 0.  The packets with other global slot-id 1~5 are
   similarly termed as g1~g5 respectively.  The figure shows the
   scheduling results of these 6 batches of packets sent by upstream
   nodes when node V continues to send them.

         0   1   2   3   4   5   0   1   2   3   4   5   0   1   2
       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
   U1  | g0| g1| g2|   |   |   |   |   |   |   |   |   |   |   |   |
       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+


          1   2   3   4   5   0   1   2   3   4   5   0   1   2   3
        +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
   U2   |   |   | g3| g4|   |   |   |   |   |   |   |   |   |   |   |
        +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+


         5   0   1   2   3   4   5   0   1   2   3   4   5   0   1
       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
   U3  | g5|   |   |   |   |   |   |   |   |   |   |   |   |   |   |
       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+


           0   1   2   3   4   5   0   1   2   3   4   5   0   1   2
         +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
   V     |   |   |   | g3| g4| g5| g0| g1| g2|   |   |   |   |   |   |
         +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+


                                  Figure 7

   In this example, the mapping relationship of the outgoing timeslot
   from U1 and the ongoing sending timeslot of V is i -> i, so the
   reserved outgoing timeslot for the incoming timeslot i is i+6.  The
   mapping relationship of the outgoing timeslot from U2 and the ongoing
   sending timeslot of V is i -> i-1, so the reserved outgoing timeslot
   for the incoming timeslot i is i.  And, the mapping relationship of
   the outgoing timeslot from U3 and the ongoing sending timeslot of V
   is i -> i+1, so the reserved outgoing timeslot for the incoming
   timeslot i is i+6-1.

   For the headend, the residence delay depends on the arrival time when
   the sub-burst arrives at the scheduler and specified global timeslot.
   Suppose that the ongoing sending timeslot is j at the arrival time
   when the sub-burst arrives at the scheduler, and there is time T_j



Peng, et al.            Expires 23 November 2023               [Page 25]


Internet-Draft           Packet Generic Timeslot                May 2023


   left before the end of the timeslot j, and the sub-burst is specified
   to use global timeslot i, then, the reserved outgoing timeslot is
   (j+o)%N, where o equals (N+i-j)%N.

   The residence delay equation for headend is similar to
   Section 3.1.4.1.

   For any other nodes, the residence delay depends on the phase
   difference of the orchestration period between upstream node (U) and
   this node (V), i.e., the difference between the orchestration period
   of the upstream node (U) and the ongoing sending period of this node
   (V).  Suppose that the incoming timeslot i mapped to the ongoing
   sending timeslot j, and there is time T_ij left before the end of the
   timeslot j.  Then, the reserved outgoing timeslot is (j+o)%N, where o
   equals (N+i-j)%N.

   The residence delay equation for intermediate node is similar to
   Section 3.1.4.2.

   For example, the packets g3 sent by upstream node U2 falls into the
   ongoing sending timeslot 2 of node V, it can be sent in outgoing
   global timeslot 3.  In this case, the residency delay in the node V
   is small.  While, the packets g5 sent by upstream node U3 falls into
   the ongoing sending timeslot 0 of node V, so it needs to wait for
   timeslot 0, 1, 2, 3, 4 to be sent in global outgoing timeslot 5.  In
   this case, the residency delay in the node V is large.

   For example, the packets g0 sent by upstream node U1 fall into the
   ongoing sending timeslot 0 of node V, the packets need to wait for
   the end of the ongoing sending period to be sent in the global
   outgoing timeslot 0 in the next round of orchestration period, which
   will introduce a large node residency delay.  It should be noted that
   in this case, the packets g0, when they fall into the ongoing sending
   timeslot 0, cannot be placed in the buffer corresponding to timeslot
   0.  Instead, it needs to be stored in a buffer prior to the packet
   timeslot scheduler (such as the buffer on the input port side) for a
   fixed latency (such as a fixed timeslot) and then released to the
   timeslot scheduler.  This fixed-latency buffer is only created for
   specific upstream nodes.  It can be determined according to the
   initial detection result of the mapping relationship between the
   outgoing timeslot of the upstream node and the ongoing sending
   timeslot of this node.  If the initial detection result is slot-id i
   -> slot-id i, it needs to be introduced, otherwise it is unnecessary.
   After the introduction of fixed-latency buffer, the new detection
   result will no longer be i -> i.

   The end-to-end delay equation for intermediate node is similar to
   Section 3.1.4.4.



Peng, et al.            Expires 23 November 2023               [Page 26]


Internet-Draft           Packet Generic Timeslot                May 2023


4.2.  Variable Timeslot Mapping Mode

   In variable timeslot mapping Mode, timeslot resources are still
   reserved based on global timeslot id on the control plane, but in
   actual forwarding, packets do not necessarily have to wait until the
   outgoing time slot before being sent.

   Variable timeslot mapping is similar to cell scheduling in ATM.
   During each hop forwarding, the packets dynamically maps to an idle
   local outgoing timeslot according to the global slot-id, according to
   the principle of minimum offset (or expected offset range) between
   the global slot-id and local slot-id, but the sending packets still
   carry the global slot-id without changed.  In this case, the delay
   performance is related to the mapping algorithm (i.e., the scheduling
   algorithm) adopted.  The suggested scheduling algorithm will be
   discussed in later versions.

   As the example shown in Figure 8, each orchestration period contains
   6 timeslots.  Node V has three connected upstream nodes U1, U2, and
   U3.  Node U1 dynamically maps the packets with global slot-id 0,1,2
   to the outgoing timeslot 3,4,5 respectively, node U2 dynamically maps
   the packets with global slot-id 3,4,5 to the outgoing timeslot 4,5,0
   respectively, and node V dynamically map the packets with global
   slot-id 0~5 to the outgoing timeslot 4,5,1,0,2,3 respectively.

         0   1   2   3   4   5   0   1   2   3   4   5   0   1   2
       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
   U1  |   |   |   | g0| g1| g2|   |   |   |   |   |   |   |   |   |
       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+


           0   1   2   3   4   5   0   1   2   3   4   5   0   1   2
         +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
   U2    |   |   |   |   | g3| g4| g5|   |   |   |   |   |   |   |   |
         +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+


            0   1   2   3   4   5   0   1   2   3   4   5   0   1   2
          +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
   V      |   |   |   |   | g0| g1| g3| g2| g4| g5|   |   |   |   |   |
          +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+


                                  Figure 8

   Because the service flow arrived at the network entry node is
   periodic, each entry node should maintain state about a fixed mapping
   relationship between global slot-id and the actual outgoing slot-id



Peng, et al.            Expires 23 November 2023               [Page 27]


Internet-Draft           Packet Generic Timeslot                May 2023


   for each flow, so that it is more likely that a fixed runtime mapping
   relationship will appear on each intermediate node to avoid jitter.
   However, the characteristics of variable timeslot mapping determine
   that this fixed runtime mapping relationship is not always
   guaranteed.  For example, with the addition or deletion of service,
   the mapping status of global slot-id to the actual outgoing slot-id
   may have to be updated on the entry node, which will correspondingly
   lead to changes in the runtime mapping relationship on the
   intermediate node.

   The main purpose of global slot-id is used in the timeslots resource
   allocation.  Within the resource planning of the controller, the
   timeslot resources identified by each global slot-id are allocated
   for multiple limited service flows without conflict.  Intuitively, if
   all service flows access the outgoing timeslot according to the fixed
   timeslot mapping mode, there is no timeslot conflict, that is, the
   total timeslot resources can meet all limited service requirements;
   Variable timeslot mapping mode is to dynamically access the nearby
   idle outgoing timeslots without introducing timeslot conflicts, and
   it will not lead to the result that the total timeslot resources are
   not enough.  How to predict whether the nearest outgoing timeslot is
   idle is the focus of the selected scheduling algorithm.

   Assume that the packets with global slot-id i accessing the outgoing
   timeslot (j+o)%N nearby do not make no resources available when the
   packets with global timeslot (j+o)%N arrive now or soon, the delay
   performance of variable timeslot mapping mode is better than fixed
   timeslot mapping mode, because o may be less than (N+i-j)%N.
   However, it will introduce jitter.

   A possible implementation of variable timeslot mapping mode is to
   schedule packets with work-conserving behavior.  That is, the sub-
   burst is still inserted into the queue corresponding to the global
   timeslot id, but that queue can be sent in advance, as long as the
   queues corresponding to all timeslots from the ongoing sending
   timeslot to the previous timeslot of this global timeslot are empty.
   It should be noted that this behavior of sending in advance cannot
   disrupt the basic timeslot rotation (round robin).

5.  Summary of Timeslot Style

   Depending on the strategy of reserving timeslot resources, different
   timeslot styles will be presented, as shown in the table below.








Peng, et al.            Expires 23 November 2023               [Page 28]


Internet-Draft           Packet Generic Timeslot                May 2023


       +===============+========================+==================+
       |  Strategy     |      Timeslot Style    |   Referrence     |
       +===============+========================+==================+
       | Flexible o    | Local timeslot style   | section 3.1.4    |
       | (1<=o<M)      |                        |                  |
           +---------------+------------------------+------------------+
       | Constant o    | Global timeslot style  | section 4        |
       | (o=(N+i-j)%N) |                        |                  |
       +---------------+------------------------+------------------+
       | Constant o    | Multi-CQF              | [Multi-CQF]      |
       | (o=1)         |                        |                  |
       +---------------+------------------------+------------------+

                                  Figure 9

6.  Queue Design

   The number of timeslot queues should be designed according to the
   number of timeslots included in the scheduling period.  Each timeslot
   corresponds to a separate queue (or queue group), in which the
   buffered packets must be able to be sent within a timeslot.

   The length of the queue, i.e., the total number of bits that can be
   reserved or sent for a timeslot, does not have to be set to be
   exactly equal to the link rate multiplied by the timeslot length.
   This is because the bandwidth requirements of other non-deterministic
   services and protocols running in the network should also be
   considered.

6.1.  Full Queues

   When the scheduling period length is equal to the orchestration
   period length, the node will implement full queues.  The advantage is
   that the actual forwarding resources are the same view as the
   resources used for reservation, so that the resource reservation
   process is simple.  However, the disadvantage is that because the
   scheduling period is generally large to cover all services
   requirements, the number of queues maintained by the node will be
   large.












Peng, et al.            Expires 23 November 2023               [Page 29]


Internet-Draft           Packet Generic Timeslot                May 2023


   For example, if the accumulated length of all queues supported by the
   hardware is 4G bytes, the queue length corresponding to a timeslot of
   10us at a port rate of 100G bps is 1M bits, then a maximum of 32K
   timeslot queues can be provided, and the maximum length of the
   orchestration period supported is 320ms.  However, considering the
   queue resource requirements of other non-deterministic services, the
   packet timeslot function can only use some of the queue resources,
   such as 10K~20K queues.  In this case, the length of the
   orchestration period supported by the node is 100~200 ms.

6.2.  Non-full Queues

   When the length of the scheduling period is less than the length of
   the orchestration period, the node will implement a non-full queues.
   The advantages and disadvantages are opposite to the full queues
   option.  The actual forwarding resources are inconsistent with the
   view of the resources reservation.  But the number of queues
   maintained by the node is small.

7.  Admission Control on the Headend

   On the network entry, traffic regulation must be performed on the
   incoming port, so that the service flow does not exceed its T-SPEC
   such as burst interval, burst size, maximum packet size, etc.  This
   kind of regulation is usually the shaping using leaky bucket combined
   with the incoming queue that receives service flow.  A service flow
   may contain discrete multiple sub-bursts within its periodic burst
   interval.  The leaky bucket depth should be larger than the maximum
   packet size, and should be consistent with the reserved burst
   resources required for the maximum sub-burst.

   The scheduling mechanism described in this document has a requirement
   on the arrival time of service flows on the network entry.  It is
   hoped that the distribution of sub-bursts (after regulation) of the
   service flow will always appear in a fixed position within the
   orchestration period.  Some amount of jitter is permited for this
   position, but the jitter cannot reach a scheduling cycle.
   Intuitively, small o (i.e., the offset between the reserved outgoing
   timeslog and ongoing sending timeslot) can tolerate large early
   arrival deviations, while large o can tolerate large late arrival
   deviations.  Only in this way can we achieve the ideal design goal
   that scheduling period is smaller than the orchestration period.
   Otherwise, for randomly arriving service flows, it can be supported
   by taking the scheduling period equal to the orchestration period
   (optoin-1), or it can be supported by introducing an explicit buffer
   put before the scheduler on the network entry to let the arrival time
   always meet the fixed position (option-2).




Peng, et al.            Expires 23 November 2023               [Page 30]


Internet-Draft           Packet Generic Timeslot                May 2023


   *  Note that due to randomness of arrival time, the packet may just
      miss the scheduling (or arrive too earlier) and need to wait in
      the scheduling queue (in the case of option-1) or the explicit
      buffer (in the case of option-2) for the next scheduling period.
      From this perspective, we suggest that it is best for service
      flows to strictly obey their arrival time, which should be the
      ideal admission control for all scheduling mechanisms that attempt
      to forward service flows in the specified time windown.

8.  Frequency Synchronization

   The basic explanation for frequency synchronization is that the
   crystal frequency of the hardware is equal, which enables all nodes
   in the network to be in the same inertial frame and have the same
   time lapse rate.  This is a prerequisite for all latency based
   scheduling mechanisms.  This frequency synchronization mechanism
   (such as syncE) is not within the scope of this document.

   Sometimes, people also refer to the frequency asynchrony as the
   timeslot rotation frequency difference caused by different node
   configurations with different timeslot lengths.  This document
   supports the interconnection between nodes with this type of
   frequency asynchrony.

9.  IANA Considerations

   TBD.

10.  Security Considerations

   TBD.

11.  Acknowledgements

   TBD.

12.  References

12.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.



Peng, et al.            Expires 23 November 2023               [Page 31]


Internet-Draft           Packet Generic Timeslot                May 2023


12.2.  Informative References

   [ATM-LATENCY]
              "Bounded Latency Scheduling Scheme for ATM Cells", 1999,
              <https://ieeexplore.ieee.org/document/780828/>.

   [CQF]      "Cyclic Queuing and Forwarding", 2017,
              <https://ieeexplore.ieee.org/document/7961303>.

   [Multi-CQF]
              "Multiple Cyclic Queuing and Forwarding", 2021,
              <https://www.ieee802.org/1/files/public/docs2021/new-finn-
              multiple-CQF-0921-v02.pdf>.

Authors' Addresses

   Shaofu Peng
   ZTE
   China
   Email: peng.shaofu@zte.com.cn


   Peng Liu
   China Mobile
   China
   Email: liupengyjy@chinamobile.com


   Kashinath Basu
   Oxford Brookes University
   United Kingdom
   Email: kbasu@brookes.ac.uk


   Aihua Liu
   ZTE
   China
   Email: liu.aihua@zte.com.cn


   Dong Yang
   Beijing Jiaotong University
   China
   Email: dyang@bjtu.edu.cn







Peng, et al.            Expires 23 November 2023               [Page 32]