rtgwg                                                          S. Bryant
Internet-Draft                                 University of Surrey 5GIC
Intended status: Informational                                  A. Clemm
Expires: April 22, 2022                     Futurewei Technologies, Inc.
                                                        October 19, 2021

                 Token Cell Routing Data Plane Concepts


   Token Cell Routing is a powerful yet hardware friendly method of
   constructing data plane packets to meet the needs of new
   applications.  It is based on the use of token cells (special kinds
   of lightly structured tokens) to provide pointers to procedures pre-
   positioned in the forwarding layer together with the parameters
   needed to provide the required processing context.  A packet can be
   composed from multiple token cells as needed to result in new new
   network processing and forwarding semantics.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on April 22, 2022.

Copyright Notice

   Copyright (c) 2021 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect

Bryant & Clemm           Expires April 22, 2022                 [Page 1]

Internet-Draft                   TCR-DP                     October 2021

   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   4
   2.  The TCR Concept . . . . . . . . . . . . . . . . . . . . . . .   4
   3.  Relationship to prior work  . . . . . . . . . . . . . . . . .   5
   4.  TCR Packet Structure  . . . . . . . . . . . . . . . . . . . .   6
   5.  Token Cell types and categories . . . . . . . . . . . . . . .   7
   6.  Token Cell Structure  . . . . . . . . . . . . . . . . . . . .   9
   7.  Token Cell Processing Model . . . . . . . . . . . . . . . . .  10
   8.  Token Cell Processing Order . . . . . . . . . . . . . . . . .  12
     8.1.  Serial Token Cell Processing  . . . . . . . . . . . . . .  13
     8.2.  Parallel Token Cell Processing  . . . . . . . . . . . . .  14
     8.3.  Combined Serial and Parallel Token Cell Processing  . . .  16
   9.  Token Cell Pushing and Token Cell Popping . . . . . . . . . .  18
   10. Selected Token Cell Type Categories . . . . . . . . . . . . .  19
     10.1.  Disposition Token Cells  . . . . . . . . . . . . . . . .  19
     10.2.  Scratchpad and Metadata Token Cells  . . . . . . . . . .  19
     10.3.  Conditional and Directive Token Cells  . . . . . . . . .  21
     10.4.  Security Token Cells . . . . . . . . . . . . . . . . . .  21
   11. Example applications of TCR . . . . . . . . . . . . . . . . .  23
     11.1.  Basic Tunneling of Payload . . . . . . . . . . . . . . .  24
     11.2.  Latency-Based Forwarding . . . . . . . . . . . . . . . .  25
     11.3.  Forwarding with Flexible Addressing  . . . . . . . . . .  26
     11.4.  Forwarding with iOAM analytics . . . . . . . . . . . . .  27
     11.5.  FRR with Latency-Based Forwarding  . . . . . . . . . . .  30
     11.6.  Segment Routing with Latency-Based Forwarding  . . . . .  32
     11.7.  Enhanced Segment Routing with Latency-Based Forwarding .  32
     11.8.  Enhanced Segment Routing with Differentiated iOAM  . . .  33
   12. Items for further discussion  . . . . . . . . . . . . . . . .  35
   13. Security Considerations . . . . . . . . . . . . . . . . . . .  36
   14. IANA Considerations . . . . . . . . . . . . . . . . . . . . .  38
   15. References  . . . . . . . . . . . . . . . . . . . . . . . . .  38
     15.1.  Normative References . . . . . . . . . . . . . . . . . .  38
     15.2.  Informative References . . . . . . . . . . . . . . . . .  38
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  39

1.  Introduction

   Advances in data plane protocols are needed to address new network
   requirements that stretch existing protocols (including MPLS, IPv6
   and Segment Routing) to their limits.  Token Cell Routing (TCR) is a
   new network layer data plane technology that provides the ability to

Bryant & Clemm           Expires April 22, 2022                 [Page 2]

Internet-Draft                   TCR-DP                     October 2021

   program the data plane to meet the needs of many new operational
   scenarios.  TCR is based on token cells which provide a flexible
   method describing the required packet action to the forwarder as well
   as carrying any parameters and other data necessary to correctly
   execute the required action.

   Packet actions are not limited to forwarding actions, and it is
   possible perform multiple packet actions at any given node.  For
   example, packet actions can include application of special QoS
   algorithms, collection of telemetry, even assessment of dynamic
   conditions before the performing of other actions.  Token Cells can
   be differentiated by type depending on the type of packet action that
   they represent.

   TCR is thus analogous to each packet carrying a stack of pointers to
   procedures together with the data needed by those procedures.  The
   structure used allows token cells to be sequenced and parallelized in
   groups, and permits the use of pointers to information in other token
   cells.  This results in a powerful method constructing advanced new
   packet types from token cells to meet network needs of new

   TCR therefore supports customizable semantics with which packets are
   to be processed by nodes encountered along a path, it accommodates
   flexible addressing semantics that do not necessarily depend on a
   single addressing format.  TCR accommodates custom "guidance" beyond
   forwarding (such as the definition of QoS treatments that are to be
   applied, the ability to differentiate behavior depending on dynamic
   context encountered at a node, and the ability to collect and pre-
   process telemetry in support of manageability applications).  TCR
   enables the direct processing via a "scaffold" that explicitly
   indicates serialization, parallelization, disposition rules.  It that
   enables a "lego-esque" composition of packet processing behavior /
   features while at the same time being hardware-friendly, and easy to
   optimize for performance at line rate general in nature and easy to

   A new TCR features is added by introducing a new procedure in the
   forwarder and then including in the packet a pointer to that
   procedure.  The procedure knows how to interpret the other
   information carried in the token cell.  In many ways this is similar
   to the way new FECs are introduced into MPLS and new instructions
   introduced to segment routing.

   TCR thus provides a general, highly extensible data plane that
   supports custom semantics which is hardware friendly.  It thus takes
   the programmability of both MPLS and Segment Routing to a new level
   of capability.

Bryant & Clemm           Expires April 22, 2022                 [Page 3]

Internet-Draft                   TCR-DP                     October 2021

   A key differentiator from earlier protocols is the ability to process
   a variable number of processing actions at each hop, at directed by
   the token cell structure.  Furthermore token cells do not need to be
   processed in the order in which they are placed in the packet, and
   may be explicitly programmed for a flow.

   We would like to note that at the time of writing the current -00
   version of the draft, this represents a sketch of an idea that
   neither we or anyone else has built.  Thus, it is likely to have bugs
   and certainly has many aspects that can be improved on.  We would be
   delighted to work with others who are interested in exploring this
   idea and developing it further.  A starter set of discussion items is
   included in Section 12 towards the end of the document.

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "OPTIONAL" in this document are to be interpreted as described in
   BCP14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

2.  The TCR Concept

   The foundation of TCR is the construction of the packet from a set of
   token cells.  A token cell is an extended length, type, value
   construct.  The type and part of the value is processed by a longest
   match engine, which operates much like an IP address lookup engine,
   but operates on arbitrary constructs rather than being confined to
   address lookup.  As part of its value, the token cell may also carry
   parameters specific to the token cell type that are needed to process
   the token cell.  Depending on the token cell type, different code
   points can be invoked that process the token cell.  The token cell
   type determines the semantics, i.e. the function to be applied.  It
   also defines any structure that may be contained in the token cell

   Token Cell processing can have generalizable packet processing
   semantics: forwarding is a common semantic, but other semantics can
   be applied, in a similar manner to the way in which an MPLS label has
   generalized semantics.  The processing of a token cell is: Input -
   Match - Effect, where "effect" is one of forwarding, token cell
   disposition, or something else (such as, conditional directives or
   QoS treatment).  Packet processing based on the first n bits of the
   token cell at a known position, which is matchable on prefix, in
   which case less significant bits serve as input.

Bryant & Clemm           Expires April 22, 2022                 [Page 4]

Internet-Draft                   TCR-DP                     October 2021

   The TCR approach to packet design is differentiated from common
   protocol designs in that it allow for processing of variable number
   of token cells per hop, as directed by token cell structure.  Which
   token cells to process, and whether token cells have
   interdependencies that require them to be processed in serial order
   or whether they allow for parallelization, is indicated by the token
   cells themselves, as the token cell structure includes length and
   serialization indicators.  Processing can be serialized per "next
   token cell" indicator.  Processing can be parallelized per "manifest
   token cell" that refers to parallel token cells in order to allow for
   optimization.  Note that there is no requirement to "must"
   parallelize.  Instead, parallelization is an optimization that nodes
   with support for parallel packet processing stages may take advantage
   of to reduce packet latency due to packet forwarding time.

   Token Cells do not necessarily need to be processed in stack order
   but can be located wherever is most efficient.  Some token cells may
   not be processed at all but can be used to carry meta-data (read-only
   or writable) that can be referred to from other token cells.

   The TCR approach allow for extensibility and programmability through:

   o  Level 1: Combining different token cells

   o  Level 2: Parameterizing token cells

   o  Level 3: Introducing new types of token cells

3.  Relationship to prior work

   In general data plane packets have been designed with a fixed
   structure plus some variable number of TLVs to provide additional
   instructions/advice to the forwarder.  In general any address (for
   example IP address) or instruction (MPLS label of SR SID) is of a
   fixed length although may be structured into a prefix and suffix
   arrangement to support aggregation and is processed using a longest
   match lookup.  Where TLVs are used there is their inclusion in the
   packet is normally free form and thus it is necessary search the
   packet for the set of TLVs that need to be processed.  MPLS has a
   very primitive parameter system in which one label may be used to
   provide context for the label that follows.  Examples of this are the
   use of the context label and the use of the ELI/EL pair.

   In deigning TCR we noted the generality and simplicity of the MPLS
   label stack model and the effectiveness of the longest match
   technique used in IP lookups.  Putting these two together we
   concluded that an LTV approach allowed the creation of a simple,
   powerful, extensible, hardware friendly packet design.

Bryant & Clemm           Expires April 22, 2022                 [Page 5]

Internet-Draft                   TCR-DP                     October 2021

   As we will see later in the document, concatenating the T(ype) and
   the V(alue) allows a single longest match look-up to resolve which
   table to look up the action in, and then to absorb as much of the V
   as is necessary to determine the specific action to take on the T.
   The stack and pointer structure means that it is not necessary to
   search for the applicable TLVs, they the forwarder is led to them
   through the token cell structure pushed onto the packet.

   The general method of processing is thus input - match - effect via a
   match processor.  The normal effect is to process one token cell at a
   time in series, but an effect might be to process multiple other
   token cells in parallel where the forwarder supports multiple
   concurrent operations (where parallization is not supported serial
   processing results in the same effect).  Token Cells can have
   interdependencies and they can allow for cross-referencing (e.g.
   meta-data, scratch-pad)

4.  TCR Packet Structure

   A TCR packet Figure 1 is a series of token cells, each token cell
   carrying a component of the packet delivery system or the payload

   <Token Cell>
   <Token Cell>
   <Token Cell>
   <Token Cell>

                    Figure 1: Structure of a TCR Packet

   Token Cells are a unit of packet processing that may include
   parameters and/or data.  The semantics, structure and processing of
   the token cell is determined by the token cell type.  A Token Cell
   can be thought of as a type of "stem cell" that can be morphed into
   any packet component, including components not yet designed,
   resulting in a highly programmable packet design.

   The preamble contains a small number of packet elements that are
   always present in all packets, such as version identification and
   TTL.  It has yet to be decided if the preamble should be carried
   conventionally or be carried within a token cell.

   There is no separate payload portion of the packet.  Instead, the
   payload is carried within a token cell, generally located at the tail
   of the packet.  This allows for different payload delivery semantics
   at the destination, including simply stripping the payload off or

Bryant & Clemm           Expires April 22, 2022                 [Page 6]

Internet-Draft                   TCR-DP                     October 2021

   applying a special type of codec.  It also allows for the possibility
   of payload-less packets that can be used for signaling and control

   A token cell MAY contain a complete TCR packet permitting
   hierarchical encapsulation.

5.  Token Cell types and categories

   Token Cells have a type.  Types themselves can be categorized,
   depending on the purpose which token cells of that type serve.

   The following token cell categories are provided as part of the TCR

   o  Forwarding: A forwarding token cell specifies the destination
      address and method of delivery of the packet.  It may also include
      the source address as a parameter, but this could also be
      specified in a separate token cell.  Different types within this
      category may differentiate between address types such as IPv6 or

   o  SLO: A Service Level Objective (SLO) token cell specifies the
      target quality of delivery, such as latency, delivery time,
      required discard properties etc., each differentiated by
      corresponding IDs as separate types.  This allows intermediate
      nodes on the path to apply special treatment to the packet, such
      as scheduling algorithms, resource reservation, or prioritization,
      in order meet SLOs as requirement.

   o  Metadata: These token cells carry metadata that can be referenced
      and accessed as other token cells are being processed.  Metadata
      can thus be decoupled from token cells that access it, allowing
      for their independent disposal, not interfering with pushing and
      popping of other token cells.

   o  Scratchpad: Scratchpad token cell are in effect writeable metadata
      token cells, a category of token cell in which the network takes
      "notes" during the packet transit.  Examples of this include
      recording the route, adding proofs-of-transit, telemetry data, or
      packet transit time of particular nodes that were traversed.

   o  Security: A security token cell signs parts of the packet with an
      agreed cryptographic signature.  It includes a signature mask that
      specifies which token cells and/or portions thereof are covered by
      the signature.  This allows for the possibility of not only the
      sender, but also nodes in transit being able to sign portions of
      the packet.  One example use would be for telemetry data that is

Bryant & Clemm           Expires April 22, 2022                 [Page 7]

Internet-Draft                   TCR-DP                     October 2021

      added to a scratchpad by a node being traversed while leaving
      other parts of the scratchpad open to modification by other nodes.

   o  Conditional: A conditional token cell is able to test one of more
      conditions and make invokation of the next token cell (NT)
      dependent on the conditions evaluating to true.  This allows to
      define more complex behavior, such as the invocation of a
      particular function depending on a dynamic condition encountered
      at a node.

   o  Directive: A directive token cell specifies some type of action
      that should be performed.  An example would be a directive to
      collect telemetry or OAM data.

   o  Manifest: A manifest token cell provides a method of specifying
      which token cells may be processed in parallel.  Parallel
      processing is optional, and the token cells can also be correctly
      processed serially.  It is up to the entity that specifies the
      manifest to ensure that the parallelism is safe.

   o  Rendezvous: A rendezvous token cell is a token cell used to ensure
      that all parallel operations have completed and that it is hence
      safe to resume serial operation of the forwarder.  A rendezvous
      token cell may specify the first serial operation to execute after
      the rendezvous, or it may simply hand off to a new token cell.

   o  Disposition: A disposition token cell describes what is to be done
      when the packet leaves the TCR domain.  Such a token cell might,
      for example specify a pseudowire [RFC3985] action (strip the TCR
      header and send the payload to interface X), or a VPN action
      (lookup the payload IP address in VRF Y).  However, the mechanism
      introduces the opportunity to attach a more sophisticated
      disposition action, for example "if the packet arrives before time
      T, forward using VRF V, otherwise drop the packet".

   o  Payload: A payload token cell simply carries the payload as its

   o  Other: This is a catch-all category to allow for token cell types
      that do not fit any of the other categories.

   Some of the mentioned categories will be described in greater detail
   in Section 10.  In addition to the mentioned categories, it is
   expected that other categories would be introduced as needed.

   The grouping of types into categories may prove useful for various
   purposes.  For example, it will allow for the articulation of packet
   grammars and "best packet practices", such as mandating that a packet

Bryant & Clemm           Expires April 22, 2022                 [Page 8]

Internet-Draft                   TCR-DP                     October 2021

   contain at least one token cell of the forwarding category, or that
   the first token cell in a TCR packet must not be a token cell of
   categories metadata, scratchpad, or security.  Types allow to further
   differentiate token cells within a given category.

   It should be noted that some of the token categories should be
   considered experimental.  Which token types and even which token
   categories to support will in depend on the needs of actual

6.  Token Cell Structure

   The structure of a token cell is shown in Figure 2:

   <Next Token>
   <Token Cell Type>  -+
       <Cat/Purpose>   |
       <ID>            +- Match Zone
   <Token Cell Blob>   |
       <Prefix>       -+

                  Figure 2: Structure of a TCR Token Cell

   Token Cells will vary in length depending on the token cell type and
   the contents of the token cell blob, although in practice a given
   token cell type MAY be a fixed size.

   Next token cell is a relative pointer (offset) to the next token cell
   to be processed as part of the group of token cells to be
   sequentially processed as a part of the packet action at the node.
   If there is no next token cell to process, its value is null.
   Processing of a packet begins with processing the first token cell.
   Whether or not any additional token cells are processed depends on
   the guidance provided via the Next Token field.

   The token cell type is used to identify the category or purpose of
   the token cell (Cat/Purpose) and the sub-type (ID) within that class.
   An example of a purpose might be "Forwarding", indicating that the
   token cell represents an instruction to forward the packet closer to
   the destination.  An example of a sub-type within that category might
   be "IPv6", indicating tht the destination is identified by the
   following IPv6 address.

   The set of IDs consists of a set of well known IDs and a set of user
   specified IDs.  This provides both an extensible, and a programmable
   mechanism for enhancing the protocol over time and within

Bryant & Clemm           Expires April 22, 2022                 [Page 9]

Internet-Draft                   TCR-DP                     October 2021

   deployments.  Token cell type, category, and sub-type ID are not
   fixed-length fields but represent conventions.

   The token cell blob carries the information needed to process the
   explicit packet that carries it, and/or is a place to record
   information about the packet for later use.  Within the token cell
   blob there is a prefix and a suffix, which may themselves each be
   structured.  The structure of the token cell blob depends on the
   token cell type, some of which may themselves be subjected to

   The token cell blob prefix, which is neither fixed nor a fixed length
   field is used to qualify the type in the lookup.  For example, if the
   sub-type was IPv6 destination address the prefix would be an IPv6
   address.  However this is a more general device than just a
   destination address and allows for token cell type categorization.
   This may prove useful for various purposes (e.g. packet grammars and
   Best Packet Practices).

   The match zone is the portion of the token cell that is subject to
   look up (see processing model section).  The model is that the lookup
   will be a longest match of the whole match zone.  The token cell
   design does not specify the length of the match zone or the length of
   either the ID or the prefix.  It is a property of longest match
   lookup that it will either consume all the bits it needs, or reject
   the lookup.  If a result is found the result can specify the
   structure of the token cell blob.  Note that this is a model, and
   there is much scope for implementor optimization without sacrificing
   the generality of the design.

   The number of bytes sent to the lookup engine is implementation
   specific.  If the attempted match is longer than needed, the longest
   match will ignore the overspill.  If more bytes are needed, it is a
   property of longest match that the lookup can be restarted from where
   it left off.

   Within the blob, and in particular within the suffix, any structure
   can be carried such as subfields, parameters.  The definition of what
   the suffix contains and how it is structured is part of the token
   cell type definition.

7.  Token Cell Processing Model

   The TCR processing model is shown in Figure 3.

Bryant & Clemm           Expires April 22, 2022                [Page 10]

Internet-Draft                   TCR-DP                     October 2021

   |<   Token Cell Match Zone    >|
   |<Token Cell type><Blob Prefix>|<Blob Suffix>
   |______________________________|    |
            ||          ||             |
            ||          ||             |
             \          /              |
              +---v---+                |
                  |                    |
                  V                    |
               Lookup                  |
               Engine                  |
                  |                    |
                  |                    |
                  V                    |
        Lookup o/p parameters          |     Pipeline
             Code Pointer              |   state register
                  |                    |       !   ^
                  |                    |       !   !
                  V                    |       !   !
                Code <-----------------+       !   !
              "callback" <~~~~~~~~~~~~~~~~~~~~~'   !
                  |                                !
                  |                                !
                  V                                !
                Effect ~~~~~~~~~~~~~~~~~~~~~~~~~~~~'

                      Figure 3: TCR Processing Model

   This operates as follows.  First the token cell match zone is fed
   into the lookup engine.  The lookup engine performs a longest match
   and returns a parameter block which includes the address of the code
   to be executed on the token cell by the forwarder.  That code knows
   how to interpret the token cell.  Readers will recognize the genesis
   of this is a hybrid between an IP address lookup which performs a
   longest match lookup and returns a parameter block to the IP
   forwarding code, and an MPLS label lookup which performs a fixed size
   look-up logically returns a pointer to the executable code (MPLS
   forward packet, pseudowire, VPN etc) and a parameter block.

   Where it cannot be clear that what the length of the blog prefix is
   from the lookup result, for example where the address is an IPv6
   address, that length needs to be encoded in the prefix in some
   convenient way, such as as a prefix to the prefix.

   From the above, it is clear that there is no constraint on the type
   and structure of the prefix and thus any address type or other
   construct may be submitted to the lookup engine.  Token Cell types

Bryant & Clemm           Expires April 22, 2022                [Page 11]

Internet-Draft                   TCR-DP                     October 2021

   and prefix sets may be added to the forwarder by adding appropriate
   data to the database queried by the lookup engine, and providing the
   corresponding callback code to the network processor, in a manner
   similar to that used in MPLS and in Network Programming.  In this
   regard the approach is compatible with proven hardware forwarding

   The callback code has access to any other element of the token cell
   that it needs, and indeed to other elements of the packet as
   required.  Implementations MAY impose a reasonable limitation on the
   number of processing cycles within which callback code needs to

   As part of the effect of processing a token cell, a pipeline state
   register can be written which can serve as input for the processing
   of subsequent token cells of the same packet.  This allows to compose
   a pipeline of token cells being processed in which the output of one
   function serve as input to the next function.  In the special case of
   rendezvous token cells that have multiple predecessors, their
   respective outputs are merged as part of the specific rendezvous

8.  Token Cell Processing Order

   It is entirely possible to require several token cells to be
   processed at any given node.  For example, one token cell may contain
   a forwarding directive, whereas another token cell might contain a
   directive related to QoS treatment of the packet for meeting a
   Service Level Objective (SLO), while a third token cell indicates
   that certain telemetry data from traversed nodes should be collected.

   In the simple case that token cells can or should be serially
   processed, the processing of token cells will simply be chained, as
   directed by the token cells' respective NT fields.  After processing
   of a token cell concludes, the reference in the NT field is resolved
   and processing continues with that token cell.  If the NT contains no
   reference (or in the special case of a conditional token cell
   evaluating to false), the processing of the packet concludes.  In
   that case, the processing of a packet will require n stages, n being
   the number of chained token cells.

   A packet processing pipeline needs to support a depth that is
   equivalent to the maximum number of token cells that can be chained.

   In some cases, optimization is possible by exploiting
   parallelization.  In the earlier example, it may be possible to
   perform some tasks in parallel, such as the application of QoS
   treatment and collection of telemetry data, while other tasks may

Bryant & Clemm           Expires April 22, 2022                [Page 12]

Internet-Draft                   TCR-DP                     October 2021

   still need to be serialized, such as determining which outgoing
   interface to use as a forwarding decision before performing the QoS
   actions against that interface.  The fact that certain token cells
   may be processed concurrently can be indicated through a special
   manifest token cell that references the token cells that follow.
   Each of those token cells can in turn have their own successors, in
   effect resulting in separate packet processing "threads" (all
   processing different token cells of the same packet).  While the
   processing of the manifest adds an additional cycle, depending on the
   complexity of the workflow this may be more than offset by the
   parallelization that ensues.

   While full exploitation of the optimization potential may require
   advances in hardware pipeline design, it should be emphasized any
   such optimization is optional and not required.  Also, to maintain
   packet ordering, the packet will generally still be required to pass
   through all n pipeline stages.  That said, the option of
   parallelization does allow for hardware pipeline designs able to
   exploit concurrency of multiple threads and accommodating a larger
   number of token cells than would otherwise be supported by pipelines
   of a given depth, respectively reduce the pipeline depth that needs
   to be supported.

8.1.  Serial Token Cell Processing

   All packets have an element of serial processing in that the preamble
   and then the token cell that follows are always processed.

   A serial group of token cells is constructed by using the next
   pointer to point to the token cell to processed on completion of the
   token cell.  The end of a serial token cell group is logically
   indicated using a NULL next token cell pointer, although none of the
   foregoing should be taken as dictating the wire format which will be
   the subject of another text.

Bryant & Clemm           Expires April 22, 2022                [Page 13]

Internet-Draft                   TCR-DP                     October 2021

   Token Cell T1 <Length>
                 <Next Token> T2
                 <Token Cell Type>
                 <Token Cell Blob>
   Token Cell T2 <Length>
                 <Next Token> T4
                 <Token Cell Type>
                 <Token Cell Blob>
   Token Cell T3 <Length>
                 <Next Token> -
                 <Token Cell Type>
                 <Token Cell Blob>
   Token Cell T4 <Length>
                 <Next Token> -
                 <Token Cell Type>
                 <Token Cell Blob>

         Figure 4: TCR Serial Token Cell Processing of Token Cells

   Figure 4 shows four token cells.  The serial processing instructed by
   this construct is that token cell 1 is to be processed, followed by
   token cell 2 and then followed by token cell 4.  There are many
   reasons why this construct is interesting and these are discussed
   later in this document.

   In order to simplify the graphical annotation, references to token
   cell cells are in the following simply indicated by a numeric
   identifier instead of being depicted by arrows.

8.2.  Parallel Token Cell Processing

   In some cases it is desirable to introduce parallel processing to the
   packet where there are token cells have no result interdependencies.
   We do this through the introduction of a manifest token cell that
   contains a set of pointers or offsets to other token cells.

Bryant & Clemm           Expires April 22, 2022                [Page 14]

Internet-Draft                   TCR-DP                     October 2021

   <Next Token>
   <Token Cell Type>  -+
       <Cat/Purpose>   |
       <ID>            +- Match Zone
   <Token Cell Blob>   |
       <Par 1>        -+ -+
       <Par 2>            |
         ..               +- Manifest
       <Par 4>            |
       <Par 5>           -+

             Figure 5: TCR Parallel Processing of Token Cells

   The structure of the manifest token cell is shown in Figure 5.  The
   initial part of the token cell is standard to all token cells.  There
   follows a list of token cell pointers to the set of token cells to be
   processed in parallel.  The end of the set of token cells to be
   processed in parallel is determined when the end of the token cell is
   reached as indicted by the token cell length.  When all the child
   token cells have completes execution the token cell pointed to by the
   next token cell field is executed.

   Note that the match zone overlaps the manifest.  The token cells in
   the manifest will however be ignored by the longest match which will
   complete with the ID.

   Also note that, as previously mentioned, parallelism is an efficiency
   issue, not a correctness issue.  A forwarder that does not support
   the parallel dispatch of token cells or that supports less
   parallelism than specified in the manifest can choose to execute
   individual token cells (respectively groups of token cells) serially.
   It is up to the sender to construct the packet as needed and make any
   token cell interdependencies explit, without requiring the network to
   second-guess whether or not there are any such interdependencies.  In
   other words, when the order in which token cells are processed might
   result in different behavior, it is the responsibility of the sender
   to specify any required serialization as needed.

   In most cases where it is used, a manifest token cell will typically
   be the first token cell after the preamble, to exploit the
   possibility of concurrent processing threads for multiple token cells
   in the same packet from the onset.  However, this is not an
   architectural requirement and manifest token cells could also occur
   later in the packet.

   It is possible to remerge concurrent token cell threads using a
   special rendezvous token cell.  A rendezvous token cell awaits for

Bryant & Clemm           Expires April 22, 2022                [Page 15]

Internet-Draft                   TCR-DP                     October 2021

   each of its predecessors to have completed before resuming
   processing.  In addition, output from predecessors (i.e. pipeline
   state register content) can be aggregated as needed.

8.3.  Combined Serial and Parallel Token Cell Processing

   Figure 6 illustrates the concept of parallel and serialized
   processing further.  Please note that this is an admittedly complex
   scenario and most scenarios in practice will be simpler.

                  / | \
                 /  |  \
               T2  T3  T4
               |      / | \
               T5    T6 T7 T8

   Figure 6: TCR Combined Serial and Parallel Processing of Token Cells

   After processing T1, three tokens (T2, T3, T4) can be processed
   concurrently.  T5 has a serialization dependency on T2.  T6, T7, T8
   can be procesed concurrently once T4 has been processed.  Finally, T9
   has a serialization dependency on T7.

   A corresponding packet is shown in Figure 7.  Manifest token cells
   are introduced to represent the fact that T2, T3, T4 respectively T6,
   T7, T8 can be processed in parallel.  A Next Token field of "-"
   indicates there is no next token.

   Token Cell T1 <Length>
                 <Next Token> M1
                 <Token Cell Type>
                 <Token Cell Blob>
   Token Cell M1 <Length>
                 <Next Token> -
                 <Token Cell Type = Manifest>
                 <Token Cell Blob>
                   <Par 1> T2
                   <Par 2> T3
                   <Par 3> T4
   Token Cell T2 <Length>
                 <Next Token> T5
                 <Token Cell Type>
                 <Token Cell Blob>
   Token Cell T3 <Length>
                 <Next Token> -

Bryant & Clemm           Expires April 22, 2022                [Page 16]

Internet-Draft                   TCR-DP                     October 2021

                 <Token Cell Type>
                 <Token Cell Blob>
   Token Cell T4 <Length>
                 <Next Token> M2
                 <Token Cell Type>
                 <Token Cell Blob>
   Token Cell M2 <Length>
                 <Next Token> -
                 <Token Cell Type = Manifest>
                   <Par 1> T6
                   <Par 2> T7
                   <Par 3> T8
   Token Cell T5 <Length>
                 <Next Token> -
                 <Token Cell Type>
                 <Token Cell Blob>
   Token Cell T6 <Length>
                 <Next Token> -
                 <Token Cell Type>
                 <Token Cell Blob>
   Token Cell T7 <Length>
                 <Next Token> T9
                 <Token Cell Type>
                 <Token Cell Blob>
   Token Cell T8 <Length>
                 <Next Token> -
                 <Token Cell Type>
                 <Token Cell Blob>
   Token Cell T9 <Length>
                 <Next Token> -
                 <Token Cell Type>
                 <Token Cell Blob>

   Figure 7: TCR Combined Serial and Parallel Processing of Token Cells

   Token Cell M1 is a manifest that indicates three children (at the
   protocol level, the number of potential children is subject only to
   packet size constraints).  From a protocol perspective all three
   children: token cell 2, token cell 3 and token cell 4 can execute
   immediately and concurrently.  When token cell 2 completes token cell
   5 runs, when token cell 5 completes that that processing branch is
   completed.  Token Cell 3 runs and when it completes that token cell
   branch is completed.  Token Cell M2 is also a manifest with three
   children: token cell 6, token cell 7 and token cell 8.  When token
   cell 6 completes that processing branch is completed.  When token
   cell 7 is completed token cell 9 runs.  When token cell 9 completes
   that processing branch is completed.  When token cell 8 is completed

Bryant & Clemm           Expires April 22, 2022                [Page 17]

Internet-Draft                   TCR-DP                     October 2021

   that processing branch is completed.  When token cells 5, 3, 6, 9 and
   8 are completed the token cell group is completed.

   It should be noted that the current design of the manifest token cell
   type includes all pointers to subsequent token cells as part of the
   token cell blob.  Alternative designs are conceivable in which (for
   example) the next token field would be populated with the first of
   the concurrent successors, with only additional token cells (beyond
   the first) to be referenced from the token cell blob.

   It should also be noted packet permutations can be accommodated, i.e.
   the order of the token cells in the token cell group can be any order
   convenient to the network application designer.  For example, the
   same token cells from Figure 7 could have been arranged also in the
   following sequence: T1 - M1 - T2 - T5 - T4 - T3 - M2 - T6 - T7 - T9 -
   T8.  The token cells do not need to be arranged in a stack in the way
   that MPLS or SRv6 arrange their labels or SIDs respectively; the
   order in which to process token cells is always resolved by the NT
   reference (respectively any manifest token cell references).  However
   it is RECOMMENDED that backward references are avoided as a packet
   with no backward references will not form a processing loop.

9.  Token Cell Pushing and Token Cell Popping

   Token Cell groups can be stacked by pushing a group of token cells
   onto a packet to encode a hierarchical set of operations to be
   executed on the packet as it journeys to its destination.  This is
   similar to IP tunneling or the pushing and popping of MPLS labels.

   In a manner similar to the pipe model described in [RFC3270] it is a
   matter for the protocol designer and the operator whether the pushed
   token cells are able to understand and interact with existing the
   tokens cells.  This will be discussed further in a future version of
   this document.

   Similarly when a token cell group has accomplished its purpose on the
   packet journey it is popped so that the forwarded can gain access to
   the next element of processing.

   Again considerations similar to [RFC3270] may apply.

   For the purposes of this discussion a Token Cell group may consist of
   a single token cell.

   Considerations regarding penultimate hop popping will be included in
   a future version of this text.

Bryant & Clemm           Expires April 22, 2022                [Page 18]

Internet-Draft                   TCR-DP                     October 2021

10.  Selected Token Cell Type Categories

   An overview of token cell type categories was given in Section 5.  In
   this section, some of these categories will be explained in further

10.1.  Disposition Token Cells

   A disposition token cell describes to the network what actions are to
   be performed as a packet egresses the TCR domain.  A forwarding token
   cell may for this purpose include a reference to a disposition token
   cell as a parameter.

   All existing IETF network protocols include a disposition semantic
   although sometimes this is implicit.  For example when IPv6 has a
   next header of "TCP" the disposition instruction is to dispose of the
   IPv6 header and hand the packet to the TCP handler.  Similarly
   pseudowires have a disposition instruction in the PW label
   instructing the forwarder how to dispose of the MPLS header and to
   reconstruct the packet in its original format.

   However as the metadata carried in the packet become more
   sophisticated there is a requirement for the disposition to become
   more sophisticated.

   Disposition token cells thus formalize the description of the egress
   behavior of the network on the packet and allow a richer egress
   semantic to be described.

   It is conceivable to define TCR such that in the absence of a
   reference to a disposition token cell, TCR will revert to implicit
   disposition behavior when the destination address of a forwarding
   token cell is reached.  That will involve popping the token cells up
   to the forwarding token cell and resuming processing with the
   subsequent token cell.  In other words, a disposition token cell is
   in that case used to specify any special semantics that would go
   beyond vanilla implicit disposition.

10.2.  Scratchpad and Metadata Token Cells

   Metadata, provided by a sender of a packet or by an ingress node, can
   provide important guidance to nodes along a path to guide processing
   of the packet.  Examples include SLOs that should be taken into
   account for QoS treatment, profiles to apply towards proessing a
   packet, and more.  Other uses include the carrying of security
   material as well as the tagging of packets for classification

Bryant & Clemm           Expires April 22, 2022                [Page 19]

Internet-Draft                   TCR-DP                     October 2021

   Scratchpads refer to writeable metadata, i.e., metadata that can not
   only be accessed but also modified or added by nodes "in flight",
   i.e. during transit.  Example uses include collection of telemetry
   and iOAM data [I-D.ietf-ippm-ioam-data], auxiliary data used for
   measurements such as intermediate time stamps, or proof-of-transit
   data and data verifying certain properties of nodes being traversed
   (such as whether from a trusted vendor or located in a certain

   Metadata and scratchpad token cells allow to carry such metadata as
   part of a packet independent of the token cells that access it.  This
   decoupling allows to dispose of metadata and scratchpads independent
   of token cells that process it.  For example, it makes it conceivable
   to collect different iOAM data along different segments of a path, as
   directed by different token cells, and to export the data only once
   an exporter or egress node is reached.  This way, scratchpad token
   cells enable applications that rely on sharing of node-specific data
   along a path to do so without the complications of having to
   introduce piggyback extensions to the underlying protocol.  In
   addition to adding and updating data items in scratchpad token cells
   along a path, also additional scratchpads can be added.  It also
   avoids the need for the same metadata to be copied across token
   cells, for example in cases where the same SLOs are applicable across
   different segments, each governed by their own respective token

   Metadata and scratchpad items can be referenced by other token cells.
   The specific format of those references is to be determined by the
   rules governing the content of the respective token cell blobs for
   the token cell type; in general the reference format will involve an
   offset from the referring token cell.

   Disposition semantics need to include defining what is to happen with
   metadata or scratchpad carried in corresponding token cells.
   Possible disposition actions include:

   o  Discard

   o  Export (possibly parameterized to specificy export mechanism as
      well as export target)

   o  Log (again, possibly parametrized with a logging target)

   Metadata and scratchpad data can also be independently authenticated
   and secured.  This allows, for example, to ensure that scratchpad
   data that is added or modified by intermediate nodes cannot be
   tampered with.  It also allows for the implementation of operations

Bryant & Clemm           Expires April 22, 2022                [Page 20]

Internet-Draft                   TCR-DP                     October 2021

   that authenticate both metadata and scratchpad data before processing
   it further.

   Metadata, let alone scratchpad data, is a concept that is not
   directly supported by token-based protocols such as SR or MPLS today.
   While it would be possible to encode such as part of a token, its
   processing would require the token itself be processed (e.g. as part
   of a forwarding operation), tied to its being pushed or popped on a
   stack.  This would make it harder to e.g. update scratchpad data
   (that should be protected from popping and may be buried underneath a
   token or label stack), to access data without effectively copying it
   across tokens, and generally to accommodate metadata and scratchpad
   processing where the lifecycle of data items does not directly
   correspond to that of segments.  With TCR, there is no need to
   process metadata in "stacked order" nor need for complex token cell
   rewrite rules in order to preserve data.

10.3.  Conditional and Directive Token Cells

   Directive Token Cells allow the specification of some type of action
   or function that should be performed as a result of the token cell
   being processed.  An example of a directive would be a request to
   collect telemetry or in-situ OAM data and record it as part of
   scratchpad.  It is expected that there can potentially be a large
   number of possible directives, each distinguished by its own ID
   respectively token cell type.  Different token cell types may impose
   their own respective structure on the token cell blob to represent
   different parameters.

   Conditional Token Cells are similar to Directive Token Cells in that
   they allow the specification of an operation to be performed.
   However, it is different from other categories of tokens in that the
   outcome of the operation determines whether the NT field will be
   processed, i.e. whether processing will subsequently resume with the
   token cell referenced by the NT field or whether it should terminate.
   This allows for the definition of functionality that should be
   performed depending on certain conditions that are being encountered.
   For example, a conditional token cell might allow for a different
   paths to be selected depending on dynamic circumstances such as load
   conditions.  Similarly, the collection of certain telemetry data
   might be made dependent on certain conditions being encountered.

10.4.  Security Token Cells

   Security token cells enable a security mechanism that allows to sign
   the invariant elements of the packet whilst avoiding signing the
   packet elements that are modified during the passage of the packet
   through the network, such as scratchpad token cells.

Bryant & Clemm           Expires April 22, 2022                [Page 21]

Internet-Draft                   TCR-DP                     October 2021

   Likewise, they allow for differentiated signing of different parts of
   the packet by different signers, such as in cases where nodes add
   data items to a scratchpad that should be independently signed.

   A security token cell allows to carry signature material pertaining
   to elements of the packet.  It includes the following items:

   o  An identification of the signer

   o  A mask indicating the parts of the packet respectively token
      cell(s) being signed

   o  The signature material itself

   The general structure of the security token cell is as follows

   <Next Token>
   <Token Cell Type>  -+
       <Cat/Purpose>   |- Match Zone
       <ID>           -+
   <Token Cell Blob>
       <Key ID>
       <Token Cell Mask>

             Figure 8: Structure of a TCR Security Token Cell

   The token purpose and ID specify that this is a security token, the
   exact structure of the security token, and the type of signature that
   has been used.  The structure used here is illustrative and used to
   explain the concept.

   The token cell blob contains the security material and the mask.  A
   possible structure is as follows:

   As described in [RFC8754] the Key ID is used to identify the
   preshared key and the algorithm, though the algorithm may be
   indicated by the token ID (this is a matter for the token designer).

Bryant & Clemm           Expires April 22, 2022                [Page 22]

Internet-Draft                   TCR-DP                     October 2021

   The Hashed Message Authentication Code (HMAC) is the hash of the
   complete TCR security token and the N TCR tokens, and within those
   tokens the token elements specified by the token mask.

   For efficiency the mask operates at a number of levels:

   o  The next N token cells

   o  The marked tokens within the next N token cells

   o  Octets or words within a specific token cell

   N specifies the number of tokens covered by this security token in
   addition to itself.  If present, Token Cell Mask is a structure that
   specifies which tokens are actually protected.  Token_Protected.NP is
   the number of tokens within the token mask.
   Token_Protected.Full_Token is a bit mask of Token_Protected.NP bits
   indicating which complete tokens are to be included in the signature.
   Token_Protected.Byte_Protected is a bit mask of Token_Protected.NP
   bits indicating which complete tokens are to be included in the

11.  Example applications of TCR

   This section contains a number of examples illustrating how the TCR
   token system is used to create or "program" packets.  The examples
   are illustrative and not exhaustive.  Only the essential features of
   the packet are shown.

   In order to simplify depiction of TCR packets, specifically the order
   in which tokens are processed and the cross-dependencies between
   tokens, we will introduce a syntax which identifies individual tokens
   by a numeric token identifier, and that allows to reference tokens
   using this identifier as opposed to packet offsets depicted using
   ASCII art pointers.  This is merely done for convenience of textually
   representing TCR packets in this draft.  It is independent of the
   actual TCR packet and token structure, in which tokens do not have
   separate identifiers and are simply referenced by offset.  This
   applies to Next Token fields, as well as for disposition.  A
   disposition token cell is referenced through a field in token cells
   whose type is of the forwarding category.

   A packet is simply represented by a sequence of tokens.  Each token
   is represented by a set of fields and their values.  In addition, a
   "pseudo field" is introduced for the numeric token identifier.  A
   reference to another token (from the <Next Token> field, from a
   manifest token, or from a rendezvous token) contains as value the
   respective token identifier.

Bryant & Clemm           Expires April 22, 2022                [Page 23]

Internet-Draft                   TCR-DP                     October 2021

11.1.  Basic Tunneling of Payload

   In this example we consider a the case where a packet is to be
   tunneled across a network as one might do in a sub-IP network.

   For illustrative purposes we show in Figure 9 an IPv6 address being
   used as a destination address, but any other address family can be
   used with the corresponding forwarding token type.

   Token Cell T1 <Length>
                 <Next Token> -
                 <Type = IPv6fwd>
                 <Prefix = Dest Add>
                 <Disp = T2>
   Token Cell T2 <Length>
                 <Next Token>
                 <Type = Disp>
                 <Disp Parameters>
   Token Cell T3 <Length>
                 <Next Token>
                 <Type = Payload>

                    Figure 9: Basic Tunnelling Over TCR

   The packet starts with a preamble followed by token T1 which is IPv6
   forwarding token, a token with the semantic that it is to deliver the
   packet as close to the target address as it can reach.  There is no
   next token cell to be processed until arrival at the destination and
   so Next Token is NULL.

   When the packet arrives at its destination the token that is pointed
   to by the disposition pointer is noted and all tokens up to the
   disposition token are popped (in this case only token 1 is popped).

   The disposition token T2 is processed and the disposition parameters
   say how the payload is to be processed.  This may be as simple as
   providing the protocol type of the payload, but it may also provide
   information other information such as the identity of a VRF table to
   use, or an interface to dispatch the packet to in the case of a
   pseudowire.  It also provides a pointer to the payload.  When the
   parser knows how to dispose of the packet, it pops Token 2 and
   processes token 3.  Token 3, in the case of containing a tunneled IP
   packet, just requires the length to be noted and corrected for the
   token overhead, the type confirmed as payload and payload to be
   forwarded as an IP packet.

Bryant & Clemm           Expires April 22, 2022                [Page 24]

Internet-Draft                   TCR-DP                     October 2021

11.2.  Latency-Based Forwarding

   In this example we consider the previous case (basic tunneling of
   payload) Section 11.1 and extend it to provide support for on-time
   delivery according to an end-to-end latency objective.  The support
   is provided using a Latency-Based Forwarding (LBF).  LBF bases the
   decision on when to sent to packet on how urgently or at what exact
   time it needs to arrive.  To do so, LBF involves an algorithm that
   determines at any given node whether the packet is "on track" to meet
   its latency objective, and matches the QoS treatment and scheduling
   of the packet against a latency "budget" that is determined from
   latency objective, the latency that was already incurred, and the
   expected remaining latency towards the destination.  For further
   details, please refer to [DOI.10.1109_NOMS47738.2020.9110431].

   To indicate that a packet should undergo LBF treatment, a
   corresponding token cell type of category "SLO" is introduced.  When
   it is processed, the packet is subject to LBF.  Parameters for LBF
   include the end-to-end latency objectives and a helper parameter to
   assess the latency being incurred.  An additional input is the egress
   interface that is obtained from processing of the fowarding token
   cell that precedes it and that can be passed using TCR's pipeline
   state register.

   A packet that enables LBF in conjunction with the preivious case is
   depicted in Figure 10.  It adds one additional token cell over the
   previous use case, in essence adding the LBF functionality as a
   "module" that is combined with the earlier functionality.  The
   ability to compose functionality by simply combining corresponding
   token cells into a packet is part of what makes TCR quite powerful.

   Processing of the packet is similar to the processing of the basic
   tunneling case except that Token 1 the IPv6fwd token contains a
   pointer to a further token that is to be sequentially processed at
   every hop, including arrival at the destination.  The Next Token
   pointer in Token 1 points to token 2 which specifies that the packet
   must arrive at a specified time, and contains information needed to
   record the time taken and hence the time at which it should optimally
   leave its current node.

   When the packet arrives at its destination a final check is made to
   see if the arrival time requirement was met and it is processed
   according to the failure to arrive on time instructions in either the
   LBF_ontime token or the disposition token.  All further processing is
   as per the above basic tunneling case.

Bryant & Clemm           Expires April 22, 2022                [Page 25]

Internet-Draft                   TCR-DP                     October 2021

   Token Cell T1 <Length>
                 <Next Token> T2
                 <Type = IPv6fwd>
                 <Prefix = Dest Add>
                 <Disp = T3>
   Token Cell T2 <Length>
                 <Next Token> -
                 <Type = LBF_ontime>
                 <Blob = LBF parameters>
   Token Cell T3 <Length>
                 <Next Token>
                 <Type = Disp>
                 <Disp Parameters>
   Token Cell T4 <Length>
                 <Next Token>
                 <Type = Payload>

                         Figure 10: LBF Using TCR

11.3.  Forwarding with Flexible Addressing

   The is an emerging need to support multiple address technologies.
   There are two cases in point, firstly where an operator wants to use
   a short address to to address the infrastructure nodes in their
   network.  This is particularly the case in segment routing where
   there is competition amongst the vendors to introduce address
   compression due to the extended size of SRv6 data plane headers.
   There is a secondary benefit to this in that if an address system
   other than IP is used within the provider network there are security
   benefits as has been found in operating MPLS.  Finally it has been
   speculated that some application environments would prefer to use
   their native addresses rather than manage the mapping between those
   addresses and IP addresses.

   This is achieved by creating a new token cell type, populating the
   FIB with the corresponding addresses and installing in the forwarder
   the necessary forwarding code.  That forwarding code may be generic
   since the action will almost certainly be address family independent.

   In the example shown in Figure 11, the address family is encoded in
   the address itself.  In an alternative model, the token cell type
   would be specific to the address family.

Bryant & Clemm           Expires April 22, 2022                [Page 26]

Internet-Draft                   TCR-DP                     October 2021

   Token Cell T1 <Length>
                 <Next Token> -
                 <Type = FlexAddr>
                 <Prefix = AddressFamily + Address>
                 <Disp> T2
   Token Cell T2 <Length>
                 <Next Token> -
                 <Type = Disp>
                 <Disp Parameters>
   Token Cell T3 <Length>
                 <Next Token>
                 <Type = Payload>

       Figure 11: Indirect LBF UsingUsing Flexible Addresses in TCR

11.4.  Forwarding with iOAM analytics

   In many cases, there is desire to collect in-situ OAM data
   [I-D.ietf-ippm-ioam-data] as a packet traverses a path.  There are
   multiple applications for this, including but not limited to
   diagnosing performance, identification of bottlenecks, and generation
   of data to feed machine-learning algorithms for service optimization.

   Using TCR, it is possible to indicate that iOAM data should be
   collected using a corresponding token cell.  The token cell can
   contain in-situ parameters, such as which data items to collect.  In-
   situ data itself can be added to a scratchpad, allowing for its
   export using a variety of means upon disposition of packet.

   One scenario is depicted in Figure 12.  In this particular example,
   we assume that the iOAM data can be collected per T3 in parallel with
   the forwarding decision being made per T2 in order to show also use
   of a manifest (T1). iOAM Data items are written to the scratchpad in
   T5.  T4 indicates disposition, T6 contains the payload.

   It should be noted that rather than simply collecting iOAM data,
   other operations could be applied to aggregate that data and result
   in more refined behavior.  In the interest of brevity, the example
   does not feature security tokens used by intermediate nodes to sign
   the scratchpad data items that they add.

Bryant & Clemm           Expires April 22, 2022                [Page 27]

Internet-Draft                   TCR-DP                     October 2021

   Token Cell T1 <Length>
                 <Next Token> -
                 <Type = Manifest>
                   <Par 1> T2
                   <Par 2> T3
   Token Cell T2 <Length>
                 <Next Token> -
                 <Type = IPv6fwd>
                 <Prefix = Dest Add>
                 <Disp> T4
   Token Cell T3 <Length>
                 <Next Token> -
                 <Type = Directive-iOAM>
                   <Ioam-parameters - data items to collect>
                   <Scratchpad> T5
   Token Cell T4 <Length>
                 <Next Token> -
                 <Type = Disp>
                 <Disp Parameters>
   Token Cell T5 <Length>
                 <Next Token> -
                 <Type = Scratchpad>
   Token Cell T6 <Length>
                 <Next Token>
                 <Type = Payload>

                          Figure 12: iOAM in TCR

   If iOAM data to be collected includes telemetry about the egress
   interface, the manifest cell can be omitted as the forwarding
   decision needs to be made prior to collecting the iOAM data.  The
   resulting packet becomes even simpler, as depicted in Figure 13.

Bryant & Clemm           Expires April 22, 2022                [Page 28]

Internet-Draft                   TCR-DP                     October 2021

   Token Cell T1 <Length>
                 <Next Token> T2
                 <Type = IPv6fwd>
                 <Prefix = Dest Add>
                 <Disp> T3
   Token Cell T2 <Length>
                 <Next Token> -
                 <Type = Directive-iOAM>
                   <Ioam-parameters - data items to collect>
                   <Scratchpad> T4
   Token Cell T3 <Length>
                 <Next Token> -
                 <Type = Disp>
                 <Disp Parameters>
   Token Cell T4 <Length>
                 <Next Token> -
                 <Type = Scratchpad>
   Token Cell T5 <Length>
                 <Next Token>
                 <Type = Payload>

                    Figure 13: iOAM in TCR - serialized

   To show the modularity that TCR enables, the third scenario shows
   iOAM data to be collected while at the same time LBF is being applied
   to the same packet (Figure 14).  Note that in this case, LBF and iOAM
   could have also been applied in parallel, in which case a manifest
   token cell would be added.  T1 would then point to the manifest token
   cell, which in turn would point to T2 and T3 as successors.

Bryant & Clemm           Expires April 22, 2022                [Page 29]

Internet-Draft                   TCR-DP                     October 2021

   Token Cell T1 <Length>
                 <Next Token> T2
                 <Type = IPv6fwd>
                 <Prefix = Dest Add>
                 <Disp> T4
   Token Cell T2 <Length>
                 <Next Token> T3
                 <Type = LBF_ontime>
                 <Blob = LBF parameters>
   Token Cell T3 <Length>
                 <Next Token> -
                 <Type = Directive-iOAM>
                   <Ioam-parameters - data items to collect>
                   <Scratchpad> T6
   Token Cell T4 <Length>
                 <Next Token> -
                 <Type = Disp>
                 <Disp Parameters>
   Token Cell T5 <Length>
                 <Next Token> -
                 <Type = Scratchpad>
   Token Cell T6 <Length>
                 <Next Token> -
                 <Type = Payload>

                      Figure 14: iOAM and LBF in TCR

11.5.  FRR with Latency-Based Forwarding

   This example depicted in Figure 15 extends the earlier LBF example to
   include a fast re-route diversion.  We show only a single token cell
   (T0) added at the point of local repair (PLR), but of course the
   repair might be more complex and need multiple intermediate staging
   counts to successfully be repaired.

Bryant & Clemm           Expires April 22, 2022                [Page 30]

Internet-Draft                   TCR-DP                     October 2021

   Token Cell T0 <Length>
                 <Next Token> T2
                 <Type = IPv6fwd>
                 <Prefix = Reroute Add>
   Token Cell T1 <Length>
                 <Next Token> T2
                 <Type = IPv6fwd>
                 <Prefix = Dest Add>
                 <Disp = T3>
   Token Cell T2 <Length>
                 <Next Token> -
                 <Type = LBF_ontime>
                 <Blob = LBF parameters>
   Token Cell T3 <Length>
                 <Next Token>
                 <Type = Disp>
                 <Disp Parameters>
   Token Cell T4 <Length>
                 <Next Token>
                 <Type = Payload>

                     Figure 15: FRR with LBF Using TCR

   The PLR was expecting to forward the packet normally and so is aware
   that the packet is latency sensitive and understands the semantics
   and hence importance of token 2.  To maintain the expected path
   quality the PLR MUST use an FRR path while also ensuring the SLO is
   still being adhered to per the LBF token cell.  The FRR path
   therefore needs to feature nodes able to support LBF token cells, and
   not incur a latency penalty that would physically prohibit being able
   to meet the SLO.  This path can be selected by the SDN controller, or
   locally through the use of path attributes applied to the normal IP-
   FRR path selection process.

   On detecting a local repairable failure of the next hop or the link
   to the next hop the PLR pushes one or more tokens as is necessary to
   deliver the packet to the destination.  In each case the next token
   pointer points to Token 2 the LBF token.  When the packet arrives at
   the intermediate node chosen by the PLR for the next stage of the
   repair, the token (in this case Token 0) at the top of the token
   stack is popped and forwarding proceeds as dictated by token 1.  The
   above operation can clearly be carried out as many times as necessary
   on the packet to provide a repair, by pushing as many tokens as are

Bryant & Clemm           Expires April 22, 2022                [Page 31]

Internet-Draft                   TCR-DP                     October 2021

   Note that the scheme in this example uses an implicit disposition
   operation by intermediate nodes as described above in Section 10.1.

11.6.  Segment Routing with Latency-Based Forwarding

   The operation described in Section 11.5 in which repair tokens are
   pushed onto the packet is identical to the operation that happens
   when a packet is configured for segment routing (SR).  Technically
   the packet depicted in Figure 15 IS a segment routed packet.

   It therefore follows that implementing segement routing and segment
   routing enhanced by features such as enhanced QoS is trivial in TCR.
   As we shall see in the next sections, TCR is capable of enhancing SR
   significantly beyond that.

11.7.  Enhanced Segment Routing with Latency-Based Forwarding

   This example illustrates how TCR can be used to add an enhanced QoS
   capability such as latency based forwarding to segment routing.  We
   saw previously Section 11.6 how all segments can be made to execute a
   common policy but it might be desirable to execute a different policy
   in different segments.  For example, some segments might underly
   their own latency objectives (just for that segment), without
   affecting the overall end-to-end objective.

   A packet that achieves this is depicted below in Figure 16.  The
   packet in the example has three segments.  Segments 1 and 3 apply LBF
   for the end-to-end latency objective, per T5.  Segment 2 applies a
   "sub- SLO" just for that particular segment, per T4.  Disposition of
   T1 and T2 is implicit (popping the token on reaching the segment
   destination), whereas disposition of T3 is explicit per T6.

Bryant & Clemm           Expires April 22, 2022                [Page 32]

Internet-Draft                   TCR-DP                     October 2021

   Token Cell T1 <Length>
                 <Next Token> T5
                 <Type = IPv6fwd>
                 <Prefix = Seg 1 Add>
   Token Cell T2 <Length>
                 <Next Token> T4
                 <Type = IPv6fwd>
                 <Prefix = Seg 2 Add>
   Token Cell T3 <Length>
                 <Next Token> T5
                 <Type = IPv6fwd>
                 <Prefix = Seg 3 Add>
                 <Disp = T6)
   Token Cell T4 <Length>
                 <Next Token> -
                 <Type = LBF_ontime (for segment)>
                 <Blob = LBF parameters>
   Token Cell T5 <Length>
                 <Next Token> -
                 <Type = LBF_ontime (for end-to-end)>
                 <Blob = LBF parameters>
   Token Cell T6 <Length>
                 <Next Token>
                 <Type = Disp>
                 <Disp Parameters>
   Token Cell T7 <Length>
                 <Next Token>
                 <Type = Payload>

               Figure 16: Enhanced Segment Routing Using TCR

11.8.  Enhanced Segment Routing with Differentiated iOAM

   The final example shows a variation of the previous example.  Instead
   of applying a specific latency objectives for particular segments, it
   is possible to also invoke other functionality, such as collecting
   certain iOAM data only for particular segments, or collecting
   additional iOAM data for one of the segments, or even for collecting
   different sets of iOAM data along different segments.  The particular
   example depicted (Figure 17) shows a packet with three segments.  One
   set of parameters is collected for segments 1 and 3 using scratchpad
   T6, another set of parameters is collected for segment 2 using
   scratchpad T7.

   If instead, segment 2 should collect additional parameters beyond
   those collected for segments 1 and 2, this could be easily

Bryant & Clemm           Expires April 22, 2022                [Page 33]

Internet-Draft                   TCR-DP                     October 2021

   accommodated by simply setting "Next Token" of T4 to T5 instead of
   null.  (This does not include a possible optimization for
   parallelization, but attests to the flexibility of the approach.)

   Token Cell T1 <Length>
                 <Next Token> T4
                 <Type = IPv6fwd>
                 <Prefix = Seg 1 Add>
   Token Cell T2 <Length>
                 <Next Token> T5
                 <Type = IPv6fwd>
                 <Prefix = Seg 2 Add>
   Token Cell T3 <Length>
                 <Next Token> T4
                 <Type = IPv6fwd>
                 <Prefix = Seg 3 Add>
                 <Disp = T8)
   Token Cell T4 <Length>
                 <Next Token> -
                 <Type = Directive-iOAM>
                   <Ioam-parameters - data items to collect>
                   <Scratchpad> T6
   Token Cell T5 <Length>
                 <Next Token> -
                 <Type = Directive-iOAM>
                   <Ioam-parameters - data items to collect>
                   <Scratchpad> T7
   Token Cell T6 <Length>
                 <Next Token> -
                 <Type = Scratchpad>
   Token Cell T7 <Length>
                 <Next Token> -
                 <Type = Scratchpad>
   Token Cell T8 <Length>
                 <Next Token>
                 <Type = Disp>
                 <Disp Parameters>
   Token Cell T9 <Length>
                 <Next Token>
                 <Type = Payload>

               Figure 17: Enhanced Segment Routing Using TCR

Bryant & Clemm           Expires April 22, 2022                [Page 34]

Internet-Draft                   TCR-DP                     October 2021

12.  Items for further discussion

   This document does not constitute a finalized design and there are
   many design decisions that will require further discussion.  The
   following is a partial list:

   o  Preamble.  The preamble needs further study.  The goal is that it
      contains only the minimum of information as it needs to be popped
      and pushed when a token cell is popped or pushed.  We need to
      understand if there is a need for a number of token cell's
      indicator, and/or a last child indicator.

   o  Token Cell identification.  As an alternative to referencing other
      token cells by offset, it is conceivable to introduce token cell
      identifiers and refer to token cells by their ID.

   o  Manifest token cells.  Rather to require processing of a full
      token cell, it is conceivable to express manifests as part of a
      token cell preamble, at least as long as the number of token cells
      to process in parallel are limited.  This could save on stage in a
      token cell processing pipeline.  However, it would result in a
      slightly longer or more complex preamble.

   o  Manifest token cells (cont'd.)  It should be noted that the
      current design of the manifest token cell type includes all
      pointers to subsequent token cells as part of the token cell blob.
      Alternative designs are conceivable.  For example, the design
      might be altered to allow for the next token to be populated and
      only require any additional token cells to be referenced from the
      token cell blob.

   o  Rendezvous token cells.  As an optimization, it is conceivable to
      combine manifests and rendezvous points into a single token cell.

   o  Security token cells.  Further investigation is needed to
      determine the most effective structure, specifically compact yet
      efficient encodings for the token cell mask that is used to
      identify the portions of the packet being signed.

   o  Disposition token cells.  There are different ways that
      disposition token cells could be referred to for processing,
      including through a DT parameter (Disposition Token Cell) as part
      of forwarding token cells, through a separate field as part of the
      token cell structure, or indirectly by virtue of popping a
      forwarding token cell when the destination is reached and
      processing the token cell behind it.

Bryant & Clemm           Expires April 22, 2022                [Page 35]

Internet-Draft                   TCR-DP                     October 2021

   o  Disposition token cells for processing by intermediate nodes.  The
      example in Section 11.5 uses an implicit disposition operation of
      forwarding token cells by intermediate nodes, in which a
      forwarding token cell is simply popped and processing simply
      resumes with the subsequent token cell.  It is conceivable to
      apply explicit disposition operations instead for the sake of more
      consistent semantics.  Explicit semantics of "pop and resume
      processing with subsequent token cell" could be incorporated into
      the semantics of a forwarding token cell itself, or potentially
      indicated by a corresponding flag in the prefix.

   o  Implementation profiles.  Implementations may impose a reasonable
      limit on the number of token cells that can be serially processed
      at any one node.  This will facilitate mapping to packet
      processing pipelines, which then support a predefined number of
      token cell processing stages per packet at line rate while
      maintaining constant packet processing delay at any given node.
      Determination of reasonable constraints is for further study.
      Analogous limitations may apply to the number of processing cycles
      that can be performed by callback functions, within which the
      processing of any given token cell needs to complete.

   o  Implementation profiles (contd.).  For further study are any
      mechanisms for the discovery of constraints as mentioned in the
      previous list item, as well as any capabilities for negotiation of
      corresponding profiles and the definition of node behavior when
      such limitations were to be breached (in all likelihood resulting
      in aborting the processing of the packet, dropping it with
      corresponding error code).

   o  The design only requires forward token cell pointers and this
      prevents processing loops.  We need to understand if this is too
      significant a restriction.  If this restriction is removed some
      form of maximum tokens to be processed limit, similar in concept
      to TTL may be needed.

13.  Security Considerations

   This section concerns itself with the security of the TCR dataplane.

   The security of the control and management plane is a matter for the
   designers of those aspects of the solution.  However, it is not
   anticipated that the securing of those components will be any more
   onerous than securing the control and management plane of other IETF
   designed dataplanes.

   Security of entry to the dataplane will depend on what entities have
   access to the dataplane.  If TCR is used as a single domain sub-IP

Bryant & Clemm           Expires April 22, 2022                [Page 36]

Internet-Draft                   TCR-DP                     October 2021

   layer in the way that MPLS is used, then it will have the same
   security properties as MPLS in that it is extremely difficult for an
   unauthorized party to inject traffic into such a network because with
   TCR such traffic is easily recognized at network ingress and dropped.
   If the traffic is a TCR packet that is to be carried across the TCR
   network it will be encapsulated and so, in the absence of a TCR
   network error, will not be able to escape the encapsulation and cause
   harm.  Only if a TCR network (or node) were to peer with another TCR
   network would there be a security concern with that third party
   having the ability to control the actions taken on the packet.  Such
   a case is for further study.  It should be noted that similar
   situations have been satisfactorily addressed in MPLS.

   We now need to consider the security of the contents of the packet.
   Clearly we could craft a token that signed the payload, and where the
   payload was a token, we have the option of including that signature
   in the token itself.  However, securing the tokens themselves is more
   interesting because we need to authenticate selected components of
   the packet header, such as single tokens, groups of tokens or even
   components/portions of tokens.  This is needed to allow the
   differentiation of metadata that must not be altered from scratchpad
   data that may be modified during packet transit.  In addition, we
   need to allow intermediate nodes along a path to authenticate data
   that they modify, e.g. scratchpad data items that they create.

   The approach used in TCR allows the authentication of tokens and
   processing guidance they contain for additional security.
   Furthermore there is flexibility for intermediate hops to provide
   their own authentication, to secure scratchpad-type data added to
   packets along a path.  This also allows for "authenticity chains" in
   which nodes verify the authenticity of data items they operate on
   before modifying and signing the update.

   It is clear that the above approach is different from earlier
   protocols where payloads are generally signed in their entirety and
   do not include support for differentiated signing, accommodating
   multiple signers of different packet aspects along a path.

   Most protocols secure their payload in its entirety, exposing only
   the packet header for processing (unless that is tunneled as well).
   TCR is a protocols that include additional packet components that may
   require more differentiated securing.  Specifically, it includes
   guidance for how to process packets, including tokens, and metadata.
   In addition, TCR includes some packet components that can be modified
   or added by intermediate nodes in transit, specifically scratchpad
   data.  This includes telemetry and iOAM data as well as data to
   indicate and verify certain properties of nodes that were traversed.
   While some data must not be modified, other data items might be added

Bryant & Clemm           Expires April 22, 2022                [Page 37]

Internet-Draft                   TCR-DP                     October 2021

   and/or be subject to modification.  An example would be data that
   aggregates or analyzes telemetry data encountered in transit, for
   example an indicator of a minimum or maximum (queue length,
   utilization, latency) encountered along a path.  TCR thus includes a
   mechanism that allows the operator to ensure the authenticity of
   packet data beyond the payload, but that allows them to do this in a
   way that exempts certain data items which are allowed to be modified
   along the way, with the option to allow the corresponding nodes to
   secure these modifications.  This allows receiving applications to
   (for example) verify the authenticity of scratchpad data, and allow
   for the modification of data items where such modification is
   permitted without compromising the authenticity of the remaining
   portions of the packet.

   The design of the security token is described in Section 10.4 . This
   can only be used to sign itself and tokens or token contents after
   the security token.  By including in the security token a mask
   structure it is possible to select what is to be signed.  The
   efficiency of this method is described in Section 10.4.

   Matters related to inter-domain security will be considered in a
   future version of this text.

14.  IANA Considerations

   This document makes no IANA requests.

15.  References

15.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

15.2.  Informative References

              Clemm, A. and T. Eckert, "High-Precision Latency
              Forwarding over Packet-Programmable Networks", NOMS 2020 -
              2020 IEEE/IFIP Network Operations and
              Management Symposium, DOI 10.1109/noms47738.2020.9110431,
              April 2020.

Bryant & Clemm           Expires April 22, 2022                [Page 38]

Internet-Draft                   TCR-DP                     October 2021

              Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields
              for In-situ OAM", draft-ietf-ippm-ioam-data-15 (work in
              progress), October 2021.

   [RFC3270]  Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen,
              P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi-
              Protocol Label Switching (MPLS) Support of Differentiated
              Services", RFC 3270, DOI 10.17487/RFC3270, May 2002,

   [RFC3985]  Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation
              Edge-to-Edge (PWE3) Architecture", RFC 3985,
              DOI 10.17487/RFC3985, March 2005,

   [RFC8754]  Filsfils, C., Ed., Dukes, D., Ed., Previdi, S., Leddy, J.,
              Matsushima, S., and D. Voyer, "IPv6 Segment Routing Header
              (SRH)", RFC 8754, DOI 10.17487/RFC8754, March 2020,

Authors' Addresses

   Stewart Bryant
   University of Surrey 5GIC

   Email: sb@stewartbryant.com

   Alexander Clemm
   Futurewei Technologies, Inc.

   Email: ludwig@clemm.org

Bryant & Clemm           Expires April 22, 2022                [Page 39]