Transport Considerations for Large-Scale Distributed Inference Networks
draft-li-tsvwg-inference-transport-00

TSVWG                                                              Z. Li
Internet-Draft                                                     Z. Du
Intended status: Standards Track                            China Mobile
Expires: 5 January 2027                                          J. Wang
                                                                W. Cheng
                                                                G. Zhang
                                                                  Centec
                                                                  X. Sun
                                                                   Inesa
                                                                 C. Zhao
                                                                    SAIA
                                                             4 July 2026

Transport Considerations for Large-Scale Distributed Inference Networks
                 draft-li-tsvwg-inference-transport-00

Abstract

   Large-scale distributed inference systems generate traffic patterns
   that differ from both traditional data center workloads and
   distributed training workloads.  Disaggregated prefill/decode serving
   transfers key-value cache state between server pools, and expert-
   parallel architectures generate all-to-all traffic among expert
   groups.  These flows are typically carried over a small number of
   RDMA connections, producing low-entropy traffic that is prone to
   uneven link utilization under Equal-Cost Multipath (ECMP) forwarding.

   This document specifies transport considerations for such networks,
   covering path load awareness, path steering through ECMP entropy
   variation, ordering tolerance at the receiver, and differentiated
   reliability for data with different loss sensitivity.  The discussion
   builds on existing IETF building blocks; this document does not
   define new protocol elements.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

Li, et al.               Expires 5 January 2027                 [Page 1]
Internet-Draft             Inference Transport                 July 2026

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 5 January 2027.

Copyright Notice

   Copyright (c) 2026 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   3
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Problem Statement . . . . . . . . . . . . . . . . . . . . . .   4
   4.  Transport Considerations  . . . . . . . . . . . . . . . . . .   4
     4.1.  Path Load Awareness . . . . . . . . . . . . . . . . . . .   4
     4.2.  Path Steering Using ECMP Entropy  . . . . . . . . . . . .   5
     4.3.  Ordering Considerations . . . . . . . . . . . . . . . . .   5
     4.4.  Differentiated Reliability  . . . . . . . . . . . . . . .   6
   5.  Deployment Considerations . . . . . . . . . . . . . . . . . .   6
   6.  Security Considerations . . . . . . . . . . . . . . . . . . .   6
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   7
   8.  Normative References  . . . . . . . . . . . . . . . . . . . .   7
   9.  Informative References  . . . . . . . . . . . . . . . . . . .   8
   Acknowledgements  . . . . . . . . . . . . . . . . . . . . . . . .   8
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   8

1.  Introduction

   Early inference serving deployed models on single servers or small
   clusters, with modest demands on the interconnection network.
   Current large-scale inference systems are different in several
   respects.  Disaggregated serving separates the prefill phase
   (processing the input prompt) from the decode phase (generating
   output tokens) onto distinct server pools.  The key-value (KV) cache

Li, et al.               Expires 5 January 2027                 [Page 2]
Internet-Draft             Inference Transport                 July 2026

   computed during prefill is transferred over the network to the decode
   pool, and the latency of this transfer directly affects time-to-
   first-token.  Mixture-of-experts models deploy expert-parallel (EP)
   groups across many servers.  Token routing between experts generates
   all-to-all communication whose scale grows with the EP group size,
   regularly crossing leaf and spine tiers of the data center fabric.

   This traffic is typically carried by RDMA transports such as RoCEv2
   over a small number of connections between any given pair of
   endpoints.  The resulting flows are large and few -- low-entropy
   traffic from the perspective of flow-based load balancing.  With
   Equal-Cost Multipath (ECMP) forwarding, the hash function maps each
   flow to one path; with few flows, multiple large flows can hash onto
   the same link while parallel links remain idle.  The operational
   issues of low-entropy traffic with flow-based load distribution are
   described in [RFC7424].  In inference fabrics, such collisions
   translate into jitter and increased tail latency for KV cache
   transfer and all-to-all exchanges.

   This document specifies transport-layer considerations for these
   networks: how endpoints can become aware of per-path load, how
   traffic can be steered across paths using existing ECMP mechanisms
   without network upgrades, what ordering tolerance is required at
   receivers, and how reliability can be differentiated for data with
   different loss sensitivity.  The intent is to document the
   considerations and map them to existing IETF building blocks.  This
   document does not define new protocol elements or data plane
   behavior.

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

2.  Terminology

   Prefill:  The inference phase that processes the input prompt and
      produces the initial KV cache.

   Decode:  The inference phase that generates output tokens
      incrementally, consuming and extending the KV cache.

   KV Cache:  Intermediate attention state (keys and values) produced
      during inference.  In disaggregated serving, the KV cache is
      transferred from prefill servers to decode servers.

Li, et al.               Expires 5 January 2027                 [Page 3]
Internet-Draft             Inference Transport                 July 2026

   Expert Parallelism (EP):  A parallelization strategy for mixture-of-
      experts models in which experts are distributed across servers,
      requiring all-to-all token exchange.

   Message:  An application-level unit of transfer (e.g., one RDMA
      operation).  A message comprises one or more packets.

   Entropy:  Variability in the packet header fields used by ECMP
      hashing to select among equal-cost paths, as discussed in
      [RFC7424].

3.  Problem Statement

   Several operational approaches are deployed today to mitigate uneven
   link utilization caused by low-entropy RDMA traffic.  Each involves
   trade-offs.  Increasing the number of connections: spreading traffic
   over more RDMA connections (queue pairs) increases entropy and
   reduces the probability that large flows collide on a single link;
   however, additional queue pairs consume NIC resources and add
   scheduling overhead.  Placement affinity: scheduling communicating
   workers under the same top-of-rack switch or leaf reduces the volume
   of traffic crossing the tiers where collisions occur; this reduces
   exposure but does not change the load-distribution behavior itself.
   Per-hop dynamic load balancing: switches can forward packets of the
   same flow across different links based on real-time link load; this
   achieves fine-grained balance but introduces packet reordering that
   receivers must tolerate.

   A complementary approach is for endpoints to steer traffic across
   paths using mechanisms that ECMP fabrics already support, informed by
   an endpoint view of per-path load.  The remainder of this document
   discusses the considerations for this approach.

4.  Transport Considerations

4.1.  Path Load Awareness

   Endpoint-driven path steering benefits from knowledge of the relative
   load of the candidate paths.  Two sources of this knowledge are
   available with existing building blocks.  On-path telemetry: in
   networks where devices support In situ Operations, Administration,
   and Maintenance (IOAM) [RFC9197], probe or data packets can collect
   per-hop information along their forwarding path, including transit
   delay and queue depth; the export of collected data is described in
   [RFC9326].  Endpoint estimation: where on-path support is not
   available, endpoints can estimate relative path load from end-to-end
   measurements such as round-trip time and delivery rate per path, in
   the manner familiar from delay-based congestion control.

Li, et al.               Expires 5 January 2027                 [Page 4]
Internet-Draft             Inference Transport                 July 2026

   A sender MAY combine both sources where available.  Load information
   is advisory input to path steering and MUST NOT be interpreted as a
   congestion signal in the sense of [RFC3168]; existing congestion
   control behavior is unchanged.

4.2.  Path Steering Using ECMP Entropy

   ECMP path selection is a function of packet header fields.  For
   RoCEv2 traffic, the UDP source port is commonly included in the hash
   input, and varying it changes the selected path without any change to
   network devices; the fabric continues to perform standard flow-based
   ECMP.  The use of header entropy for load distribution is discussed
   in [RFC7424], and an analogous technique using the IPv6 Flow Label is
   described in [RFC6438].

   A sender that observes uneven path load MAY change the entropy value
   (e.g., the UDP source port) used for subsequent traffic on a
   connection, causing that traffic to be hashed onto a different path.
   Senders SHOULD rate-limit such changes; frequent repathing can itself
   induce load oscillation across the fabric.

4.3.  Ordering Considerations

   Changing the path of in-flight traffic reorders packets across the
   change.  The disruption can be confined by aligning path changes with
   application-level message boundaries: all packets of a given message
   SHOULD carry the same entropy value, so that each message traverses a
   single path and arrives in order within itself; the entropy value MAY
   differ between messages, distributing successive messages across
   paths.  With this alignment, the receiver observes reordering only
   between messages, not within a message.  Receivers of multipath
   traffic MUST tolerate inter-message arrival reordering.  For
   transports where each message is independently placed in receiver
   memory (as with RDMA operations carrying explicit placement
   information), inter-message reordering does not require reassembly
   buffering.

   The message size determines the granularity of load distribution.
   Smaller messages distribute load more evenly but increase per-message
   overhead; larger messages reduce overhead but coarsen the
   distribution.  A sender MAY adjust message sizing based on observed
   path balance, preferring larger messages on lightly loaded paths.

Li, et al.               Expires 5 January 2027                 [Page 5]
Internet-Draft             Inference Transport                 July 2026

4.4.  Differentiated Reliability

   Inference traffic is not uniformly sensitive to loss.  The
   sensitivity of model state to perturbation varies with position in
   the model; loss affecting early-layer data propagates through all
   subsequent computation, while loss affecting late-layer data has more
   bounded effect on output quality.  This creates an opportunity for
   differentiated reliability, for which the IETF has established
   precedents: partial reliability in SCTP [RFC3758] allows a sender to
   abandon delivery of selected data, and the QUIC DATAGRAM extension
   [RFC9221] provides unreliable delivery within a reliable connection.

   When the application indicates the loss sensitivity of the data it
   submits (for example, by model layer), the transport MAY apply full
   retransmission to loss-sensitive data and bounded or no
   retransmission to loss-tolerant data, particularly under high path
   load.  The mapping from data category to reliability level is an
   application policy decision; it SHOULD be set so that service quality
   objectives (such as response accuracy and token latency) are
   preserved.  How the application communicates this indication to the
   transport is a local interface matter outside the scope of this
   document.

5.  Deployment Considerations

   The path steering requires only standard ECMP in the fabric and is
   therefore deployable incrementally: endpoints that implement it
   coexist with endpoints that do not.  On-path telemetry is an
   optimization, not a dependency; endpoint estimation suffices where
   IOAM support is absent.  Path steering and per-hop dynamic load
   balancing should not operate on the same traffic simultaneously
   without coordination, as independent repathing decisions at both the
   endpoint and the fabric can interact unpredictably.  Differentiated
   reliability should be introduced conservatively, with loss-tolerant
   treatment applied only where its effect on inference quality has been
   validated for the model in use.

6.  Security Considerations

   Path load information, whether collected via IOAM or estimated at
   endpoints, reveals aspects of fabric topology and utilization.  The
   security considerations of [RFC9197] and [RFC9326] apply to telemetry
   collection and export; access to collected data SHOULD be restricted
   to authorized components.

   Entropy-based path steering uses header fields that are part of
   normal traffic; it does not introduce new spoofing surface beyond
   that of the underlying transport.  However, an endpoint that repaths

Li, et al.               Expires 5 January 2027                 [Page 6]
Internet-Draft             Inference Transport                 July 2026

   aggressively can concentrate load deliberately; fabrics serving
   multiple tenants SHOULD apply the usual per-tenant rate and resource
   isolation.  Differentiated reliability relies on application
   indications of loss sensitivity.  A compromised or misconfigured
   application could mark loss-sensitive data as tolerant, degrading its
   own service quality; this is contained within the application's own
   traffic.

7.  IANA Considerations

   This document has no IANA actions.

8.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC3168]  Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
              of Explicit Congestion Notification (ECN) to IP",
              RFC 3168, DOI 10.17487/RFC3168, September 2001,
              <https://www.rfc-editor.org/info/rfc3168>.

   [RFC3758]  Stewart, R., Ramalho, M., Xie, Q., Tuexen, M., and P.
              Conrad, "Stream Control Transmission Protocol (SCTP)
              Partial Reliability Extension", RFC 3758,
              DOI 10.17487/RFC3758, May 2004,
              <https://www.rfc-editor.org/info/rfc3758>.

   [RFC6438]  Carpenter, B. and S. Amante, "Using the IPv6 Flow Label
              for Equal Cost Multipath Routing and Link Aggregation in
              Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011,
              <https://www.rfc-editor.org/info/rfc6438>.

   [RFC7424]  Krishnan, R., Yong, L., Ghanwani, A., So, N., and B.
              Khasnabish, "Mechanisms for Optimizing Link Aggregation
              Group (LAG) and Equal-Cost Multipath (ECMP) Component Link
              Utilization in Networks", RFC 7424, DOI 10.17487/RFC7424,
              January 2015, <https://www.rfc-editor.org/info/rfc7424>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

Li, et al.               Expires 5 January 2027                 [Page 7]
Internet-Draft             Inference Transport                 July 2026

   [RFC9197]  Brockners, F., Bhandari, S., and T. Mizrahi, "Data Fields
              for In Situ Operations, Administration, and Maintenance
              (IOAM)", RFC 9197, DOI 10.17487/RFC9197, May 2022,
              <https://www.rfc-editor.org/info/rfc9197>.

   [RFC9221]  Pauly, T., Kinnear, E., and D. Schinazi, "An Unreliable
              Datagram Extension to QUIC", RFC 9221,
              DOI 10.17487/RFC9221, March 2022,
              <https://www.rfc-editor.org/info/rfc9221>.

9.  Informative References

   [RFC9326]  Song, H., Gafni, B., Brockners, F., Bhandari, S., Mizrahi,
              T., Sivakolundu, R., Li, Z., and T. Zhou, "In Situ
              Operations, Administration, and Maintenance (IOAM) Direct
              Exporting", RFC 9326, DOI 10.17487/RFC9326, November 2022,
              <https://www.rfc-editor.org/info/rfc9326>.

Acknowledgements

   The authors acknowledge ongoing IETF discussion of AI workload
   networking, including problem statements on training-network load
   balancing and congestion, which provides context for the inference-
   specific considerations in this document.

Authors' Addresses

   Zhiqiang Li
   China Mobile
   Beijing
   100053
   China
   Email: lizhiqiangyjy@chinamobile.com

   Zongpeng Du
   China Mobile
   Beijing
   100053
   China
   Email: duzongpeng@chinamobile.com

   Junjie Wang
   Centec
   Shanghai
   201203
   China

Li, et al.               Expires 5 January 2027                 [Page 8]
Internet-Draft             Inference Transport                 July 2026

   Email: wangjj@centec.com

   Wei Cheng
   Centec
   Shanghai
   201203
   China
   Email: chengw@centec.com

   Guoying Zhang
   Centec
   Shanghai
   201203
   China
   Email: zhanggy@centec.com

   Xun Sun
   Inesa
   Shanghai
   200030
   China
   Email: sunxun@inesa.com

   Chunhao Zhao
   SAIA
   Shanghai
   200125
   China
   Email: chunhao.zhao@sh-aia.com

Li, et al.               Expires 5 January 2027                 [Page 9]