Token Operation Problem Statement
draft-fu-nmop-tokenops-probelem-statement-00
This document is an Internet-Draft (I-D).
Anyone may submit an I-D to the IETF.
This I-D is not endorsed by the IETF and has no formal standing in the
IETF standards process.
| Document | Type | Active Internet-Draft (individual) | |
|---|---|---|---|
| Authors | Yu Fu , Sun Qiong , Xin Song , Chongfeng Xie | ||
| Last updated | 2026-07-06 | ||
| RFC stream | (None) | ||
| Intended RFC status | (None) | ||
| Formats | |||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | I-D Exists | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-fu-nmop-tokenops-probelem-statement-00
nmop Y. Fu
Internet-Draft Q. Sun
Intended status: Standards Track X. Song
Expires: 7 January 2027 C. Xie
China Telecom
6 July 2026
Token Operation Problem Statement
draft-fu-nmop-tokenops-probelem-statement-00
Abstract
Distributed LLM inference relies heavily on high-performance
networking to synchronize states across accelerators (e.g., GPUs) and
nodes. Unlike traditional web services, inference workloads
particularly those involving Mixture-of-Experts (MoE) models and
long-context windows exhibit unique traffic patterns characterized by
massive east-west traffic and strict latency constraints. Current
network infrastructures and scheduling methods often treat compute
resources and network paths independently, leading to suboptimal
performance and degraded Quality of Experience (QoE). This document
elaborates on these issues to guide potential protocol enhancements
within the IETF.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 7 January 2027.
Copyright Notice
Copyright (c) 2026 IETF Trust and the persons identified as the
document authors. All rights reserved.
Fu, et al. Expires 7 January 2027 [Page 1]
Internet-Draft Token Operation July 2026
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4
1.2. Definition and Terminology . . . . . . . . . . . . . . . 4
2. Architecture for Distrubuted Inference . . . . . . . . . . . 5
3. Problem Statement . . . . . . . . . . . . . . . . . . . . . . 7
3.1. Amplified Inter-GPU Communication Tail Latency Degrades
Overall Inference Efficiency . . . . . . . . . . . . . . 7
3.2. Disaggregated Prefill-Decode Deployment Causes Emerging KV
Cache Transmission Bottlenecks . . . . . . . . . . . . . 7
3.3. Decoupled Modeling and Network Scheduling Leads to
Suboptimal Inference Service Experience . . . . . . . . . 8
3.4. Isolated Inference and Network Metrics Impair Fault
Localization and Experience Assurance . . . . . . . . . . 9
4. Security Considerations . . . . . . . . . . . . . . . . . . . 9
5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9
6. Normative References . . . . . . . . . . . . . . . . . . . . 9
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 10
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10
1. Introduction
Prior to 2023, the artificial intelligence (AI) industry was
predominantly focused on model training. The primary objective was
to build larger models by aggregating massive computational
resources, where network infrastructure served as a peripheral
utility to interconnect GPU clusters. During this era, the
optimization goals for network services were strictly confined to
maximizing collective communication efficiency and cluster
scalability to ensure high utilization of training compute resources.
Fu, et al. Expires 7 January 2027 [Page 2]
Internet-Draft Token Operation July 2026
However, by 2026, AI deployment has entered a phase of large-scale
commercialization. The AI operation has decisively shifted from
training to inference. The sustainable, stable, low-cost token
generation capability directly determines the commercial viability of
AI services. Consequently, the network is no longer merely a support
facility for compute; it has evolved into the critical backbone of
the token production pipeline. The network directly dictates key
business metrics, including Time to First Token (TTFT) etc., and
overall operational expenditure.
The transition to inference-centric AI is accompanied by a
fundamental transformation in application architecture, specifically
the emergence of Agentic AI. This shift introduces novel traffic
patterns that legacy network designs fail to accommodate.
• Traditional Chatbots, characterized by stateless, single-turn
interactions with short contexts. These workloads are compute-
intensive but network-light, as the volume of data transferred is
minimal relative to the computation performed.
• Agentic AI, characterized by long-lived sessions, multi-turn
interactions, extensive tool calling, and high reuse rates of KV-
Cache (Key-Value Cache).
In Agentic workflows, agents frequently reference historical context
to perform complex reasoning. This dynamic shifts the network
bottleneck: "computation decreases while data movement increases."
Since the KV-Cache constitutes a significant portion of the session
state and must persist across interactions, the network becomes the
primary conduit for state retrieval and sharing. In this context,
the network is no longer an insignificant "pipe" but the dominant
factor determining inference performance and latency..
The split between training-era and inference-era network positioning
creates incompatible design requirements for token operation traffic
steering
• Network for Training: The network acts as supporting infrastructure
exclusively for interconnecting GPU compute clusters. Its sole
design objective is to eliminate communication bottlenecks so that
GPU arithmetic units can operate at full utilization. Traffic
decisions prioritize bulk synchronous collective communication,
batch-level load balancing, and long-timescale cluster resource
planning. Token generation, if present, is a minor side workload
without dedicated network optimization logic.
Fu, et al. Expires 7 January 2027 [Page 3]
Internet-Draft Token Operation July 2026
• Network for Inference & Agent: The network becomes the core
backbone of the end-to-end token production pipeline. All critical
commercial metrics，such as TTFT etc., directly depend on dynamic
network scheduling of KV-cache state, agent context, and sequential
token streams. Network decisions must jointly evaluate real-time
compute capacity, cross-node state transfer overhead, path latency,
session affinity, and persistent cache locality to select optimal
service instances for each agent request.
To address these issues, traffic steering mechanisms must evolve to
consider Token-specific metrics. The process of selecting service
instances and routing traffic based on the state of KV-Caches, the
locality of context, and the specific requirements of the inference
phase (prefill vs. decode) is essential. This draft defines this
emerging requirement as Token-Aware Traffic Steering (TATS), a
necessary evolution to support the next generation of AI
infrastructure.
1.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
1.2. Definition and Terminology
• AI：Artificial intelligence
• KV-Cache：Key-Value Cache
• AI Gateway：AI GW
• QoE：Quality of Experience
• QoS：Quality of Service
• TTFT：Time To First Token
• TE：Traffic Engineering
• TPOT：Traffic Engineering
Fu, et al. Expires 7 January 2027 [Page 4]
Internet-Draft Token Operation July 2026
2. Architecture for Distrubuted Inference
The distrubuted inference network architecture characterizes three
distinct scenorios of communication interactions, which differ in
traffic characteristics, performance requirements.
Scenorio 1: Client-to-Ingress Service Communication.
This scenorio carries the service traffic between end-users and the
inference service ingress. Its performance evaluation focuses
primarily on user-perceived latency, request success rate, Time-To-
First-Token (TTFT), and the stability of streaming output (e.g.,
token inter-arrival jitter). Consequently, the network path should
exhibit low latency and high availability to ensure a satisfactory
Quality of Experience (QoE).
Scenorio 2: Intra-Cluster High-Frequency Communication.
This scenorio refers to high-bandwidth, low-latency interactions
within the inference cluster. Traffic patterns include inter-GPU
tensor synchronization, pipeline stage handovers, Mixture-of-Experts
(MoE) routing, and KV Cache read/write operations as well as live
migration. Due to the fine-grained nature of these transactions,
this scenorio imposes stringent requirements on the fabric, mandating
minimal latency, high bandwidth, and ultra-low jitter.
Scenorio 3: Cross-Cluster, Cross-Domain, and Cloud-Edge Coordination
Communication.
This scenorio refers to communication across clusters, geographical
regions, and cloud-edge infrastructures. Use cases include task
scheduling across heterogeneous compute centers, cross-site KV Cache
migration, disaster recovery mechanisms for model services, and edge-
localized inference offloading. Unlike intra-cluster communication,
which prioritizes raw throughput, the efficiency of this
communication class relies heavily on the capabilities of the carrier
network, specifically regarding intelligent path selection, Traffic
Engineering (TE), and robust Quality of Service (QoS) assurance
mechanism.
Fu, et al. Expires 7 January 2027 [Page 5]
Internet-Draft Token Operation July 2026
+-------------+
| Client |
| |
+-------------+
│
+-------------v---------------+
| AI GW |
| |
+-----------------------------+
|
|
+-----------------------v-------------------------+
( ISP Network )
( )
( )
( +----------+ +---------+ +------+ )
( | Routing | | Traffic | | SLA | )
( |Forwarding| | Steering| | | )
( +---------+ +---------+ +------+ )
( )
( )
( )
+--------------------------------------------------+
| |
| |
| |
+-------------v-------------+ +---------------v------------+
| | | |
| CLoud 1 | | Cloud 2 |
| | | |
| +--------------------+ | | +---------------------+ |
| | | | | | | |
| | KV Cache | | | | KV Cache | |
| | | | | | | |
| +-----------^--------+ | | +----------^----------+ |
| | | | | |
| | | | | |
| +---------+-------+----<----->----+--------+--------+ |
| | | | | | | | | |
| | | | | | | | | |
|+---v---+ +---v--+ +--v--+ | | +---v---+ +--v---+ +--v--+ |
||Prefill| |Decode| | GPU | | | |Prefill| |Decode| | GPU | |
|+-------+ +------+ +-----+ | | +-------+ +------+ +-----+ |
| | | |
+---------------------------+ +----------------------------+
Fu, et al. Expires 7 January 2027 [Page 6]
Internet-Draft Token Operation July 2026
3. Problem Statement
As inference architectures evolve towards tensor/pipeline/expert
parallelism, prefill/decode disaggregation, and geo-distributed
serving, the network has transitioned from a passive transport medium
to a critical performance bottleneck. This section outlines four
specific problems: amplified tail latency in GPU clusters, KV Cache
transfer bottlenecks, the disconnect between compute and network
scheduling, and the lack of observability correlation between
application-level metrics and network telemetry. These problems
motivate the need for enhanced protocols and metrics to support AI-
driven traffic engineering.
3.1. Amplified Inter-GPU Communication Tail Latency Degrades Overall
Inference Efficiency
Modern large model inference relies on tensor parallelism, pipeline
parallelism, and MoE expert parallelism, which require collaborative
computation across multiple GPU nodes and network devices.
* The Problem: Distributed training and inference rely heavily on
collective communication. In these patterns, the straggler effect
is pronounced; a delay or jitter on a single node or link forces
all other participating nodes to wait. Consequently, network tail
latency is amplified, directly degrading the token generation
rate. Existing network mechanisms lack dedicated optimization for
fine-grained, synchronous AI collective communication flows.
Conventional cluster network designs focus on average latency
rather than tail latency metrics, and static task scheduling fails
to adapt to dynamic network jitter and congestion. No
standardized metric system exists for characterizing inference-
specific network performance, nor are there defined telemetry
indicators for tail latency, jitter, and collective communication
completion status targeted at GPU cluster inference workloads.
3.2. Disaggregated Prefill-Decode Deployment Causes Emerging KV Cache
Transmission Bottlenecks
To optimize resource utilization, modern inference architectures
adopt separated Prefill and Decode deployment modes. In this model,
the compute-intensive Prefill phase and the memory-bandwidth-
intensive Decode phase are scheduled on different sets of resources.
While this improves compute efficiency, it introduces a heavy
dependency on network bandwidth.
* The Problem: with the rapid growth of long-context inference,
multi-turn dialogue interactions, and concurrent inference
requests, the volume of cross-node KV Cache transmission increases
Fu, et al. Expires 7 January 2027 [Page 7]
Internet-Draft Token Operation July 2026
exponentially, introducing severe bandwidth pressure and queuing
latency. Traditional network scheduling and task placement
strategies ignore session-level context continuity and KV Cache
transmission requirements. Current task scheduling policies
prioritize GPU resource occupancy while neglecting end-to-end
network path quality between Prefill and Decode nodes. Existing
schedulers often select Decode nodes based solely on GPU
availability, ignoring the network "distance" or available
bandwidth required to transfer the KV Cache, leading to increased
Time-To-First-Token (TTFT) and inter-token latency. There is no
standardized mechanism for exposing application-layer KV Cache
transmission demands, including context length, session
stickiness, latency tolerance, and cache migration requirements,
to the network plane. The absence of unified priority marking and
QoS guarantee mechanisms for KV Cache flows further exacerbates
transmission instability in large-scale concurrent scenarios.
3.3. Decoupled Modeling and Network Scheduling Leads to Suboptimal
Inference Service Experience
Current orchestration systems typically treat compute/model
scheduling and network routing as independent silos.
* The Problem: the distributed inference scheduling systems
primarily optimize computing/modeling resource indicators,
including GPU idle rate, memory footprint, and model loading
status, while neglecting network path quality between users and
computing nodes. A computing node with sufficient idle resources
may still deliver poor user experience due to high end-to-end
latency, persistent packet loss, cross-domain transmission
instability, or link congestion. The decoupling of computing/
modeling scheduling and network scheduling results in a common
paradox: idle computing/modeling resources cannot be fully
utilized due to network constraints, while high-quality network
paths cannot be matched with latency-sensitive inference tasks.
No standardized compute-network joint scheduling model is
available for AI inference services. The existing IETF framework
lacks explicit definitions for inference-specific joint metrics,
maybe CATS (Compute-Aware Traffic Scheduling)
[I-D.ietf-cats-metric-definition] related? There is no unified
interface for computing nodes to expose real-time computational
load status or for network devices to feedback path quality
indicators, making it impossible to implement fine-grained,
experience-oriented collaborative scheduling for agent
collaboration, real-time dialogue, and multimedia inference
services.
Fu, et al. Expires 7 January 2027 [Page 8]
Internet-Draft Token Operation July 2026
3.4. Isolated Inference and Network Metrics Impair Fault Localization
and Experience Assurance
In current industrial deployment, inference service metrics and
network performance metrics are collected and maintained by
independent platforms without unified correlation and association.
* The Problem: user-facing experience indicators (e.g., Time-To-
First-Token (TTFT), Time-Per-Token (TPOT), end-to-end inference
latency, request success rate) are decoupled from underlying
network metrics (e.g., packet loss, jitter, retransmission rate,
link congestion). When service degradation occurs, operators
cannot accurately locate root causes among model computation
overhead, task queuing delay, KV Cache transmission latency, or
network link anomalies. The lack of a cross-layer unified
observability system breaks the end-to-end inference service link.
No standardized flow identification, marker fields, or telemetry
reporting formats are defined for AI inference traffic. There is
no unified specification for associating request IDs, session IDs,
model instances, GPU node information, and network path telemetry
data, which hinders automated fault diagnosis, service quality
auditing, and refined experience assurance for large-scale
intelligent agent and LLM inference services.
4. Security Considerations
TBD
5. IANA Considerations
This document has no IANA actions.
6. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
Fu, et al. Expires 7 January 2027 [Page 9]
Internet-Draft Token Operation July 2026
[I-D.ietf-cats-metric-definition]
Yao, K., Li, C., Contreras, L. M., Ros-Giralt, J., and G.
Zeng, "CATS Metrics Definition", Work in Progress,
Internet-Draft, draft-ietf-cats-metric-definition-10, 22
June 2026, <https://datatracker.ietf.org/doc/html/draft-
ietf-cats-metric-definition-10>.
Acknowledgements
TBD
Authors' Addresses
Yu Fu
China Telecom
Beijing
China
Email: fuy44@chinatelecom.cn
Qiong Sun
China Telecom
Beijing
China
Email: sunqiong@chinatelecom.cn
Xin Song
China Telecom
Beijing
China
Email: songx18@chinatelecom.cn
Chongfeng Xie
China Telecom
Beijing
China
Email: xiechf@chinatelecom.cn
Fu, et al. Expires 7 January 2027 [Page 10]