KV Cache Distribution for Distributed LLM Inference: Use Case and Requirements
draft-li-cats-kv-cache-distribution-00

CATS                                                               Z. Li
Internet-Draft                                                     Z. Du
Intended status: Informational                              China Mobile
Expires: 5 January 2027                                          J. Wang
                                                                W. Cheng
                                                                G. Zhang
                                                                  Centec
                                                                  X. Sun
                                                                   Inesa
                                                                 C. Zhao
                                                                    SAIA
                                                             4 July 2026

   KV Cache Distribution for Distributed LLM Inference: Use Case and
                              Requirements
                 draft-li-cats-kv-cache-distribution-00

Abstract

   In large language model (LLM) inference, the key-value (KV) cache
   holds the attention state computed from previously processed tokens.
   Reusing cached state across requests avoids repeated prefill
   computation and reduces time-to-first-token.  In distributed
   inference deployments, the KV cache becomes a network-distributed
   resource: the effectiveness of steering a request to a service
   instance depends not only on computing and network metrics but also
   on whether reusable cached state is available at or near that
   instance.

   This document describes the KV cache distribution use case for
   Computing-Aware Traffic Steering (CATS), identifies the gaps relative
   to the existing CATS framework and metrics, and states requirements
   for cache-state metric exposure and for the distribution and
   synchronization of cached content across multiple cache tiers.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

Li, et al.               Expires 5 January 2027                 [Page 1]
Internet-Draft            KV Cache Distribution                July 2026

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 5 January 2027.

Copyright Notice

   Copyright (c) 2026 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   3
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Problem Statement . . . . . . . . . . . . . . . . . . . . . .   3
   4.  Cache-State Metric Exposure . . . . . . . . . . . . . . . . .   4
   5.  Distribution and Synchronization of Cached Content  . . . . .   4
   6.  Security Considerations . . . . . . . . . . . . . . . . . . .   4
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   4
   8.  Normative References  . . . . . . . . . . . . . . . . . . . .   4
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   5

1.  Introduction

   LLM applications routinely require knowledge beyond what is contained
   in model parameters: conversation history for personalized
   assistants, enterprise data for domain-specific agents, retrieved
   documents for search-augmented responses.  Two established approaches
   address this need, each with trade-offs.  In-context approaches
   (prompt construction and retrieval-augmented generation) supply
   external knowledge at inference time without modifying the model, but
   are constrained by the model's context window and incur per-request
   retrieval and prefill computation.  Fine-tuning adapts model
   parameters to new knowledge, but requires training resources and
   labeled data, and binds the result to specific tasks.

Li, et al.               Expires 5 January 2027                 [Page 2]
Internet-Draft            KV Cache Distribution                July 2026

   KV cache reuse occupies a useful middle ground.  The key-value (KV)
   cache holds the attention state computed from previously processed
   tokens.  Reusing cached state across requests avoids repeated prefill
   computation and reduces time-to-first-token.  In distributed
   inference deployments, the KV cache becomes a network-distributed
   resource: the effectiveness of steering a request to a service
   instance depends not only on computing and network metrics but also
   on whether reusable cached state is available at or near that
   instance.

   This document describes the KV cache distribution use case for
   Computing-Aware Traffic Steering (CATS), identifies the gaps relative
   to the existing CATS framework and metrics, and states requirements
   for cache-state metric exposure and for the distribution and
   synchronization of cached content across multiple cache tiers.

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

2.  Terminology

   KV Cache:  Intermediate attention state (keys and values) produced
      during inference.  In disaggregated serving, the KV cache is
      transferred from prefill servers to decode servers.

   Prefill:  The inference phase that processes the input prompt and
      produces the initial KV cache.

   Decode:  The inference phase that generates output tokens
      incrementally, consuming and extending the KV cache.

   Expert Parallelism (EP):  A parallelization strategy for mixture-of-
      experts models in which experts are distributed across servers,
      requiring all-to-all token exchange.

3.  Problem Statement

   In distributed LLM inference, the KV cache computed during prefill is
   a valuable resource that can be reused to avoid recomputation.  When
   a request arrives, steering it to an instance that already holds the
   relevant KV cache state avoids prefill latency.  However, the KV
   cache is distributed across service instances and cache tiers; a CATS
   decision that considers only computing load and network metrics

Li, et al.               Expires 5 January 2027                 [Page 3]
Internet-Draft            KV Cache Distribution                July 2026

   cannot account for cache availability.

   This creates a gap: the existing CATS metrics do not expose cache
   state, and the CATS distribution framework does not address how
   cached content is distributed or synchronized across instances.  This
   document states requirements to close this gap.

4.  Cache-State Metric Exposure

   A service instance SHOULD expose cache-state metrics that allow a
   CATS decision function to determine whether reusable KV cache state
   is available.  These metrics include: the set of cache keys (e.g.,
   token-prefix identifiers or content hashes) present at the instance;
   the cache entry sizes; and the expected retrieval cost.  The metrics
   MUST be expressible in the CATS metric framework so that they can be
   combined with computing and network metrics in steering decisions.

5.  Distribution and Synchronization of Cached Content

   For cache reuse to be effective across instances, cached content may
   need to be distributed or migrated.  This document identifies the
   following requirements: (a) a mechanism to advertise cache
   availability across instances; (b) a mechanism to transfer KV cache
   state between instances or cache tiers, respecting latency and
   bandwidth constraints; (c) consistency handling for cache entries
   that are updated or evicted.  The specific transfer protocol is
   outside the scope of this document but should build on existing
   transport building blocks.

6.  Security Considerations

   Cache-state metrics and cache content may reveal information about
   the inputs being processed.  Exposure of cache metrics MUST be
   restricted to authorized CATS components.  Transfer of KV cache state
   between instances SHOULD be protected for integrity and
   confidentiality.

7.  IANA Considerations

   This document has no IANA actions.

8.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

Li, et al.               Expires 5 January 2027                 [Page 4]
Internet-Draft            KV Cache Distribution                July 2026

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

Authors' Addresses

   Zhiqiang Li
   China Mobile
   Beijing
   100053
   China
   Email: lizhiqiangyjy@chinamobile.com

   Zongpeng Du
   China Mobile
   Beijing
   100053
   China
   Email: duzongpeng@chinamobile.com

   Junjie Wang
   Centec
   Shanghai
   201203
   China
   Email: wangjj@centec.com

   Wei Cheng
   Centec
   Shanghai
   201203
   China
   Email: chengw@centec.com

   Guoying Zhang
   Centec
   Shanghai
   201203
   China
   Email: zhanggy@centec.com

Li, et al.               Expires 5 January 2027                 [Page 5]
Internet-Draft            KV Cache Distribution                July 2026

   Xun Sun
   Inesa
   Shanghai
   200030
   China
   Email: sunxun@inesa.com

   Chunhao Zhao
   SAIA
   Shanghai
   200125
   China
   Email: chunhao.zhao@sh-aia.com

Li, et al.               Expires 5 January 2027                 [Page 6]