Speculative Decoding Coordination Protocol (SDCP) for Distributed Inference with Multi-Level Caches
draft-li-mlc-inference-acceleration-00
This document is an Internet-Draft (I-D).
Anyone may submit an I-D to the IETF.
This I-D is not endorsed by the IETF and has no formal standing in the
IETF standards process.
| Document | Type | Active Internet-Draft (individual) | |
|---|---|---|---|
| Authors | Zhiqiang Li , Zongpeng Du , Junjie Wang , Wei Cheng , Ying Zhao , Chunhao Zhao , Xun Sun | ||
| Last updated | 2026-11-02 | ||
| RFC stream | (None) | ||
| Intended RFC status | (None) | ||
| Formats | |||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | I-D Exists | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-li-mlc-inference-acceleration-00
Network Working Group Z. Li
Internet-Draft Z. Du
Intended status: Standards Track China Mobile
Expires: 6 May 2027 J. Wang
W. Cheng
Y. Zhao
Centec
C. Zhao
SAIA
X. Sun
Inesa
2 November 2026
Speculative Decoding Coordination Protocol (SDCP) for Distributed
Inference with Multi-Level Caches
draft-li-mlc-inference-acceleration-00
Abstract
Speculative decoding accelerates language model inference: a small
draft model proposes several tokens and the large target model
verifies them in one forward pass. In large deployments, draft
models and the target model run on separate, independently scaled
instances that exchange state through a multi-level cache. This
document specifies the Speculative Decoding Coordination Protocol
(SDCP), with which a router assigns requests to draft instances by
load, draft instances submit proposals that reference draft state in
the cache, and a verifier, the only party that commits output, checks
proposals under a registered acceptance rule, returns the accepted
tokens and the next-round speculation limit, and escalates
persistently rejected sessions to a reduced speculation length,
another draft model, or decoding by the target model itself depending
on its load. SDCP defines a session contract, a committed-prefix
digest chain, CBOR messages carried over HTTP, idempotent rounds, and
epoch fencing for recovery.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Li, et al. Expires 6 May 2027 [Page 1]
Internet-Draft SDCP November 2026
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 6 May 2027.
Copyright Notice
Copyright (c) 2026 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Background . . . . . . . . . . . . . . . . . . . . . . . 3
1.2. Problem Statement . . . . . . . . . . . . . . . . . . . . 3
1.3. Protocol Overview . . . . . . . . . . . . . . . . . . . . 4
1.4. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2. Conventions and Definitions . . . . . . . . . . . . . . . . . 5
3. Architecture . . . . . . . . . . . . . . . . . . . . . . . . 6
4. Session Model . . . . . . . . . . . . . . . . . . . . . . . . 6
4.1. Session Contract . . . . . . . . . . . . . . . . . . . . 6
4.2. Committed-Prefix Digest Chain . . . . . . . . . . . . . . 8
5. Messages and Resources . . . . . . . . . . . . . . . . . . . 8
5.1. HTTP Usage . . . . . . . . . . . . . . . . . . . . . . . 8
5.2. Message Definitions . . . . . . . . . . . . . . . . . . . 9
6. Procedures . . . . . . . . . . . . . . . . . . . . . . . . . 11
6.1. Session Creation and Assignment . . . . . . . . . . . . . 11
6.2. Drafting and Proposal . . . . . . . . . . . . . . . . . . 11
6.3. Verification and Commit . . . . . . . . . . . . . . . . . 12
6.4. Acceptance Rules . . . . . . . . . . . . . . . . . . . . 13
6.5. Speculation Control and Escalation . . . . . . . . . . . 14
6.5.1. Speculation Limit . . . . . . . . . . . . . . . . . . 14
6.5.2. Low Rounds and Escalation . . . . . . . . . . . . . . 14
6.6. Sampling Policy Updates . . . . . . . . . . . . . . . . . 15
6.7. Output Delivery . . . . . . . . . . . . . . . . . . . . . 15
6.8. Commit Log . . . . . . . . . . . . . . . . . . . . . . . 16
6.9. Failure Handling and Recovery . . . . . . . . . . . . . . 16
Li, et al. Expires 6 May 2027 [Page 2]
Internet-Draft SDCP November 2026
7. Error Handling . . . . . . . . . . . . . . . . . . . . . . . 16
8. Operational Considerations . . . . . . . . . . . . . . . . . 17
9. Security Considerations . . . . . . . . . . . . . . . . . . . 18
10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18
10.1. Media Type . . . . . . . . . . . . . . . . . . . . . . . 18
10.2. SDCP Registries . . . . . . . . . . . . . . . . . . . . 19
10.2.1. SDCP Problem Types . . . . . . . . . . . . . . . . . 19
10.2.2. SDCP Acceptance Rules . . . . . . . . . . . . . . . 19
10.2.3. SDCP Actions . . . . . . . . . . . . . . . . . . . . 19
11. References . . . . . . . . . . . . . . . . . . . . . . . . . 19
11.1. Normative References . . . . . . . . . . . . . . . . . . 19
11.2. Informative References . . . . . . . . . . . . . . . . . 21
Appendix A. Test Vectors . . . . . . . . . . . . . . . . . . . . 21
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 22
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22
1. Introduction
1.1. Background
Autoregressive decoding produces one token per forward pass of the
target model. Each pass reads all model weights and the whole KV
cache, so decoding is limited by memory bandwidth and leaves compute
idle. Speculative decoding [LEVIATHAN] [CHEN] lets a cheaper draft
model propose M tokens, which the target model then scores in a
single pass. The longest acceptable prefix of the proposal is kept,
and the target model contributes one more token, so each target pass
yields between 1 and M+1 tokens. With the speculative sampling rule,
the output follows exactly the distribution of the target model.
Some models are trained with built-in multi-token prediction heads
that serve as drafters [DEEPSEEKV3].
1.2. Problem Statement
When drafting and verification run in one process, the coordination
is internal. Large deployments separate them: several draft model
instances, possibly of different sizes, serve many concurrent
requests and feed one or more target-model verifier instances, and
both exchange token state and KV caches through a multi-level cache
spanning accelerator memory, host memory, and networked storage
[I-D.li-kvcache-efficient-storage]. In that setting the following
must be agreed between independently implemented components:
* how requests are assigned to draft instances according to their
load, so that the requests of one batch can use different
drafters;
Li, et al. Expires 6 May 2027 [Page 3]
Internet-Draft SDCP November 2026
* how many tokens a drafter proposes, given its history of accepted
proposals, its load, and the bandwidth available to the cache;
* how a proposal refers to draft state in the cache instead of
carrying it, so that the verifier can assemble large verification
batches from many drafters;
* which acceptance rule the verifier applies, and how the rule can
follow user preferences;
* what happens when proposals keep being rejected: reduce the
speculation length, switch to another drafter, let the target
model decode by itself if it has spare capacity, or fail; and
* how one authoritative output is maintained across retries,
concurrent proposals, and instance failures.
1.3. Protocol Overview
SDCP addresses these needs with the following elements:
* an immutable Session Contract that fixes the models, the
acceptance rule, limits, and the escalation policy (Section 4.1);
* a committed-prefix digest chain that every proposal must extend,
which makes stale or forked proposals detectable (Section 4.2);
* assignment, proposal, verdict, and output messages in CBOR
[RFC8949], described in CDDL [RFC8610] and exchanged over HTTP
(Section 5);
* a verification procedure with registered acceptance rules
(Section 6.3); and
* speculation-length control and load-aware escalation
(Section 6.5).
The verifier is the sole commit authority. Drafters, routers, and
caches can fail or misbehave without changing the committed output,
except as discussed in Section 9.
Li, et al. Expires 6 May 2027 [Page 4]
Internet-Draft SDCP November 2026
1.4. Scope
SDCP does not define model architectures, sampling kernels, batching
algorithms, or client-facing inference APIs. It assumes a multi-
level cache that stores and serves opaque objects named by URI and
supports leases, such as KVSP [I-D.li-kvcache-efficient-storage].
Draft-model training, tree-structured speculation, and multiple
verifiers per session are out of scope for this version.
2. Conventions and Definitions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
Target model: The model whose output distribution the service
promises.
Draft model: A cheaper model, or auxiliary prediction head, that
proposes tokens.
Router: The component that admits requests, creates sessions, and
assigns them to drafters.
Drafter: An instance that runs a draft model.
Verifier: An instance that runs the target model and commits output.
Multi-Level Cache (MLC): Storage for prompts, KV caches, and draft
artifacts across memory and storage tiers, reachable by all
instances.
Session: One generation task, identified by a 16-octet identifier.
Committed prefix: The prompt followed by all output tokens committed
so far; its length in output tokens is n.
Round: One proposal and its verdict. Rounds of a session are
numbered from 1.
Speculation length (M): The number of tokens in a proposal.
Acceptance rule: The algorithm that decides how many proposed tokens
are kept (Section 6.4).
Epoch: A counter that identifies the current verifier authority of a
Li, et al. Expires 6 May 2027 [Page 5]
Internet-Draft SDCP November 2026
session (Section 6.9).
3. Architecture
requests output tokens
Client -----------> +--------+ +-------------> Client
| Router | |
+--------+ |
assignment / | \ events |
v | \ |
+-----------+ | +-----------+-+---+
| Drafter 1 | | | Verifier |
+-----------+ | | (target model) |
+-----------+ | +--------+--------+
| Drafter 2 |--+--proposal->|verdict
+-----+-----+ <---------+
| |
v draft KV, q rows v target KV, commits
+-------------------------------------+
| Multi-Level Cache |
+-------------------------------------+
Figure 1: SDCP Roles
The router creates a session at a verifier, which computes or loads
the KV cache of the prompt, and assigns the session to a drafter.
The drafter proposes tokens to the verifier. The verifier answers
each proposal with a verdict that both commits output and steers the
next round. Draft state that is large, such as the draft model's KV
cache and probability rows, stays in the MLC and is referenced by
URI.
4. Session Model
4.1. Session Contract
The Session Contract is fixed when the session is created:
Li, et al. Expires 6 May 2027 [Page 6]
Internet-Draft SDCP November 2026
contract = {
target: hash32, ; SHA-256 of target model weights
tokenizer: hash32, ; SHA-256 of tokenizer definition
? drafters: [+ hash32], ; permitted draft models
rule: uint, ; acceptance rule (registry), fixed
? sampling: sampling,
m_max: uint .le 64, ; upper bound on M
max_new: uint, ; maximum output tokens
stop: [* uint], ; stop tokens
low_rounds: uint .ge 1, ; R, escalation period
? low_ratio: uint .le 100, ; percent, default 0
fallback: [+ ladder], ; escalation ladder
? self_load: uint .le 100, ; max verifier load for self-decode
? self_max: uint, ; max tokens per self-decode period
? scope: bstr .size 16 ; KVSP Scope ID of cache objects
}
sampling = {
? temperature_milli: uint, ; temperature x 1000
? top_k: uint,
? top_p_milli: uint .le 1000,
? seed: uint
}
hash32 = bstr .size 32
action = a-continue / a-reduce / a-switch /
a-self-decode / a-finish / a-fail
ladder = a-reduce / a-switch / a-self-decode / a-fail
a-continue = 0
a-reduce = 1
a-switch = 2
a-self-decode = 3
a-finish = 4
a-fail = 5
The contract digest is the SHA-256 of the contract encoded with the
core deterministic encoding of Section 4.2.1 of [RFC8949], prefixed
with a domain separator:
contract_digest = SHA-256( "SDCP-CONTRACT-1" || 0x00 ||
deterministic_cbor(contract) )
The sampling profile defines the target distribution p used by the
acceptance rule: p is the target model's next-token distribution
after applying temperature, top-k, and top-p in that order; a
temperature of zero selects the most probable token (ties broken in
favor of the smallest token identifier). An absent "drafters" member
permits any draft model that the router's local policy allows. The
router can change the sampling profile at a round boundary
(Section 6.6); such updates are carried separately from the contract,
Li, et al. Expires 6 May 2027 [Page 7]
Internet-Draft SDCP November 2026
which, including the acceptance rule, is never modified. The
contract digest is computed once, over the contract as created. A
contract that contains a key not defined by this document MUST be
rejected with problem type "sdcp-bad-contract", so that all parties
compute the contract digest over the same content.
4.2. Committed-Prefix Digest Chain
The verifier maintains a digest over the committed prefix. For a
prompt of P tokens t[0..P-1] and committed output tokens o[1..n]:
d[0] = SHA-256( "SDCP-PREFIX-1" || 0x00 ||
contract_digest || uint32(P) ||
uint32(t[0]) || ... || uint32(t[P-1]) )
d[k] = SHA-256( d[k-1] || uint32(o[k]) ) for k = 1..n
uint32() is four octets in network byte order and SHA-256 is defined
in [RFC6234]. A proposal states the length n and digest d[n] of the
prefix it extends. Because d[n] depends on every committed token, a
proposal built on a different or outdated prefix is detected without
comparing token lists.
5. Messages and Resources
5.1. HTTP Usage
SDCP is an HTTP application following BCP 56 [RFC9205]. Each role
has a base URI configured by the operator; SDCP defines no well-known
URI. HTTP/2 [RFC9113] or HTTP/3 [RFC9114] over TLS 1.3 [RFC8446]
with mutual certificate authentication is REQUIRED. Message bodies
use the media type "application/sdcp+cbor". Errors use Problem
Details [RFC9457]; clients handle unlisted status codes by class
[RFC9110].
Li, et al. Expires 6 May 2027 [Page 8]
Internet-Draft SDCP November 2026
+========+=============================+============+=============+
|Role | Method and Path |Body |Response |
+========+=============================+============+=============+
|Verifier| POST sessions |session-open|session-state|
+--------+-----------------------------+------------+-------------+
|Verifier| PUT sessions/{id}/drafter |assignment |session-state|
+--------+-----------------------------+------------+-------------+
|Verifier| POST proposals |proposal |verdict |
+--------+-----------------------------+------------+-------------+
|Verifier| GET |none |output |
| | sessions/{id}/output?from=n | | |
+--------+-----------------------------+------------+-------------+
|Verifier| POST sessions/{id}/policy |policy |session-state|
+--------+-----------------------------+------------+-------------+
|Drafter | POST assignments |assignment |none (204) |
+--------+-----------------------------+------------+-------------+
|Router | PUT instances/{id}/status |status |none (204) |
+--------+-----------------------------+------------+-------------+
|Router | POST events |event |none (204) |
+--------+-----------------------------+------------+-------------+
Table 1: SDCP Resources
{id} in session paths is the base64url encoding without padding
[RFC4648] of the session identifier.
5.2. Message Definitions
session-id = bstr .size 16
cache-ref = tstr ; URI of an MLC object
session-open = {
session: session-id, epoch: uint, contract: contract,
prompt: [* uint] / cache-ref, ; tokens or MLC reference
? prompt_kv: cache-ref, ; target KV of the prompt
commit_log: cache-ref, ; durable commit records
? resume: { ; present on failover
round: uint, committed: uint, digest: hash32,
committed_tokens: cache-ref,
? target_kv: cache-ref
}
}
session-state = {
session: session-id, epoch: uint, round: uint,
committed: uint, digest: hash32, ? state: uint
}
assignment = {
session: session-id, epoch: uint, contract: contract,
Li, et al. Expires 6 May 2027 [Page 9]
Internet-Draft SDCP November 2026
verifier: tstr, ; verifier base URI
drafter: tstr, ; drafter base URI
prompt: [* uint] / cache-ref,
round: uint, ; last round with a verdict
? sampling: sampling, ; current profile, if updated
committed: uint, digest: hash32,
? committed_tokens: cache-ref, ; committed output so far
m_init: uint
}
proposal = {
session: session-id, epoch: uint, round: uint,
drafter: tstr, ; drafter base URI
base: uint, base_digest: hash32,
tokens: [+ uint], ; M proposed tokens
? q_rows: cache-ref, ; draft rows, for rule 2
? draft_kv: cache-ref,
status: status
}
verdict = {
session: session-id, epoch: uint, round: uint,
accepted: uint, ; a, 0 <= a <= M
emitted: [* uint], ; tokens committed this round
committed: uint, digest: hash32, ; new n and d[n]
action: action, m_cap: uint,
low: uint, ; consecutive low rounds
load: uint .le 100 ; verifier load, percent
}
output = {
session: session-id, epoch: uint, from: uint, tokens: [* uint],
final: bool, ? finish: action
}
status = {
instance: tstr, role: 1 / 2, ; 1 drafter, 2 verifier
load: uint .le 100, queue: uint,
? accept_milli: uint .le 1000, ; EWMA of a/M x 1000
? cache_mbps: uint, ; available MLC bandwidth
ttl_ms: uint
}
event = {
session: session-id, epoch: uint,
kind: action, committed: uint, digest: hash32
}
policy = { session: session-id, from_round: uint,
sampling: sampling }
Li, et al. Expires 6 May 2027 [Page 10]
Internet-Draft SDCP November 2026
The contract, sampling, hash32, and action types are defined in
Section 4.1. Unknown map keys MUST be ignored, except in the
contract (Section 4.1). A receiver that encounters an unknown action
value MUST treat it as "fail".
The "q_rows" object holds, for each proposed position, the draft
distribution from which the proposed token was sampled, as a row of V
IEEE 754 binary32 values in little-endian order, where V is the
vocabulary size; rows are concatenated in position order. The
drafter MUST sample each proposed token from exactly the row it
publishes.
6. Procedures
6.1. Session Creation and Assignment
The router selects a verifier and sends session-open with epoch 1 and
a commit_log reference in the MLC. The verifier validates the
contract (it MUST refuse a contract whose target digest does not
match its model), computes or loads the KV cache of the prompt,
computes d[0], and answers with session-state (round 0, committed 0).
The router then selects a drafter whose model is permitted by the
contract. It first informs the verifier with PUT
sessions/{id}/drafter, and then sends the assignment, carrying the
current round, committed length, and digest, to the drafter. The
verifier accepts proposals only from the drafter named in the latest
assignment it has received. Drafter selection is a local policy
based on the drafters' status reports, for example preferring
drafters with low load and queue and high recent acceptance. The
router MUST NOT assign sessions to an instance whose status has
expired. Each instance reports status at intervals shorter than its
ttl_ms.
6.2. Drafting and Proposal
A drafter holding an assignment prepares its own KV cache for the
committed prefix (by computation or by loading from the MLC) and
generates M tokens, with 1 <= M <= min(m_max, m_cap), where m_cap is
the limit from the last verdict (m_init before the first). It writes
the draft artifacts the verifier may need to the MLC and holds leases
on them until it receives the verdict of the round. It then sends a
proposal with base and base_digest equal to the committed length and
digest it last learned, and round equal to the last verdict's round
plus one.
Li, et al. Expires 6 May 2027 [Page 11]
Internet-Draft SDCP November 2026
A drafter MUST NOT have more than one outstanding proposal per
session. It MAY prepare the next proposal speculatively, assuming
full acceptance, but sends it only after the verdict confirms that
assumption.
Choice of M is local to the drafter. The following rule is
RECOMMENDED. Let alpha be the drafter's exponentially weighted
estimate of the per-token acceptance probability for the session, t_d
the time to draft one token (including writing its artifacts at the
cache bandwidth currently available), and t_v the expected
verification time. The expected number of tokens committed per round
is (1 - alpha^(M+1)) / (1 - alpha) for alpha < 1, and M + 1 for alpha
= 1 [LEVIATHAN]; the drafter chooses the M within its limits that
maximizes that quantity divided by (M * t_d + t_v). A loaded drafter
thereby proposes fewer tokens, and a drafter whose cache bandwidth is
scarce does the same.
6.3. Verification and Commit
On receiving a proposal, the verifier performs these steps:
1. Authorization: the authenticated client MUST be the drafter named
in the latest assignment the verifier received for the session;
otherwise respond 403 with problem type "sdcp-not-assigned".
2. Fencing: if epoch differs from the session epoch, respond 409
with problem type "sdcp-stale-epoch".
3. Idempotency: if a verdict for this round already exists, return
it unchanged; if the round is being verified for an identical
earlier request, wait for that verdict and return it.
4. Base check: if base is not the committed length or base_digest is
not d[base], respond 409 with problem type "sdcp-stale-base" and
the current session-state, so that the drafter can resynchronize.
If round is not the last round plus one, respond 409 likewise.
5. Preconditions of the rule: if the acceptance rule needs draft
probabilities, the verifier MUST have obtained the rows for every
proposed position, and checked that q[i](x[i]) is not zero for
every position, before drawing any random value. If it cannot
obtain them, it rejects the round with problem type "sdcp-
q-unavailable" and commits nothing. Obtaining them first is
necessary because abandoning a round only after a rejection was
observed would bias the output distribution. The verifier MAY
fetch the rows concurrently with scoring.
Li, et al. Expires 6 May 2027 [Page 12]
Internet-Draft SDCP November 2026
6. Scoring: the verifier computes the target distributions p[1..M+1]
for the positions following the committed prefix, normally by a
single forward pass over the proposed tokens using the target KV
cache of the committed prefix. It MAY batch proposals of many
sessions into one pass.
7. Acceptance: it applies the contract's acceptance rule
(Section 6.4), yielding a accepted tokens and one additional
token. It truncates the emitted tokens after the first stop
token and at max_new.
8. Commit: it appends the emitted tokens to the committed prefix,
updates n and d[n], and appends a commit record (epoch, round,
emitted tokens, n, and d[n]) to the session's commit_log
(Section 6.8); only after that append succeeds does it send the
verdict and make the tokens available as output. If the append
fails because the log has been fenced, the verifier MUST discard
the round, release no output, and stop serving the session. The
commit and the recording of the verdict MUST be atomic with
respect to concurrent proposals for the session.
9. Control: it determines action and m_cap as specified in
Section 6.5.
After the verdict, the drafter MUST discard its draft state for
proposed positions beyond the accepted ones and MAY release its
leases. Its KV cache for the accepted tokens remains valid. The
verifier MAY publish the target KV cache of committed tokens to the
MLC so that another verifier can take over the session.
6.4. Acceptance Rules
Rules are registered (Section 10.2.2). This document defines two.
In both, p[i] is the target distribution at proposed position i, x[i]
the proposed token, and positions are evaluated in order.
Rule 1, greedy:
x[i] is accepted if it equals argmax p[i], where ties are broken
in favor of the smallest token identifier. Evaluation stops at
the first position that is not accepted. The additional token is
argmax p[a+1]. The output is identical to greedy decoding of the
target model as computed by the verifier's scoring passes,
whatever the drafter proposes.
Rule 2, speculative sampling:
For each position, draw r uniformly from [0, 1). Accept x[i] if r
< min(1, p[i](x[i]) / q[i](x[i])), where q[i] is the row of q_rows
for position i, that is, the draft distribution from which x[i]
Li, et al. Expires 6 May 2027 [Page 13]
Internet-Draft SDCP November 2026
was sampled. At the first rejected position, the additional token
is drawn from the normalized residual distribution max(0, p[i] -
q[i]). If all M tokens are accepted, the additional token is
drawn from p[M+1]. With exact q values, the output has the
distribution of the target model [LEVIATHAN] [CHEN]. The proposal
MUST carry q_rows; a proposal for which q[i](x[i]) is zero is
rejected, before any random draw, with problem type "sdcp-bad-
proposal".
The verifier's random values MUST be independent of those the
drafter used. When the sampling profile contains a seed, the
verifier derives each value from HMAC-SHA-256 [RFC2104] keyed with
the seed (as eight octets in network byte order) over "SDCP-
VERIFY-1" || 0x00 || session || uint32(round) || uint32(i) ||
uint8(purpose), where purpose is 0 for the acceptance test and 1
for drawing the additional token, taking the first 53 bits of the
result divided by 2^53. Drafters MUST NOT use this derivation for
their own sampling. Without a seed, the verifier uses a
cryptographically secure random number generator.
6.5. Speculation Control and Escalation
6.5.1. Speculation Limit
Each verdict carries m_cap, the largest M the drafter may use in the
next round, with 1 <= m_cap <= m_max. The verifier lowers m_cap when
its own load makes long proposals costly, and as specified for the
"reduce" action below. It SHOULD restore m_cap gradually (for
example by one per round) when rounds are no longer low and its load
permits.
6.5.2. Low Rounds and Escalation
A round is low if 100 * a < low_ratio * M, or, when low_ratio is zero
or absent, if a = 0. The verifier keeps three counters per session,
all starting at zero: c, the number of consecutive low rounds; g, the
number of consecutive rounds that are not low; and j, the position in
the contract's fallback list. After a low round, c is incremented
and g reset; after a round that is not low, c is reset and g
incremented, and when g reaches R, where R is low_rounds, j is reset
to zero. When c reaches R, the verifier applies entry j of the
fallback list (skipping entries that are not permitted, as described
for self_decode, and advancing j past every entry it applies or
skips), then resets c to zero. If no entry remains, it applies
"fail". Otherwise it applies "continue". The fallback list MUST
contain only the actions reduce, switch, self_decode, and fail; j
persists across a change of drafter and across a self-decoding
period. The actions are:
Li, et al. Expires 6 May 2027 [Page 14]
Internet-Draft SDCP November 2026
continue (0): The drafter proceeds with the next round.
reduce (1): The verifier sets m_cap to max(1, floor(m_cap / 2)) and
the drafter proceeds.
switch (2): The verifier sends an event to the router, which assigns
the session to another drafter with a permitted model; the current
drafter MUST stop proposing for the session.
self_decode (3): The verifier decodes the session itself with the
target model. This action is permitted only if the contract
contains self_load and the verifier's current load does not exceed
it; otherwise the verifier skips it and applies the next entry of
the fallback list. Each token committed during self-decoding is
recorded as a round of its own, with no proposal, so round numbers
stay consecutive. During self-decoding the verifier commits
tokens one at a time, publishes them as output, and stops after
self_max tokens (if present) or when its load exceeds self_load;
it then sends an event so that the router assigns a drafter again.
finish (4): Generation has ended because of a stop token or max_new.
fail (5): The verifier ends the session with an error.
Because self_decode is conditional on the verifier's load, the same
fallback list leads to self-decoding on a lightly loaded verifier and
to the next entry, such as switch, on a busy one.
6.6. Sampling Policy Updates
The router can change the sampling profile of a session, for example
to follow a preference stated by the user. It sends a policy message
with from_round greater than the current round. The verifier applies
the new profile to rounds numbered from_round and later, and the
router includes it in the "sampling" member of subsequent
assignments; the contract in those assignments is unchanged. The
acceptance rule cannot be changed. Committed output is never changed
by a policy update.
6.7. Output Delivery
Committed tokens are delivered with GET output?from=n, which returns
the tokens committed at output indices n and later, waiting up to a
server-chosen time if none are available. Only tokens whose commit
record has been written are returned. Indices are stable and every
output message carries the session epoch, so a front end that retries
never duplicates or loses tokens, and a front end that receives
output with a lower epoch than one it has seen discards it. The
Li, et al. Expires 6 May 2027 [Page 15]
Internet-Draft SDCP November 2026
final output message has final set to true and finish set to "finish"
or "fail".
6.8. Commit Log
The commit_log is an MLC object to which records are appended with a
conditional operation, for example an HTTP PUT with If-Match on the
current entity tag [RFC9110], so that an append succeeds only if no
other append has happened since the writer last read the log. Each
record carries the epoch of its writer. A commit record appended by
a verifier MUST carry the session's current epoch and follow the last
record it wrote. A fence record carries only an epoch; after a fence
record with epoch E+1, appends carrying an epoch lower than E+1 MUST
fail. If the MLC cannot enforce this rule, the verifier MUST read
the log after its conditional append and treat its append as failed
if a fence record with a higher epoch precedes its own record.
6.9. Failure Handling and Recovery
* Drafter failure: when a drafter's status expires, the router
reassigns its sessions. The new drafter starts from the committed
prefix in the assignment. Proposals from the failed drafter are
rejected because it is no longer the assigned drafter.
* Verifier failure: the router first appends a fence record with the
new epoch to the commit_log, so that the old verifier, even if it
is only partitioned, can no longer commit or release output; it
then reads the last commit record and opens the session at a new
verifier with that epoch and the "resume" member carrying that
round, committed length, digest, and the target KV cache from the
MLC if available. It then informs the verifier of the drafter and
sends a new assignment. Proposals carrying an older epoch are
rejected, and front ends discard output carrying an older epoch,
so a recovered old verifier cannot extend the session.
* Lost messages: proposals and their verdicts are idempotent per
round, so a drafter retries a proposal whose verdict it did not
receive.
7. Error Handling
+=====================+========+==============================+
| Type suffix | Status | Meaning |
+=====================+========+==============================+
| sdcp-stale-epoch | 409 | Epoch is not current |
+---------------------+--------+------------------------------+
| sdcp-stale-base | 409 | Proposal does not extend the |
| | | committed prefix |
Li, et al. Expires 6 May 2027 [Page 16]
Internet-Draft SDCP November 2026
+---------------------+--------+------------------------------+
| sdcp-q-unavailable | 409 | Draft probabilities required |
| | | by the rule are unavailable |
+---------------------+--------+------------------------------+
| sdcp-bad-contract | 422 | Contract invalid or not |
| | | supported |
+---------------------+--------+------------------------------+
| sdcp-limit | 422 | Proposal violates m_cap or |
| | | m_max |
+---------------------+--------+------------------------------+
| sdcp-session-closed | 410 | Session has finished or |
| | | failed |
+---------------------+--------+------------------------------+
| sdcp-bad-proposal | 422 | Proposal inconsistent with |
| | | its draft distributions |
+---------------------+--------+------------------------------+
| sdcp-not-assigned | 403 | Drafter is not assigned to |
| | | the session |
+---------------------+--------+------------------------------+
Table 2: SDCP Problem Types
The type URI is "https://iana.org/assignments/sdcp#" followed by the
suffix. These types are specific to SDCP and are therefore
registered in an SDCP registry rather than in the "HTTP Problem
Types" registry (Section 4.2 of [RFC9457]). Problem details for
"sdcp-stale-base" include the member "session_state" carrying the
current committed length and digest.
8. Operational Considerations
* Batch windows: a verifier that waits to fill a batch adds that
wait to every round. Windows of a fraction of the target model's
forward-pass time are typical.
* Cache placement: draft artifacts are short-lived and SHOULD be
placed in the fastest tier that both drafter and verifier can
reach.
* Monitoring: implementations SHOULD report the mean accepted
length, the distribution of M, escalation counts by action, stale-
base rejections, and self-decoded tokens.
* Incremental deployment: a verifier can serve sessions without
drafters by decoding itself, so SDCP can be introduced one drafter
pool at a time.
Li, et al. Expires 6 May 2027 [Page 17]
Internet-Draft SDCP November 2026
9. Security Considerations
All exchanges use mutual TLS, and every role authorizes requests by
authenticated identity: only the router opens sessions and assigns
drafters, and only the assigned drafter submits proposals.
Integrity of output. Under the greedy rule, the committed output
depends only on the target model and the committed prefix; a
malicious drafter can waste verifier capacity but cannot change the
output. Under speculative sampling, the drafter supplies q, and a
drafter that reports false probabilities can bias the output (for
example, by reporting a small q for tokens it wants accepted).
Speculative sampling MUST therefore be used only with drafters that
are trusted to the same degree as the verifier; with other drafters
the greedy rule MUST be used.
Denial of service. Proposals cost a forward pass. Verifiers MUST
enforce m_cap and SHOULD rate-limit rounds per session and per
drafter. The escalation ladder also bounds the work a poorly matched
or hostile drafter can cause.
Cache objects. References in proposals point to the MLC; the
verifier MUST fetch them only from caches it trusts for the session's
scope, so that a proposal cannot direct the verifier to arbitrary
URIs.
Privacy. Prompts, KV caches, and outputs are user data. The privacy
considerations of [I-D.li-kvcache-efficient-storage] apply to all MLC
objects of a session.
10. IANA Considerations
10.1. Media Type
IANA is requested to register the media type "application/sdcp+cbor"
in the "Media Types" registry [RFC6838]: Type name: application;
Subtype name: sdcp+cbor; Required parameters: N/A; Optional
parameters: N/A; Encoding considerations: binary; Security
considerations: see Section 9 of RFC XXXX; Interoperability
considerations: none; Published specification: RFC XXXX; Applications
that use this media type: distributed inference systems using
speculative decoding; Fragment identifier considerations: the syntax
and semantics of fragment identifiers specified for "application/
cbor" apply; Additional information: Deprecated alias names for this
type: N/A, Magic number(s): N/A, File extension(s): N/A, Macintosh
file type code(s): N/A; Person and email address to contact: IETF
<iesg@ietf.org>; Intended usage: COMMON; Restrictions on usage: none;
Author: IETF; Change controller: IETF.
Li, et al. Expires 6 May 2027 [Page 18]
Internet-Draft SDCP November 2026
10.2. SDCP Registries
IANA is requested to create a registry group named "Speculative
Decoding Coordination Protocol (SDCP)" at
"https://www.iana.org/assignments/sdcp" with the registries below.
For the numeric registries, values 0-191 are assigned by
Specification Required [RFC8126] and values 192-255 are reserved for
Private Use. The designated experts should check that a new
acceptance rule states whether it preserves the target distribution
and what trust it requires in drafters.
10.2.1. SDCP Problem Types
Columns: Type Suffix, Title, Recommended HTTP Status Code, Reference.
The type URI is "https://iana.org/assignments/sdcp#" followed by the
suffix. Initial contents: the entries of Table 2, with reference RFC
XXXX. Policy: Specification Required.
10.2.2. SDCP Acceptance Rules
Initial contents: 0 Reserved; 1 greedy; 2 speculative-sampling; each
with reference RFC XXXX. A registration MUST specify the acceptance
test, the additional token, and the information a proposal must
carry.
10.2.3. SDCP Actions
Initial contents: 0 continue; 1 reduce; 2 switch; 3 self_decode; 4
finish; 5 fail.
11. References
11.1. Normative References
[RFC2104] Krawczyk, H., Bellare, M., and R. Canetti, "HMAC: Keyed-
Hashing for Message Authentication", RFC 2104,
DOI 10.17487/RFC2104, February 1997,
<https://www.rfc-editor.org/info/rfc2104>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC4648] Josefsson, S., "The Base16, Base32, and Base64 Data
Encodings", RFC 4648, DOI 10.17487/RFC4648, October 2006,
<https://www.rfc-editor.org/info/rfc4648>.
Li, et al. Expires 6 May 2027 [Page 19]
Internet-Draft SDCP November 2026
[RFC6234] Eastlake 3rd, D. and T. Hansen, "US Secure Hash Algorithms
(SHA and SHA-based HMAC and HKDF)", RFC 6234,
DOI 10.17487/RFC6234, May 2011,
<https://www.rfc-editor.org/info/rfc6234>.
[RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type
Specifications and Registration Procedures", BCP 13,
RFC 6838, DOI 10.17487/RFC6838, January 2013,
<https://www.rfc-editor.org/info/rfc6838>.
[RFC8126] Cotton, M., Leiba, B., and T. Narten, "Guidelines for
Writing an IANA Considerations Section in RFCs", BCP 26,
RFC 8126, DOI 10.17487/RFC8126, June 2017,
<https://www.rfc-editor.org/info/rfc8126>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8446] Rescorla, E., "The Transport Layer Security (TLS) Protocol
Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018,
<https://www.rfc-editor.org/info/rfc8446>.
[RFC8610] Birkholz, H., Vigano, C., and C. Bormann, "Concise Data
Definition Language (CDDL): A Notational Convention to
Express Concise Binary Object Representation (CBOR) and
JSON Data Structures", RFC 8610, DOI 10.17487/RFC8610,
June 2019, <https://www.rfc-editor.org/info/rfc8610>.
[RFC8949] Bormann, C. and P. Hoffman, "Concise Binary Object
Representation (CBOR)", STD 94, RFC 8949,
DOI 10.17487/RFC8949, December 2020,
<https://www.rfc-editor.org/info/rfc8949>.
[RFC9110] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke,
Ed., "HTTP Semantics", STD 97, RFC 9110,
DOI 10.17487/RFC9110, June 2022,
<https://www.rfc-editor.org/info/rfc9110>.
[RFC9113] Thomson, M., Ed. and C. Benfield, Ed., "HTTP/2", RFC 9113,
DOI 10.17487/RFC9113, June 2022,
<https://www.rfc-editor.org/info/rfc9113>.
[RFC9114] Bishop, M., Ed., "HTTP/3", RFC 9114, DOI 10.17487/RFC9114,
June 2022, <https://www.rfc-editor.org/info/rfc9114>.
Li, et al. Expires 6 May 2027 [Page 20]
Internet-Draft SDCP November 2026
[RFC9205] Nottingham, M., "Building Protocols with HTTP", BCP 56,
RFC 9205, DOI 10.17487/RFC9205, June 2022,
<https://www.rfc-editor.org/info/rfc9205>.
[RFC9457] Nottingham, M., Wilde, E., and S. Dalal, "Problem Details
for HTTP APIs", RFC 9457, DOI 10.17487/RFC9457, July 2023,
<https://www.rfc-editor.org/info/rfc9457>.
11.2. Informative References
[CHEN] Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre,
L., and J. Jumper, "Accelerating Large Language Model
Decoding with Speculative Sampling", arXiv:2302.01318,
February 2023, <https://arxiv.org/abs/2302.01318>.
[DEEPSEEKV3]
DeepSeek-AI, "DeepSeek-V3 Technical Report",
arXiv:2412.19437, December 2024,
<https://arxiv.org/abs/2412.19437>.
[I-D.li-kvcache-efficient-storage]
Li, Z., Du, Z., Wang, J., Cheng, W., Zhao, Y., Zhao, C.,
and X. Sun, "Key-Value (KV) Cache Storage Protocol (KVSP):
Indexing, Placement, Prefetch, and Migration for
Distributed Inference", Work in Progress, Internet-Draft,
draft-li-kvcache-efficient-storage-00, 2 November 2026,
<https://datatracker.ietf.org/doc/html/draft-li-kvcache-
efficient-storage-00>.
[LEVIATHAN]
Leviathan, Y., Kalman, M., and Y. Matias, "Fast Inference
from Transformers via Speculative Decoding", Proceedings
of the 40th International Conference on Machine Learning
(ICML), PMLR 202, July 2023,
<https://proceedings.mlr.press/v202/leviathan23a.html>.
Appendix A. Test Vectors
A contract with greedy rule, m_max 8, max_new 256, stop token 2,
low_rounds 3, and fallback ladder [reduce, switch, self_decode,
fail], with target and tokenizer digests equal to the KVSP test
vector digests, has the following deterministic CBOR encoding (143
octets):
Li, et al. Expires 6 May 2027 [Page 21]
Internet-Draft SDCP November 2026
a8 64 72 75 6c 65 01 64 73 74 6f 70 81 02 65 6d
5f 6d 61 78 08 66 74 61 72 67 65 74 58 20 25 4e
52 8c 66 9a 20 84 c9 72 72 98 ca cb 83 e8 85 4e
0a 6f f8 0e 2b f1 80 96 b9 dc e0 f7 eb da 67 6d
61 78 5f 6e 65 77 19 01 00 68 66 61 6c 6c 62 61
63 6b 84 01 02 03 05 69 74 6f 6b 65 6e 69 7a 65
72 58 20 9f 42 a7 a2 9d ca 30 63 ad 26 d7 be 7f
e6 a9 30 fa 38 af b1 a7 03 e4 f7 ff 53 8d b4 e1
33 b1 9c 6a 6c 6f 77 5f 72 6f 75 6e 64 73 03
For the prompt [1, 2, 3, 4] and committed output tokens 17 and 42:
contract_digest = bcae60a92c622b03fa1c229bac9f5654
076957e2f1a8bedcb3549ce4a6328e1f
d[0] = 186b373698d5baf27716830811ffa641
4887dd0a4f9d13db6c00e19bdb57b76e
d[1] = 3372e0fd91b77611c3502da5b0a5c8a5
1540e055d7abdd0d1bcf91110e142c1c
d[2] = 469bbe77ceb5b1540a4d61d826a3117e
357e097c5ac20271306633559fccdf1f
Example round under the greedy rule: committed n = 2; the drafter
proposes [7, 9, 11] (M = 3); the target argmax tokens at the three
positions are 7, 9, and 13, and at the fourth position 5. Then a =
2, the additional token is 13 (the correction at the first mismatch),
emitted = [7, 9, 13], and the new committed length is 5. The round
is not low, so c is reset and the action is "continue".
Acknowledgments
The authors thank colleagues working on disaggregated inference
serving for discussions on separating drafting from verification.
Authors' Addresses
Zhiqiang Li
China Mobile
Beijing
100053
China
Email: lizhiqiangyjy@chinamobile.com
Zongpeng Du
China Mobile
Beijing
100053
China
Li, et al. Expires 6 May 2027 [Page 22]
Internet-Draft SDCP November 2026
Email: duzongpeng@chinamobile.com
Junjie Wang
Centec
Shanghai
201203
China
Email: wangjj@centec.com
Wei Cheng
Centec
Shanghai
201203
China
Email: chengw@centec.com
Ying Zhao
Centec
Shanghai
200000
China
Email: zhaoying@centec.com
Chunhao Zhao
SAIA
Shanghai
200125
China
Email: chunhao.zhao@sh-aia.com
Xun Sun
Inesa
Shanghai
200030
China
Email: sunxun@inesa.com
Li, et al. Expires 6 May 2027 [Page 23]