Communication Optimization for MoE Expert Parallelism in Distributed Training
draft-li-moe-ep-communication-optimization-00
This document is an Internet-Draft (I-D).
Anyone may submit an I-D to the IETF.
This I-D is not endorsed by the IETF and has no formal standing in the
IETF standards process.
| Document | Type | Active Internet-Draft (individual) | |
|---|---|---|---|
| Authors | Zhiqiang Li , Zongpeng Du , Junjie Wang , Wei Cheng , Guoying Zhang , Xun Sun , Chunhao Zhao | ||
| Last updated | 2026-07-04 | ||
| RFC stream | (None) | ||
| Intended RFC status | (None) | ||
| Formats | |||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | I-D Exists | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-li-moe-ep-communication-optimization-00
Network Working Group Z. Li
Internet-Draft Z. Du
Intended status: Informational China Mobile
Expires: 5 January 2027 J. Wang
W. Cheng
G. Zhang
Centec
X. Sun
Inesa
C. Zhao
SAIA
4 July 2026
Communication Optimization for MoE Expert Parallelism in Distributed
Training
draft-li-moe-ep-communication-optimization-00
Abstract
This document describes a communication optimization mechanism for
Mixture of Experts (MoE) Expert Parallelism (EP) in distributed
training environments. It defines two core mechanisms: (1) an
adaptive communication mode selection that dynamically chooses
between Data-Centric and Expert-Centric All-to-All communication
patterns based on a comparison of expert parameter size versus token
data size, minimizing cross-node communication volume; (2) a
priority-based co-scheduling strategy for All-to-All and AllReduce
collective communications, combined with chunked transmission and in-
network aggregation acceleration, to eliminate bandwidth contention
and reduce overall training latency.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 5 January 2027.
Li, et al. Expires 5 January 2027 [Page 1]
Internet-Draft MoE EP Comm Optimization July 2026
Copyright Notice
Copyright (c) 2026 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Requirements Language . . . . . . . . . . . . . . . . . . 3
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Problem Statement . . . . . . . . . . . . . . . . . . . . . . 4
3.1. All-to-All Communication Bottleneck . . . . . . . . . . . 4
3.2. Static Communication Mode Limitation . . . . . . . . . . 5
3.3. All-to-All and AllReduce Bandwidth Contention . . . . . . 5
4. Adaptive Communication Mode Selection . . . . . . . . . . . . 5
4.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 5
4.2. Mode Selection Criteria . . . . . . . . . . . . . . . . . 5
4.3. Data-Centric Mode Execution . . . . . . . . . . . . . . . 6
5. All-to-All and AllReduce Co-Scheduling . . . . . . . . . . . 6
5.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 6
5.2. Scheduling Procedure . . . . . . . . . . . . . . . . . . 6
5.3. Communication Hierarchy . . . . . . . . . . . . . . . . . 7
6. Protocol Operations . . . . . . . . . . . . . . . . . . . . . 7
6.1. Collective Communication Library Extensions . . . . . . . 7
6.2. Mode Re-evaluation Triggers . . . . . . . . . . . . . . . 7
6.3. Compatibility . . . . . . . . . . . . . . . . . . . . . . 7
7. Security Considerations . . . . . . . . . . . . . . . . . . . 8
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8
9. Normative References . . . . . . . . . . . . . . . . . . . . 8
10. Informative References . . . . . . . . . . . . . . . . . . . 8
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 9
Li, et al. Expires 5 January 2027 [Page 2]
Internet-Draft MoE EP Comm Optimization July 2026
1. Introduction
Mixture of Experts (MoE) [MoE] is a sparse neural network
architecture that significantly scales model parameters without
proportionally increasing computational cost. In a MoE architecture,
each input token activates only a small number of expert sub-networks
(typically 1 to 2) for processing, enabling efficient utilization of
compute resources.
Expert Parallelism (EP) is a key parallelism strategy for MoE
training. By distributing different experts across multiple compute
nodes (e.g., GPUs), EP achieves parallel computation across nodes,
significantly reducing per-node load and improving training
throughput.
However, EP relies heavily on All-to-All collective communication to
dispatch tokens to the correct experts and gather results.
Experiments have shown that All-to-All communication accounts for
over 34.1% of total training time [DeepSpeed-MoE], constituting the
primary scalability bottleneck. Furthermore, during the backward
pass, All-to-All communication contends for bandwidth with AllReduce
communication used for gradient aggregation in data parallelism.
This document proposes two mechanisms to address these challenges:
(1) Adaptive Communication Mode Selection: Dynamically selects
between Data-Centric and Expert-Centric All-to-All patterns based on
model and cluster parameters. (2) Priority-Based Co-Scheduling:
Coordinates All-to-All and AllReduce communications through priority
scheduling, chunked transmission, and in-network aggregation to
eliminate bandwidth contention.
1.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
2. Terminology
MoE (Mixture of Experts): A sparse neural network architecture where
each input activates only a subset of expert sub-networks.
EP (Expert Parallelism): A parallelism strategy that distributes
different experts across multiple compute nodes.
All-to-All: A collective communication primitive where each node
Li, et al. Expires 5 January 2027 [Page 3]
Internet-Draft MoE EP Comm Optimization July 2026
sends distinct data to every other node.
AllReduce: A collective communication primitive that performs a
reduction operation across all nodes and distributes the result.
Token: The smallest unit of input data processed by the model.
Worker: A single compute unit participating in training, typically
corresponding to one GPU.
H: The hidden dimension size of the model.
T: Total number of tokens to process, determined by Batch Size (B)
and Sequence Length (S).
E: Total number of experts in the MoE model.
Top-K: The number of experts selected per token by the MoE router.
GDR (GPU Direct RDMA): A technology enabling direct memory access
between GPUs across a network without CPU involvement.
NVLink: A high-speed interconnect bus between NVIDIA GPUs within a
node.
NVSwitch: A high-speed switching chip for inter-GPU communication
within a node.
ToR Switch: Top-of-Rack switch, used for inter-node communication.
In-Network Aggregation: The capability of performing reduction
operations on network switching devices.
3. Problem Statement
3.1. All-to-All Communication Bottleneck
MoE models achieve computational efficiency through dynamic expert
activation. However, the All-to-All communication required for
Expert Parallelism constitutes the primary bottleneck in scaling
distributed training. Empirical studies demonstrate that All-to-All
communication accounts for over 34.1% of total training time. The
All-to-All communication exhibits a "barrel effect": overall
communication latency is determined by the slowest node. Since
processing capacity and token count vary across nodes, individual
slow nodes can degrade overall training throughput.
Li, et al. Expires 5 January 2027 [Page 4]
Internet-Draft MoE EP Comm Optimization July 2026
3.2. Static Communication Mode Limitation
Existing implementations typically employ a fixed All-to-All
communication pattern, either exchanging data or exchanging expert
parameters. However, the communication cost of each pattern varies
significantly under different model configurations and cluster
topologies. A static approach cannot adapt to all scenarios.
3.3. All-to-All and AllReduce Bandwidth Contention
During the backward pass of training, All-to-All communication (for
EP gradient synchronization) and AllReduce communication (for data-
parallel gradient aggregation) share physical link bandwidth.
Concurrent execution of both communication types leads to bandwidth
contention, causing significant increases in overall communication
latency and degraded training performance.
4. Adaptive Communication Mode Selection
4.1. Overview
This mechanism dynamically selects the optimal All-to-All
communication mode by comparing expert parameter size against token
data size. Data-Centric Mode: When expert parameter size is smaller
than token data size, workers dynamically exchange expert parameters
while keeping data stationary. Suitable for scenarios with smaller
experts and larger data volumes. Expert-Centric Mode: When expert
parameter size is larger than token data size, workers exchange token
data while keeping expert parameters stationary. Suitable for
scenarios with larger experts and smaller data volumes.
4.2. Mode Selection Criteria
The collective communication library SHOULD automatically select the
communication mode through the following procedure: Step 1 - Compute
Sizes: Expert parameter size S_expert = 8 * H^2; Token data size
S_data = 2 * H * T (where T = B * S * K). Step 2 - Detect Cluster
Topology: The library MUST automatically detect node configuration,
GPU type, network topology, and inter-node interconnect bandwidth.
Step 3 - Compute Cross-Node Communication Volume: Data-Centric volume
C_dc = 8 * H * E * m * (n - 1); Expert-Centric volume C_ec = 2 * m *
H * T * (n - 1) / n (E = expert count, n = node count, m = workers
per node). Step 4 - Select Mode: If C_dc < C_ec, select Data-Centric
mode. Otherwise, select Expert-Centric mode.
Li, et al. Expires 5 January 2027 [Page 5]
Internet-Draft MoE EP Comm Optimization July 2026
4.3. Data-Centric Mode Execution
When Data-Centric mode is selected, the following procedure MUST be
executed: (1) Each worker fetches remote expert parameters to local
node CPU memory via GDR NIC; fetch targets SHOULD be staggered across
workers to avoid link contention. (2) The fetch operations are
coordinated by the collective communication library or scheduled by
in-network compute switches. (3) Intra-node expert transfer between
workers is performed via NVLink, scheduled by the collective
communication library or NVSwitch. (4) Expert parameters are loaded
into workers (GPUs) via the PCIe bus in parallel. (5) Inter-node
Ethernet transfers and intra-node PCIe/NVLink transfers SHOULD
proceed concurrently to achieve pipelining.
5. All-to-All and AllReduce Co-Scheduling
5.1. Overview
During the backward pass, this mechanism eliminates bandwidth
contention between All-to-All and AllReduce communications through:
Chunked Transmission (AllReduce gradient data is split into fixed-
size chunks); Priority Scheduling (All-to-All traffic is assigned
high priority; AllReduce chunks are assigned low priority); Gap
Filling (AllReduce chunks are transmitted during idle intervals
between All-to-All bursts); In-Network Aggregation (AllReduce traffic
passing through ToR switches and NVSwitch is accelerated by
performing aggregation operations on network devices).
5.2. Scheduling Procedure
The collective communication library MUST execute the following
scheduling rules: (1) AllReduce gradient data MUST be split into
fixed-length chunks; chunk size MAY be statically configured or
dynamically adjusted; each chunk is assigned low priority and
enqueued. (2) All-to-All gradient synchronization data MUST be
assigned high priority and enqueued. (3) The library MUST schedule
the send queue in priority order, transmitting All-to-All data first
and filling idle gaps with AllReduce chunks. (4) The library MAY
monitor or predict contention in real time, dynamically adjusting
chunk size to optimize throughput. (5) AllReduce traffic traversing
ToR switches and NVSwitch SHOULD leverage in-network aggregation for
acceleration.
Li, et al. Expires 5 January 2027 [Page 6]
Internet-Draft MoE EP Comm Optimization July 2026
5.3. Communication Hierarchy
This mechanism follows an "intra-node first, inter-node second"
communication principle. Intra-node communication (typical
bandwidth: 600 GB/s via NVLink) completes local All-to-All and
AllReduce operations first. Inter-node communication (typical
bandwidth: 200 Gbps via Ethernet/InfiniBand) then completes cross-
node synchronization. For AllReduce, inter-node traffic is
aggregated via ToR switches, and intra-node traffic is aggregated via
NVSwitch.
6. Protocol Operations
6.1. Collective Communication Library Extensions
To support this mechanism, the collective communication library
SHOULD provide the following extension interfaces: Mode Query
Interface (returns the currently recommended communication mode);
Topology Detection Interface (returns cluster physical topology
information); Chunk Size Configuration Interface (sets AllReduce
chunk size); Priority Configuration Interface (sets priority levels
for different communication task types); In-Network Aggregation
Control Interface (enables or disables in-network aggregation).
6.2. Mode Re-evaluation Triggers
Communication mode selection is performed during training
initialization. The library MUST re-evaluate mode selection upon:
training initialization; cluster topology changes (e.g., node
addition or removal); model configuration changes (e.g., changes to
expert count, hidden dimension, or other relevant parameters).
6.3. Compatibility
This mechanism is designed as an incremental extension to existing
collective communication libraries (e.g., NCCL, HCCL). It does not
alter existing All-to-All and AllReduce semantic interfaces. Mode
selection and scheduling strategies are implemented as internal
optimizations, supporting seamless integration with existing training
frameworks (e.g., PyTorch, MindSpore).
Li, et al. Expires 5 January 2027 [Page 7]
Internet-Draft MoE EP Comm Optimization July 2026
7. Security Considerations
This mechanism involves communication optimization across multiple
nodes and the following security aspects MUST be considered. Cluster
Topology Protection: Auto-detected cluster topology information
constitutes sensitive infrastructure data; the library MUST ensure
this information is not accessible to unauthorized parties.
Communication Integrity: In Data-Centric mode, expert parameters are
transmitted over the network; implementations SHOULD ensure the
integrity of parameter data during transmission to prevent man-in-
the-middle tampering that could compromise training results. Denial
of Service Protection: Misuse of in-network aggregation capabilities
could exhaust switch resources; network devices SHOULD implement rate
limiting and resource isolation for in-network aggregation tasks.
Priority Abuse Prevention: Improper use of the priority mechanism
could cause persistent starvation of low-priority traffic; the
library SHOULD enforce minimum guaranteed bandwidth to ensure
AllReduce communication is not indefinitely deferred.
8. IANA Considerations
This document has no IANA actions.
9. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
10. Informative References
[DeepSpeed-MoE]
Rajbhandari, S., Li, C., Yao, Z., Zhang, M., Aminabadi,
R.Y., Awan, A.A., Rasley, J., and Y. He, "DeepSpeed-MoE:
Advancing Mixture-of-Experts Inference and Training to
Power Next-Generation AI Scale", ICML 2022, 2022.
[MoE] Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le,
Q., Hinton, G., and J. Dean, "Outrageously Large Neural
Networks: The Sparsely-Gated Mixture-of-Experts Layer",
ICLR 2017, 2017.
Li, et al. Expires 5 January 2027 [Page 8]
Internet-Draft MoE EP Comm Optimization July 2026
Authors' Addresses
Zhiqiang Li
China Mobile
Beijing
100053
China
Email: lizhiqiangyjy@chinamobile.com
Zongpeng Du
China Mobile
Beijing
100053
China
Email: duzongpeng@chinamobile.com
Junjie Wang
Centec
Shanghai
201203
China
Email: wangjj@centec.com
Wei Cheng
Centec
Shanghai
201203
China
Email: chengw@centec.com
Guoying Zhang
Centec
Shanghai
201203
China
Email: zhanggy@centec.com
Xun Sun
Inesa
Shanghai
200030
China
Email: sunxun@inesa.com
Li, et al. Expires 5 January 2027 [Page 9]
Internet-Draft MoE EP Comm Optimization July 2026
Chunhao Zhao
SAIA
Shanghai
200125
China
Email: chunhao.zhao@sh-aia.com
Li, et al. Expires 5 January 2027 [Page 10]