Fully Adaptive Routing Ethernet in Multi-Plane Scale-Out Networks
draft-xu-rtgwg-fare-in-mp-son-00
This document is an Internet-Draft (I-D).
Anyone may submit an I-D to the IETF.
This I-D is not endorsed by the IETF and has no formal standing in the
IETF standards process.
| Document | Type | Active Internet-Draft (individual) | |
|---|---|---|---|
| Authors | Xiaohu Xu , Zongying He , Nan Wang , Wei Wan , Hua Wang , Jian Guo , Xiang Li , Tianyou Zhou , Yongtao Yang , Yinben Xia , Weifeng Zhang , Peilong Wang , Yan Zhuang , Fajie Yang , Chao Li , Xiaojun Wang , Roman Glebov | ||
| Last updated | 2026-06-10 | ||
| RFC stream | (None) | ||
| Intended RFC status | (None) | ||
| Formats | |||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | I-D Exists | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-xu-rtgwg-fare-in-mp-son-00
Network Working Group X. Xu
Internet-Draft China Mobile
Intended status: Standards Track Z. He
Expires: 12 December 2026 Broadcom
N. Wang
Intel
N. Wang
Hygon
W. Wan
Sugon
H. Wang
Moore Threads
J. Guo
Biren Technology
X. Li
Enflame Technology
T. Zhou
Resnics Technology
Y. Yang
Centec
Y. Xia
W. Zhang
Tencent
P. Wang
Baidu
Y. Zhuang
Huawei Technologies
F. Yang
Cloudnine Information Technologies
C. Li
Metanet Networking Technology
X. Wang
Ruijie Networks
R. Glebov
Yandex
10 June 2026
Fully Adaptive Routing Ethernet in Multi-Plane Scale-Out Networks
draft-xu-rtgwg-fare-in-mp-son-00
Abstract
FARE-BGP enables weighted ECMP load balancing using a path-bandwidth
extended community. FARE-in-SUN extends this mechanism from switches
to GPUs for scale-up networks, which are typically multi-plane.
Large AI training clusters increasingly adopt multi-plane scale-out
network topologies. This document further extends FARE-BGP from
Xu, et al. Expires 12 December 2026 [Page 1]
Internet-Draft FARE in SUN June 2026
switches to RoCE NICs (RNICs) for such multi-plane scale-out
networks. The document also presents two techniques to address route
scalability concerns caused by the injection of numerous host routes.
Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 12 December 2026.
Copyright Notice
Copyright (c) 2026 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5
3. Solution Description . . . . . . . . . . . . . . . . . . . . 5
Xu, et al. Expires 12 December 2026 [Page 2]
Internet-Draft FARE in SUN June 2026
3.1. Route Aggregation with Explicit Unreachable Host Route
Advertisement . . . . . . . . . . . . . . . . . . . . . . 6
3.2. Prefix-ORF-Based Route Filtering . . . . . . . . . . . . 7
4. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 7
5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7
6. Security Considerations . . . . . . . . . . . . . . . . . . . 7
7. References . . . . . . . . . . . . . . . . . . . . . . . . . 7
7.1. Normative References . . . . . . . . . . . . . . . . . . 7
7.2. Informative References . . . . . . . . . . . . . . . . . 8
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 8
1. Introduction
Large AI training clusters (beyond 100,000 GPUs) increasingly use
multi-plane scale-out network topologies (see below) to reduce the
total number of switches and links. In such a topology, a high-speed
RNIC is split into multiple lower-speed lanes, each connected to an
independent CLOS fabric (a “plane”). Because there are no links
between planes, the RNIC itself must decide which plane to use for
each packet or flow. In other words, the RNIC must know the
reachability of each plane and then perform global load balancing
across planes.
Xu, et al. Expires 12 December 2026 [Page 3]
Internet-Draft FARE in SUN June 2026
=========================================
# +----+ +----+ +----+ +----+ #
# | S1 | | S2 | | S3 | | S4 | (Spine) #
# +----+ +----+ +----+ +----+ #
# Plane-1 #
# +----+ +----+ +----+ +----+ #
# | L1 | | L2 | | L3 | | L4 | (Leaf) #
# +----+ +----+ +----+ +----+ #
=========================================
=================================== ===================================
# +-----+ +-----+ +-----+ +-----+ # # +-----+ +-----+ +-----+ +-----+ #
# |RNIC1| |RNIC2| |RNIC3| |RNIC4| # # |RNIC1| |RNIC2| |RNIC3| |RNIC4| #
# +-----+ +-----+ +-----+ +-----+ # # +-----+ +-----+ +-----+ +-----+ #
# Server-1 # # Server-n #
#================================== ... ===================================
=========================================
# +----+ +----+ +----+ +----+ #
# | L1 | | L2 | | L3 | | L4 | (Leaf) #
# +----+ +----+ +----+ +----+ #
# Plane-2 #
# +----+ +----+ +----+ +----+ #
# | S1 | | S2 | | S3 | | S4 | (Spine) #
# +----+ +----+ +----+ +----+ #
=========================================
Figure 1
(For simplicity, the diagram above omits the connections between
RNICs and leaf switches. In practice, each RNIC is multi-homed to
one leaf switch in every plane.)
FARE-in-SUN [I-D.xu-rtgwg-fare-in-sun] describes how to extend the
FARE-BGP protocol [I-D.xu-idr-fare] from switches to GPUs for
scale-up networks. Because scale-up shares the same multi-plane
architectural pattern as multi-plane scale-out networks, the adaptive
routing approach defined in FARE-in-SUN can be applied directly to
multi-plane scale-out networks.
The solution described in this document is almost identical to
FARE-in-SUN, with the following two essential differences. First,
FARE-BGP is extended from switches to RNICs rather than to GPUs.
Second, in a scale-up network, the number of route entries is small
(typically a few hundred) and can be installed directly on GPUs. In
an isolated multi-plane scale-out network with 100,000 GPUs and four
planes, each plane may propagate up to 100,000 host routes – a total
Xu, et al. Expires 12 December 2026 [Page 4]
Internet-Draft FARE in SUN June 2026
of 400,000 routes. Storing all these routes on an RNIC is
impractical. Therefore, the RNIC must suppress the routing table
using the techniques described in Section 4.
This document describes how to extend the Fully Adaptive Routing
Ethernet (FARE) using BGP (FARE-BGP in short) as described in , which
was originally designed for scale-out netowrks, to scale-up networks.
2. Terminology
This memo makes use of the terms defined in [RFC2119].
3. Solution Description
In an isolated multi-plane scale-out network, an RNIC connects to
each plane and is configured as a stub BGP speaker per plane. It
establishes separate BGP sessions with the attached leaf switches of
each plane. The BGP neighbor discovery
[I-D.xu-idr-neighbor-autodiscovery] can be used to simplify
configuration.
Through these sessions, the RNIC learns routes to remote GPUs
together with the path-bandwidth extended community. Because the
RNIC participates in BGP with each plane independently, it aggregates
per-plane path-bandwidth information and performs weighted load
balancing across planes. The RNIC thus performs the same Weighted
Equal-Cost Multi-Path (WECMP) functions as a FARE-capable switch,
distributing traffic in proportion to the path bandwidth of each ECMP
route.
Two modes of WECMP are supported:
Per-flow WECMP (for RNICs that cannot handle disordered packet
delivery): The RNIC establishes at least one QP per plane. The
number of QPs allocated to a plane is proportional to the plane’s
weight. All packets of a given flow go through the same plane,
preserving order.
Per-packet WECMP (for RNICs that support out-of-order packet
delivery): A single QP per (source, destination) RNIC pair
suffices. The RNIC sprays each packet of that QP across all
available planes according to the weights.
In an isolated multi-plane scale-out network with 100,000 GPUs and
four planes, each plane may propagate up to 100,000 host routes – a
total of 400,000 routes. Storing all these routes on an RNIC is
impractical. Two complementary approaches can reduce the number of
routes the RNIC must store.
Xu, et al. Expires 12 December 2026 [Page 5]
Internet-Draft FARE in SUN June 2026
3.1. Route Aggregation with Explicit Unreachable Host Route
Advertisement
It's straightfoward to resort to route aggregation mechanism, i.e.,
aggregating host routes when advertising them from leaf to spine.
However, naive aggregation can cause route blackholes: if a specific
host within an aggregate becomes unreachable, the aggregated route
still points to that plane. Consequently, traffic destined for that
host will still be forwarded according to the aggregated route and
then dropped.
To address this issue, the switches MUST explicitly advertises
unreachable host routes for a given RNIC to the other RNICs. When a
RNIC becomes unreachable via a particular plane, the leaf switch
advertises this unreachability to the RNIC using one of two methods:
Path bandwidth value of 0: The leaf switch advertises the host
route (NLRI) with the BGP path-bandwidth extended community set
to 0. The RNIC interprets this as “unreachable” and excludes that
plane from the next-hop set for that destination.
Specific BGP unreachability advertisement: The leaf switch sends a
dedicated BGP unreachability message. This is distinct from a
standard BGP route withdrawal. It explicitly marks the host as
unreachable via that plane while keeping the aggregated route
intact.
Upon receiving such an advertisement, the RNIC updates its forwarding
table as follows:
It locates the longest-matching aggregated route that covers the
unreachable host (e.g., a default route or a supernet prefix).
From that aggregated route’s set of next-hops (which originally
included multiple planes), it removes the next-hop corresponding
to the plane where the host is unreachable.
It then installs a host-specific route for the unreachable
destination, with the remaining next-hops from the aggregated
route.
Example: Suppose an RNIC has a default route (0.0.0.0/0) with
next-hops pointing to planes A, B, C, and D. Host X (a specific /32)
becomes unreachable via plane A. The RNIC learns an unreachable
advertisement for X. It then creates a host route for X with
next-hops set to {B, C, D} – i.e., the original aggregated next-hops
minus the next-hop associated with plane A. Traffic to X will never
be sent to plane A, avoiding blackholes.
Xu, et al. Expires 12 December 2026 [Page 6]
Internet-Draft FARE in SUN June 2026
This technique dramatically reduces BGP table size on the RNIC: the
RNIC only needs to store aggregated routes (e.g., a handful of
default routes per plane) plus explicit unreachable host routes for
the small number of hosts that are actually unreachable. The
majority of reachable hosts are covered by aggregates and require no
per-host state. The approach is especially effective when
unreachability is rare, which is typical in well-managed clusters.
Switches within each plane does not need to install the unreachable
host route into their FIB tables.
3.2. Prefix-ORF-Based Route Filtering
Since a given RNIC communicates only with a limited subset of GPUs
(due to AI training parallelism patterns), it’s possible for the
RNIC to filter routes to retain only those it actually needs.
The RNIC sends Address Prefix ORF entries to its BGP peer (leaf
switch) per plane. These entries indicate the host routes for remote
RNICs the local RNIC is interested in. The peer filters outbound
route updates accordingly, sending only the requested routes. In
this way, the RNIC stores only a limited number of routes.
For switches, there is no need install host routes for remote RNICs.
Therefore, the FIB-suppression mechanism as described in Virtual
Aggregation Auto-configuration [I-D.ietf-grow-va-auto] could be
reused.
4. Acknowledgements
TBD.
5. IANA Considerations
TBD.
6. Security Considerations
TBD.
7. References
7.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
Xu, et al. Expires 12 December 2026 [Page 7]
Internet-Draft FARE in SUN June 2026
7.2. Informative References
[I-D.ietf-grow-va-auto]
Francis, P., Xu, X., Ballani, H., Jen, D., Raszuk, R., and
L. Zhang, "Auto-Configuration in Virtual Aggregation",
Work in Progress, Internet-Draft, draft-ietf-grow-va-auto-
05, 30 December 2011,
<https://datatracker.ietf.org/doc/html/draft-ietf-grow-va-
auto-05>.
[I-D.xu-idr-fare]
Xu, X., Hegde, S., Patel, K., He, Z., Wang, J., Huang, H.,
Zhang, Q., Wu, H., Liu, Y., Xia, Y., Wang, P., Tiezheng,
and R. Glebov, "Fully Adaptive Routing Ethernet using
BGP", Work in Progress, Internet-Draft, draft-xu-idr-fare-
05, 1 June 2026, <https://datatracker.ietf.org/doc/html/
draft-xu-idr-fare-05>.
[I-D.xu-idr-neighbor-autodiscovery]
Xu, X., Talaulikar, K., Bi, K., Tantsura, J.,
Triantafillis, N., and X. Chen, "BGP Neighbor Discovery",
Work in Progress, Internet-Draft, draft-xu-idr-neighbor-
autodiscovery-13, 28 January 2026,
<https://datatracker.ietf.org/doc/html/draft-xu-idr-
neighbor-autodiscovery-13>.
[I-D.xu-rtgwg-fare-in-sun]
Xu, X., He, Z., Wang, N., Wang, H., Guo, J., Li, X., Zhou,
T., Yang, Y., Xia, Y., Zhang, W., Wang, P., Zhuang, Y.,
Yang, F., Li, C., and X. Wang, "Fully Adaptive Routing
Ethernet in Scale-Up Networks", Work in Progress,
Internet-Draft, draft-xu-rtgwg-fare-in-sun-02, 26 February
2026, <https://datatracker.ietf.org/doc/html/draft-xu-
rtgwg-fare-in-sun-02>.
[RFC7306] Shah, H., Marti, F., Noureddine, W., Eiriksson, A., and R.
Sharp, "Remote Direct Memory Access (RDMA) Protocol
Extensions", RFC 7306, DOI 10.17487/RFC7306, June 2014,
<https://www.rfc-editor.org/info/rfc7306>.
Authors' Addresses
Xiaohu Xu
China Mobile
Email: xuxiaohu_ietf@hotmail.com
Xu, et al. Expires 12 December 2026 [Page 8]
Internet-Draft FARE in SUN June 2026
Zongying He
Broadcom
Email: zongying.he@broadcom.com
Nan Wang
Intel
Email: nan.wang@intel.com
Nan Wang
Hygon
Email: wangn@hygon.cn
Wei Wan
Sugon
Email: wanwei@sugon.com
Hua Wang
Moore Threads
Email: wh@mthreads.com
Jian Guo
Biren Technology
Email: jguo@birentech.com
Xiang Li
Enflame Technology
Email: xiang.li@enflame-tech.com
Tianyou Zhou
Resnics Technology
Email: tzhou@resnics.com
Yongtao Yang
Centec
Email: yangyt@centec.com
Yinben Xia
Tencent
Email: forestxia@tencent.com
Xu, et al. Expires 12 December 2026 [Page 9]
Internet-Draft FARE in SUN June 2026
Weifeng Zhang
Tencent
Email: wikkizhang@tencent.com
Peilong Wang
Baidu
Email: wangpeilong01@baidu.com
Yan Zhuang
Huawei Technologies
Email: zhuangyan.zhuang@huawei.com
Fajie Yang
Cloudnine Information Technologies
Email: yangfajie@cloudnineinfo.com
Chao Li
Metanet Networking Technology
Email: lichao22@ieisystem.com
Wang Xiaojun
Ruijie Networks
Email: wxj@ruijie.com.cn
Roman Glebov
Yandex
Email: kitaro630@yandex.ru
Xu, et al. Expires 12 December 2026 [Page 10]