Use Case and Requirements for Flow Control and Congestion Control Coordination in WAN-based AI DC Interconnect
draft-dong-fann-fc-cc-coordinate-00

FANN Working Group                                               J. Dong
Internet-Draft                                       Huawei Technologies
Intended status: Informational                           27 October 2026
Expires: 30 April 2027

   Use Case and Requirements for Flow Control and Congestion Control
              Coordination in WAN-based AI DC Interconnect
                  draft-dong-fann-fc-cc-coordinate-00

Abstract

   AI services are moving from single-site data center (DC) toward a
   multi-site distributed architecture.  This requires to provide high-
   throughput, lossless, and deterministic network connectivity for AI
   workloads that span Data Center Networks (DCNs) over the Wide Area
   Network (WAN).  Due to the long distance introduced by WAN, end-to-
   end congestion control across the DCN and WAN networks become a
   challenge, as the reaction time increases from microseonds to
   millisend level, which correspondingly increases the risk of packet
   loss both at the receiver side and in the network.  This document
   describes the use case of coordinated flow control and congestion
   control to minimize the packet loss in WAN-based AI DCI networks.
   The requirements on Fast Network Notifications (FANN) mechanisms in
   this scenario is also described.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 30 April 2027.

Copyright Notice

   Copyright (c) 2026 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

Dong                      Expires 30 April 2027                 [Page 1]
Internet-Draft             DCI CC Coordination              October 2026

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   3
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Use Cases of Cordinated FC and CC . . . . . . . . . . . . . .   4
     3.1.  Congestion in WAN . . . . . . . . . . . . . . . . . . . .   4
     3.2.  Congestion in Egress DC . . . . . . . . . . . . . . . . .   7
   4.  Requirements on Fast Network Notifications  . . . . . . . . .  10
   5.  Security Considerations . . . . . . . . . . . . . . . . . . .  11
   6.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  11
   7.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  11
   8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  11
     8.1.  Normative References  . . . . . . . . . . . . . . . . . .  11
     8.2.  Informative References  . . . . . . . . . . . . . . . . .  11
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  12

1.  Introduction

   Driven by the need for larger model scales, higher compute density,
   and improved resource utilization, AI services are moving from
   single-site data center (DC) toward a multi-site distributed
   architecture.  This requires to provide high-throughput, lossless,
   and deterministic network connectivity for AI workloads that span
   Data Center Networks (DCNs) over the Wide Area Network (WAN).  Any
   packet loss or tail latency directly translates into stalled training
   pipelines, degraded job completion times (JCT), and wasted GPU
   cycles.

   A technical challenge arises from the fundamental difference between
   DCN and DCI scenarios is in congestion control and flow control.
   Within a single data center, the short physical distance keeps the
   Round-Trip Time (RTT) low, typically in the order of tens of
   microseconds, allowing the control loop to react quickly and maintain
   a stable, lossless fabric through mechanisms such as Explicit
   Congestion Notification (ECN), Congestion Notification Packet (CNP)
   and Priority Flow Control (PFC).  In the distributed DCI scenario,
   the physical distance between DCs stretches the RTT to milliseconds
   or even tens of milliseconds.  This significantly slows down the

Dong                      Expires 30 April 2027                 [Page 2]
Internet-Draft             DCI CC Coordination              October 2026

   control loop, making it difficult to promptly detect and react to
   congestion in the network.  As a result, queue buildup and packet
   loss become far more likely, and the tight lossless guarantee
   achievable in DCN cannot be easily extended to the WAN domain.  This
   motivates the need for coordinated congestion control and flow
   control across the WAN and the data center.
   [I-D.ietf-fann-problem-statement] describes the problem and
   limitations of existing approaches and documents the need for a fast
   notification mechanism.  [I-D.song-fann-framework] provides the
   overall architecture, the functional role and information carried in
   fast network notifications.

   Rather than treating DC and WAN as independent network domains, AI
   DCI requires the congestion control and flow control mechanisms in
   both domains to work in an coordinated manner to preserve end-to-end
   lossless, high bandwidth utilization characteristics, so as to meet
   the performance demands of distributed AI workloads.

   This document describes the use case of coordinated congestion
   control and flow control in WAN based AI DCI to minimize the packet
   loss in both DCN and WAN networks.  The requirements on a fast
   network notification mechanism in this scenraio is also described.

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

2.  Terminology

   The following terminologies are used in this document:

   *  PFC: Priority-based Flow Control

   *  ECN: Explicit Congestion Notification

   *  CNP: Congestion Notification Packet

   *  RoCEv2: RDMA over Converged Ethernet Version 2

Dong                      Expires 30 April 2027                 [Page 3]
Internet-Draft             DCI CC Coordination              October 2026

3.  Use Cases of Cordinated FC and CC

   Figure 1 shows a typical topology of distributed DC interconnect, in
   which two or more DCs are interconnected via the WAN network.  The
   WAN network may be used only for AI DCI traffic, or it may also be
   used to carry other AI or non-AI services.  Congestion may happen
   either within the DCs or in the WAN.  The following of this section
   describes different congestion scenarios, and the coordination
   mechanisms in flow control and congestion control.

       +-------------------------+       +------------------------+
       |   Data Center A         |       |   Data Center B        |
       +----------+--------------+       +----------+-------------+
                  |                                 |
            ------|---------------------------------|-------
           |      |     +---+                       |       |
           |      |     | R2|-----------------\     |       |
           |      |    /+---+\                 \    |       |
           |      |   /       \                 \   |       |
           |    +-+-+/         +---+             \+-+-+     |
           |    | R1 +---------+ R3+--------------+ R5|     |
           |    +---+\         +---+             /+---+     |
           |          \       /                 /           |
           |           \+---+/                 /            |
           |            | R4|-----------------/             |
           |            +-|-+  DC Inter-connect Network     |
            --------------|---------------------------------
                          |
                     +----+--------------------+
                     |      Data Center C      |
                     +-------------------------+

                Figure 1. WAN-based AI DC Interconnect

3.1.  Congestion in WAN

   Assume AI traffic is sent from DC-A (the ingress DC) to DC-B (the
   egress DC) taking the path R1-R3-R5 in WAN network.  If a traffic
   burst or surge causes congestion on link R3-R5 (as shown in
   Figure 1), node R3 has the following options for handling congestion:

   Option 1: Receiver-Based Congestion Notification (Standard ECN)

Dong                      Expires 30 April 2027                 [Page 4]
Internet-Draft             DCI CC Coordination              October 2026

   R3 marks the ECN [RFC3168] field in the data packet as "Congestion
   Experienced (CE)" and forwards it to the receiver in the egress DC
   (DC-B).  Upon detecting the CE mark, the receiver in DC-B generates a
   congestion notification message and sends it back to the traffic
   source in the ingress DC (DC-A), instructing the source to reduce its
   sending rate for the affected flows.

   This is the standard congestion notification mechanism used in DCN.
   However, in a WAN-based DC Interconnect scenario, the control loop
   spans the full WAN RTT, which is typically in the millisecond range.
   This latency may prevent timely rate reduction, leading R3 to drop
   packets due to buffer overflow.

                          AI Traffic
          ------------------------------------------>
     +----------+               Congestion     +----------+
     |   Data   |   +----+   +----+   +----+   |   Data   |
     | center A |---| R1 |---| R3 |-=-| R5 |---| center B |
     +----------+   +----+   +----+   +----+   +----------+
                                 Packet with ECN Set
                                |-------------------->
                 Congestion Notification Message
          <-----------------------------------------|

           Figure 2. Receiver-Based Notification

   Option 2: Direct to Sender Congestion Notification

   R3 directly sends the congestion information to the traffic source in
   the ingress DC (DC-A).

   This approach is considered a "fast" congestion notification because
   it reduces the feedback path to approximately half-RTT.  Upon
   receiving the notification, the sender can immediately reduce its
   sending rate for the identified flow.

   Although the feedback loop is reduced from one-RTT (Option 1) to
   half-RTT, the absolute latency in a WAN environment remains in the
   millisecond range.  Consequently, the sending rate reduction may
   still not take effect quickly enough at the congestion node R3, and
   packet drops due to buffer overflow may still occur.

   Sending direct congestion notifications from WAN nodes presents two
   main challenges:

   *  Packet Parsing: WAN nodes must parse data packets deeply to
      extract flow identification information.

Dong                      Expires 30 April 2027                 [Page 5]
Internet-Draft             DCI CC Coordination              October 2026

   *  State Maintenance: For RoCEv2 flows, WAN nodes must obtain and
      maintain the mapping of source and destination Queue Pairs (QPs)
      in advance.

                          AI Traffic
          ------------------------------------------>
     +----------+               Congestion     +----------+
     |   Data   |   +----+   +----+   +----+   |   Data   |
     | center A |---| R1 |---| R3 |-=-| R5 |---| center B |
     +----------+   +----+   +----+   +----+   +----------+
          <---------------------|
      Congestion Notification Message

        Figure 3. Direct to Sender Notification

   Option 3: Coordinated Flow Control and Congestion Control
   Notification

   Instead of sending the congestion information directly to the
   receiver in the egress DC or the source in the egress DC, R3 sends a
   WAN flow control notification to its upstream node R1 on the data
   path, notifying R1 to pause the specific flow which caused
   congestion.  If R1 detects local congestion due to the pause of the
   flow, it can send a flow control notification to its upstream node on
   the data path.  This hop-by-hop notification mechanism continues
   until the flow control signal reaches the gateway of the ingress DC
   (DC-A).

   The DC gateway pauses the flow according to the flow control
   notification.  If this causes local congestion on the DC gateway, it
   may generate a congestion notification to the traffic source.  For
   RoCEv2 flows, this requires the DC gateway to maintain the source and
   destination Queue Pair (QP) information.

   The advantage of this approach is, the control loop latency is
   comparable to that in Option 2, while it leverages hop-by-hop flow
   control within the WAN network.  By pausing the traffic at each
   upstream node, the buffer resources on WAN nodes along the data path
   can absorb burst traffic before the source reduces its sending rate.
   This mechanism can effectively avoid or alleviate packet loss in the
   WAN network, even if the rate adjustment at the sender is delayed due
   to the long distance.

Dong                      Expires 30 April 2027                 [Page 6]
Internet-Draft             DCI CC Coordination              October 2026

                          AI Traffic
          ------------------------------------------>
     +----------+               Congestion     +----------+
     |   Data   |   +----+   +----+   +----+   |   Data   |
     | center A |---| R1 |---| R3 |-=-| R5 |---| center B |
     +----------+   +----+   +----+   +----+   +----------+
          <-----|<-----|<-------|
                   WAN     WAN
       Congestion  Flow    Flow
     Notification Control Control

   Figure 4. Coordinated Flow Control and Congestion Notification

3.2.  Congestion in Egress DC

   Assume traffic is sent from DC-A (the ingress DC) to DC-B (the egress
   DC) through the WAN network, taking the path R1-R3-R5.  If a traffic
   burst or surge causes congestion within the egress DC (e.g., on link
   S1-L2 in Figure 5), the node detecting the congestion (node S1) has
   the following options for handling it:

   Option 1: Receiver-Based Congestion Notification (Standard ECN)

   S1 marks the ECN field in the data packet as "Congestion Experienced"
   and forwards it to the receiver in the egress DC (DC-B).  Upon
   detecting the ECN mark, the receiver generates a congestion
   notification message and sends it back to the traffic source in DC-A,
   instructing the source to reduce its sending rate for the affected
   flows.

   This is the standard congestion notification mechanism used in DCN.
   However, in a DC interconnect scenario, the control loop spans the
   full WAN RTT, which is typically in the millisecond range.  This
   latency may prevent timely rate reduction.  Consequently, Priority-
   based Flow Control (PFC) may be used within DC-B to prevent S1 from
   dropping packets while waiting for the sending rate adjustment.  If
   the burst is too large, S1 may still drop packets due to buffer
   overflow.

Dong                      Expires 30 April 2027                 [Page 7]
Internet-Draft             DCI CC Coordination              October 2026

                          AI Traffic
          ------------------------------------------>
                                                   DC-B
     +----------+                              +------------+
     |          |   +----+   +----+   +----+   |   S1  S2   |
     |   DC-A   |---| R1 |---| R3 |---| R5 |---|  / \ /  \  |
     |          |   +----+   +----+   +----+   | L1  L2  L3 |
     +----------+                              +------------+
                                                 Congestion

                                             Packet with ECN Set
                                                     --->
                 Congestion Notification Message
          <---------------------------------------------|

           Figure 5. Receiver-Based Notification

   Option 2: Direct to Sender Notification

   S1 directly sends the congestion information to the traffic source in
   the ingress DC (DC-A).

   This approach is considered a "fast" congestion notification as it
   eliminates the delay of sending the notification to the receiver
   first.  However, since the transmission time in the WAN is much
   larger than in the DC, the time for the source to receive the
   notification and for the rate reduction to take effect at the
   congestion node S1 is still at the WAN RTT (milliseconds) level.
   Thus, packet drops may still occur at S1 due to buffer overflow.
   Consequently, PFC may be used in DC-B to prevent S1 from dropping
   packets during this latency period.

                          AI Traffic
          ------------------------------------------>
                                                   DC-B
     +----------+                              +------------+
     |          |   +----+   +----+   +----+   |   S1  S2   |
     |   DC-A   |---| R1 |---| R3 |---| R5 |---|  /  \/  \  |
     |          |   +----+   +----+   +----+   | L1  L2  L3 |
     +----------+                              +------------+
                                                  Congestion
                  Congestion Notification Message
         <------------------------------------------|

           Figure 6. Direct to Sender Notification

   Option 3: Coordinated Flow Control and Congestion Control
   Notification

Dong                      Expires 30 April 2027                 [Page 8]
Internet-Draft             DCI CC Coordination              October 2026

   This option introduces a coordinated flow control and congestion
   control mechanism across the DC and WAN networks.

   When congestion is detected at node S1 within DC-B, it initiates a
   PFC pause frame towards the DC-B Gateway.  This step is critical for
   immediate packet loss prevention within the DC fabric.  The DC-B
   Gateway receives the PFC pause frame and ensures that no packets are
   dropped on the S1-L2 link while the congestion signal is being
   processed and propagated.

   At the same time, the DC-B Gateway convert the congestion status into
   an fine-grained flow control signal and injects it into the WAN path
   at the first hop, R5.  Unlike PFC, which is confined to Layer 2, the
   fine-grained flow control message can traverse the Layer-3
   boundaries.  Similar to the case described in section 3.1, option 3,
   the fine-grained flow control propagates hop-by-hop along the path,
   from R5 to R3, then to R1, and finally reaches the DC-A Gateway.
   Each WAN node receiving the flow control signal pauses the
   corresponding flow on its outgoing interface, which effectively
   throttles the traffic upstream before it reaches the congested point.

   Upon receiving the fine-grained flow control signal, the DC-A Gateway
   pauses the flow at the source edge.  If this causes local congestion
   on the DC-A side, the gateway may trigger congestion notification
   message to the traffic source in DC-A for further rate reduction.
   For RoCEv2 flows, this requires the DC gateway to maintain source and
   destination Queue Pair (QP) mappings to ensure precise flow
   identification.

   The advantage of this coordinated approach is that, during the
   propagation of the congestion information to the source in DC-A, DC-B
   can be protected from packet loss, and the WAN links are protected
   from consequent congestion and buffer overflow, as reaction to the
   congestion can be performed at the intermediate WAN and DC nodes
   respectively.

Dong                      Expires 30 April 2027                 [Page 9]
Internet-Draft             DCI CC Coordination              October 2026

                           AI Traffic
          ------------------------------------------>
                                                   DC-B
     +----------+                              +------------+
     |          |   +----+   +----+   +----+   |   S1  S2   |
     |   DC-A   |---| R1 |---| R3 |---| R5 |---|  /  \/  \  |
     |          |   +----+   +----+   +----+   | L1  L2  L3 |
     +----------+                              +------------+
          <-----|<-----|<-------|<----- |<------ |<-----
                   WAN     WAN     WAN     WAN       DC
      Congestion   Flow    Flow    Flow    Flow     Flow
     Notification Control Control Control Control  Control

   Figure 7. Coordinated Flow Control and Congestion Notification

4.  Requirements on Fast Network Notifications

   Section 3 demonstrates that in WAN-based DC interconnect scenarios,
   the coordination of flow control and congestion control between the
   WAN and the DC provides an effective approach to avoid packet loss
   due to congestion happened in either the WAN or the DCN.  This
   coordination introduces new requirements for fast network
   notifications for flow control in WAN.  The specific requirements are
   summarized as follows:

   *  Fine Granularity: Flow control notifications in the WAN MUST NOT
      stop all traffic on a specific link or port.  The pause signal
      SHOULD operate at a fine granularity, targeting specific flows,
      groups of flows, or traffic belonging to a specific tenant, etc.

   *  Directed Delivery: Flow control notifications in the WAN MUST NOT
      be sent via broadcast or multicast.  They SHOULD be delivered hop-
      by-hop along the reverse direction of the data path to upstream
      nodes, and MAY ultimately reach the ingress node of the specific
      traffic flows within the WAN.

   *  Transport: Since flow control notifications need to be consumed by
      WAN network nodes in the data plane, it is RECOMMENDED that the
      notification message be carried within IP packets.

   *  Coordination: Flow control notifications in the WAN MAY be
      generated either upon detecting congestion in the WAN network, or
      by converting flow control notifications received from downstream
      DCNs.  Additionally, a DC gateway MAY use these notifications as a
      trigger to send a congestion notification to the source of the
      traffic flow causing the congestion.

Dong                      Expires 30 April 2027                [Page 10]
Internet-Draft             DCI CC Coordination              October 2026

5.  Security Considerations

   The security considerations in [I-D.song-fann-framework] applies to
   this document.  The Flow control notifications generated in WAN MUST
   be kept in the WAN network domain and MUST NOT be sent directly
   across the network boundary with DC networks.  The fast flow control
   notification MAY contain nodes and path information of the WAN
   network, which MUST NOT be exposed to entities outside the WAN
   domain.

6.  IANA Considerations

   This document does not require any IANA actions.

7.  Acknowledgements

   The authors would like to thank Xuesong Geng and Xingjian He for
   their review and comments.

8.  References

8.1.  Normative References

   [I-D.song-fann-framework]
              Song, H., Dong, J., Ali, Z., Cheng, W., Zhu, Y., Pang, R.,
              Contreras, L. M., and L. Guo, "A Framework for Fast
              Network Notifications", Work in Progress, Internet-Draft,
              draft-song-fann-framework-03, 26 October 2026,
              <https://datatracker.ietf.org/doc/html/draft-song-fann-
              framework-03>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC3168]  Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
              of Explicit Congestion Notification (ECN) to IP",
              RFC 3168, DOI 10.17487/RFC3168, September 2001,
              <https://www.rfc-editor.org/info/rfc3168>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

8.2.  Informative References

Dong                      Expires 30 April 2027                [Page 11]
Internet-Draft             DCI CC Coordination              October 2026

   [I-D.ietf-fann-problem-statement]
              Dong, J., McBride, M., Clad, F., Zhang, Z. J., Zhu, Y.,
              Xu, X., Zhuang, R., Pang, R., Lu, H., Liu, Y., Contreras,
              L. M., Mehmet, D., and R. Rahman, "Fast Network
              Notifications Problem Statement", Work in Progress,
              Internet-Draft, draft-ietf-fann-problem-statement-01, 26
              October 2026, <https://datatracker.ietf.org/doc/html/
              draft-ietf-fann-problem-statement-01>.

Author's Address

   Jie Dong
   Huawei Technologies
   Huawei Campus, No.156 Beiqing Road
   Beijing, 100095
   China
   Email: jie.dong@huawei.com

Dong                      Expires 30 April 2027                [Page 12]