Convergence of Congestion Control from Retained State
draft-ietf-tsvwg-careful-resume-10
| Document | Type | Active Internet-Draft (tsvwg WG) | |
|---|---|---|---|
| Authors | Nicolas Kuhn , Emile Stephan , Gorry Fairhurst , Raffaello Secchi , Christian Huitema | ||
| Last updated | 2024-07-08 | ||
| Replaces | draft-kuhn-tsvwg-careful-resume | ||
| RFC stream | Internet Engineering Task Force (IETF) | ||
| Intended RFC status | Proposed Standard | ||
| Formats | |||
| Additional resources |
GitHub Repository
Mailing list discussion |
||
| Stream | WG state | WG Document | |
| Associated WG milestone |
|
||
| Document shepherd | Marten Seemann | ||
| IESG | IESG state | I-D Exists | |
| Consensus boilerplate | Yes | ||
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | martenseemann@gmail.com |
draft-ietf-tsvwg-careful-resume-10
Internet Engineering Task Force N. Kuhn
Internet-Draft Thales Alenia Space
Intended status: Standards Track E. Stephan
Expires: 9 January 2025 Orange
G. Fairhurst
R. Secchi
University of Aberdeen
C. Huitema
Private Octopus Inc.
8 July 2024
Convergence of Congestion Control from Retained State
draft-ietf-tsvwg-careful-resume-10
Abstract
This document specifies a cautious method for IETF transports that
enables fast startup of congestion control for a wide range of
connections. It reuses a set of computed congestion control
parameters that are based on previously observed path characteristics
between the same pair of transport endpoints. These parameters are
saved, allowing them to be later used to modify the congestion
control behavior of a subsequent connection.
It describes assumptions and defines requirements for how a sender
utilizes these parameters to provide opportunities for a connection
to more rapidly get up to speed and rapidly utilize available
capacity. It discusses how use of Careful Resume impacts the
capacity at a shared network bottleneck and the safe response that is
needed after any indication that the new rate is inappropriate.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 9 January 2025.
Kuhn, et al. Expires 9 January 2025 [Page 1]
Internet-Draft Congestion Control Convergence July 2024
Copyright Notice
Copyright (c) 2024 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Use of saved CC parameters by a Sender . . . . . . . . . 4
1.2. Receiver Preference . . . . . . . . . . . . . . . . . . . 5
1.3. Transport Protocol Interaction . . . . . . . . . . . . . 5
1.4. Examples of Scenarios of Interest . . . . . . . . . . . . 5
2. Language, Notation and Terms . . . . . . . . . . . . . . . . 6
2.1. Requirements Language . . . . . . . . . . . . . . . . . . 6
2.2. Notation and Terms . . . . . . . . . . . . . . . . . . . 6
3. The Phases of CC using Careful Resume . . . . . . . . . . . . 7
3.1. Observing . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2. Reconnaissance Phase . . . . . . . . . . . . . . . . . . 8
3.3. Unvalidated Phase . . . . . . . . . . . . . . . . . . . . 10
3.4. Validating Phase . . . . . . . . . . . . . . . . . . . . 11
3.5. Safe Retreat Phase . . . . . . . . . . . . . . . . . . . 12
3.5.1. Loss Recovery after entering Safe Retreat . . . . . . 13
3.6. RTO Expiry while using Careful Resume . . . . . . . . . . 14
3.7. Normal Phase . . . . . . . . . . . . . . . . . . . . . . 14
4. The Endpoint Token . . . . . . . . . . . . . . . . . . . . . 14
4.1. Creating an Endpoint Token . . . . . . . . . . . . . . . 14
5. Implementation Notes and Guidelines . . . . . . . . . . . . . 15
5.1. Observing the Path Capacity . . . . . . . . . . . . . . . 15
5.2. Confirming the Path in the Reconnaissance Phase . . . . . 16
5.2.1. Confirming the Path . . . . . . . . . . . . . . . . . 16
5.3. Safety for the Unvalidated Phase . . . . . . . . . . . . 17
5.3.1. Lifetime of CC Parameters . . . . . . . . . . . . . . 17
5.3.2. Pacing in the Unvalidated Phase . . . . . . . . . . . 18
5.3.3. Exit from the Unvalidated Phase because of Variable
Network Conditions . . . . . . . . . . . . . . . . . 18
5.4. The Validating Phase . . . . . . . . . . . . . . . . . . 19
5.5. Safety in the Safe Retreat Phase . . . . . . . . . . . . 19
5.6. Returning to Normal Congestion Control . . . . . . . . . 20
5.7. Limitations from Transport Protocols . . . . . . . . . . 20
Kuhn, et al. Expires 9 January 2025 [Page 2]
Internet-Draft Congestion Control Convergence July 2024
6. QLOG support for QUIC . . . . . . . . . . . . . . . . . . . . 21
6.1. cr_phase Event . . . . . . . . . . . . . . . . . . . . . 21
7. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 23
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 23
9. Security Considerations . . . . . . . . . . . . . . . . . . . 23
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 23
10.1. Normative References . . . . . . . . . . . . . . . . . . 23
10.2. Informative References . . . . . . . . . . . . . . . . . 24
Appendix A. Notes on the Careful Resume Phases . . . . . . . . . 26
A.1. Example with No Loss . . . . . . . . . . . . . . . . . . 28
A.2. Example with No Loss, Rate-Limited . . . . . . . . . . . 29
A.3. Example with Loss detected in the Reconnaissance Phase . 29
A.4. Example with Loss detected in the Validating Phase . . . 30
Appendix B. Internet Draft Revision details . . . . . . . . . . 31
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 32
1. Introduction
All Internet transports are required to either use a Congestion
Control (CC) algorithm, or to constrain their rate of transmission
[RFC8085]. In 2010, a survey of alternative CC algorithms [RFC5783],
noted that there are challenges when a CC algorithm operates across
an Internet path with a high and/or varying Bandwidth-Delay Product
(BDP). This mechanism targets a solution for these challenges.
A CC algorithm typically takes time to ramp-up the sending rate,
called the "Slow-Start phase", informally known as the time to "Get
up to speed". This defines a time in which a sender intentionally
uses less capacity than might be available, with the intention to
avoid or limit overshoot of the available capacity for the path.
This can increase queuing (latency or jitter) and/or congestion
packet loss for the flow. Any overshoot can have a detrimental
effect on other flows sharing a common bottleneck. A sender can use
a method that observes the rate of acknowledged data, and seek to
avoid an overshoot of the bottleneck capacity (e.g., Hystart++
[RFC9406]). In the extreme case, an overshoot can result in
persistent congestion with unwanted starvation of other flows
[RFC8867] (i.e., preventing other flows from successfully sharing the
capacity at a common bottleneck).
Kuhn, et al. Expires 9 January 2025 [Page 3]
Internet-Draft Congestion Control Convergence July 2024
The present document specifies a CC mechanism, called Careful Resume,
which is expected to reduce the time to complete a transfer when the
transfer sends significantly more data than allowed by the Initial
congestion Window (IW), and where the BDP of the path is also
significantly more than the IW. It introduces an alternative
mechanism to select initial CC parameters, that seek to more rapidly
and safely grow the sending rate controlled by the congestion window
(CWND). CC algorithms that are rate-based can make similar
adjustments to their target sending rate.
Careful Resume is based on temporal sharing (sometimes known as
caching) of a saved set of CC parameters that relate to previous
observations of the same path. The parameters include: the
saved_cwnd for the path and the minimum Round Trip Time (RTT). These
parameters are saved and used to modify the CC behavior of a
subsequent connection between the same endpoints. Some congestion
control algorithms may use other parameters. For example,
implementations using BBR also retain the value of the bottleneck
bandwidth required to reach the capacity available to the flow
(BBR.max_bw, see [I-D.cardwell-iccrg-bbr-congestion-control]).
When used with the QUIC transport, this provides transport services
that resemble those that could be implemented in TCP, using methods
such as TCP Control Block (TCB) [RFC9040] caching.
1.1. Use of saved CC parameters by a Sender
CC parameters are used by Careful Resume for three functions:
1. Information to confirm whether a saved path corresponds to the
current path.
2. Information about the utilised path capacity to set CC
parameters.
3. Information to check the CC parameters are not too old.
"Generally, implementations are advised to be cautious when using
saved CC parameters on a new path", as stated in [RFC9000]. While
this statement has been proposed in the context of QUIC
standardization, this advice is appropriate for any IETF transport
protocol. Care is therefore needed to assure safe use and to be
robust to changes in traffic patterns, network routing, and link/node
conditions. There are cases where using the saved parameters of a
previous connection is not appropriate (see Section 3.2).
Kuhn, et al. Expires 9 January 2025 [Page 4]
Internet-Draft Congestion Control Convergence July 2024
1.2. Receiver Preference
Whilst a sender could take optimization decisions without considering
the receiver's preference, there are cases where a receiver could
have information that is not available at the sender, or might
benefit from understanding that Careful Resume might be used. In
these cases, a receiver could explicitly ask to enable or inhibit
Careful Resume when an application initiates a new connection.
Examples where a receiver might request to inhibit use Careful Resume
include:
1. a receiver that can predict the pattern of traffic (e.g., insight
into the volume of data to be sent, the expected length of a
connection, or the requested maximum transfer rate);
2. a receiver with a local indication that a path/local interface
has changed since the CC parameters were saved;
3. knowledge of the current hardware limitations at a receiver;
4. a receiver that can predict additional capacity will be needed
for other concurrent or later flows (i.e., prefers to activate
Careful Resume for a different connection).
A related document proposes an extension for QUIC that allows sender-
generated CC parameters to be stored at the receiver
[I-D.kuhn-quic-bdpframe-extension]. This avoids the need for a
sender to retain transport state for each receiver. It also allows
the receiver to express a preference for whether a sender ought use
Careful Resume.
1.3. Transport Protocol Interaction
The CWND is one factor that limits the sending rate of a transport
protocol. Other mechanisms also constrain the maxmimum sending rate.
These include the sender pacing rate and the receiver-advertised
window (or flow credit), see Section 5.7.
1.4. Examples of Scenarios of Interest
This section provides a set of examples where Careful Resume is
expected to improve performance. Either endpoint can assume the role
of a sender or a receiver. Careful Resume also supports a
bidirectional data transfer, where both endpoints simultaneously send
data (e.g., remote execution of an application, or a bidirectional
video conference call).
Kuhn, et al. Expires 9 January 2025 [Page 5]
Internet-Draft Congestion Control Convergence July 2024
In one example, an application uses a series of connections over a
path. Without a new method, each connection would need to
individually discover appropriate CC parameters, whereas Careful
Resume allows the flow to use a rate based on the previously observed
CC parameters.
In another example, an application connects after a disruption had
temporarily reduced the path capacity. When the endpoint returns to
use the path using Careful Resume, the sending rate can be based on
the previously observed CC parameters.
There is particular benefit for any path with an RTT that is much
larger than typical Internet paths. In a specific example, an
application connected via a satellite access network [IJSCN] could
take 9 seconds to complete a 5.3 MB transfer using standard CC,
whereas a sender using Careful Resume could be reduce this transfer
time to 4 seconds. The time to complete a 1 MB transfer could
similarly be reduced by 62 % [MAPRG111]. This benefit is also
expected for other sizes of transfer and for different path
characteristics when a path has a large BDP.
2. Language, Notation and Terms
This subsection provides a brief summary of key terms and the
requirements language.
2.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
2.2. Notation and Terms
The document uses language drawn from a range of IETF RFCs. The
following terms are defined:
Careful Resume (CR): The method specified in this document to
select initial CC parameters and to more rapidly and safely
increase the initial sending rate.
CC parameters: A set of saved congestion control parameters from a
observing the capacity of an established connection (see
Section 1.1).
Kuhn, et al. Expires 9 January 2025 [Page 6]
Internet-Draft Congestion Control Convergence July 2024
CWND: The congestion window, or equivalent CC variable limiting
the maximum sending rate;
current_endpoint_token: The Endpoint Token of the current
receiver;
current_rtt: A sample measurement of the current RTT;
Endpoint Token: A token identifying a path to a receiver
Section 4;
flight_size: The current volume of unacknowledged data;
jump_cwnd: The resumed CWND, used in the Unvalidated Phase.
LifeTime: The time for which the saved CC parameters can be safely
re-used.
max_jump: The configured maximum jump_cwnd;
PipeSize: A measure of the validated available capacity based on
the acknowledged data;
saved_cwnd: The preserved capacity derived from observation of a
previous connection (see Section 5.1);
saved_endpoint_token: The Endpoint Token associated with a set of
CC parameters;
saved_rtt: The preserved minimum RTT (see Section 5.1).
Unvalidated Packet: A packet sent when the CWND has been increased
beyond the size normally permitted by the congestion control
algorithm; if such a packet is acknowledged, it contributes to the
PipeSize, but if congestion is detected, it triggers entry to the
Safe Retreat Phase.
3. The Phases of CC using Careful Resume
This section defines a series of phases that the congestion
controller moves through as a connection uses Careful Resume.
Kuhn, et al. Expires 9 January 2025 [Page 7]
Internet-Draft Congestion Control Convergence July 2024
Observing ...> Connect -> Reconnaissance --------------------> Normal
(Normal) | ^
v |
Unvalidated --------------------------+
| | |
| +--> Validating --------------+
| | |
| | |
+---------------+--> Safe Retreat ---+
Figure 1: Key transitions between Phases in Careful Resume
An established connection in the Normal Phase is permitted to start
observing CC parameters. The key phases of Careful are illustrated
in Figure 1. Examples of the transitions between phases are provided
in Appendix A.
3.1. Observing
An established connection in the Normal Phase, can save a set of CC
parameters for the specific path to the current endpoint. Each set
of CC parameters includes the saved_endpoint_token and the LifeTime
(e.g., as a timestamp after which the parameters must not be used).
* Observing (saved_cwnd): The saved_cwnd is a measure of the
currently utilised capacity for the connection, measured as the
volume of bytes sent during an RTT. This could be computed by
measuring the volume of data acknowledged in one RTT. If the
measured CWND is less than four times the Initial Window (IW) a
sender can choose to not save the CC parameters, because the
additional actions associated with performing Careful Resume for a
small CWND would not justify its use.
* Observing (saved_rtt): The minimum RTT is saved as the saved_RTT.
Implementation notes are provided in Section 5.1.
3.2. Reconnaissance Phase
A sender enters the Reconnaissance Phase after connection setup. In
this phase, the CWND is initialised to the IW, and the sender
transmits initial data. The CWND MAY be increased using normal CC as
each ACK confirms delivery of previously unacknowledged data (i.e.,
the CC is unchanged).
Kuhn, et al. Expires 9 January 2025 [Page 8]
Internet-Draft Congestion Control Convergence July 2024
The phase seeks to determine if the path is consistent with a
previously observed path (saved as a set of CC parameters). The
following conditions need to be confirmed before the sender enters
the Reconnaissance Phase:
* Reconnaissance Phase (Endpoint change): If the
current_endpoint_token is not the same as one of the
saved_endpoint_tokens, the sender MUST enter the Normal Phase. (A
difference in the Endpoint Token indicates a the network path was
different to one that was observed.)
* Reconnaissance Phase (Lifetime of saved CC parameters): The CC
parameters are temporal. If the LifeTime of the observed CC
parameters is exceeded, the CC parameters are not used and the
sender enters the Normal Phase.
The following actions are performed during the Reconnaissance Phase:
* Reconnaissance Phase (Confirming the RTT): During this phase, a
sender MUST record the minimum RTT for the current connection as
the current_rtt.
* Reconnaissance Phase (Detected congestion): If the sender detects
congestion (e.g., packet loss or ECN-CE marking), the sender MUST
enter the Normal Phase to respond to the detected congestion.
* Reconnaissance Phase (Using saved_cwnd): Only one connection can
use a specific set of saved CC parameters. If another connection
has already started to use the saved_cwnd, the sender MUST enter
the Normal Phase.
* Reconnaissance Phase (Path confirmed): When a sender has confirmed
the RTT and also has received an acknowledgement for the initial
data without reported congestion, it MAY then enter the
Unvalidated Phase. This transition occurs when a sender has more
data than permitted by the current CWND.
If a sender is rate-limited [RFC7661], it might send insufficient
data to be able to validate transmission at the higher rate. A
sender is allowed to remain in the Reconnaissance Phase and to not
transition to the Unvalidated Phase until there is more data in the
transmission buffer than can be sent using the current CWND. In some
implementations, the decision to enter the Unvalidated Phase could
need coordination with the management of buffers in the interface to
the upper layers.
When a path is not confirmed, Careful Resume is not used and the
sender enters the Normal Phase.
Kuhn, et al. Expires 9 January 2025 [Page 9]
Internet-Draft Congestion Control Convergence July 2024
Implementation notes are provided in Section 5.2.
3.3. Unvalidated Phase
The Unvalidated Phase is designed to enable the CWND to more rapidly
get up to speed by using paced transmission of a tenatively increased
CWND. The following conditions need to be confirmed before the
sender enters the Unvalidated Phase:
* Unvalidated Phase (Confirming the path on entry): If the
current_rtt is greater than or equal to (saved_rtt / 2) or the
current_rtt is less than or equal to (saved_rtt x 10) (see
Section 5.2.1), the sender MUST enter the Normal Phase (see
trigger rtt_not_validated in Section 6). The calculation of a
sending rate from a saved_cwnd is directly impacted by the RTT,
therefore a significant change in the RTT is a strong indication
that the previously observed CC parameters are not be valid for
the current path.
On entry to the Unvalidated Phase, the sender:
* Unvalidated Phase (Initialising PipeSize): The variable PipeSize
if initialised to CWND on entry to the Unvalidated Phase. This
records the CWND before the jump is applied.
* Unvalidated Phase (Setting the jump_cwnd): To avoid starving other
flows that could have either started or increased their use of
capacity after the Observation Phase, the jump_cwnd MUST be no
more than half of the saved_cwnd. Hence, jump_cwnd is less than
or equal to Min(max_jump,(saved_cwnd/2)). CWND = jump_cwnd.
The following actions are performed during the Unvalidated Phase:
* Unvalidated Phase (Pacing transmission): All packet sent in the
Unvalidated Phase MUST use based on the current_rtt.
* Unvalidated Phase (Confirming the path during transmission): If a
sender determines that the previous CC parameters are not valid
(due to a detected path change), the Safe Retreat Phase is
entered. (In the Unvalidated Phase, insufficient time has passed
for a sender to receive feedback validating the the jump in CWND.
Therefore, any detected congestion must have resulted from packets
sent before the Unvalidated Phase.)
* Unvalidated Phase (Tracking PipeSize): The variable PipeSize is
increased by the volume of data acknowledged by each received ACK.
(This indicates a previously unvalidated packet has been
succesfuly sent over the path.)
Kuhn, et al. Expires 9 January 2025 [Page 10]
Internet-Draft Congestion Control Convergence July 2024
* Unvalidated Phase (Receiving acknowledgement for an unvalidated
packet): The sender enters the Validating Phase when an
acknowledgement is received for the first packet number (or
higher) that was sent in the Unvalidated Phase (see
first_unvalidated_packet_acknowledged in Section 6).
When the flow is controlled using BBR, Careful Resume is implemented
by setting the pacing rate from the saved congestion control
parameters, with the following precautions:
* The flag "carefully-resuming" is added to the BBR state, and
initialized to "False" when the BBR flow starts;
* Careful Resume is only activated if a BBR flow is in the Startup
state;
* The probing rate is set to 1/2 of the bottleneck bandwidth in the
saved congestion control parameters.
* The sender starts the Unvalidated Phase at the beginning of a
round, and sets the "carefully-resuming" flags to "True";
* When the "carefully-resuming" flag is set, the sender sets the BBR
pacing rate to the larger of the nominal pacing rate (BBR.bw times
BBRStartupPacingGain) and the probing rate. The CWND is set to
the largest of BBR.bw and the probing rate, multiplied by
BBR.rtt_min times BBRStartupCwndGain;
* The "carefully-resuming" flag is reset to False two rounds after
it is set, i.e., after all the packets sent in the first round of
"carefully resuming" have been received and acknowledged by the
peer. At that stage (after the capacity has been validated), the
measured delivery rate is expected to reflect the probing rate.
Implementation notes are provided in Section 5.3.
3.4. Validating Phase
The Validating Phase checks that all packets sent in the Unvalidated
Phase were received without inducing congestion. The CWND remains
unvalidated and the sender typically remains in this phase for one
RTT. On entry to the Validating Phase, the sender:
* Validating Phase (Check flight_size on entry): On entry to the
Validating Phase, if the flight_size is less equal to the
PipeSize, the Normal Phase is entered with the CWND reset to the
PipeSize. (The unvalidated part of the jump_cwnd was not
utilised).
Kuhn, et al. Expires 9 January 2025 [Page 11]
Internet-Draft Congestion Control Convergence July 2024
* Validating Phase (Limiting CWND on entry): On entry to the
Validating Phase, the CWND is set to the flight_size.
During the Validating Phase, the sender performs the following
actions:
* Validating Phase (Tracking PipeSize): The PipeSize is increased by
the volume of acknowledged data for each received ACK that
indicates a packet was successfully sent over the path.
* Validating Phase (Updating CWND): The CWND is updated using the
normal rules for the current congestion controller, this typically
allows CWND to be increased for each received acknowledgement that
indicates a packet has been successfully sent across the path.
* Validating Phase (Congestion indication): If a sender determines
that congestion was experienced (e.g., packet loss or ECN-CE
marking), Careful Resume enters the Safe Retreat Phase (see
trigger packet_loss and ECN_CE in Section 6).
* Validating Phase (Receiving acknowledgement for all unvalidated
packets): The sender enters the Normal Phase when an
acknowledgement is received for the last packet number (or higher)
that was sent in the Unvalidated Phase (see
last_unvalidated_packet_acknowledged in Section 6).
When using BBR, validation is performed using the regular BBR rules
for exiting Startup. The measured delivery rate will reflect the
actual capacity of the network. If congestion was experienced and
packet losses were observed, BBR will exit the Startup state and
enter the Drain state while the "carefully-resuming" flag is still
True, which will trigger the "safe retreat" option of the Drain
state.
3.5. Safe Retreat Phase
This phase is entered when the first loss/ECN-CE marking is detected
for an unvalidated packet. It drains the path of other unvalidated
packets. (This trigger is the same as used by a QUIC sender to
transition from Slow-Start to Recovery [RFC9002].)
On entry to the Safe Retreat Phase, the sender:
* Safe Retreat Phase (Removing saved information): The set of saved
CC parameters for the path are deleted, to prevent these from
being used again by other flows.
Kuhn, et al. Expires 9 January 2025 [Page 12]
Internet-Draft Congestion Control Convergence July 2024
* Safe Retreat Phase (Re-initializing CWND): The CWND MUST be
reduced to no more than (PipeSize/2). This avoids persistent
starvation by allowing capacity for other flows to regain their
share of the total capacity. The minimum CWND in QUIC is 2
packets (see: [RFC9002] section 4.8).
* Safe Retreat Phase (QUIC recovery): When the CWND is reduced, a
QUIC sender can immediately send a single packet prior to the
reduction [RFC9002]. (This speeds up loss recovery if the data in
the lost packet is retransmitted and is similar to TCP as
described in Section 5 of [RFC6675].)
In the Safe Retreat Phase, the sender performs the following actions:
* Safe Retreat Phase (Tracking PipeSize): The sender continues to
update the PipeSize after processing each acknowledgement. (The
PipeSize is used to reset the ssthresh when leaving this phase, it
does not modify CWND.)
* Safe Retreat Phase (Maintaining CWND): The CWND MUST NOT be
increased in the Safe Retreat Phase.
* Safe Retreat Phase (Acknowledgement of all unvalidated packets):
The sender enters Normal Phase when the last packet (or a later
packet) sent during the Unvalidated Phase has been acknowledged,
and if required adjusts the ssthresh (see exit_recovery in
Section 6). The value of ssthresh on leaving the Safe Retreat
Phase MUST NOT be more than the PipeSize.
When using BBR, the Safe Retreat Phase is entered if the Drain state
is entered while the "carefully-resuming" flag is still True, i.e.,
if less than 2 full rounds have elapsed after the sender entered the
Unvalidated Phase. The delivery rates measured in these conditions
are tainted, because packets sent during the attempt are still queued
at the bottleneck and may have "pushed out" competing traffic. The
delivery rates measured in Drain state MUST be discarded if the
"carefully-resuming" flag is set to True. This flag is cleared upon
exiting the Drain state.
Implementation notes are provided in Section 5.5.
3.5.1. Loss Recovery after entering Safe Retreat
Unacknowledged packets that were sent in the Unvalidated Phase can be
lost when there is congestion. Loss recovery commences using the
reduced CWND that was set on entry to the Safe Retreat Phase.
Kuhn, et al. Expires 9 January 2025 [Page 13]
Internet-Draft Congestion Control Convergence July 2024
* Loss Recovery (Receiving acknowledgement for all unvalidated
packets): The sender leaves the Safe Retreat Phase when the last
packet number (or a later packet) sent in the Unvalidated Phase is
acknowledged. (Note that if the last packet number is not
cumulatively acknowledged, then additional packets might need to
be retransmitted.)
3.6. RTO Expiry while using Careful Resume
A sender that experiences a Retransmission Time Out (RTO) expiry
ceases to use Careful Resume. The sender enters the Normal Phase.
If using BBR, the normal processing of packet losses will cause it to
enter the Drain state while the "carefully-resuming" flag is set to
True, which will force the Safe Retreat mode.
As in loss recovery, data sent in the Unvalidated Phase could be
later acknowledged after an RTO event (see Section 3.5.1).
3.7. Normal Phase
In the Normal Phase, the sender transitions to using the normal CC
algorithm (e.g., in congestion avoidance if CWND is more than
ssthresh). (Note that when the sender did not use the entire
jump_cwnd the CWND was reduced on entering the Validating Phase.)
Implementation notes are provided in Section 5.6.
4. The Endpoint Token
The Endpoint Token is an implementation-dependent token that allows a
sender to identify its own view of the network path being used to
connect to a specific remote endpoint. This internal identifier is
used by Careful Resume to match the current path with a set of CC
parameters associated with a previously observed path.
4.1. Creating an Endpoint Token
When computing the Endpoint Token, the sender includes information to
identify the path on which it sends, this needs to include:
* a unique identifier for its own sending interface (e.g., a
globally assigned address/prefix or other local identifier);
* when multiple interfaces are in use, this unique identifier should
include an interface identifier (e.g., an index value or a MAC
address to associate the endpoint with the interface used for
sending);
Kuhn, et al. Expires 9 January 2025 [Page 14]
Internet-Draft Congestion Control Convergence July 2024
* an identifier for the destination (e.g., a name or a destination
IP address used to connect to the receiver);
The Endpoint Token could include other information such as the sender
DSCP, the transport ports, a flow label, etc and other information
(e.g., including PvD information [RFC8801] or information relating to
its public-facing IP address). However, such additional information
needs to be set consistently for a resumed connection to the same
remote endpoint. Although additional information could improve the
path differentiation, it could also reduce the re-usability of the
token for resumed connections.
5. Implementation Notes and Guidelines
This section provides guidance for implementation and use.
5.1. Observing the Path Capacity
There are various approaches to measuring the capacity used by a
connection. Congestion controllers, such as CUBIC or Reno, can
estimate the capacity by utilizing the CWND or flight_size. A
different approach could estimate the same parameters for a rate-
based congestion controller, such as BBR
[I-D.cardwell-iccrg-bbr-congestion-control], or by observing the rate
at which data is acknowledged by the remote endpoint.
Implementations are required to calculate a saved_rtt, measuring the
minimum RTT while observing the capacity. For example, this could be
the minimum of a set RTT of measurements measured over the previous 5
minutes.
Implementations are expected to include a LifeTime parameter in the
CC parameters that can be used to remove old CC parameters when no
longer needed, or the CC parameters are out of date.
* There are cases where the current CWND does not reflect the path
capacity. At the end of slow start, the CWND can be significantly
larger than needed to fully utilize the path (i.e., a CWND
overshoot). It is inappropriate to use an overshoot in the CWND
as a basis for estimating the capacity. In most cases, the CWND
will converge to a stable value after several more RTTs. One
mitigation could be to set the saved_cwnd based on the
flight_size, or an averaged CWND.
Kuhn, et al. Expires 9 January 2025 [Page 15]
Internet-Draft Congestion Control Convergence July 2024
* When a sender is rate-limited, or in the RTT following a burst of
transmission, a sender typically transmits less data than allowed
by the CWND. Such observations could to be discounted when
estimating the saved_cwnd (e.g., when a previous observation
recorded a higher value.)
5.2. Confirming the Path in the Reconnaissance Phase
In the Reconnaissance Phase, a sender initiates a connection and
starts sending initial data, while measuring the current_rtt. The CC
is not modified. A sender therefore needs to limit the initial data,
sent in the first RTT of transmitted data, to not more than the IW
[RFC9000]. This transmission using the IW is assumed to be a safe
starting point for any path to avoid adding excessive load to a
potentially congested path.
Careful Resume does not permit multiple concurrent reuse of the saved
CC parameters. When multiple new concurrent connections are made to
a server, each can have a valid saved_endpoint_token, but the
saved_cwnd can once (i.e., if two connections start simultaneously
they cannot both use the saved_cwnd to perform a jump). This is to
prevent a sender from performing multiple jumps in the CWND, each
individually based on the same saved_cwnd, and hence creating an
excessive aggregate load at the bottleneck.
The method that is used to prevent re-use of the saved CC parameters
will depend upon the design of the server (e.g., if all connections
from a given client IP arrive at the same server process, then the
server process could use a hash table, whereas when using some types
of load balancing, a distributed system might be needed to ensure
this invariant when the load balancing hashes connections by 4-tuple
and hence multiple connections from the same client device are served
by different server processes.
5.2.1. Confirming the Path
Path characteristics can change over time for many reasons. This can
result in the previously observed CC parameters becoming irrelevant.
To help confirm the path, the sender compares the saved_RTT with each
of a series of current_rtt samples. If the current_rtt sample is
less than a half of the saved_RTT, this is regarded as too small, and
is an indicator of a path change. (This factor of two arises,
because the rate should not exceed the observed rate when the
saved_cwnd was measured, because the jump_cwnd is calculated as half
the measured saved_cwnd.)
Kuhn, et al. Expires 9 January 2025 [Page 16]
Internet-Draft Congestion Control Convergence July 2024
If the current RTT is larger than saved_rtt (when the saved_cwnd was
measured), this results in a proportionally lower resumed rate,
because the transmission using Careful Resume is paced based on the
current_rtt (i.e., a larger RTT sample in the Unvalidated Phase would
reduce the paced sending rate ,and hence is still safe). If the
current_rtt is incorrectly measured as larger than the actual path
RTT, the sender will receive an ACK for an unvalidated packet before
it would have completed the Unvalidated Phase, Careful Resume uses
this ACK to reset the CWND to reflect the flight_size, and the sender
then enters the Validating Phase.
A current_rtt more than ten times the saved_RTT is indicative of a
path change. (The value of ten was chosen to accommodate both
increases in latency from buffering on a path, and any variation
between RTT samples). A sender also verifies that the initial data
was acknowledged. (i.e., both coukd otherwise could be indicative of
persistent congestion).
A sender in Reconnaissance Phase reverts to the Normal Phase if
congestion is detected. Some transport protocols implement CC
mechanisms that infer potential congestion from an increase in the
current_rtt. In the Reconnaissance Phase, this indication can occur
earlier than congestion that is reported by loss or by ECN marking.
Designs need to consider if such an indication is a suitable trigger
to revert to the Normal Phase.
5.3. Safety for the Unvalidated Phase
This section considers the safety for using saved CC parameters to
tentatively update the CWND. This is designed to mitigate the risk
of adding excessive congestion to an already congested path.
A connection must not directly use the previously saved_cwnd to
directly initialize a new flow causing it to resume sending at the
same rate. The jump_cwnd must therefore be no more than half the
previously saved_cwnd.
5.3.1. Lifetime of CC Parameters
The long-term use of the previously observed parameters is not
appropriate, a lifetime therefore needs to be specified during which
the saved CC parameters can be safely re-used.
[RFC9040] provides guidance on the implementation of TCP Control
Block Interdependence, but does not specify how long a saved
parameter can safely be reused.
Kuhn, et al. Expires 9 January 2025 [Page 17]
Internet-Draft Congestion Control Convergence July 2024
[RFC7661] specifies a method for managing an unvalidated CWND. This
states: "After a fixed period of time (the non-validated period
(NVP)), the sender adjusts the cwnd (Section 4.4.3). The NVP SHOULD
NOT exceed five minutes." Section 5 of [RFC7661] discusses the
rationale for choosing that period. However, RFC 7661 targets rate-
limited connections using normal CC. Careful Resume includes
additional mechanisms to avoid and mitigate the effects of overshoot,
and therefore this can be used to justify a longer lifetime of the
saved_cwnd using Careful Resume.
5.3.2. Pacing in the Unvalidated Phase
A QUIC sender must avoid sending a burst of packets greater than IW
as a result of a step-increase in the CWND. This is consistent with
[RFC8085], [RFC9000].
Pacing packets as a function of the current_rtt, rather than the
saved_RTT provides an additional safety during the Unvalidated Phase,
because it avoids a smaller saved_RTT inflating the sending rate.
Pacing also places a limitation on the minimum acceptable current_RTT
to avoid sending at a rate higher than was previously observed.
The following example provides a relevant pacing rhythm using the RTT
and the saved_cwnd. The Inter-packet Transmission Time (ITT) is
determined by using the current Maximum Message Size (MMS), the
saved_cwnd and the current_RTT. A safety margin can be configured to
avoid sending more than a maximum (max_jump):
jump_cwnd = Min(max_jump,saved_cwnd/2)
ITT = (current_RTT x MMS)/jump_cwnd
This follows the idea presented in [RFC4782],
[I-D.irtf-iccrg-sallantin-initial-spreading] and [CONEXT15]. Other
sender mitigations have also been suggested to avoid line-rate bursts
(e.g., [I-D.hughes-restart]).
5.3.3. Exit from the Unvalidated Phase because of Variable Network
Conditions
* Careful Resume has been designed to be robust to changes in
network conditions due to variations in the forwarding path, such
as reconfiguration of equipment, or changes in the link
conditions. This is mitigated by path confirmation.
Kuhn, et al. Expires 9 January 2025 [Page 18]
Internet-Draft Congestion Control Convergence July 2024
* Careful Resume has been designed to be robust to changes in
network traffic, including the arrival of new flows that compete
for capacity at a shared bottleneck. This is mitigated by jumping
to no more than a half of the saved_cwnd and by using pacing.
* Careful Resume has been designed to avoid unduly suppressing flows
that used the capacity since the available capacity was measured.
This is further mitigated by bounding the duration of the
Unvalidated Phase (and the following Validating Phase), and the
conservative design of the Safe Retreat Phase.
5.4. The Validating Phase
The purpose of the Validating Phase is to trigger an entry to the
Safe Retreat Phase if the capacity is not validated.
When a sender completes the Unvalidated Phase, either by sending a
jump_cwnd of data or after one RTT, it ceases to use the unvalidated
CWND. That is, CWND is reset to the flight_size, and the sender
awaits reception of ACKs to validate the use of this capacity. New
packets are sent when previously sent data is newly acknowledged.
The CWND is increased during the Validating Phase, based on received
ACKs. This allows new data to be sent, but this does not have any
final impact on the CWND if congestion is subsequently detected.
5.5. Safety in the Safe Retreat Phase
This section considers the safety after congestion has been detected
for unvalidated packets.
The Safe Retreat Phase sets a safe CWND value to drain any
unvalidated packets from the path after a packet loss has been
detected or ACKs that indicate sent packets were ECN CE-marked. The
CC parameters that were used are invalid, and are removed.
The Safe Retreat reaction differs from a traditional reaction to
detected congestion, because a jump_cwnd can result in a
significantly higher rate than would be allowed by Slow-Start. This
jump could aggressively feed a congested bottleneck, resulting in
overshoot where a disproportionate number of packets from existing
flows are displaced from the buffer at the congested bottleneck. For
this reason, a sender in the Safe Retreat Phase needs to react to
detected congestion by reducing CWND significantly below the
saved_cwnd.
During loss recovery, a receiver can cumulatively acknowledge data
that was previously sent in the Unvalidated Phase in addition to
acknowledging successful retransmission of data. [RFC3465]
Kuhn, et al. Expires 9 January 2025 [Page 19]
Internet-Draft Congestion Control Convergence July 2024
describes how to appropriately account for such ACKs. ACKS
received for unvalidated packets are tracked to measure the
maximum available capacity, called the PipeSize (The first
unvalidated packet can be determined by recording the sequence
number of the first packet sent in the Unvalidated Phase.) This
calculated PipeSize is later used to reset the ssthresh. However,
note that this is not a safe measure of the currently available
share of the capacity whenever there was also a significant
overshoot at the bottleneck, and must not be used to reinitialise
the CWND.
The Proportional Rate Reduction (PRR) [RFC6937] assumes that it is
safe to reduce the rate gradually when in congestion avoidance.
PRR is therefore not appropriate when there might be significant
overshoot in the use of the capacity, which can be the case when
the Safe Retreat Phase is entered.
The recovery from loss depends on the design of a transport
protocol. A TCP or SCTP sender is required to retransmit all lost
data [RFC5681]. For QUIC and DCCP, the need for loss recovery
depends on the sender policy for retransmission. On entry to the
Safe Retreat Phase, the CWND can be significantly reduced, when
there was multiple loss, a sender recovering all lost data could
take multiple RTTs to complete.
5.6. Returning to Normal Congestion Control
After using Careful Resume, the CC controller returns to the Normal
Phase. The implementation details for different transports depend on
the design of the transport. In the Normal Phase, a sender is
permitted to start Observing the capacity of the path.
5.7. Limitations from Transport Protocols
The CWND is one factor that limits the sending rate of a sender.
Other mechanisms can also constrain the maximum sending rate of a
transport protocol. A transport protocol might need to update these
mechanisms to fully utilise the CWND made available by Careful
Resume:
A TCP sender is limited by the receiver window (rwnd). Unless
configured at a receiver, the rwnd constrains the rate of increase
for a connection and reduces the benefit of Careful Resume.
Kuhn, et al. Expires 9 January 2025 [Page 20]
Internet-Draft Congestion Control Convergence July 2024
QUIC this includes flow control mechanisms and mechanisms to
prevent amplification attacks. In particular, a QUIC receiver
might need to issue proactive MAX_DATA frames to increase the flow
control limits of a connection that is started when using Careful
Resume to gain the expected benefit.
6. QLOG support for QUIC
This section provides definitions that enable a Careful Resume
implementation to generate qlog events when using QUIC. It
introduces an event to report the current phase of a sender, and an
associated description.
The event and data structure definitions in this section are
expressed in the Concise Data Definition Language (CDDL) [RFC8610]
and its extensions described in [I-D.ietf-quic-qlog-quic-events].
The current convention is to use long names for variables. For
example, "CWND" is expanded as "congestion_window" and "saved_cwnd"
is expanded as "saved_congestion_window".
6.1. cr_phase Event
Importance: Extra
When the CC algorithm changes the Careful Resume Phase described in
Section 3 of this specification.
Definition:
Kuhn, et al. Expires 9 January 2025 [Page 21]
Internet-Draft Congestion Control Convergence July 2024
RecoveryCarefulResumePhaseUpdated = {
? old_phase: CarefulResumePhase,
new_phase: CarefulResumePhase,
state_data: CarefulResumeStateParameters,
? restored_data: CarefulResumeRestoredParameters,
? trigger:
; for the Unvalidated phase, when no unvalidated packets
"congestion_window_limited" /
; for the Validating phase
"first_unvalidated_packet_acknowledged" /
; for the Normal phase
; and no remaining unvalidated packets to be acknowledged
"last_unvalidated_packet_acknowledged" /
; for the Normal phase, when CR not allowed
"rtt_not_validated" /
; for the Normal phase,
; when sending fewer unvalidated packets than CWND permits
"rate_limited" /
; for the Safe Retreat phase, when loss detected
"packet_loss" /
; for the Safe Retreat phase,
; when ECN congestion experienced reported
"ECN_CE" /
; for the Normal phase 1 RTT after a congestion event
"exit_recovery"
}
CarefulResumePhase =
"reconnaissance" /
"unvalidated" /
"validating" /
"normal" /
"safe_retreat"
CarefulResumeStateParameters = {
pipesize: uint,
first_unvalidated_packet: uint,
last_unvalidated_packet: uint,
? congestion_window: uint,
? ssthresh: uint
}
CarefulResumeRestoredParameters = {
saved_congestion_window: uint,
saved_rtt: float32
}
Figure 2
Kuhn, et al. Expires 9 January 2025 [Page 22]
Internet-Draft Congestion Control Convergence July 2024
7. Acknowledgments
The authors would like to thank John Border, Gabriel Montenegro,
Patrick McManus, Ian Swett, Igor Lubashev, Robin Marx, Roland Bless,
Franklin Simo, Kazuho Oku, Tong, Ana Custura, Neal Cardwell, and
Joerg Deutschmann for their fruitful comments on earlier versions of
this document.
The authors would like to particularly thank Tom Jones for co-
authoring several previous versions of this document. Ana Custura
and Robin Marx developed the qlog support.
8. IANA Considerations
No current parameters are required to be registered by IANA.
9. Security Considerations
This document does not exhibit specific security considerations.
Security considerations for the interactions with the receiver are
discussed in [I-D.kuhn-quic-bdpframe-extension].
10. References
10.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC8085] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage
Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085,
March 2017, <https://www.rfc-editor.org/info/rfc8085>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8610] Birkholz, H., Vigano, C., and C. Bormann, "Concise Data
Definition Language (CDDL): A Notational Convention to
Express Concise Binary Object Representation (CBOR) and
JSON Data Structures", RFC 8610, DOI 10.17487/RFC8610,
June 2019, <https://www.rfc-editor.org/info/rfc8610>.
Kuhn, et al. Expires 9 January 2025 [Page 23]
Internet-Draft Congestion Control Convergence July 2024
[RFC8801] Pfister, P., Vyncke, É., Pauly, T., Schinazi, D., and W.
Shao, "Discovering Provisioning Domain Names and Data",
RFC 8801, DOI 10.17487/RFC8801, July 2020,
<https://www.rfc-editor.org/info/rfc8801>.
[RFC9000] Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based
Multiplexed and Secure Transport", RFC 9000,
DOI 10.17487/RFC9000, May 2021,
<https://www.rfc-editor.org/info/rfc9000>.
10.2. Informative References
[CONEXT15] Li, Q., Dong, M., and P B. Godfrey, "Halfback: Running
Short Flows Quickly and Safely", ACM CoNEXT , 2015.
[I-D.cardwell-iccrg-bbr-congestion-control]
Cardwell, N., Cheng, Y., Yeganeh, S. H., Swett, I., and V.
Jacobson, "BBR Congestion Control", Work in Progress,
Internet-Draft, draft-cardwell-iccrg-bbr-congestion-
control-02, 7 March 2022,
<https://datatracker.ietf.org/doc/html/draft-cardwell-
iccrg-bbr-congestion-control-02>.
[I-D.hughes-restart]
Hughes, A., Touch, J., and J. Heidemann, "Issues in TCP
Slow-Start Restart After Idle", Work in Progress,
Internet-Draft, draft-hughes-restart-00, December 2001,
<https://www.ietf.org/archive/id/draft-hughes-restart-
00.txt>.
[I-D.ietf-quic-qlog-quic-events]
Marx, R., Niccolini, L., Seemann, M., and L. Pardue, "QUIC
event definitions for qlog", Work in Progress, Internet-
Draft, draft-ietf-quic-qlog-quic-events-07, 4 March 2024,
<https://datatracker.ietf.org/doc/html/draft-ietf-quic-
qlog-quic-events-07>.
[I-D.irtf-iccrg-sallantin-initial-spreading]
Sallantin, R., Baudoin, C., Arnal, F., Dubois, E., Chaput,
E., and A. Beylot, "Safe increase of the TCP's Initial
Window Using Initial Spreading", Work in Progress,
Internet-Draft, draft-irtf-iccrg-sallantin-initial-
spreading-00, 15 January 2014,
<https://datatracker.ietf.org/doc/html/draft-irtf-iccrg-
sallantin-initial-spreading-00>.
Kuhn, et al. Expires 9 January 2025 [Page 24]
Internet-Draft Congestion Control Convergence July 2024
[I-D.kuhn-quic-bdpframe-extension]
Kuhn, N., Emile, S., Fairhurst, G., Secchi, R., and C.
Huitema, "Signalling CC Parameters for Careful Resume
using QUIC", Work in Progress, Internet-Draft, draft-kuhn-
quic-bdpframe-extension-05, 4 March 2024,
<https://datatracker.ietf.org/doc/html/draft-kuhn-quic-
bdpframe-extension-05>.
[IJSCN] Thomas, L., Dubois, E., Kuhn, N., and E. Lochin, "Google
QUIC performance over a public SATCOM access",
International Journal of Satellite Communications and
Networking 10.1002/sat.1301, 2019.
[MAPRG111] Kuhn, N., Stephan, E., Fairhurst, G., Jones, T., and C.
Huitema, "Feedback from using QUIC's 0-RTT-BDP extension
over SATCOM public access", IETF 111 - MAPRG meeting ,
2022.
[RFC3465] Allman, M., "TCP Congestion Control with Appropriate Byte
Counting (ABC)", RFC 3465, DOI 10.17487/RFC3465, February
2003, <https://www.rfc-editor.org/info/rfc3465>.
[RFC4782] Floyd, S., Allman, M., Jain, A., and P. Sarolahti, "Quick-
Start for TCP and IP", RFC 4782, DOI 10.17487/RFC4782,
January 2007, <https://www.rfc-editor.org/info/rfc4782>.
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
<https://www.rfc-editor.org/info/rfc5681>.
[RFC5783] Welzl, M. and W. Eddy, "Congestion Control in the RFC
Series", RFC 5783, DOI 10.17487/RFC5783, February 2010,
<https://www.rfc-editor.org/info/rfc5783>.
[RFC6675] Blanton, E., Allman, M., Wang, L., Jarvinen, I., Kojo, M.,
and Y. Nishida, "A Conservative Loss Recovery Algorithm
Based on Selective Acknowledgment (SACK) for TCP",
RFC 6675, DOI 10.17487/RFC6675, August 2012,
<https://www.rfc-editor.org/info/rfc6675>.
[RFC6937] Mathis, M., Dukkipati, N., and Y. Cheng, "Proportional
Rate Reduction for TCP", RFC 6937, DOI 10.17487/RFC6937,
May 2013, <https://www.rfc-editor.org/info/rfc6937>.
[RFC7661] Fairhurst, G., Sathiaseelan, A., and R. Secchi, "Updating
TCP to Support Rate-Limited Traffic", RFC 7661,
DOI 10.17487/RFC7661, October 2015,
<https://www.rfc-editor.org/info/rfc7661>.
Kuhn, et al. Expires 9 January 2025 [Page 25]
Internet-Draft Congestion Control Convergence July 2024
[RFC8867] Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test
Cases for Evaluating Congestion Control for Interactive
Real-Time Media", RFC 8867, DOI 10.17487/RFC8867, January
2021, <https://www.rfc-editor.org/info/rfc8867>.
[RFC9002] Iyengar, J., Ed. and I. Swett, Ed., "QUIC Loss Detection
and Congestion Control", RFC 9002, DOI 10.17487/RFC9002,
May 2021, <https://www.rfc-editor.org/info/rfc9002>.
[RFC9040] Touch, J., Welzl, M., and S. Islam, "TCP Control Block
Interdependence", RFC 9040, DOI 10.17487/RFC9040, July
2021, <https://www.rfc-editor.org/info/rfc9040>.
[RFC9406] Balasubramanian, P., Huang, Y., and M. Olson, "HyStart++:
Modified Slow Start for TCP", RFC 9406,
DOI 10.17487/RFC9406, May 2023,
<https://www.rfc-editor.org/info/rfc9406>.
Appendix A. Notes on the Careful Resume Phases
The table below is provided to illustrate the operation of Careful
Resume. This table is informative, please refer to the body of the
document for the normative specification. The description is based
on a Normal CC that uses Reno or Cubic. The PipeSize tracks the
validated CWND.
Kuhn, et al. Expires 9 January 2025 [Page 26]
Internet-Draft Congestion Control Convergence July 2024
+------+---------+---------+------------+-----------+------------+
|Phase |Normal |Recon. |Unvalidated |Validating |Safe Retreat|
+------+---------+---------+------------+-----------+------------+
| |Observing|Confirm |Send faster |Validate |Drain path; |
| |CC params|path |using saved |new CWND; |Update PS |
| | | |_cwnd |Update PS | |
+------+---------+---------+------------+-----------+------------+
|On | - |CWND=IW |PS=CWND; |If (FS>PS) |CWND=(PS/2) |
|entry:| | |jump_cwnd |{CWND=FS} | |
| | | |=saved_cwnd |else | |
| | | |/2; |{CWND=PS; | |
| | | |CWND |enter |
| | | |=jump_cwnd |Normal} | |
+------+---------+---------+------------+-----------+------------+
|CWND: |When in |CWND |CWND is not |CWND can |CWND is not |
| |observe, |increases|increased |increase |increased |
| |measure |using SS | |using SS | |
| |saved | | | | |
| |_cwnd | | | | |
+------+---------+---------+------------+-----------+------------+
|PS: | - | - | PS+=ACked |
+------+---------+---------+------------+-----------+------------+
|RTT: |Measure |Measure | - | - | - |
| |saved_rtt|current | | | |
| | |_rtt | | | |
+------+---------+---------+------------+-----------+------------+
|If |Normal |Normal | Enter | - |
|loss |CC |CC; | Safe | |
|or | |CR is not| Retreat | |
|ECNCE:| |allowed | | |
+------+---------+---------+------------+-----------+------------+
|Next |Observing|If ( |If (FS=CWND |If (ACK |If (ACK |
|Phase:|(as |FS=CWND, |or >1 RTT |>= last |>= last |
| |needed) |Lifetime,|has passed |unvalidated|unvalidated |
| | |and RTT |or ACK for |packet), |packet), |
| | |confirmed|>= last |enter |{ssthresh=PS|
| | |), enter |unvalidated |Normal |and enter |
| | |Unvalidat|packet), | |Normal} |
| | |ing else |enter | | |
| | |enter |Validating | | |
| | |Normal | | | |
+------+---------+---------+------------+-----------+------------+
Figure 3: Illustration of the operation of Careful Resume
Kuhn, et al. Expires 9 January 2025 [Page 27]
Internet-Draft Congestion Control Convergence July 2024
The following abbreviations are used SS = Slow-Start FS =
flight_size; PS = PipeSize; ACK = acknowledgement. The PipeSize
tracks the validated part of the cwnd. It is set to the CWND on
entry to the Unvalidated Phase and is updated as each additional
packet is acknowledged.
Note: For an implementation that keeps track of transmitted data in
terms of packets: In the Unvalidated Phase, the first unvalidated
packet corresponds to the highest sent packet recorded on entry to
this phase. In the Validating Phase and Safe Retreat Phase, this
corresponds to the last unvalidated packet. It is also the highest
sent packet number recorded on entry to this phase.
The remaining subsections provide informative examples of use.
Note: Although the QLOG variables are expressed in bytes, to simplify
the description, these examples are described in term of packet
numbers.
A.1. Example with No Loss
In the first example of using Careful Resume, the sender starts by
sending IW packets, assumed to be 10 packets, in the Reconnaissance
Phase, and then continues in a subsequent RTT to send more packets
until the sender becomes CWND-limited (i.e., flight_size = CWND).
The sender in the Reconaissance Phase then confirms the RTT and other
conditions for using Careful Resume. In this example, this is
confirmed when the sender has 29 packets in flight.
The sender then enters the Unvalidated Phase. (This path
confirmation could have happened earlier if data had been available
to send.) The sender initialises the PipeSize to the CWND (at this
time this is the same as the flight_size, i.e., 29 packets) and then
sets the CWND to 150 packets (based upon half of the previously
observed saved_cwnd of 300 packets).
The sender now sends 121 unvalidated packets (the unused portion of
the current CWND). Each time a packet is sent, the sender checks
whether 1 RTT has passed since entering the Unvalidated Phase
(otherwise, the Validating Phase is entered). This check triggers
only for cases where the sender is rate-limited, see the following
example.
The PipeSize increases after each ACK is received.
Kuhn, et al. Expires 9 January 2025 [Page 28]
Internet-Draft Congestion Control Convergence July 2024
When the first unvalidated packet is acknowledged (packet number 30)
the sender enters the Validating Phase. (This transition would also
occur if the flight_size increased to equal CWND.) During this
phase, the CWND can be increased for each ACK that acknowledges an
unvalidated packet, because this indicates that the packet was indeed
validated.
When an ACK is received for the last packet that was sent in the
Unvalidated Phase, the sender completes using Careful Resume. It
then enters the Normal Phase. If CWND is less than ssthresh, a Reno
or Cubic sender in the Normal Phase is permitted to use Slow-Start to
grow the CWND towards the ssthresh, and will then enter congestion
avoidance.
A.2. Example with No Loss, Rate-Limited
A rate-limited sender will not fully utilize the available CWND when
using Careful Resume, and CWND is therefore reset on entry to the
Validating Phase, as described below.
The sender starts by sending IW packets (10) in the Reconnaissance
Phase. It commences as described in the first example, transitioning
to the Unvalidated Phase. This sets the CWND to 150 packets, and the
PipeSize to the flight_size (i.e., 29 packets).
The sender then becomes rate-limited because it only sends 50
unvalidated packets.
After about one RTT (detected by using local timestamps or by
receiving an ACK for the first unvalidated packet), the sender will
still not have fully used the CWND. It then enters the Validating
Phase and resets the CWND to the current flight_size, (i.e., 50
packets). During this phase, the CWND can be increased for each
received ACK that validates reception of an unvalidated packet. The
PipeSize also increases with each ACK received, to reflect the
discovered capacity.
When an ACK is received for the last packet sent in the Unvalidated
Phase, the sender has completed using Careful Resume. It then enters
the Normal Phase, as in the example with no loss.
A.3. Example with Loss detected in the Reconnaissance Phase
When a packet is lost in the Reconnaissance Phase, the sender will
enter the Normal Phase and recovers this using the normal method.
(There is no change to the CC method, because the sender has
discovered a potential capacity limit and is not allowed to continue
to use Careful Resume.)
Kuhn, et al. Expires 9 January 2025 [Page 29]
Internet-Draft Congestion Control Convergence July 2024
A.4. Example with Loss detected in the Validating Phase
As in the first example, the sender enters the Unvalidated Phase and
sets the CWND to 150 packets with the PipeSize initialized to the
flight_size (i.e., 29 packets).
The sender now sends 121 unvalidated packets (the remaining unused
CWND). This example considers the case when one of the unvalidated
packet is lost, which we choose to be packet 64 (the 35th packet sent
in the Unvalidated Phase).
ACKs confirm the first 34 unvalidated packets are received without
loss. The PipeSize at this point is equal to 63 (29 + 34) packets.
The loss is then detected (by a timer or by receiving three ACKs that
do not cover packet number 35), the sender then enters the Safe
Retreat Phase because the window was not validated. The PipeSize at
this point is equal to 66 (29 + 34) packets. Assuming that the IW
was 10 packets, the CWND is reset to Max(10,PS/2) = Max(10,66/2) = 33
packets. This CWND is used during the Safe Retreat Phase, because
congestion was detected and the sender still does not yet know if the
remaining unvalidated packets will be successfully acknowledged. A
conservative CWND calculation ensures the sender drains the path
after this potentially severe congestion event. There is no further
increase in CWND in this phase.
The sender continues to receive ACKs for the remaining 86 (121-35)
unvalidated packets. Recall that the 35th unvalidated packet was
lost and had packet number 64 (29+35). The PipeSize tracks the
capacity discovered by acknowledgments for the unvalidated packets
and continues to be further increased for each received ACK
acknowledges new data. Although the PipeSize cannot be used to
safety initialise the CWND (because it was measured when the sender
had aggressively created overload), the estimated PipeSize (which, in
this case, is 121-1 = 120 packets) can be used to set the ssthresh on
exit from Safe Retreat, since it does indicate an upper limit to the
current capacity.
At the point where all packets sent in the Unvalidated Phase have
been either acknowledged or have been declared lost, the sender
updates ssthresh and enters the Normal Phase. Because CWND will now
now be less than ssthresh, a sender in the Normal Phase is permitted
to use Slow-Start to grow the CWND towards the ssthresh, after which
it will enter congestion avoidance.
Kuhn, et al. Expires 9 January 2025 [Page 30]
Internet-Draft Congestion Control Convergence July 2024
Appendix B. Internet Draft Revision details
Previous individual submissions were discussed in TSVWG and QUIC.
WG -00 included clarifications and restructuring to form the 1st
WG draft.
WG -01 included review comments and suggestions from John Border,
and follows the setting of the TSVWG milestone with an intended
status of "Proposed Standard".
WG -02 includes steps to complete the spec. In particular,
consideration of rate-limited senders; selection of reasoned
parameters; specification of the Safe Retreat Phase; and
improvements to the consistency throughout. Added the Validating
Phase.
WG -03, explain entry to Validating Phase, editorial tidy.
WG -04, update based on review comments from Kazuho Oku.
WG-05, update based on review comments from Neal Cardwell. WG
feedback from IETF-118. Reviewed the requirements v. guidelines;
clarified that CC is not changed in recon., but the recon. info is
used to steer the next phase; clarified saved_cwnd can be computed
from ACK rate; use jump once; that real server platforms are
complex. Clarified lifetime for saved CC params. Incorporates
comments from Tong.
WG-06, SR updated following Hackathon comments from Kazuho Oku,
and rework of use of PipeSize. Added an informative summary of
actions, on suggestion by Tong. Added examples based on text by
Ana Custura.
WG-07, Use "rate-limited" uniformly instead of application and
data limited.
Updated to exit early when the unvalidated CWND not utilised,
detected in tests by Q Misell. Change pipe_size to be PipeSize.
WG-08, Updated CDDL, and made constraints to Observing into
guidance, they say what makes sense - but do not need to be
followed for conformance. Updated table in the appendix to align
with text.
WG-09, Cleaning text to separate guidelines and specification and
adjust wording to improve clarity based on questions received
during implementation.
Kuhn, et al. Expires 9 January 2025 [Page 31]
Internet-Draft Congestion Control Convergence July 2024
WG-10, CH developed text to explain expected operation with BBR.
This also fixed some typos introduced in previous edits. Fix XML
and fix CDDL bugs for submission.
Authors' Addresses
Nicolas Kuhn
Thales Alenia Space
Email: nicolas.kuhn.ietf@gmail.com
Emile Stephan
Orange
Email: emile.stephan@orange.com
Godred Fairhurst
University of Aberdeen
Department of Engineering
Fraser Noble Building
Aberdeen
AB24 3UE
United Kingdom
Email: gorry@erg.abdn.ac.uk
Raffaello Secchi
University of Aberdeen
Department of Engineering
Fraser Noble Building
Aberdeen
AB24 3UE
United Kingdom
Email: r.secchi@erg.abdn.ac.uk
Christian Huitema
Private Octopus Inc.
Email: huitema@huitema.net
Kuhn, et al. Expires 9 January 2025 [Page 32]