Convergence of Congestion Control from Retained State
draft-ietf-tsvwg-careful-resume-08
| Document | Type | Active Internet-Draft (tsvwg WG) | |
|---|---|---|---|
| Authors | Nicolas Kuhn , Stephan Emile , Gorry Fairhurst , Raffaello Secchi , Christian Huitema | ||
| Last updated | 2024-04-22 | ||
| Replaces | draft-kuhn-tsvwg-careful-resume | ||
| RFC stream | Internet Engineering Task Force (IETF) | ||
| Intended RFC status | Proposed Standard | ||
| Formats | |||
| Additional resources |
GitHub Repository
Mailing list discussion |
||
| Stream | WG state | WG Document | |
| Associated WG milestone |
|
||
| Document shepherd | Marten Seemann | ||
| IESG | IESG state | I-D Exists | |
| Consensus boilerplate | Yes | ||
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | martenseemann@gmail.com |
draft-ietf-tsvwg-careful-resume-08
Internet Engineering Task Force N. Kuhn
Internet-Draft Thales Alenia Space
Intended status: Standards Track E. Stephan
Expires: 24 October 2024 Orange
G. Fairhurst
R. Secchi
University of Aberdeen
C. Huitema
Private Octopus Inc.
22 April 2024
Convergence of Congestion Control from Retained State
draft-ietf-tsvwg-careful-resume-08
Abstract
This document specifies a cautious method for IETF transports that
enables fast startup of congestion control for a wide range of
connections. It reuses a set of computed congestion control
parameters that are based on previously observed path characteristics
between the same pair of transport endpoints. These parameters are
stored, allowing them to be later used to modify the congestion
control behavior of a subsequent connection.
It describes assumptions and defines requirements for how a sender
utilizes these parameters to provide opportunities for a connection
to more rapidly get up to speed and rapidly utilize available
capacity. It discusses how use of the method impacts the capacity at
a shared network bottleneck and the safe response that is needed
after any indication that the new rate is inappropriate.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 24 October 2024.
Kuhn, et al. Expires 24 October 2024 [Page 1]
Internet-Draft Congestion Control Convergence April 2024
Copyright Notice
Copyright (c) 2024 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Use of saved CC parameters by a Sender . . . . . . . . . 4
1.2. Receiver Preference . . . . . . . . . . . . . . . . . . . 4
1.3. Examples of Scenarios of Interest . . . . . . . . . . . . 5
2. Language, Notation and Terms . . . . . . . . . . . . . . . . 6
2.1. Requirements Language . . . . . . . . . . . . . . . . . . 6
2.2. Notation and Terms . . . . . . . . . . . . . . . . . . . 6
3. The Phases of CC using Resume . . . . . . . . . . . . . . . . 7
3.1. Observe Phase . . . . . . . . . . . . . . . . . . . . . . 8
3.2. Reconnaissance Phase . . . . . . . . . . . . . . . . . . 8
3.3. Unvalidated Phase . . . . . . . . . . . . . . . . . . . . 10
3.4. Validating Phase . . . . . . . . . . . . . . . . . . . . 11
3.5. Safe Retreat Phase . . . . . . . . . . . . . . . . . . . 12
3.5.1. Loss Recovery after entering Safe Retreat . . . . . . 13
3.6. Normal Phase . . . . . . . . . . . . . . . . . . . . . . 13
3.7. RTO Expiry while using Careful Resume . . . . . . . . . . 13
4. Congestion Control Guidelines and Requirements . . . . . . . 14
4.1. Determining the Current Path Capacity in the Observe
Phase . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.2. Confirming the Path in the Reconnaissance Phase . . . . . 14
4.2.1. Confirming the RTT . . . . . . . . . . . . . . . . . 15
4.3. Safety Requirements for the Unvalidated Phase . . . . . . 16
4.3.1. Lifetime of CC Parameters . . . . . . . . . . . . . . 16
4.3.2. Pacing in the Unvalidated Phase . . . . . . . . . . . 17
4.3.3. Exit from the Unvalidated Phase because of Variable
Network Conditions . . . . . . . . . . . . . . . . . 17
4.4. Safety Requirements for the Validating Phase . . . . . . 18
4.5. Safety Requirements for the Safe Retreat Phase . . . . . 18
4.6. Returning to Normal Congestion Control . . . . . . . . . 19
4.7. Limitations from Transport Protocols . . . . . . . . . . 19
5. QLOG support for QUIC . . . . . . . . . . . . . . . . . . . . 19
5.1. cr_phase Event . . . . . . . . . . . . . . . . . . . . . 19
Kuhn, et al. Expires 24 October 2024 [Page 2]
Internet-Draft Congestion Control Convergence April 2024
6. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 21
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21
8. Security Considerations . . . . . . . . . . . . . . . . . . . 21
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 21
9.1. Normative References . . . . . . . . . . . . . . . . . . 21
9.2. Informative References . . . . . . . . . . . . . . . . . 22
Appendix A. Notes on the Careful Resume Phases . . . . . . . . . 24
A.1. Example with No Loss . . . . . . . . . . . . . . . . . . 26
A.2. Example with No Loss, Rate-Limited . . . . . . . . . . . 27
A.3. Example with Loss detected in the Reconnaissance Phase . 27
A.4. Example with Loss detected in the Validating Phase . . . 27
Appendix B. Appendix: An Endpoint Token . . . . . . . . . . . . 28
B.1. Creating an Endpoint Token . . . . . . . . . . . . . . . 28
Appendix C. Appendix: Revision details . . . . . . . . . . . . . 29
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 30
1. Introduction
All Internet transports are required to either use a Congestion
Control (CC) method, or to constrain their rate of transmission
[RFC8085]. In 2010, a survey of alternative CC methods [RFC5783],
noted that there are challenges when a CC method operates across an
Internet path with a high and/or varying Bandwidth-Delay Product
(BDP). This mechanism targets a solution for these challenges.
A CC method typically takes time to ramp-up the sending rate, called
the "slow-start phase", informally known as the time to "Get up to
speed". This slow-start phase defines a time in which a sender
intentionally uses less capacity than might be available, with the
intention to avoid or limit overshooting the available capacity for
the path. The slow-start design can increase queuing (latency or
jitter) and/or congestion packet loss for the flow. Any overshoot
can have a detrimental effect on other flows sharing a common
bottleneck. A sender can use a method to observe the rate of
acknowledged data, and seek to avoid overshooting the bottleneck
capacity (e.g., Hystart++ [RFC9406]). In the extreme case, an
overshoot can result in persistent congestion with unwanted
starvation of other flows [RFC8867] (i.e., preventing other flows
from successfully sharing the capacity at a common bottleneck).
This document proposes a CC method that is expected to reduce the
time to complete a transfer when the transfer sends significantly
more data than allowed by the Initial congestion Window (IW), and
where the BDP of the path is also significantly more than the IW. It
introduces an alternative method to select initial CC parameters,
that seek to more rapidly and safely grow the sending rate controlled
by the congestion window (CWND). CC methods that are rate-based can
make similar adjustments to their target sending rate.
Kuhn, et al. Expires 24 October 2024 [Page 3]
Internet-Draft Congestion Control Convergence April 2024
This method is based on temporal sharing (sometimes known as caching)
of a saved set of CC parameters that relate to previous observations
of the same path. The parameters include: the saved_cwnd for the
path and the minimum Round Trip Time (RTT). These parameters are
stored and used to modify the CC behavior of a subsequent connection
between the same endpoints.
When used with the QUIC transport, this provides transport services
that resemble those currently available in TCP, using methods such as
TCP Control Block (TCB) [RFC9040] caching.
1.1. Use of saved CC parameters by a Sender
CC parameters are used by Careful Resume for three functions:
1. Information about the utilised path capacity (saved_cwnd) to
determine an appropriate set of CC parameters for re-using the
path.
2. Information to characterize the saved path to confirm whether the
current path is consistent with a saved path.
3. Information to check the validity of the saved CC parameters,
including the time for which the parameters remain valid.
"Generally, implementations are advised to be cautious when using
saved CC parameters on a new path", as stated in [RFC9000]. While
this statement has been proposed in the context of QUIC
standardization, this advice is appropriate for any IETF transport
protocol. Care is therefore needed to assure safe use and to be
robust to changes in traffic patterns, network routing, and link/node
conditions. There are cases where using the saved parameters of a
previous connection is not appropriate (e.g., Section 3.2).
1.2. Receiver Preference
Whilst a sender could take optimization decisions without considering
the receiver's preference, there are cases where a receiver could
have information that is not available at the sender, or might
benefit from understanding that Careful Resume might be used. In
these cases, a receiver could explicitly ask to enable or inhibit
tuning of the CC when an application initiates a new session or
resume an existing one. A receiver could also tune policies for
using the connection (e.g., managing the receiver window or flow
credit).
Examples where a receiver might request not to use Careful Resume
include:
Kuhn, et al. Expires 24 October 2024 [Page 4]
Internet-Draft Congestion Control Convergence April 2024
1. a receiver that can predict the pattern of traffic (e.g., insight
into the volume of data to be sent, the expected length of a
session, or the required maximum transfer rate);
2. a receiver with a local indication that a path/local interface
has changed since the CC parameters were stored;
3. knowledge of the current hardware limitations at a receiver;
4. a receiver that can predict additional capacity will be needed
for other concurrent or later flows (i.e., prefers to activate
Careful Resume for a different connection).
A related document proposes an extension for QUIC that allows sender-
generated CC parameters to be stored at the receiver
[I-D.kuhn-quic-bdpframe-extension]. Transferring the information to
a receiver releases the need for a sender to retain transport state
for each receiver, and allows a receiver to express a preference for
whether to use the method.
1.3. Examples of Scenarios of Interest
This section provides a set of examples where Careful Resume is
expected to improve performance.
Either endpoint can assume the role of a sender or a receiver.
Careful Resume also supports a bidirectional data transfer, where
both endpoints simultaneously send data (e.g., remote execution of an
application, or a bidirectional video conference call).
In one example, an application uses a series of connections over a
path (i.e., resumes a connection to the same endpoint). Without a
new method, each connection would need to individually discover
appropriate CC parameters, whereas Careful Resume allows the flow to
use a rate that is based on the previously observed CC parameters.
In another example, an application connects after a disruption had
temporarily reduced the path capacity. When the endpoint returns to
use a path with the original characteristics, using a rate that is
based on the previously observed CC parameters.
Kuhn, et al. Expires 24 October 2024 [Page 5]
Internet-Draft Congestion Control Convergence April 2024
There is particular benefit for any path with an RTT that is much
larger than typical Internet paths. In a specific example, an
application connected via a satellite access network [IJSCN] could
require 9 seconds to complete a 5.3 MB transfer using standard CC,
whereas a sender using Careful Resume could reduce this transfer time
to 4 seconds. The time to complete a 1 MB transfer could similarly
be reduced by 62 % [MAPRG111]. This benefit is also expected for
other sizes of transfer and for different path characteristics when a
path has a large BDP.
2. Language, Notation and Terms
This subsection provides a brief summary of key terms and the
requirements language.
2.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
2.2. Notation and Terms
The document uses language drawn from a range of IETF RFCs. It
defines current, and saved values for a set of CC parameters:
CC parameters: A set of saved congestion control parameters from a
previously observed connection (see Section 1.1).
Careful Resume (CR): The method specified in this document to
select initial CC parameters, that seeks to more rapidly and
safely increase the initial sending rate.
current_endpoint_token: The Endpoint Token of the current
receiver;
current_rtt: A sample measurement of the current RTT;
endpoint_token: An Endpoint Token identifying a path to a
receiver;
flight_size: The currently unacknowledged volume of data sent by
the CC method;
jump_cwnd: The resumed CWND, used in the Unvalidated Phase.
Kuhn, et al. Expires 24 October 2024 [Page 6]
Internet-Draft Congestion Control Convergence April 2024
LifeTime: The time for which the saved CC parameters can be safely
re-used.
max_jump : The maximum configured jump_cwnd;
PipeSize: A measure of the validated available capacity based on
the acknowledged data;
saved_cwnd: A value of CWND derived from observation of a previous
connection, which reflects capacity that was utilised by the
observed connection;
saved_endpoint_token: The Endpoint Token of a previous connection
to a receiver;
saved_rtt: The preserved minimum RTT, e.g., corresponding to the
minimum of a set RTT of measurements over the last 5 minutes of a
connection.
Unvalidated Packet: A packet sent when the CWND has been increased
beyond the size normally permitted by the congestion control
algorithm; if such a packet is acknowledged, it contributes to the
PipeSize, but if congestion is detected, it triggers entry to the
Safe Retreat Phase.
The Endpoint Token is described in Appendix B.
3. The Phases of CC using Resume
This section defines a series of phases that the CC algorithm moves
through as a connection uses Careful Resume, as shown in Figure 1.
Observe ...> Connect -> Reconnaissance --------------------> Normal
(Normal) | ^
v |
Unvalidated --------------------------+
| | |
| +--> Validating --------------+
| | |
| | |
+---------------+--> Safe Retreat ---+
Figure 1: Key transistions between Phases in Careful Resume
Kuhn, et al. Expires 24 October 2024 [Page 7]
Internet-Draft Congestion Control Convergence April 2024
Figure 1: Phases when a connection uses the Careful Resume. The
Observe Phase is performed by an established connection as an action
within the Normal Phase. Examples of the transitions between phases
are provided in Appendix A.
3.1. Observe Phase
During a previous established connection, the CC parameters for the
specific path to an endpoint are saved. This characterizes the path
and determines the saved_cwnd. The saved_cwnd is a measure of the
currently utilised capacity for the connection, measured as the
number of bytes sent over a RTT. This could be computed by measuring
the volume of data acknowledged in one RTT. The CC parameters also
include the minimum RTT (saved_rtt) and the receiver Endpoint Token
(saved_endpoint_token).
An implementation can store the CC parameters at the server (or could
exchange this information with a receiver
[I-D.kuhn-quic-bdpframe-extension]).
* Observe Phase: The sender updates the stored CC parameters and/or
sends the updated CC parameter information after each observation.
If the measured CWND is less than four times the Initial Window
(IW) (i.e., CWND less than IW*4), a sender can choose to not store
and/or send CC parameters, because the additional actions
associated with performing Careful Resume for a small CWND would
not justify the use of the method.
* Observe Phase (Sending CC Parameters): When sending the CC
parameters to a receiver, these ought to be updated if there are
significant changes in the saved CC parameters; The frequency of
update SHOULD be less than one update for several RTTs
[I-D.kuhn-quic-bdpframe-extension].
Implementation notes are provided in Section 4.1.
3.2. Reconnaissance Phase
A sender enters the Reconnaissance Phase after connection setup. In
this phase, the CWND is initialised to the IW, and the sender
transmits initial data. The CWND MAY be increased using normal CC as
each acknowledgment confirms delivery of a packet (i.e., the CC is
unchanged).
The phase seeks to determine if the path is consistent with a
previously observed path (saved in the CC parameters). There are a
set of conditions that need to be confirmed before the sender is
permitted to enter the Unvalidated Phase:
Kuhn, et al. Expires 24 October 2024 [Page 8]
Internet-Draft Congestion Control Convergence April 2024
* Reconnaissance Phase (Endpoint change): If the current remote
endpoint is not the same as a saved endpoint, the sender MUST
enter the Normal Phase. If the Endpoint Token differs (i.e., the
saved_endpoint_token is different from the
current_endpoint_token), it is assumed to represent a different
network path. The sender also enters the Normal Phase when there
are no corresponding saved CC parameters.
* Reconnaissance Phase (Lifetime of saved CC parameters): The CC
parameters are temporal. If the LifeTime of the observed CC
parameters is exceeded Section 4.3.1, the CC parameters are no
longer used and sender enters the Normal Phase.
* Reconnaissance Phase (Confirming the RTT): During this phase, a
sender MUST record the minimum RTT for the current connection.
* Reconnaissance Phase (Avoiding using Careful Resume): A receiver
can use a method (e.g., [I-D.kuhn-quic-bdpframe-extension]) to
request that the sender instead enters the Normal Phase.
* Reconnaissance Phase (Detected congestion): If the sender detects
congestion (e.g., packet loss or ECN-CE marking), the sender does
not use the Careful Resume method and MUST enter the Normal Phase
to respond to the detected congestion.
* Reconnaissance Phase (Using saved_cwnd): Only one connection can
use a specific set of saved CC parameters. If another connection
has already started to use the saved_cwnd, the sender MUST enter
the Normal Phase.
* Reconnaissance Phase (Rate-limited sender): If the sender is rate-
limited [RFC7661], it might send insufficient data to be able to
validate transmission at the higher rate. Careful Resume is
allowed to remain in the Reconnaissance Phase and to not
transition to the Unvalidated Phase until the sender has more data
ready to send in the transmission buffer than is permitted by the
current CWND. In some implementations, the decision to enter the
Unvalidated Phase could require coordination with the management
of buffers in the interface to the higher layers.
When a sender confirms the path and it receives an acknowledgement
for the initial data without reported congestion, it MAY then enter
the Unvalidated Phase. This transition occurs when a sender has more
data than permitted by the current CWND.
When a path is not confirmed, Careful Resume is not used and the
sender enters the Normal Phase.
Kuhn, et al. Expires 24 October 2024 [Page 9]
Internet-Draft Congestion Control Convergence April 2024
Implementation requirements are provided in Section 4.2.
3.3. Unvalidated Phase
The Unvalidated Phase is designed to enable the CWND to more rapidly
get up to speed.
The sender only enters this phase when the saved CC parameters are
confirmed:
Unvalidated Phase (Confirming the path on entry): The calculation
of a sending rate from a saved_cwnd is directly impacted by the
RTT, therefore a significant change in the RTT is a strong
indication that the previously observed CC parameters may not be
valid for the current path. If the RTT measurement is not
confirmed, i.e., the current_rtt is greater than or equal to
(saved_rtt / 2) or the current_rtt is less than or equal to
(saved_rtt x 10) (see Section 4.2.1), the sender MUST enter the
Normal Phase (see trigger rtt_not_validated in Section 5).
When the RTT is confirmed:
* Unvalidated Phase (Initialising PipeSize): The variable PipeSize
is initialised to CWND on entry to the Unvalidated Phase. This
records the value before the jump is applied.
* Unvalidated Phase (Setting the jump_cwnd): To avoid starving other
flows that could have either started or increased their used
capacity after the Observation Phase, the jump_cwnd MUST be no
more than half of the saved_cwnd. Hence, jump_cwnd is less than
or equal to the (saved_cwnd/2). CWND = jump_cwnd.
* Unvalidated Phase (Pacing trandmission): Transmission using an
unvalidated CWND MUST use pacing.
* Unvalidated Phase (Confirming the path during transmission) If a
sender determines that the previous CC parameters are not valid
(due to a detected path change), the Safe Retreat Phase is
entered. (The sender has not yet received feedback for the jump
in CWND, because less than an RTT has passed before the
Unvalidated Phase was entered. Therefore, any detected congestion
must have resulted from packets sent before the Unvalidated
Phase.)
Kuhn, et al. Expires 24 October 2024 [Page 10]
Internet-Draft Congestion Control Convergence April 2024
* Unvalidated Phase (Receiving acknowledgements for reconnaissance
packets): The variable PipeSize is increased by the amount of data
that is acknowledged by each acknowledgment (in bytes). This
indicates a previously unacknowledged packet has been succesfully
sent over the path.
* Unvalidated Phase (Receiving acknowledgements for an unvalidated
packet): The sender enters the Validating Phase when an
acknowledgement is received for the first packet number (or
higher) that was sent in the Unvalidated Phase (see
first_unvalidated_packet_acknowledged in Section 5).
Implementation notes are provided in Section 4.3.
3.4. Validating Phase
The Validating Phase checks that the packets sent in the Unvalidated
Phase were received without inducing congestion. The CWND remains
unvalidated and the sender typically remains in this phase for one
RTT.
* Validating Phase (Check flight_size on entry): On entry to the
Validating Phase, if the flight_size is less equal to the PipeSize
(the validated window), the CWND is reset to the PipeSize and the
Normal Phase is then immediately entered. There is no need to
validate the current data in flight.
* Validating Phase (Limiting CWND on entry): On entry to the
Validating Phase, the CWND is set to the flight_size. This
corresponds to the capacity being validated (see trigger
rate_limited in Section 5).
* Validating Phase (Updating CWND): The CWND is updated using the
normal rules for the current congestion controller, this typically
allows CWND to be increased for each ACK received that indicates a
packet has been successfully sent across the path.
* Validating Phase (Receiving acknowledgements for unvalidated
packets): The variable PipeSize is increased upon each
acknowledgment that indicates a packet has been successfully sent
over the path. This records the validated PipeSize in bytes.
* Validating Phase (Congestion indication): If a sender determines
that congestion was experienced (e.g., packet loss or ECN-CE
marking), Careful Resume enters the Safe Retreat Phase (see
trigger packet_loss and ECN_CE in Section 5).
Kuhn, et al. Expires 24 October 2024 [Page 11]
Internet-Draft Congestion Control Convergence April 2024
* Validating Phase (Receiving acknowledgement for all unvalidated
packets): The sender enters the Normal Phase when an
acknowledgement is received for the last packet number (or higher)
that was sent in the Unvalidated Phase (see
last_unvalidated_packet_acknowledged in Section 5).
3.5. Safe Retreat Phase
This phase is entered when the first loss/ECN-CE marking is detected
for unvalidated packets. On entry to the Safe Retreat Phase, the
sender MUST drain the path of any other unvalidated packets before
entering the Normal Phase. (This trigger is the same as used by a
QUIC sender to transition from Slow Start to Recovery [RFC9002].)
* Safe Retreat Phase (Removing saved information): The set of saved
CC parameters for the path are deleted, to prevent these from
being used again by other flows. To avoid persistent overshoot,
on entering the Normal Phase, the congestion control algorithm
needs to itself determine the path capacity (e.g., using SS or
bandwidth probing), which MUST NOT consider capacity information
deduced from packets sent in the Unvalidated Phase.
* Safe Retreat Phase (Re-initializing CC): On entry, the CWND MUST
be reduced to no more than the (PipeSize/2). This avoids
persistent starvation by allowing capacity for other flows to
regain their share of the total capacity.
* Note: The minimum CWND in QUIC is 2 packets (see: [RFC9002]
section 4.8).
* Safe Retreat Phase (QUIC recovery): When the CWND is reduced, a
QUIC sender can immediately send a single packet prior to the
reduction [RFC9002]. (This speeds up loss recovery if the data in
the lost packet is retransmitted and is similar to TCP as
described in Section 5 of [RFC6675].)
* Safe Retreat Phase (Increasing CWND): The CWND MUST NOT be
increased in the Safe Retreat Phase.
* Safe Retreat Phase (Tracking PipeSize): The sender continues to
update the PipeSize after processing each ACK. This value is used
to reset the ssthresh when leaving this phase, it does not modify
CWND.
Kuhn, et al. Expires 24 October 2024 [Page 12]
Internet-Draft Congestion Control Convergence April 2024
* Safe Retreat Phase (Receiving acknowledgement for all unvalidated
packets): The sender enters Normal Phase when the last packet (or
later) sent during the Unvalidated Phase has been acknowledged.
The sender MUST set the ssthresh to no more than the PipeSize (see
exit_recovery in Section 5).
Implementation requirements are provided in Section 4.5.
3.5.1. Loss Recovery after entering Safe Retreat
Unacknowledged packets that were sent in the Unvalidated Phase can be
lost when there is congestion. Loss recovery commences using the
reduced CWND that was set on entry to the Safe Retreat Phase.
NOTE: A TCP or SCTP sender is required to retransmit all lost
data. For QUIC and DCCP, the need for loss recovery depends on
the sender policy for retransmission.
NOTE: During loss recovery, a receiver can cumulatively
acknowledge data that was previously sent in the Unvalidated Phase
in addition to acknowledging successful retransmission of data.
[RFC3465] describes how to appropriately account for such
acknowledgments.
NOTE: On entry to the Safe Retreat Phase, the CWND can be
significantly reduced, when there was multiple loss, recovery of
all lost data could require multiple RTTs to complete.
The sender leaves the Safe Retreat Phase when the last packet number
(or higher) sent in the Unvalidated Phase is acknowledged. If the
last packet number is not cumulatively acknowledged, then additional
packets might need to be retransmitted.
3.6. Normal Phase
In the Normal Phase, the sender transitions to using the normal CC
method (e.g., in congestion avoidance if CWND is more than ssthresh).
(Note that when the sender did not use the entire jump_cwnd the CWND
was reduced on entering the Validating Phase.
Implementation requirements are provided in Section 4.6.
3.7. RTO Expiry while using Careful Resume
A sender that experiences a Retransmission Time Out (RTO) expiry
ceases to use Careful Resume. The sender continues enters the Normal
Phase.
Kuhn, et al. Expires 24 October 2024 [Page 13]
Internet-Draft Congestion Control Convergence April 2024
NOTE: As in loss recovery, data sent in the Unvalidated Phase
could be later acknowledged after an RTO event (see
Section 3.5.1).
4. Congestion Control Guidelines and Requirements
This section provides guidance for implementation and use.
4.1. Determining the Current Path Capacity in the Observe Phase
There are various approaches to measuring the capacity used by a
connection. Congestion controllers, such as Reno or CUBIC [RFC9438],
can estimate the capacity by utilizing the CWND or flight_size. A
different approach could estimate the same parameters for a rate-
based congestion controller, such as BBR
[I-D.cardwell-iccrg-bbr-congestion-control], or by observing the rate
at which data is acknowledged by the remote endpoint.
Implementations are expected to include a LifeTime parameter in the
CC parameters that can be used to remove old CC parameters when no
longer needed, or the CC parameters are out of date.
* Observe Phase: There are cases where the current CWND does not
reflect the path capacity. At the end of slow start, the CWND can
be significantly larger than needed to fully utilize the path
(i.e., a CWND overshoot). It is inappropriate to use an overshoot
in the CWND as a basis for estimating the capacity. In most
cases, the CWND will converge to a stable value after several more
RTTs. One mitigation could be to set the saved_cwnd based on the
flight_size, or an averaged CWND.
* Observe Phase (Rate-limited sender): When the sender is rate-
limited, or in an RTT following a burst of transmission, a sender
typically transmits much less data than allowed. Such
observations could be discounted when estimating the saved_cwnd
(e.g., when a previous observation recorded a higher value.)
4.2. Confirming the Path in the Reconnaissance Phase
The CC is not modified during the Reconnaissance Phase. A sender
therefore needs to limit the initial data, sent in the first RTT of
transmitted data, to not more than the IW [RFC9000]. This
transmission using the IW is assumed to be a safe starting point for
any path to avoid adding excessive load to a potentially congested
path. (When used in a controlled network, additional information
about local path characteristics could be known, which might be used
to configure a non-standard IW.)
Kuhn, et al. Expires 24 October 2024 [Page 14]
Internet-Draft Congestion Control Convergence April 2024
The method does not permit multiple concurrent reuse of the saved CC
parameters. When multiple new concurrent connections are made to a
server, each can have a valid endpoint_token, but the saved_cwnd can
only be used for one new connection. This is designed to prevent a
sender from performing multiple jumps in the cwnd, each individually
based on the same saved_cwnd, and hence creating an excessive
aggregate load at the bottleneck.
The method used to prevent re-use of the saved CC parameters will
depend on the design of the server that is being used (e.g., if all
connections from a given client IP arrive at the same server process,
then the server process could use a hash table). A distributed
system might be required when using some types of load balancing, to
ensure this invariant when the load balancing hashes connections by
4-tuple and hence multiple connections from the same client device
are served by different server processes.
In the Reconnaissance Phase a sender initiates a connection and
starts sending initial data. This measures the current minimum RTT.
If a decision is made to use Careful Resume, this is used to confirm
the path.
4.2.1. Confirming the RTT
Path characteristics can change over time for many reasons, resulting
in the previously observed CC parameters becoming irrelevant. The
sender therefore compares the saved_RTT with each of a series of
measured RTT samples.
If the current RTT sample is less than a half of the saved_RTT, this
is regarded as too small, and is an indicator of a path change.
(This factor of two arises, because the rate should not exceed the
observed rate when the saved_cwnd was measured, because the jump_cwnd
is calculated as half the measured saved_cwnd.)
A current RTT larger than that at the time the saved_cwnd was
measured results in a proportionally lower resumed rate, because the
transmission using the CR method is paced based on the current RTT
(i.e., the larger RTT samples reduces the paced sending rate in the
Unvalidated Phase - and hence is still safe).
Also note that when the RTT is incorrectly measured as larger than
the actual path RTT, the sender will receive an acknowledgment for an
unvalidated packet before it might have completed the Unvalidated
Phase, this acknowledgment resets the CWND to reflect the
flight_size, and the sender enters the Validating Phase.
Kuhn, et al. Expires 24 October 2024 [Page 15]
Internet-Draft Congestion Control Convergence April 2024
an RTT sample more than ten times the saved_RTT is indicative of a
path change. (The value of ten was chosen to accommodate both
increases in latency from buffering on a path, and any variation
between RTT samples).
A sender in Reconnaissance Phase reverts to the Normal Phase if
congestion is detected. Some transport protocols implement methods
that infer potential congestion from an increase in the RTT. In the
Reconnaissance Phase, this indication occurs earlier than congestion
which is reported by loss or by ECN marking. Designs need to
consider if this is a suitable trigger to revert to the Normal Phase.
4.3. Safety Requirements for the Unvalidated Phase
This section defines the safety requirements for using saved CC
parameters to tentatively update the CWND. These safety requirements
mitigate the risk of adding excessive congestion to an already
congested path.
* Unvalidated Phase (Jump): A connection must not directly use the
previously saved_cwnd to directly initialize a new flow causing it
to resume sending at the same rate. The jump_cwnd must be no more
than half the previously saved_cwnd.
4.3.1. Lifetime of CC Parameters
The long-term use of the previously observed parameters is not
appropriate, a lifetime therefore needs to be specified during which
the saved CC parameters can be safely re-used.
[RFC9040] provides guidance on the implementation of TCP Control
Block Interdependence, but does not specify how long a saved
parameter can safely be reused.
[RFC7661] specifies a method for managing an unvalidated CWND. This
states: "After a fixed period of time (the non-validated period
(NVP)), the sender adjusts the cwnd (Section 4.4.3). The NVP SHOULD
NOT exceed five minutes." Section 5 of [RFC7661] discusses the
rationale for choosing that period. However, RFC 7661 targets rate-
limited connections using normal CC. The method described in the
present specification includes additional mechanisms to avoid and
mitigate the effects of overshoot, and therefore this can be used to
justify a longer lifetime of the saved_cwnd using the Careful Resume
method.
Kuhn, et al. Expires 24 October 2024 [Page 16]
Internet-Draft Congestion Control Convergence April 2024
4.3.2. Pacing in the Unvalidated Phase
The sender must avoid sending a burst of packets greater than IW as a
result of a step-increase in the CWND. (This is consistent with
[RFC8085], [RFC9000]). Pacing sent packets as a function of the
current RTT, rather than the saved_RTT, provides an additional safety
during the Unvalidated Phase. Other sender mitigations have also
been suggested to avoid line-rate bursts (e.g.,
[I-D.hughes-restart]).
Pacing places a limitation on the minimum acceptable current_RTT to
avoid sending at a rate higher than was previously observed.
The following example provides a relevant pacing rhythm using the RTT
and the saved_cwnd. The Inter-packet Transmission Time (ITT) is
determined by using the current Maximum Message Size (MMS), the
saved_cwnd and the RTT. A safety margin can avoid sending more than
a recommended maximum (max_jump):
jump_cwnd = min(max_jump,saved_cwnd/2)
ITT = (current_RTT x MMS)/jump_cwnd
This follows the idea presented in [RFC4782],
[I-D.irtf-iccrg-sallantin-initial-spreading] and [CONEXT15].
4.3.3. Exit from the Unvalidated Phase because of Variable Network
Conditions
* Careful Resume has been designed to be robust to changes in
network conditions due to variations in the forwarding path, such
as reconfiguration of equipment, or changes in the link
conditions. This is mitigated by path confirmation.
* Careful Resume has been designed to be robust to changes in
network traffic, including the arrival of new flows that compete
for capacity at a shared bottleneck. This is mitigated by jumping
to no more than a half of the saved_cwnd and by using pacing.
* Careful Resume has been designed to prevent unduly suppressing
flows that used the capacity since the available capacity was
measured. This is further mitigated by bounding the duration of
the Unvalidated Phase (and the following Validating Phase).
Kuhn, et al. Expires 24 October 2024 [Page 17]
Internet-Draft Congestion Control Convergence April 2024
4.4. Safety Requirements for the Validating Phase
When a sender completes the Unvalidated Phase, either by sending a
jump_cwnd of data or after one RTT, it ceases to use the unvalidated
CWND. That is, CWND is reset to the flight_size, and the sender
awaits reception of the acknowledgments to validate the use of this
capacity. New packets are sent when previously sent data is newly
acknowledged. The purpose is to trigger a entry to the Safe Retreat
Phase when the capacity is not validated.
Note that the CWND is increased during the Validating Phase, based on
received ACKs. This allows new data to be sent, but this does not
have any final impact on the CWND if congestion is detected.
4.5. Safety Requirements for the Safe Retreat Phase
This section defines the safety requirements after congestion has
been detected during the Unvalidated Phase.
The Safe Retreat Phase sets a safe CWND value to drain unvalidated
packets from the path when a packet loss is detected or
acknowledgments indicate sent packets were ECN CE-marked.
The Safe Retreat reaction differs from a traditional reaction to
detected congestion, because the jump_cwnd can result in a
significantly higher rate than would be allowed by the slow-start
mechanism. This could aggressively feed a congested bottleneck,
resulting in overshoot where a disproportionate number of packets
from existing flows are displaced from the buffer at the congested
bottleneck. For this reason, a sender needs to react to detected
congestion by reducing CWND significantly below the saved_cwnd.
Note: Proportional Rate Reduction (PRR) [RFC6937] assumes that it is
safe to reduce the rate gradually when in congestion avoidance. PRR
is therefore not appropriate when there might be significant
overshoot in the use of the capacity, which can be the case when the
Safe Retreat Phase is entered.
Acknowledgements for unvalidated packets are tracked to measure the
maximum capacity, called the PipeSize (The first unvalidated packet
can be determined by recording the sequence number of the first
packet sent in this phase.) This PipeSize is not a safe measure of
the currently available share of the capacity whenever there was also
a significant overshoot at the bottleneck, but may be used to reset
ssthresh.
Kuhn, et al. Expires 24 October 2024 [Page 18]
Internet-Draft Congestion Control Convergence April 2024
4.6. Returning to Normal Congestion Control
After using Careful Resume, the CC controller returns to the Normal
Phase. The implementation details for different transports depend on
the design of the transport.
In the Normal Phase, a sender can enter the Observation Phase to
perform observation of the path.
{XXX-Editor note: A future revision should discuss updating the saved
parameters, whether used or not, after reaching normal operation for
use the next time even if that update is to just refresh the
expiration time.}
4.7. Limitations from Transport Protocols
A sender is limited by any rate-limitation of the transport protocol
that is used.
For QUIC this includes flow control mechanisms or preventing
amplification attacks. In particular, a QUIC receiver might need
to issue proactive MAX_DATA frames to increase the flow control
limits of a connection that is started when using Careful Resume
to gain the expected benefit.
A TCP sender is limited by the receiver window (rwnd). Unless
configured at a receiver, the rwnd constrains the rate of increase
for a connection and reduces the benefit of Careful Resume.
5. QLOG support for QUIC
This section provides definitions that enable the Careful Resume
method to generate qlog events when using QUIC. It introduces an
event to report the current phase of a sender, and the associated
description.
The event and data structure definitions in this section are
expressed in the Concise Data Definition Language (CDDL) [RFC8610]
and its extensions described in [I-D.ietf-quic-qlog-quic-events].
The current convention is to use long names for variables. For
example, "cwnd" is expanded as "congestion_window" and "saved_cwnd"
is expanded as "saved_congestion_window" below.
5.1. cr_phase Event
Importance: Extra
Kuhn, et al. Expires 24 October 2024 [Page 19]
Internet-Draft Congestion Control Convergence April 2024
When the CC algorithm changes the Careful Resume Phase described in
Section 3 of this specification.
Definition:
RecoveryCarefulResumePhaseUpdated = {
? old_phase: CarefulResumePhase,
new_phase: CarefulResumePhase,
state_data: CarefulResumeStateParameters,
? restored_data: CarefulResumeRestoredParameters,
? trigger:
; for the Unvalidated Phase, when no unvalidated packets
"congestion_window_limited" /
; for Validating Phase
"first_unvalidated_packet_acknowledged" /
; for Normal Phase, no unvalidated packets to be acknowledged
"last_unvalidated_packet_acknowledged" /
; for Normal Phase, when CR not allowed
"rtt_not_validated" /
; for Normal Phase, fewer packets than CWND permits
"rate_limited" /
; for Safe Retreat Phase, when loss detected
"packet_loss" /
; for Safe Retreat Phase, when ECN congestion experienced
"ECN_CE" /
; for Normal Phase 1 RTT after a congestion event
"exit_recovery"
}
CarefulResumePhase =
"reconnaissance" /
"unvalidated" /
"validating" /
"normal" /
"safe_retreat"
CarefulResumeStateParameters = {
pipesize: uint,
first_unvalidated_packet: uint,
last_unvalidated_packet: uint,
? congestion_window: uint,
? ssthresh: uint
}
CarefulResumeRestoredParameters = {
saved_congestion_window: uint,
saved_rtt: float32
}
Kuhn, et al. Expires 24 October 2024 [Page 20]
Internet-Draft Congestion Control Convergence April 2024
Figure 2: Definitions of qlog events.
6. Acknowledgments
The authors would like to thank John Border, Gabriel Montenegro,
Patrick McManus, Ian Swett, Igor Lubashev, Robin Marx, Roland Bless,
Franklin Simo, Kazuho Oku, Tong, Ana Custura, Neal Cardwell, and
Joerg Deutschmann for their fruitful comments on earlier versions of
this document.
The authors would like to particularly thank Tom Jones for co-
authoring several previous versions of this document. Ana Custura
and Robin Marx developed the qlog support.
7. IANA Considerations
No current parameters are required to be registered by IANA.
8. Security Considerations
This document does not exhibit specific security considerations.
Security considerations for the interactions with the receiver are
discussed in [I-D.kuhn-quic-bdpframe-extension].
9. References
9.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC8085] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage
Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085,
March 2017, <https://www.rfc-editor.org/info/rfc8085>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
[RFC8610] Birkholz, H., Vigano, C., and C. Bormann, "Concise Data
Definition Language (CDDL): A Notational Convention to
Express Concise Binary Object Representation (CBOR) and
JSON Data Structures", RFC 8610, DOI 10.17487/RFC8610,
June 2019, <https://www.rfc-editor.org/info/rfc8610>.
Kuhn, et al. Expires 24 October 2024 [Page 21]
Internet-Draft Congestion Control Convergence April 2024
[RFC8801] Pfister, P., Vyncke, É., Pauly, T., Schinazi, D., and W.
Shao, "Discovering Provisioning Domain Names and Data",
RFC 8801, DOI 10.17487/RFC8801, July 2020,
<https://www.rfc-editor.org/info/rfc8801>.
[RFC9000] Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based
Multiplexed and Secure Transport", RFC 9000,
DOI 10.17487/RFC9000, May 2021,
<https://www.rfc-editor.org/info/rfc9000>.
[RFC9438] Xu, L., Ha, S., Rhee, I., Goel, V., and L. Eggert, Ed.,
"CUBIC for Fast and Long-Distance Networks", RFC 9438,
DOI 10.17487/RFC9438, August 2023,
<https://www.rfc-editor.org/info/rfc9438>.
9.2. Informative References
[CONEXT15] Li, Q., Dong, M., and P B. Godfrey, "Halfback: Running
Short Flows Quickly and Safely", ACM CoNEXT , 2015.
[I-D.cardwell-iccrg-bbr-congestion-control]
Cardwell, N., Cheng, Y., Yeganeh, S. H., Swett, I., and V.
Jacobson, "BBR Congestion Control", Work in Progress,
Internet-Draft, draft-cardwell-iccrg-bbr-congestion-
control-02, 7 March 2022,
<https://datatracker.ietf.org/doc/html/draft-cardwell-
iccrg-bbr-congestion-control-02>.
[I-D.hughes-restart]
Hughes, A., Touch, J., and J. Heidemann, "Issues in TCP
Slow-Start Restart After Idle", Work in Progress,
Internet-Draft, draft-hughes-restart-00, December 2001,
<https://www.ietf.org/archive/id/draft-hughes-restart-
00.txt>.
[I-D.ietf-quic-qlog-quic-events]
Marx, R., Niccolini, L., Seemann, M., and L. Pardue, "QUIC
event definitions for qlog", Work in Progress, Internet-
Draft, draft-ietf-quic-qlog-quic-events-07, 4 March 2024,
<https://datatracker.ietf.org/doc/html/draft-ietf-quic-
qlog-quic-events-07>.
Kuhn, et al. Expires 24 October 2024 [Page 22]
Internet-Draft Congestion Control Convergence April 2024
[I-D.irtf-iccrg-sallantin-initial-spreading]
Sallantin, R., Baudoin, C., Arnal, F., Dubois, E., Chaput,
E., and A. Beylot, "Safe increase of the TCP's Initial
Window Using Initial Spreading", Work in Progress,
Internet-Draft, draft-irtf-iccrg-sallantin-initial-
spreading-00, 15 January 2014,
<https://datatracker.ietf.org/doc/html/draft-irtf-iccrg-
sallantin-initial-spreading-00>.
[I-D.kuhn-quic-bdpframe-extension]
Kuhn, N., Emile, S., Fairhurst, G., Secchi, R., and C.
Huitema, "Signalling CC Parameters for Careful Resume
using QUIC", Work in Progress, Internet-Draft, draft-kuhn-
quic-bdpframe-extension-05, 4 March 2024,
<https://datatracker.ietf.org/doc/html/draft-kuhn-quic-
bdpframe-extension-05>.
[IJSCN] Thomas, L., Dubois, E., Kuhn, N., and E. Lochin, "Google
QUIC performance over a public SATCOM access",
International Journal of Satellite Communications and
Networking 10.1002/sat.1301, 2019.
[MAPRG111] Kuhn, N., Stephan, E., Fairhurst, G., Jones, T., and C.
Huitema, "Feedback from using QUIC's 0-RTT-BDP extension
over SATCOM public access", IETF 111 - MAPRG meeting ,
2022.
[RFC3465] Allman, M., "TCP Congestion Control with Appropriate Byte
Counting (ABC)", RFC 3465, DOI 10.17487/RFC3465, February
2003, <https://www.rfc-editor.org/info/rfc3465>.
[RFC4782] Floyd, S., Allman, M., Jain, A., and P. Sarolahti, "Quick-
Start for TCP and IP", RFC 4782, DOI 10.17487/RFC4782,
January 2007, <https://www.rfc-editor.org/info/rfc4782>.
[RFC5783] Welzl, M. and W. Eddy, "Congestion Control in the RFC
Series", RFC 5783, DOI 10.17487/RFC5783, February 2010,
<https://www.rfc-editor.org/info/rfc5783>.
[RFC6675] Blanton, E., Allman, M., Wang, L., Jarvinen, I., Kojo, M.,
and Y. Nishida, "A Conservative Loss Recovery Algorithm
Based on Selective Acknowledgment (SACK) for TCP",
RFC 6675, DOI 10.17487/RFC6675, August 2012,
<https://www.rfc-editor.org/info/rfc6675>.
[RFC6937] Mathis, M., Dukkipati, N., and Y. Cheng, "Proportional
Rate Reduction for TCP", RFC 6937, DOI 10.17487/RFC6937,
May 2013, <https://www.rfc-editor.org/info/rfc6937>.
Kuhn, et al. Expires 24 October 2024 [Page 23]
Internet-Draft Congestion Control Convergence April 2024
[RFC7661] Fairhurst, G., Sathiaseelan, A., and R. Secchi, "Updating
TCP to Support Rate-Limited Traffic", RFC 7661,
DOI 10.17487/RFC7661, October 2015,
<https://www.rfc-editor.org/info/rfc7661>.
[RFC8867] Sarker, Z., Singh, V., Zhu, X., and M. Ramalho, "Test
Cases for Evaluating Congestion Control for Interactive
Real-Time Media", RFC 8867, DOI 10.17487/RFC8867, January
2021, <https://www.rfc-editor.org/info/rfc8867>.
[RFC9002] Iyengar, J., Ed. and I. Swett, Ed., "QUIC Loss Detection
and Congestion Control", RFC 9002, DOI 10.17487/RFC9002,
May 2021, <https://www.rfc-editor.org/info/rfc9002>.
[RFC9040] Touch, J., Welzl, M., and S. Islam, "TCP Control Block
Interdependence", RFC 9040, DOI 10.17487/RFC9040, July
2021, <https://www.rfc-editor.org/info/rfc9040>.
[RFC9406] Balasubramanian, P., Huang, Y., and M. Olson, "HyStart++:
Modified Slow Start for TCP", RFC 9406,
DOI 10.17487/RFC9406, May 2023,
<https://www.rfc-editor.org/info/rfc9406>.
Appendix A. Notes on the Careful Resume Phases
The table below is provided to illustrate the operation of Careful
Resume. This table is informative, please refer to the body of the
document for the normative specification. The description is based
on a Normal CC that uses Reno or CUBIC. The PipeSize tracks the
validated CWND.
Kuhn, et al. Expires 24 October 2024 [Page 24]
Internet-Draft Congestion Control Convergence April 2024
+------+---------+---------+------------+-----------+------------+
|Phase |Normal |Recon. |Unvalidated |Validating |Safe Retreat|
+------+---------+---------+------------+-----------+------------+
| |Observe |Confirm |Send faster |Validate |Drain path; |
| |CC params|path |using saved |new CWND; |Update PS |
| | | | |Update PS | |
+------+---------+---------+------------+-----------+------------+
|On | - |CWND=IW |PS=CWND; |If (FS>PS) |CWND=(PS/2) |
|entry:| | |CWND |{CWND=FS} | |
| | | |=jump_cwnd |else | |
| | | | |{CWND=PS; | |
| | | | |enter |
| | | | |Normal} | |
+------+---------+---------+------------+-----------+------------+
|CWND: |When in |CWND |CWND is not |CWND can |CWND is not |
| |observe, |increases|increased |increase |increased |
| |measure |using | |using | |
| |sav_cwnd |normal CC| |normal CC | |
| | |methods | |methods | |
+------+---------+---------+------------+-----------+------------+
|PS: | - | - | PS+=ACked |
+------+---------+---------+------------+-----------+------------+
|RTT: |Measure |Measure | - | - | - |
| |saved_rtt|current | | | |
| | |_rtt | | | |
+------+---------+---------+------------+-----------+------------+
|If |Normal |Normal | Enter | - |
|loss |CC method|CC method| Safe | |
|or | |CR is not| Retreat | |
|ECNCE:| |allowed | | |
+------+---------+---------+------------+-----------+------------+
|Next |Observe |If ( |If (FS=CWND |If (ACK |If (ACK |
|Phase:|(as |FS=CWND |or >1 RTT |>= last |>= last |
| |needed) |and CR |has passed |unvalidated|unvalidated |
| | |confirmed|or ACK |packet), |packet), |
| | |), enter |>= last |enter |{ssthresh=PS|
| | |Unvalidat|unvalidated |Normal |and enter |
| | |-ing else|packet), | |Normal} |
| | |enter |enter | | |
| | |Normal |Validating | | |
+------+---------+---------+------------+-----------+------------+
Figure 3: Illustration of the operation of Careful Resume
The following abbreviations are used SS = slow start FS= flight_size;
PS = PipeSize. The PipeSize is set to the CWND on entry to the
Unvalidated Phase. This tracks the validated portion of the CWND and
is updated as each additional packet is acknowledged.
Kuhn, et al. Expires 24 October 2024 [Page 25]
Internet-Draft Congestion Control Convergence April 2024
Note: For an implementation that keeps track of transmitted data in
terms of packets: In the Unvalidated Phase, the first unvalidated
packet corresponds to the highest sent packet recorded on entry to
this phase. In the Validating and Safe Retreat Phases, corresponds
to the last unvalidated packet is also the highest sent packet
recorded on entry to this phase.
The remaining subsections provide informative examples of use.
Although the QLOG variables are expressed in bytes, to simplify the
description, these examples are described in term of packet numbers.
A.1. Example with No Loss
In the first example of using the Careful Resume method, the sender
starts by sending IW (10) packets in the Reconnaissance Phase, and
then continues in a subsequent RTT to send more packets until the
sender becomes CWND-limited (i.e., flight_size = CWND).
The sender then confirms the RTT and other conditions for using
Careful Resume. In this example, this is confirmed when the sender
has 29 packets in flight. The sender enters the Unvalidated Phase.
This confirmation could have happened earlier if data was available
to send. The sender initialises the PipeSize to the CWND (the same
as the flight_size, i.e., 29 packets) and sets the CWND to 150
packets (based upon a previously observed saved_cwnd of 300 packets).
It now sends 121 unvalidated packets. This is the unused portion of
the CWND. Each time a packet is sent, the sender checks whether 1
RTT has passed since entering the Unvalidated Phase (otherwise, the
Validating Phase is entered). This check triggers only for cases
where the sender is rate-limited, see the following example.
The PipeSize will increase after each ACK is received.
When the first unvalidated packet is acknowledged (packet number 30)
the sender enters the Validating Phase. This transition would also
occur if the flight_size increases to equal CWND. During this phase,
the CWND can be increased for each ACK for an unvalidated packet,
because this indicates that the packet was indeed validated.
When an ACK is received for the last packet sent in the Unvalidated
Phase, the sender completes using Careful Resume. It enters the
Normal Phase. If CWND is less than ssthresh, a Reno or CUBIC sender
in the Normal Phase is permitted to use slow start to grow the CWND
towards the ssthresh, and will then enter congestion avoidance.
Kuhn, et al. Expires 24 October 2024 [Page 26]
Internet-Draft Congestion Control Convergence April 2024
A.2. Example with No Loss, Rate-Limited
A rate-limited sender will not fully utilize the available CWND when
using Careful Resume, and CWND is therefore reset on entry to the
Validating Phase, as described below.
The sender in this example starts by sending IW packets (10) in the
Reconnaissance Phase, and sends as described in the first example,
transitioning to the Unvalidated Phase. This sets the CWND to 150
packets, and the PipeSize is set to the flight_size (i.e., 29
packets).
The sender then becomes rate-limited because it only sends 50
unvalidated packets.
After ~1 RTT (detected by using local timestamps or by receiving an
ACK for the first unvalidated packet), the sender will still not have
fully used the CWND. It then enters the Validating Phase and resets
the CWND to the current flight_size, (i.e., 50 packets). During this
phase, the CWND can be increased for each ACK that validates
reception of a packet. The PipeSize also increases with each ACK
received, to reflect the discovered capacity.
When an ACK is received for the last packet sent in the Unvalidated
Phase, the sender has completed using Careful Resume. It enters the
Normal Phase. If CWND is less than ssthresh, a Reno or CUBIC sender
in the Normal Phase is permitted to use slow start to grow the CWND
towards the ssthresh, and will then enter congestion avoidance.
A.3. Example with Loss detected in the Reconnaissance Phase
When a packet is lost in the Reconnaissance Phase, the sender enters
the Normal Phase and recovers this using normal method. There is no
change to the CC method, because the sender has discovered a
potential capacity limit and is not allowed to use Careful Resume.
A.4. Example with Loss detected in the Validating Phase
As in the first Example the sender enters the Unvalidated Phase it
sets the CWND to 150 packets with the PipeSize initialized to the
flight_size (i.e., 29 packets).
The sender now sends 121 unvalidated packets (the remaining unused
CWND). This example considers the case when one of the unvalidated
packet is lost, which we choose to be packet 64 (the 35th packet in
the Unvalidated Phase).
Kuhn, et al. Expires 24 October 2024 [Page 27]
Internet-Draft Congestion Control Convergence April 2024
Acknowledgements confirm the first 34 unvalidated packets are
received without loss. The PipeSize at this point is equal to 29 +
34 = 63 packets.
The loss is then detected (by receiving three ACKs that do not cover
packet number 35), the sender then enters the Safe Retreat Phase
because the window was not validated. The PipeSize at this point is
equal to 29 + 34 = 66 packets. Assuming IW=10. The CWND is reset to
max(10,ps/2) = max(10,66/2) = 33 packets. This CWND is used during
the Safe Retreat Phase, because congestion was detected and the
sender still does not know if the remaining unvalidated packets will
be successfully acknowledged. A conservative CWND calculation
ensures the sender drains the path after this potentially severe
congestion event. There is no further increase in CWND in this
phase.
The sender continues to receives ACKs for the remaining 86 (121-35)
unvalidated packets (recall that the 35th unvalidated packet was lost
and had packet number 29+35=64). The PipeSize continues to further
increase as each ACK acknowledges new data, because this tracks the
size of the pipe discovered by the unvalidated packets. Although
this cannot be used to safety initialise CWND (because was measured
when the sender aggressively created overload), the estimated
PipeSize (which, in this case, is 121-1 = 120 packets) can be used to
set the ssthresh on exit from Safe Retreat, since it indicates an
upper limit to the current capacity.
At the point where all packets sent in the Unvalidated Phase have
been either acknowledged or declared lost, the sender enters the
Normal Phase, but first updates ssthresh. Because CWND will now be
less than ssthresh, a sender in the Normal Phase is permitted to use
slow start to grow the CWND towards the ssthresh, after which it will
enter congestion avoidance.
Appendix B. Appendix: An Endpoint Token
This annex proposes an Endpoint Token to allow a sender to identify
its own view of the network path that it is using. In
[I-D.kuhn-quic-bdpframe-extension] this Endpoint Token could be
shared and used as an opaque path identifier to other parties and the
sender can verify if this is one of its current paths.
B.1. Creating an Endpoint Token
When computing the Endpoint Token, the sender includes information to
identify the path on which it sends, for example, this:
Kuhn, et al. Expires 24 October 2024 [Page 28]
Internet-Draft Congestion Control Convergence April 2024
* needs to include a unique identifier for itself (e.g., a globally
assigned address/prefix; or randomly chosen value such as a
nonce);
* needs to include an identifier for the destination (e.g., a
destination IP address or name);
* needs to include an interface identifier (e.g., an index value or
a MAC address to associate the endpoint with the interface on
which the path starts);
* could include other information such as the DSCP, ports, flow
label, etc (recognising that this additional information might
improve the path differentiation, but that this can reduce the re-
usability of the token);
* this could include any other information the sender chooses to
include, and potentially including PvD information [RFC8801] or
information relating to its public-facing IP address;
When creating an Endpoint Token, the sender has to ensure the
following:
1. To reduce the likelihood of misuse of the Endpoint Token, the
value ought to be encoded in a way that hides the component
information from the recipient and any eavesdropper on the path
(this could already protected by methods such as TLS).
2. The sender can recalculate the Endpoint Token to validate a
previously issued token; and can be included in the computed
integrity check for any path information it provides.
3. The Endpoint Token is designed so that if shared, it prevents
another party from deriving private data from the token, or to
use the token to perform unwanted likability with other
information. Therefore, the Endpoint Token MUST necessarily be
different when used to identify paths using different interfaces.
Appendix C. Appendix: Revision details
Previous individual submissions were discussed in TSVWG and QUIC.
WG -00 included clarifications and restructuring to form the 1st
WG draft.
WG -01 included review comments and suggestions from John Border,
and follows the setting of the TSVWG milestone with an intended
status of "Proposed Standard".
Kuhn, et al. Expires 24 October 2024 [Page 29]
Internet-Draft Congestion Control Convergence April 2024
WG -02 includes steps to complete the spec. In particular,
consideration of rate-limited senders; selection of reasoned
parameters; specification of the Safe Retreat Phase; and
improvements to the consistency throughout. Added the Validating
Phase.
WG -03, explain entry to Validating Phase, editorial tidy.
WG -04, update based on review comments from Kazuho Oku.
WG-05, update based on review comments from Neal Cardwell. WG
feedback from IETF-118. Reviewed the requirements v. guidelines;
clarified that CC is not changed in recon., but the recon. info is
used to steer the next phase; clarified saved_cwnd can be computed
from ACK rate; use jump once; that real server platforms are
complex. Clarified lifetime for saved CC params. Incorporates
comments from Tong.
WG-06, SR updated following Hackathon comments from Kazuho Oku,
and rework of use of PipeSize. Added an informative summary of
actions, on suggestion by Tong. Added examples based on text by
Ana Custura.
WG-07, Use "rate-limited" uniformaly instead of application and
data limited.
Updated to exit early when unvalidated CWND not utilised, detected
in tests by Q Misell. Change pipe_size to be PipeSize.
WG-08, Updated CDDL (AC), and made constraints in the Observe
Phase into guidance, they say what makes sense - but do not need
to be followed for conformance. Updated table in annexe to align
with text. Formatted CDDL and added cross-refs to triggers in the
text (from a comments by JD and AC). Fixed typos and consistency
(JD, RS, GF).
Authors' Addresses
Nicolas Kuhn
Thales Alenia Space
Email: nicolas.kuhn.ietf@gmail.com
Emile Stephan
Orange
Email: emile.stephan@orange.com
Kuhn, et al. Expires 24 October 2024 [Page 30]
Internet-Draft Congestion Control Convergence April 2024
Godred Fairhurst
University of Aberdeen
Department of Engineering
Fraser Noble Building
Aberdeen
AB24 3UE
United Kingdom
Email: gorry@erg.abdn.ac.uk
Raffaello Secchi
University of Aberdeen
Department of Engineering
Fraser Noble Building
Aberdeen
AB24 3UE
United Kingdom
Email: r.secchi@erg.abdn.ac.uk
Christian Huitema
Private Octopus Inc.
Email: huitema@huitema.net
Kuhn, et al. Expires 24 October 2024 [Page 31]