Identifying Modified Explicit Congestion Notification (ECN) Semantics for Ultra-Low Queuing Delay (L4S)
draft-ietf-tsvwg-ecn-l4s-id-09
| The information below is for an old version of the document |
| Document |
Type |
|
Active Internet-Draft (tsvwg WG)
|
|
Authors |
|
Koen De Schepper
,
Bob Briscoe
|
|
Last updated |
|
2020-02-20
(latest revision 2019-11-04)
|
|
Replaces |
|
draft-briscoe-tsvwg-ecn-l4s-id
|
|
Stream |
|
Internet Engineering Task Force (IETF)
|
|
Formats |
|
pdf
htmlized (tools)
htmlized
bibtex
|
| Stream |
WG state
|
|
WG Document
(wg milestone:
Oct 2021 - Submit "Identifying ...
)
|
|
Document shepherd |
|
Wesley Eddy
|
| IESG |
IESG state |
|
I-D Exists
|
|
Consensus Boilerplate |
|
Unknown
|
|
Telechat date |
|
|
|
Responsible AD |
|
(None)
|
|
Send notices to |
|
Wesley Eddy <wes@mti-systems.com>
|
Transport Services (tsv) K. De Schepper
Internet-Draft Nokia Bell Labs
Intended status: Experimental B. Briscoe, Ed.
Expires: August 23, 2020 Independent
February 20, 2020
Identifying Modified Explicit Congestion Notification (ECN) Semantics
for Ultra-Low Queuing Delay (L4S)
draft-ietf-tsvwg-ecn-l4s-id-09
Abstract
This specification defines the identifier to be used on IP packets
for a new network service called low latency, low loss and scalable
throughput (L4S). It is similar to the original (or 'Classic')
Explicit Congestion Notification (ECN). 'Classic' ECN marking was
required to be equivalent to a drop, both when applied in the network
and when responded to by a transport. Unlike 'Classic' ECN marking,
for packets carrying the L4S identifier, the network applies marking
more immediately and more aggressively than drop, and the transport
response to each mark is reduced and smoothed relative to that for
drop. The two changes counterbalance each other so that the
throughput of an L4S flow will be roughly the same as a non-L4S flow
under the same conditions. Nonetheless, the much more frequent
control signals and the finer responses to them result in ultra-low
queuing delay (sub-millisecond) for L4S traffic without compromising
link utilization, and this low delay can be maintained during high
traffic load.
The L4S identifier defined in this document is the key piece that
distinguishes L4S from TCP-Reno-Friendly (or 'Classic') traffic. It
gives an incremental migration path so that suitably modified network
bottlenecks can distinguish and isolate existing Classic traffic from
L4S traffic to prevent it from degrading the ultra-low queuing delay
and loss of the new scalable transports, without harming Classic
performance. Examples of new active queue management (AQM) marking
algorithms and examples of new transports (whether TCP-like or real-
time) are specified separately.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
De Schepper & Briscoe Expires August 23, 2020 [Page 1]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 23, 2020.
Copyright Notice
Copyright (c) 2020 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Latency, Loss and Scaling Problems . . . . . . . . . . . 4
1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 7
1.3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2. Consensus Choice of L4S Packet Identifier: Requirements . . . 9
3. L4S Packet Identification at Run-Time . . . . . . . . . . . . 10
4. Prerequisite Transport Layer Behaviour . . . . . . . . . . . 10
4.1. Prerequisite Codepoint Setting . . . . . . . . . . . . . 10
4.2. Prerequisite Transport Feedback . . . . . . . . . . . . . 10
4.3. Prerequisite Congestion Response . . . . . . . . . . . . 11
5. Prerequisite Network Node Behaviour . . . . . . . . . . . . . 13
5.1. Prerequisite Classification and Re-Marking Behaviour . . 13
5.2. The Meaning of L4S CE Relative to Drop . . . . . . . . . 14
5.3. Exception for L4S Packet Identification by Network Nodes
with Transport-Layer Awareness . . . . . . . . . . . . . 15
5.4. Interaction of the L4S Identifier with other Identifiers 15
5.4.1. DualQ Examples of Other Identifiers Complementing L4S
Identifiers . . . . . . . . . . . . . . . . . . . . . 15
5.4.1.1. Inclusion of Additional Traffic with L4S . . . . 15
5.4.1.2. Exclusion of Traffic From L4S Treatment . . . . . 17
5.4.1.3. Generalized Combination of L4S and Other
De Schepper & Briscoe Expires August 23, 2020 [Page 2]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
Identifiers . . . . . . . . . . . . . . . . . . . 17
5.4.2. Per-Flow Queuing Examples of Other Identifiers
Complementing L4S Identifiers . . . . . . . . . . . . 19
6. L4S Experiments . . . . . . . . . . . . . . . . . . . . . . . 19
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20
8. Security Considerations . . . . . . . . . . . . . . . . . . . 20
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 20
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 20
10.1. Normative References . . . . . . . . . . . . . . . . . . 20
10.2. Informative References . . . . . . . . . . . . . . . . . 21
Appendix A. The 'Prague L4S Requirements' . . . . . . . . . . . 27
A.1. Requirements for Scalable Transport Protocols . . . . . . 28
A.1.1. Use of L4S Packet Identifier . . . . . . . . . . . . 28
A.1.2. Accurate ECN Feedback . . . . . . . . . . . . . . . . 28
A.1.3. Fall back to Reno-friendly congestion control on
packet loss . . . . . . . . . . . . . . . . . . . . . 29
A.1.4. Fall back to Reno-friendly congestion control on
classic ECN bottlenecks . . . . . . . . . . . . . . . 30
A.1.5. Reduce RTT dependence . . . . . . . . . . . . . . . . 30
A.1.6. Scaling down to fractional congestion windows . . . . 31
A.1.7. Measuring Reordering Tolerance in Time Units . . . . 31
A.2. Scalable Transport Protocol Optimizations . . . . . . . . 34
A.2.1. Setting ECT in TCP Control Packets and
Retransmissions . . . . . . . . . . . . . . . . . . . 34
A.2.2. Faster than Additive Increase . . . . . . . . . . . . 34
A.2.3. Faster Convergence at Flow Start . . . . . . . . . . 35
Appendix B. Alternative Identifiers . . . . . . . . . . . . . . 35
B.1. ECT(1) and CE codepoints . . . . . . . . . . . . . . . . 36
B.2. ECN Plus a Diffserv Codepoint (DSCP) . . . . . . . . . . 39
B.3. ECN capability alone . . . . . . . . . . . . . . . . . . 41
B.4. Protocol ID . . . . . . . . . . . . . . . . . . . . . . . 41
B.5. Source or destination addressing . . . . . . . . . . . . 41
B.6. Summary: Merits of Alternative Identifiers . . . . . . . 42
Appendix C. Potential Competing Uses for the ECT(1) Codepoint . 42
C.1. Integrity of Congestion Feedback . . . . . . . . . . . . 43
C.2. Notification of Less Severe Congestion than CE . . . . . 44
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 44
1. Introduction
This specification defines the identifier to be used on IP packets
for a new network service called low latency, low loss and scalable
throughput (L4S). It is similar to the original (or 'Classic')
Explicit Congestion Notification (ECN [RFC3168]). RFC 3168 required
an ECN mark to be equivalent to a drop, both when applied in the
network and when responded to by a transport. Unlike Classic ECN
marking, the network applies L4S marking more immediately and more
aggressively than drop, and the transport response to each mark is
De Schepper & Briscoe Expires August 23, 2020 [Page 3]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
reduced and smoothed relative to that for drop. The two changes
counterbalance each other so that the throughput of an L4S flow will
be roughly the same as a non-L4S flow under the same conditions.
Nonetheless, the much more frequent control signals and the finer
responses to them result in ultra-low queuing delay without
compromising link utilization, and this low delay can be maintained
during high load. Ultra-low queuing delay means less than 1
millisecond (ms) on average and less than about 2 ms at the 99th
percentile.
An example of a scalable congestion control that would enable the L4S
service is Data Center TCP (DCTCP), which until now has been
applicable solely to controlled environments like data centres
[RFC8257], because it is too aggressive to co-exist with existing
TCP-Reno-Friendly traffic. The DualQ Coupled AQM, which is defined
in a complementary experimental specification
[I-D.ietf-tsvwg-aqm-dualq-coupled], is an AQM framework that enables
scalable congestion controls like DCTCP to co-exist with existing
traffic, each getting roughly the same flow rate when they compete
under similar conditions. Note that a transport such as DCTCP is
still not safe to deploy on the Internet unless it satisfies the
requirements listed in Section 4.
L4S is not only for elastic (TCP-like) traffic - there are scalable
congestion controls for real-time media, such as the L4S variant of
the SCReAM [RFC8298] real-time media congestion avoidance technique
(RMCAT). The factor that distinguishes L4S from Classic traffic is
its behaviour in response to congestion. The transport wire
protocol, e.g. TCP, QUIC, SCTP, DCCP, RTP/RTCP, is orthogonal (and
therefore not suitable for distinguishing L4S from Classic packets).
The L4S identifier defined in this document is the key piece that
distinguishes L4S from TCP-Reno-Friendly (or 'Classic') traffic. It
gives an incremental migration path so that suitably modified network
bottlenecks can distinguish and isolate existing Classic traffic from
L4S traffic to prevent it from degrading the ultra-low delay and loss
of the new scalable transports, without harming Classic performance.
Initial implementation of the separate parts of the system has been
motivated by the performance benefits.
1.1. Latency, Loss and Scaling Problems
Latency is becoming the critical performance factor for many (most?)
applications on the public Internet, e.g. interactive Web, Web
services, voice, conversational video, interactive video, interactive
remote presence, instant messaging, online gaming, remote desktop,
cloud-based applications, and video-assisted remote control of
machinery and industrial processes. In the 'developed' world,
De Schepper & Briscoe Expires August 23, 2020 [Page 4]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
further increases in access network bit-rate offer diminishing
returns, whereas latency is still a multi-faceted problem. In the
last decade or so, much has been done to reduce propagation time by
placing caches or servers closer to users. However, queuing remains
a major intermittent component of latency.
The Diffserv architecture provides Expedited Forwarding [RFC3246], so
that low latency traffic can jump the queue of other traffic.
However, on access links dedicated to individual sites (homes, small
enterprises or mobile devices), often all traffic at any one time
will be latency-sensitive. Then, given nothing to differentiate
from, Diffserv makes no difference. Instead, we need to remove the
causes of any unnecessary delay.
The bufferbloat project has shown that excessively-large buffering
('bufferbloat') has been introducing significantly more delay than
the underlying propagation time. These delays appear only
intermittently--only when a capacity-seeking (e.g. TCP) flow is long
enough for the queue to fill the buffer, making every packet in other
flows sharing the buffer sit through the queue.
Active queue management (AQM) was originally developed to solve this
problem (and others). Unlike Diffserv, which gives low latency to
some traffic at the expense of others, AQM controls latency for _all_
traffic in a class. In general, AQM methods introduce an increasing
level of discard from the buffer the longer the queue persists above
a shallow threshold. This gives sufficient signals to capacity-
seeking (aka. greedy) flows to keep the buffer empty for its intended
purpose: absorbing bursts. However, RED [RFC2309] and other
algorithms from the 1990s were sensitive to their configuration and
hard to set correctly. So, this form of AQM was not widely deployed.
More recent state-of-the-art AQM methods, e.g. fq_CoDel [RFC8290],
PIE [RFC8033], Adaptive RED [ARED01], are easier to configure,
because they define the queuing threshold in time not bytes, so it is
invariant for different link rates. However, no matter how good the
AQM, the sawtoothing sending window of a Classic congestion control
will either cause queuing delay to vary or cause the link to be
under-utilized. Even with a perfectly tuned AQM, the additional
queuing delay will be of the same order as the underlying speed-of-
light delay across the network.
If a sender's own behaviour is introducing queuing delay variation,
no AQM in the network can "un-vary" the delay without significantly
compromising link utilization. Even flow-queuing (e.g. [RFC8290]),
which isolates one flow from another, cannot isolate a flow from the
delay variations it inflicts on itself. Therefore those applications
De Schepper & Briscoe Expires August 23, 2020 [Page 5]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
that need to seek out high bandwidth but also need low latency will
have to migrate to scalable congestion control.
Altering host behaviour is not enough on its own though. Even if
hosts adopt low latency behaviour (scalable congestion controls),
they need to be isolated from the behaviour of existing Classic
congestion controls that induce large queue variations. L4S enables
that migration by providing latency isolation in the network and
distinguishing the two types of packets that need to be isolated: L4S
and Classic. L4S isolation can be achieved with a queue per flow
(e.g. [RFC8290]) but a DualQ [I-D.ietf-tsvwg-aqm-dualq-coupled] is
sufficient, and actually gives better tail latency. Both approaches
are addressed in this document.
The DualQ solution was developed to make ultra-low latency available
without requiring per-flow queues at every bottleneck. This was
because FQ has well-known downsides - not least the need to inspect
transport layer headers in the network, which makes it incompatible
with privacy approaches such as IPSec VPN tunnels, and incompatible
with link layer queue management, where transport layer headers can
be hidden, e.g. 5G.
Latency is not the only concern addressed by L4S: It was known when
TCP congestion avoidance was first developed that it would not scale
to high bandwidth-delay products (footnote 6 of Jacobson and Karels
[TCP-CA]). Given regular broadband bit-rates over WAN distances are
already [RFC3649] beyond the scaling range of Reno TCP, 'less
unscalable' Cubic [RFC8312] and Compound [I-D.sridharan-tcpm-ctcp]
variants of TCP have been successfully deployed. However, these are
now approaching their scaling limits. Unfortunately, fully scalable
congestion controls such as DCTCP [RFC8257] cause Classic ECN
congestion controls sharing the same queue to starve themselves,
which is why they have been confined to private data centres or
research testbeds (until now).
It turns out that a congestion control algorithm like DCTCP that
solves the latency problem also solves the scalability problem of
Classic congestion controls. The finer sawteeth in the congestion
window have low amplitude, so they cause very little queuing delay
variation and the average time to recover from one congestion signal
to the next (the average duration of each sawtooth) remains
invariant, which maintains constant tight control as flow-rate
scales. A background paper [DCttH15] gives the full explanation of
why the design solves both the latency and the scaling problems, both
in plain English and in more precise mathematical form. The
explanation is summarised without the maths in the L4S architecture
document [I-D.ietf-tsvwg-l4s-arch].
De Schepper & Briscoe Expires August 23, 2020 [Page 6]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
1.2. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
[RFC2119]. In this document, these words will appear with that
interpretation only when in ALL CAPS. Lower case uses of these words
are not to be interpreted as carrying RFC-2119 significance.
Classic Congestion Control: A congestion control behaviour that can
co-exist with standard TCP Reno [RFC5681] without causing flow
rate starvation. With Classic congestion controls, as flow rate
scales, the number of round trips between congestion signals
(losses or ECN marks) rises with the flow rate. So it takes
longer and longer to recover after each congestion event.
Therefore control of queuing and utilization becomes very slack,
and the slightest disturbance prevents a high rate from being
attained [RFC3649].
For instance, with 1500 byte packets and an end-to-end round trip
time (RTT) of 36 ms, over the years, as Reno flow rate scales from
2 to 100 Mb/s the number of round trips taken to recover from a
congestion event rises proportionately, from 4 to 200. Cubic was
developed to be less unscalable, but it is approaching its scaling
limit; with the same RTT of 36ms, at 100Mb/s it takes over 300
round trips to recover, and at 800 Mb/s its recovery time doubles
to over 600 round trips, or more than 20 seconds.
Scalable Congestion Control: A congestion control where the average
time from one congestion signal to the next (the recovery time)
remains invariant as the flow rate scales, all other factors being
equal. This maintains the same degree of control over queueing
and utilization whatever the flow rate, as well as ensuring that
high throughput is robust to disturbances. For instance, DCTCP
averages 2 congestion signals per round-trip whatever the flow
rate. See Section 4.3 for more explanation.
Classic service: The Classic service is intended for all the
congestion control behaviours that co-exist with Reno [RFC5681]
(e.g. Reno itself, Cubic [RFC8312], Compound
[I-D.sridharan-tcpm-ctcp], TFRC [RFC5348]).
Low-Latency, Low-Loss Scalable throughput (L4S) service: The 'L4S'
service is intended for traffic from scalable congestion control
algorithms, such as Data Center TCP [RFC8257]. The L4S service is
for more general traffic than just DCTCP--it allows the set of
congestion controls with similar scaling properties to DCTCP to
De Schepper & Briscoe Expires August 23, 2020 [Page 7]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
evolve (e.g. Relentless TCP [Mathis09], TCP Prague [LinuxPrague]
and the L4S variant of SCREAM for real-time media [RFC8298]).
Both Classic and L4S services can cope with a proportion of
unresponsive or less-responsive traffic as well, as long as it
does not build a queue (e.g. DNS, VoIP, game sync datagrams,
etc).
The terms Classic or L4S can also qualify other nouns, such
'queue', 'codepoint', 'identifier', 'classification', 'packet',
'flow'. For example: a 'Classic queue', means a queue providing
the Classic service; an L4S packet means a packet with an L4S
identifier sent from an L4S congestion control.
Classic ECN: The original Explicit Congestion Notification (ECN)
protocol [RFC3168], which requires ECN signals to be treated the
same as drops, both when generated in the network and when
responded to by the sender. The names used for the four
codepoints of the 2-bit IP-ECN field are as defined in [RFC3168]:
Not ECT, ECT(0), ECT(1) and CE, where ECT stands for ECN-Capable
Transport and CE stands for Congestion Experienced.
1.3. Scope
The new L4S identifier defined in this specification is applicable
for IPv4 and IPv6 packets (as for Classic ECN [RFC3168]). It is
applicable for the unicast, multicast and anycast forwarding modes.
The L4S identifier is an orthogonal packet classification to the
Differentiated Services Code Point (DSCP) [RFC2474]. Section 5.4
explains what this means in practice.
This document is intended for experimental status, so it does not
update any standards track RFCs. Therefore it depends on [RFC8311],
which is a standards track specification that:
o updates the ECN proposed standard [RFC3168] to allow experimental
track RFCs to relax the requirement that an ECN mark must be
equivalent to a drop (when the network applies markings and/or
when the sender responds to them);
o changes the status of the experimental ECN nonce [RFC3540] to
historic;
o makes consequent updates to the following additional proposed
standard RFCs to reflect the above two bullets:
* ECN for RTP [RFC6679];
De Schepper & Briscoe Expires August 23, 2020 [Page 8]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
* the congestion control specifications of various DCCP
congestion control identifier (CCID) profiles [RFC4341],
[RFC4342], [RFC5622].
This document is about identifiers that are used for interoperation
between hosts and networks. So the audience is broad, covering
developers of host transports and network AQMs, as well as covering
how operators might wish to combine various identifiers, which would
require flexibility from equipment developers.
2. Consensus Choice of L4S Packet Identifier: Requirements
This subsection briefly records the process that led to a consensus
choice of L4S identifier, selected from all the alternatives in
Appendix B.
The identifier for packets using the Low Latency, Low Loss, Scalable
throughput (L4S) service needs to meet the following requirements:
o it SHOULD survive end-to-end between source and destination
applications: across the boundary between host and network,
between interconnected networks, and through middleboxes;
o it SHOULD be visible at the IP layer
o it SHOULD be common to IPv4 and IPv6 and transport-agnostic;
o it SHOULD be incrementally deployable;
o it SHOULD enable an AQM to classify packets encapsulated by outer
IP or lower-layer headers;
o it SHOULD consume minimal extra codepoints;
o it SHOULD be consistent on all the packets of a transport layer
flow, so that some packets of a flow are not served by a different
queue to others.
Whether the identifier would be recoverable if the experiment failed
is a factor that could be taken into account. However, this has not
been made a requirement, because that would favour schemes that would
be easier to fail, rather than those more likely to succeed.
It is recognised that the chosen identifier is unlikely to satisfy
all these requirements, particularly given the limited space left in
the IP header. Therefore a compromise will be necessary, which is
why all the above requirements are expressed with the word 'SHOULD'
not 'MUST'. Appendix B discusses the pros and cons of the
De Schepper & Briscoe Expires August 23, 2020 [Page 9]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
compromises made in various competing identification schemes against
the above requirements.
On the basis of this analysis, "ECT(1) and CE codepoints" is the best
compromise. Therefore this scheme is defined in detail in the
following sections, while Appendix B records the rationale for this
decision.
3. L4S Packet Identification at Run-Time
The L4S treatment is an experimental track alternative packet marking
treatment [RFC4774] to the Classic ECN treatment in [RFC3168], which
has been updated by [RFC8311] to allow experiments such as the one
defined in the present specification. Like Classic ECN, L4S ECN
identifies both network and host behaviour: it identifies the marking
treatment that network nodes are expected to apply to L4S packets,
and it identifies packets that have been sent from hosts that are
expected to comply with a broad type of sending behaviour.
For a packet to receive L4S treatment as it is forwarded, the sender
sets the ECN field in the IP header to the ECT(1) codepoint. See
Section 4 for full transport layer behaviour requirements, including
feedback and congestion response.
A network node that implements the L4S service normally classifies
arriving ECT(1) and CE packets for L4S treatment. See Section 5 for
full network element behaviour requirements, including
classification, ECN-marking and interaction of the L4S identifier
with other identifiers and per-hop behaviours.
4. Prerequisite Transport Layer Behaviour
4.1. Prerequisite Codepoint Setting
A sender that wishes a packet to receive L4S treatment as it is
forwarded, MUST set the ECN field in the IP header (v4 or v6) to the
ECT(1) codepoint.
4.2. Prerequisite Transport Feedback
For a transport protocol to provide scalable congestion control it
MUST provide feedback of the extent of CE marking on the forward
path. When ECN was added to TCP [RFC3168], the feedback method
reported no more than one CE mark per round trip. Some transport
protocols derived from TCP mimic this behaviour while others report
the accurate extent of ECN marking. This means that some transport
protocols will need to be updated as a prerequisite for scalable
De Schepper & Briscoe Expires August 23, 2020 [Page 10]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
congestion control. The position for a few well-known transport
protocols is given below.
TCP: Support for the accurate ECN feedback requirements [RFC7560]
(such as that provided by AccECN [I-D.ietf-tcpm-accurate-ecn]) by
both ends is a prerequisite for scalable congestion control in
TCP. Therefore, the presence of ECT(1) in the IP headers even in
one direction of a TCP connection will imply that both ends must
be supporting accurate ECN feedback. However, the converse does
not apply. So even if both ends support AccECN, either of the two
ends can choose not to use a scalable congestion control, whatever
the other end's choice.
SCTP: A suitable ECN feedback mechanism for SCTP could add a chunk
to report the number of received CE marks (e.g.
[I-D.stewart-tsvwg-sctpecn]), and update the ECN feedback protocol
sketched out in Appendix A of the standards track specification of
SCTP [RFC4960].
RTP over UDP: A prerequisite for scalable congestion control is for
both (all) ends of one media-level hop to signal ECN support
[RFC6679] and use the new generic RTCP feedback format of
[I-D.ietf-avtcore-cc-feedback-message]. The presence of ECT(1)
implies that both (all) ends of that media-level hop support ECN.
However, the converse does not apply. So each end of a media-
level hop can independently choose not to use a scalable
congestion control, even if both ends support ECN.
QUIC: Support for sufficiently fine-grained ECN feedback is provided
by the first IETF QUIC transport [I-D.ietf-quic-transport].
DCCP: The ACK vector in DCCP [RFC4340] is already sufficient to
report the extent of CE marking as needed by a scalable congestion
control.
4.3. Prerequisite Congestion Response
As a condition for a host to send packets with the L4S identifier
(ECT(1)), it SHOULD implement a congestion control behaviour that
ensures that, in steady state, the average time from one ECN
congestion signal to the next (the 'recovery time') does not increase
as flow rate scales, all other factors being equal. This is termed a
scalable congestion control. This is necessary to ensure that queue
variations remain small as flow rate scales, without having to
sacrifice utilization. For instance, for DCTCP, the average recovery
time is always half a round trip, whatever the flow rate.
De Schepper & Briscoe Expires August 23, 2020 [Page 11]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
The condition 'all other factors being equal', allows the recovery
time to be different for different round trip times, as long as it
does not increase with flow rate for any particular RTT.
Saying that the recovery time remains roughly invariant is equivalent
to saying that the number of ECN CE marks per round trip remains
invariant as flow rate scales, all other factors being equal. For
instance, DCTCP's average recovery time of half of 1 RTT is
equivalent to 2 ECN marks per round trip. For those who understand
steady-state congestion response functions, it is also equivalent to
say that, the congestion window is inversely proportional to the
proportion of bytes in packets marked with the CE codepoint (see
section 2 of [PI2]).
As well as DCTCP, TCP Prague [LinuxPrague] and the L4S variant of
SCReAM [RFC8298] are examples of scalable congestion controls.
As with all transport behaviours, a detailed specification (probably
an experimental RFC) will need to be defined for each type of
transport or application, including the timescale over which the
proportionality is averaged, and control of burstiness. The recovery
time requirement above is worded as a 'SHOULD' rather than a 'MUST'
to allow reasonable flexibility when defining these specifications.
Each sender in a session can use a scalable congestion control
independently of the congestion control used by the receiver(s) when
they send data. Therefore there might be ECT(1) packets in one
direction and ECT(0) or Not-ECT in the other.
In order to coexist safely with other Internet traffic, a scalable
congestion control MUST NOT tag its packets with the ECT(1) codepoint
unless it complies with the following bulleted requirements. The
specification of a particular scalable congestion control MUST
describe in detail how it satisfies each requirement and, for any
non-mandatory requirements, it MUST justify why it does not comply:
o As well as responding to ECN markings, a scalable congestion
control MUST react to packet loss in a way that will coexist
safely with a TCP Reno congestion control [RFC5681] (see
Appendix A.1.3 for rationale).
o A scalable congestion control MUST react to ECN marking from a
non-L4S but ECN-capable bottleneck in a way that will coexist with
a TCP Reno congestion control [RFC5681] (see Appendix A.1.4 for
rationale).
Note that a scalable congestion control is not expected to change
to setting ECT(0) while it falls back to coexist with Reno.
De Schepper & Briscoe Expires August 23, 2020 [Page 12]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
o A scalable congestion control MUST reduce or eliminate RTT bias
over as wide a range of RTTs as possible, or at least over the
typical range of RTTs that will interact in the intended
deployment scenario (see Appendix A.1.5 for rationale).
o A scalable congestion control MUST remain responsive to congestion
when the RTT is significantly smaller than in the public Internet
(see Appendix A.1.6 for rationale).
o A scalable congestion control intended for reordering-prone
networks SHOULD detect loss by counting in time-based units, which
is scalable, as opposed to counting in units of packets (as in the
3 DupACK rule of RFC 5681 TCP), which is not scalable (see
Appendix A.1.7 for rationale). This requirement is scoped to
'reordering-prone networks' in order to exclude congestion
controls that are solely used in controlled environments where the
network introduces hardly any reordering.
As well as traffic controlled by a scalable congestion control, a
reasonable level of smooth unresponsive traffic at a low rate
relative to typical broadband capacities is likely to be acceptable
(see "'Safe' Unresponsive Traffic" in Section 5.4.1.1.1).
5. Prerequisite Network Node Behaviour
5.1. Prerequisite Classification and Re-Marking Behaviour
A network node that implements the L4S service MUST classify arriving
ECT(1) packets for L4S treatment and, other than in the exceptional
case referred to next, it MUST classify arriving CE packets for L4S
treatment as well. CE packets might have originated as ECT(1) or
ECT(0), but the above rule to classify them as if they originated as
ECT(1) is the safe choice (see Appendix B.1 for rationale). The
exception is where some flow-aware in-network mechanism happens to be
available for distinguishing CE packets that originated as ECT(0), as
described in Section 5.3, but there is no implication that such a
mechanism is necessary.
An L4S AQM treatment follows similar codepoint transition rules to
those in RFC 3168. Specifically, the ECT(1) codepoint MUST NOT be
changed to any other codepoint than CE, and CE MUST NOT be changed to
any other codepoint. An ECT(1) packet is classified as ECN-capable
and, if congestion increases, an L4S AQM algorithm will increasingly
mark the ECN field as CE, otherwise forwarding packets unchanged as
ECT(1). Necessary conditions for an L4S marking treatment are
defined in Section 5.2. Under persistent overload an L4S marking
treatment SHOULD turn off ECN marking, using drop as a congestion
signal until the overload episode has subsided, as recommended for
De Schepper & Briscoe Expires August 23, 2020 [Page 13]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
all AQM methods in [RFC7567] (Section 4.2.1), which follows the
similar advice in RFC 3168 (Section 7).
For backward compatibility in uncontrolled environments, a network
node that implements the L4S treatment MUST also implement an AQM
treatment for the Classic service as defined in Section 1.2. This
Classic AQM treatment need not mark ECT(0) packets, but if it does,
it will do so under the same conditions as it would drop Not-ECT
packets [RFC3168]. It MUST classify arriving ECT(0) and Not-ECT
packets for treatment by the Classic AQM (see the discussion of the
classifier for the dual-queue coupled AQM in
[I-D.ietf-tsvwg-aqm-dualq-coupled]).
5.2. The Meaning of L4S CE Relative to Drop
The likelihood that an AQM drops a Not-ECT Classic packet (p_C) MUST
be roughly proportional to the square of the likelihood that it would
have marked it if it had been an L4S packet (p_L). That is
p_C ~= (p_L / k)^2
The constant of proportionality (k) does not have to be standardised
for interoperability, but a value of 2 is RECOMMENDED. The term
'likelihood' is used above to allow for marking and dropping to be
either probabilistic or deterministic.
This formula ensures that Scalable and Classic flows will converge to
roughly equal congestion windows, for the worst case of Reno
congestion control. This is because the congestion windows of
Scalable and Classic congestion controls are inversely proportional
to p_L and sqrt(p_C) respectively. So squaring p_C in the above
formula counterbalances the square root that characterizes TCP-Reno-
friendly flows.
[I-D.ietf-tsvwg-aqm-dualq-coupled] specifies the essential aspects of
an L4S AQM, as well as recommending other aspects. It gives example
implementations in appendices.
Note that, contrary to RFC 3168, a Coupled Dual Queue AQM
implementing the L4S and Classic treatments does not mark an ECT(1)
packet under the same conditions that it would have dropped a Not-ECT
packet, as allowed by [RFC8311], which updates RFC 3168. However, if
it marks ECT(0) packets, it does so under the same conditions that it
would have dropped a Not-ECT packet.
De Schepper & Briscoe Expires August 23, 2020 [Page 14]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
5.3. Exception for L4S Packet Identification by Network Nodes with
Transport-Layer Awareness
To implement the L4S treatment, a network node does not need to
identify transport-layer flows. Nonetheless, if an implementer is
willing to identify transport-layer flows at a network node, and if
the most recent ECT packet in the same flow was ECT(0), the node MAY
classify CE packets for Classic ECN [RFC3168] treatment. In all
other cases, a network node MUST classify all CE packets for L4S
treatment. Examples of such other cases are: i) if no ECT packets
have yet been identified in a flow; ii) if it is not desirable for a
network node to identify transport-layer flows; or iii) if the most
recent ECT packet in a flow was ECT(1).
If an implementer uses flow-awareness to classify CE packets, to
determine whether the flow is using ECT(0) or ECT(1) it only uses the
most recent ECT packet of a flow (this advice will need to be
verified as part of L4S experiments). This is because a sender might
switch from sending ECT(1) (L4S) packets to sending ECT(0) (Classic
ECN) packets, or back again, in the middle of a transport-layer flow
(e.g. it might manually switch its congestion control module mid-
connection, or it might be deliberately attempting to confuse the
network).
5.4. Interaction of the L4S Identifier with other Identifiers
The examples in this section concern how additional identifiers might
complement the L4S identifier to classify packets between class-based
queues. Firstly considering two queues, L4S and Classic, as in the
Coupled DualQ AQM [I-D.ietf-tsvwg-aqm-dualq-coupled], then more
complex structures within a larger queuing hierarchy.
5.4.1. DualQ Examples of Other Identifiers Complementing L4S
Identifiers
5.4.1.1. Inclusion of Additional Traffic with L4S
In a typical case for the public Internet a network element that
implements L4S might want to classify some low-rate but unresponsive
traffic (e.g. DNS, LDAP, NTP, voice, game sync packets) into the low
latency queue to mix with L4S traffic. Such non-ECN-based packet
types MUST be safe to mix with L4S traffic without harming the low
latency service, where 'safe' is explained in Section 5.4.1.1.1
below.
In this case it would not be appropriate to call the queue an L4S
queue, because it is shared by L4S and non-L4S traffic. Instead it
De Schepper & Briscoe Expires August 23, 2020 [Page 15]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
will be called the low latency or L queue. The L queue then offers
two different treatments:
o The L4S treatment, which is a combination of the L4S AQM treatment
and a priority scheduling treatment;
o The low latency treatment, which is solely the priority scheduling
treatment, without ECN-marking by the AQM.
To identify packets for just the scheduling treatment, it would be
inappropriate to use the L4S ECT(1) identifier, because such traffic
is unresponsive to ECN marking. Therefore, a network element that
implements L4S MAY classify additional packets into the L queue if
they carry certain non-ECN identifiers. For instance:
o addresses of specific applications or hosts configured to be safe
(or perhaps they comply with L4S behaviour and can respond to ECN
feedback, but perhaps cannot set the ECN field for some reason);
o certain protocols that are usually lightweight (e.g. ARP, DNS);
o specific Diffserv codepoints that indicate traffic with limited
burstiness such as the EF (Expedited Forwarding [RFC3246]), Voice-
Admit [RFC5865] or proposed NQB (Non-Queue-Building
[I-D.ietf-tsvwg-nqb]) service classes or equivalent local-use
DSCPs (see [I-D.briscoe-tsvwg-l4s-diffserv]).
Of course, a packet that carried both the ECT(1) codepoint and a non-
ECN identifier associated with the L queue would be classified into
the L queue.
For clarity, non-ECN identifiers, such as the examples itemized
above, might be used by some network operators who believe they
identify non-L4S traffic that would be safe to mix with L4S traffic.
They are not alternative ways for a host to indicate that it is
sending L4S packets. Only the ECT(1) ECN codepoint indicates to a
network element that a host is sending L4S packets (and CE indicates
that it could have originated as ECT(1)). Specifically ECT(1)
indicates that the host claims its behaviour satisfies the
prerequisite transport requirements in Section 4.
To include additional traffic with L4S, a network element only reads
identifiers such as those itemized above. It MUST NOT alter these
non-ECN identifiers, so that they survive for any potential use later
on the network path.
De Schepper & Briscoe Expires August 23, 2020 [Page 16]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
5.4.1.1.1. 'Safe' Unresponsive Traffic
The above section requires unresponsive traffic to be 'safe' to mix
with L4S traffic. Ideally this means that the sender never sends any
sequence of packets at a rate that exceeds the available capacity of
the bottleneck link. However, typically an unresponsive transport
does not even know the bottleneck capacity of the path, let alone its
available capacity. Nonetheless, an application can be considered
safe enough if it paces packets out (not necessarily completely
regularly) such that its maximum instantaneous rate from packet to
packet stays well below a typical broadband access rate.
This is a vague but useful definition, because many low latency
applications of interest, such as DNS, voice, game sync packets, RPC,
ACKs, keep-alives, could match this description.
5.4.1.2. Exclusion of Traffic From L4S Treatment
To extend the above example, an operator might want to exclude some
traffic from the L4S treatment for a policy reason, e.g. security
(traffic from malicious sources) or commercial (e.g. initially the
operator may wish to confine the benefits of L4S to business
customers).
In this exclusion case, the operator MUST classify on the relevant
locally-used identifiers (e.g. source addresses) before classifying
the non-matching traffic on the end-to-end L4S ECN identifier.
The operator MUST NOT alter the end-to-end L4S ECN identifier from
L4S to Classic, because its decision to exclude certain traffic from
L4S treatment is local-only. The end-to-end L4S identifier then
survives for other operators to use, or indeed, they can apply their
own policy, independently based on their own choice of locally-used
identifiers. This approach also allows any operator to remove its
locally-applied exclusions in future, e.g. if it wishes to widen the
benefit of the L4S treatment to all its customers.
5.4.1.3. Generalized Combination of L4S and Other Identifiers
L4S concerns low latency, which it can provide for all traffic
without differentiation and without affecting bandwidth allocation.
Diffserv provides for differentiation of both bandwidth and low
latency, but its control of latency depends on its control of
bandwidth. The two can be combined if a network operator wants to
control bandwidth allocation but it also wants to provide low latency
- for any amount of traffic within one of these allocations of
bandwidth (rather than only providing low latency by limiting
bandwidth) [I-D.briscoe-tsvwg-l4s-diffserv].
De Schepper & Briscoe Expires August 23, 2020 [Page 17]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
The DualQ examples so far have been framed in the context of
providing the default Best Efforts Per-Hop Behaviour (PHB) using two
queues - a Low Latency (L) queue and a Classic (C) Queue. This
single DualQ structure is expected to be the most common and useful
arrangement. But, more generally, an operator might choose to
control bandwidth allocation through a hierarchy of Diffserv PHBs at
a node, and to offer one (or more) of these PHBs with a low latency
and a Classic variant.
In the first case, if we assume that there are no other PHBs except
the DualQ, if a packet carries ECT(1) or CE, a network element would
classify it for the L4S treatment irrespective of its DSCP. And, if
a packet carried (say) the EF DSCP, the network element could
classify it into the L queue irrespective of its ECN codepoint.
However, where the DualQ is in a hierarchy of other PHBs, the
classifier would classify some traffic into other PHBs based on DSCP
before classifying between the low latency and Classic queues (based
on ECT(1), CE and perhaps also the EF DSCP or other identifiers as in
the above example). [I-D.briscoe-tsvwg-l4s-diffserv] gives a number
of examples of such arrangements to address various requirements.
[I-D.briscoe-tsvwg-l4s-diffserv] describes how an operator might use
L4S to offer low latency for all L4S traffic as well as using
Diffserv for bandwidth differentiation. It identifies two main types
of approach, which can be combined: the operator might split certain
Diffserv PHBs between L4S and a corresponding Classic service. Or it
might split the L4S and/or the Classic service into multiple Diffserv
PHBs. In either of these cases, a packet would have to be classified
on its Diffserv and ECN codepoints.
In summary, there are numerous ways in which the L4S ECN identifier
(ECT(1) and CE) could be combined with other identifiers to achieve
particular objectives. The following categorization articulates
those that are valid, but it is not necessarily exhaustive. Those
tagged 'Recommended-standard-use' could be set by the sending host or
a network. Those tagged 'Local-use' would only be set by a network:
1. Identifiers Complementing the L4S Identifier
A. Including More Traffic in the L Queue
(Could use Recommended-standard-use or Local-use identifiers)
B. Excluding Certain Traffic from the L Queue
(Local-use only)
2. Identifiers to place L4S classification in a PHB Hierarchy
(Could use Recommended-standard-use or Local-use identifiers)
De Schepper & Briscoe Expires August 23, 2020 [Page 18]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
A. PHBs Before L4S ECN Classification
B. PHBs After L4S ECN Classification
5.4.2. Per-Flow Queuing Examples of Other Identifiers Complementing L4S
Identifiers
At a node with per-flow queueing (e.g. FQ-CoDel [RFC8290]), the L4S
identifier could complement the Layer-4 flow ID as a further level of
flow granularity (i.e. Not-ECT and ECT(0) queued separately from
ECT(1) and CE packets). "Risk of reordering Classic CE packets" in
Appendix B.1 discusses the resulting ambiguity if packets originally
marked ECT(0) are marked CE by an upstream AQM before they arrive at
a node that classifies CE as L4S. It argues that the risk of re-
ordering is vanishingly small and the consequence of such a low level
of re-ordering is minimal.
Alternatively, it could be assumed that it is not in a flow's own
interest to mix Classic and L4S identifiers. Then the AQM could use
the ECN field to switch itself between a Classic and an L4S AQM
behaviour within one per-flow queue. For instance, for ECN-capable
packets, the AQM might consist of a simple marking threshold and an
L4S ECN identifier might simply select a shallower threshold than a
Classic ECN identifier would.
6. L4S Experiments
[I-D.ietf-tsvwg-aqm-dualq-coupled] sets operational and management
requirements for experiments with DualQ Coupled AQMs. General
operational and management requirements for experiments with L4S
congestion controls are given in Section 4 and Section 5 above, e.g.
co-existence and scaling requirements, incremental deployment
arrangements.
The specification of each scalable congestion control will need to
include protocol-specific requirements for configuration and
monitoring performance during experiments. Appendix A of [RFC5706]
provides a helpful checklist.
Monitoring for harm to other traffic, specifically bandwidth
starvation or excess queuing delay, will need to be conducted
alongside all early L4S experiments. It is hard, if not impossible,
for an individual flow to measure its impact on other traffic. So
such monitoring will need to be conducted using bespoke monitoring
across flows and/or across classes of traffic.
De Schepper & Briscoe Expires August 23, 2020 [Page 19]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
7. IANA Considerations
This specification contains no IANA considerations.
8. Security Considerations
Approaches to assure the integrity of signals using the new identifer
are introduced in Appendix C.1. See the security considerations in
the L4S architecture [I-D.ietf-tsvwg-l4s-arch] for further discussion
of mis-use of the identifier.
The recommendation to detect loss in time units prevents the ACK-
splitting attacks described in [Savage-TCP].
9. Acknowledgements
Thanks to Richard Scheffenegger, John Leslie, David Taeht, Jonathan
Morton, Gorry Fairhurst, Michael Welzl, Mikael Abrahamsson and Andrew
McGregor for the discussions that led to this specification. Ing-jyh
(Inton) Tsang was a contributor to the early drafts of this document.
And thanks to Mikael Abrahamsson, Lloyd Wood, Nicolas Kuhn, Greg
White, Tom Henderson, David Black, Gorry Fairhurst, Brian Carpenter,
Jake Holland, Rod Grimes and Richard Scheffenegger for providing help
and reviewing this draft and to Ingemar Johansson for reviewing and
providing substantial text. Appendix A listing the Prague L4S
Requirements is based on text authored by Marcelo Bagnulo Braun that
was originally an appendix to [I-D.ietf-tsvwg-l4s-arch]. That text
was in turn based on the collective output of the attendees listed in
the minutes of a 'bar BoF' on DCTCP Evolution during IETF-94
[TCPPrague].
The authors' contributions were part-funded by the European Community
under its Seventh Framework Programme through the Reducing Internet
Transport Latency (RITE) project (ICT-317700). Bob Briscoe was also
funded partly by the Research Council of Norway through the TimeIn
project, partly by CableLabs and partly by the Comcast Innovation
Fund. The views expressed here are solely those of the authors.
10. References
10.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
De Schepper & Briscoe Expires August 23, 2020 [Page 20]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
of Explicit Congestion Notification (ECN) to IP",
RFC 3168, DOI 10.17487/RFC3168, September 2001,
<https://www.rfc-editor.org/info/rfc3168>.
[RFC4774] Floyd, S., "Specifying Alternate Semantics for the
Explicit Congestion Notification (ECN) Field", BCP 124,
RFC 4774, DOI 10.17487/RFC4774, November 2006,
<https://www.rfc-editor.org/info/rfc4774>.
[RFC6679] Westerlund, M., Johansson, I., Perkins, C., O'Hanlon, P.,
and K. Carlberg, "Explicit Congestion Notification (ECN)
for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August
2012, <https://www.rfc-editor.org/info/rfc6679>.
10.2. Informative References
[A2DTCP] Zhang, T., Wang, J., Huang, J., Huang, Y., Chen, J., and
Y. Pan, "Adaptive-Acceleration Data Center TCP", IEEE
Transactions on Computers 64(6):1522-1533, June 2015,
<http://ieeexplore.ieee.org/xpl/
articleDetails.jsp?arnumber=6871352>.
[Ahmed19] Ahmed, A., "Extending TCP for Low Round Trip Delay",
Masters Thesis, Uni Oslo , August 2019,
<https://www.duo.uio.no/handle/10852/70966>.
[Alizadeh-stability]
Alizadeh, M., Javanmard, A., and B. Prabhakar, "Analysis
of DCTCP: Stability, Convergence, and Fairness", ACM
SIGMETRICS 2011 , June 2011.
[ARED01] Floyd, S., Gummadi, R., and S. Shenker, "Adaptive RED: An
Algorithm for Increasing the Robustness of RED's Active
Queue Management", ACIRI Technical Report , August 2001,
<http://www.icir.org/floyd/red.html>.
[DCttH15] De Schepper, K., Bondarenko, O., Briscoe, B., and I.
Tsang, "'Data Centre to the Home': Ultra-Low Latency for
All", RITE Project Technical Report , 2015,
<http://riteproject.eu/publications/>.
[I-D.briscoe-tsvwg-l4s-diffserv]
Briscoe, B., "Interactions between Low Latency, Low Loss,
Scalable Throughput (L4S) and Differentiated Services",
draft-briscoe-tsvwg-l4s-diffserv-02 (work in progress),
November 2018.
De Schepper & Briscoe Expires August 23, 2020 [Page 21]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
[I-D.ietf-avtcore-cc-feedback-message]
Sarker, Z., Perkins, C., Singh, V., and M. Ramalho, "RTP
Control Protocol (RTCP) Feedback for Congestion Control",
draft-ietf-avtcore-cc-feedback-message-05 (work in
progress), November 2019.
[I-D.ietf-quic-transport]
Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed
and Secure Transport", draft-ietf-quic-transport-25 (work
in progress), January 2020.
[I-D.ietf-tcpm-accurate-ecn]
Briscoe, B., Kuehlewind, M., and R. Scheffenegger, "More
Accurate ECN Feedback in TCP", draft-ietf-tcpm-accurate-
ecn-09 (work in progress), July 2019.
[I-D.ietf-tcpm-generalized-ecn]
Bagnulo, M. and B. Briscoe, "ECN++: Adding Explicit
Congestion Notification (ECN) to TCP Control Packets",
draft-ietf-tcpm-generalized-ecn-05 (work in progress),
November 2019.
[I-D.ietf-tcpm-rack]
Cheng, Y., Cardwell, N., Dukkipati, N., and P. Jha, "RACK:
a time-based fast loss detection algorithm for TCP",
draft-ietf-tcpm-rack-07 (work in progress), January 2020.
[I-D.ietf-tsvwg-aqm-dualq-coupled]
Schepper, K., Briscoe, B., and G. White, "DualQ Coupled
AQMs for Low Latency, Low Loss and Scalable Throughput
(L4S)", draft-ietf-tsvwg-aqm-dualq-coupled-10 (work in
progress), July 2019.
[I-D.ietf-tsvwg-ecn-encap-guidelines]
Briscoe, B., Kaippallimalil, J., and P. Thaler,
"Guidelines for Adding Congestion Notification to
Protocols that Encapsulate IP", draft-ietf-tsvwg-ecn-
encap-guidelines-13 (work in progress), May 2019.
[I-D.ietf-tsvwg-l4s-arch]
Briscoe, B., Schepper, K., Bagnulo, M., and G. White, "Low
Latency, Low Loss, Scalable Throughput (L4S) Internet
Service: Architecture", draft-ietf-tsvwg-l4s-arch-04 (work
in progress), July 2019.
De Schepper & Briscoe Expires August 23, 2020 [Page 22]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
[I-D.ietf-tsvwg-nqb]
White, G. and T. Fossati, "A Non-Queue-Building Per-Hop
Behavior (NQB PHB) for Differentiated Services", draft-
ietf-tsvwg-nqb-00 (work in progress), November 2019.
[I-D.sridharan-tcpm-ctcp]
Sridharan, M., Tan, K., Bansal, D., and D. Thaler,
"Compound TCP: A New TCP Congestion Control for High-Speed
and Long Distance Networks", draft-sridharan-tcpm-ctcp-02
(work in progress), November 2008.
[I-D.stewart-tsvwg-sctpecn]
Stewart, R., Tuexen, M., and X. Dong, "ECN for Stream
Control Transmission Protocol (SCTP)", draft-stewart-
tsvwg-sctpecn-05 (work in progress), January 2014.
[LinuxPacedChirping]
Misund, J. and B. Briscoe, "Paced Chirping - Rethinking
TCP start-up", Proc. Linux Netdev 0x13 , March 2019,
<https://www.netdevconf.org/0x13/session.html?talk-chirp>.
[LinuxPrague]
Briscoe, B., De Schepper, K., Albisser, O., Misund, J.,
Tilmans, O., Kuehlewind, M., and A. Ahmed, "Implementing
the `TCP Prague' Requirements for Low Latency Low Loss
Scalable Throughput (L4S)", Proc. Linux Netdev 0x13 ,
March 2019, <https://www.netdevconf.org/0x13/
session.html?talk-tcp-prague-l4s>.
[Mathis09]
Mathis, M., "Relentless Congestion Control", PFLDNeT'09 ,
May 2009, <http://www.hpcc.jp/pfldnet2009/
Program_files/1569198525.pdf>.
[Paced-Chirping]
Misund, J., "Rapid Acceleration in TCP Prague", Masters
Thesis , May 2018,
<https://riteproject.files.wordpress.com/2018/07/
misundjoakimmastersthesissubmitted180515.pdf>.
[PI2] De Schepper, K., Bondarenko, O., Tsang, I., and B.
Briscoe, "PI^2 : A Linearized AQM for both Classic and
Scalable TCP", Proc. ACM CoNEXT 2016 pp.105-119, December
2016,
<http://dl.acm.org/citation.cfm?doid=2999572.2999578>.
De Schepper & Briscoe Expires August 23, 2020 [Page 23]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
[QV] Briscoe, B. and P. Hurtig, "Up to Speed with Queue View",
RITE Technical Report D2.3; Appendix C.2, August 2015,
<https://riteproject.files.wordpress.com/2015/12/rite-
deliverable-2-3.pdf>.
[RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering,
S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G.,
Partridge, C., Peterson, L., Ramakrishnan, K., Shenker,
S., Wroclawski, J., and L. Zhang, "Recommendations on
Queue Management and Congestion Avoidance in the
Internet", RFC 2309, DOI 10.17487/RFC2309, April 1998,
<https://www.rfc-editor.org/info/rfc2309>.
[RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black,
"Definition of the Differentiated Services Field (DS
Field) in the IPv4 and IPv6 Headers", RFC 2474,
DOI 10.17487/RFC2474, December 1998,
<https://www.rfc-editor.org/info/rfc2474>.
[RFC2983] Black, D., "Differentiated Services and Tunnels",
RFC 2983, DOI 10.17487/RFC2983, October 2000,
<https://www.rfc-editor.org/info/rfc2983>.
[RFC3246] Davie, B., Charny, A., Bennet, J., Benson, K., Le Boudec,
J., Courtney, W., Davari, S., Firoiu, V., and D.
Stiliadis, "An Expedited Forwarding PHB (Per-Hop
Behavior)", RFC 3246, DOI 10.17487/RFC3246, March 2002,
<https://www.rfc-editor.org/info/rfc3246>.
[RFC3540] Spring, N., Wetherall, D., and D. Ely, "Robust Explicit
Congestion Notification (ECN) Signaling with Nonces",
RFC 3540, DOI 10.17487/RFC3540, June 2003,
<https://www.rfc-editor.org/info/rfc3540>.
[RFC3649] Floyd, S., "HighSpeed TCP for Large Congestion Windows",
RFC 3649, DOI 10.17487/RFC3649, December 2003,
<https://www.rfc-editor.org/info/rfc3649>.
[RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram
Congestion Control Protocol (DCCP)", RFC 4340,
DOI 10.17487/RFC4340, March 2006,
<https://www.rfc-editor.org/info/rfc4340>.
[RFC4341] Floyd, S. and E. Kohler, "Profile for Datagram Congestion
Control Protocol (DCCP) Congestion Control ID 2: TCP-like
Congestion Control", RFC 4341, DOI 10.17487/RFC4341, March
2006, <https://www.rfc-editor.org/info/rfc4341>.
De Schepper & Briscoe Expires August 23, 2020 [Page 24]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
[RFC4342] Floyd, S., Kohler, E., and J. Padhye, "Profile for
Datagram Congestion Control Protocol (DCCP) Congestion
Control ID 3: TCP-Friendly Rate Control (TFRC)", RFC 4342,
DOI 10.17487/RFC4342, March 2006,
<https://www.rfc-editor.org/info/rfc4342>.
[RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol",
RFC 4960, DOI 10.17487/RFC4960, September 2007,
<https://www.rfc-editor.org/info/rfc4960>.
[RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP
Friendly Rate Control (TFRC): Protocol Specification",
RFC 5348, DOI 10.17487/RFC5348, September 2008,
<https://www.rfc-editor.org/info/rfc5348>.
[RFC5562] Kuzmanovic, A., Mondal, A., Floyd, S., and K.
Ramakrishnan, "Adding Explicit Congestion Notification
(ECN) Capability to TCP's SYN/ACK Packets", RFC 5562,
DOI 10.17487/RFC5562, June 2009,
<https://www.rfc-editor.org/info/rfc5562>.
[RFC5622] Floyd, S. and E. Kohler, "Profile for Datagram Congestion
Control Protocol (DCCP) Congestion ID 4: TCP-Friendly Rate
Control for Small Packets (TFRC-SP)", RFC 5622,
DOI 10.17487/RFC5622, August 2009,
<https://www.rfc-editor.org/info/rfc5622>.
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
Control", RFC 5681, DOI 10.17487/RFC5681, September 2009,
<https://www.rfc-editor.org/info/rfc5681>.
[RFC5706] Harrington, D., "Guidelines for Considering Operations and
Management of New Protocols and Protocol Extensions",
RFC 5706, DOI 10.17487/RFC5706, November 2009,
<https://www.rfc-editor.org/info/rfc5706>.
[RFC5865] Baker, F., Polk, J., and M. Dolly, "A Differentiated
Services Code Point (DSCP) for Capacity-Admitted Traffic",
RFC 5865, DOI 10.17487/RFC5865, May 2010,
<https://www.rfc-editor.org/info/rfc5865>.
[RFC5925] Touch, J., Mankin, A., and R. Bonica, "The TCP
Authentication Option", RFC 5925, DOI 10.17487/RFC5925,
June 2010, <https://www.rfc-editor.org/info/rfc5925>.
De Schepper & Briscoe Expires August 23, 2020 [Page 25]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
[RFC6077] Papadimitriou, D., Ed., Welzl, M., Scharf, M., and B.
Briscoe, "Open Research Issues in Internet Congestion
Control", RFC 6077, DOI 10.17487/RFC6077, February 2011,
<https://www.rfc-editor.org/info/rfc6077>.
[RFC6660] Briscoe, B., Moncaster, T., and M. Menth, "Encoding Three
Pre-Congestion Notification (PCN) States in the IP Header
Using a Single Diffserv Codepoint (DSCP)", RFC 6660,
DOI 10.17487/RFC6660, July 2012,
<https://www.rfc-editor.org/info/rfc6660>.
[RFC7560] Kuehlewind, M., Ed., Scheffenegger, R., and B. Briscoe,
"Problem Statement and Requirements for Increased Accuracy
in Explicit Congestion Notification (ECN) Feedback",
RFC 7560, DOI 10.17487/RFC7560, August 2015,
<https://www.rfc-editor.org/info/rfc7560>.
[RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF
Recommendations Regarding Active Queue Management",
BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015,
<https://www.rfc-editor.org/info/rfc7567>.
[RFC7713] Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx)
Concepts, Abstract Mechanism, and Requirements", RFC 7713,
DOI 10.17487/RFC7713, December 2015,
<https://www.rfc-editor.org/info/rfc7713>.
[RFC8033] Pan, R., Natarajan, P., Baker, F., and G. White,
"Proportional Integral Controller Enhanced (PIE): A
Lightweight Control Scheme to Address the Bufferbloat
Problem", RFC 8033, DOI 10.17487/RFC8033, February 2017,
<https://www.rfc-editor.org/info/rfc8033>.
[RFC8257] Bensley, S., Thaler, D., Balasubramanian, P., Eggert, L.,
and G. Judd, "Data Center TCP (DCTCP): TCP Congestion
Control for Data Centers", RFC 8257, DOI 10.17487/RFC8257,
October 2017, <https://www.rfc-editor.org/info/rfc8257>.
[RFC8290] Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys,
J., and E. Dumazet, "The Flow Queue CoDel Packet Scheduler
and Active Queue Management Algorithm", RFC 8290,
DOI 10.17487/RFC8290, January 2018,
<https://www.rfc-editor.org/info/rfc8290>.
[RFC8298] Johansson, I. and Z. Sarker, "Self-Clocked Rate Adaptation
for Multimedia", RFC 8298, DOI 10.17487/RFC8298, December
2017, <https://www.rfc-editor.org/info/rfc8298>.
De Schepper & Briscoe Expires August 23, 2020 [Page 26]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
[RFC8311] Black, D., "Relaxing Restrictions on Explicit Congestion
Notification (ECN) Experimentation", RFC 8311,
DOI 10.17487/RFC8311, January 2018,
<https://www.rfc-editor.org/info/rfc8311>.
[RFC8312] Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and
R. Scheffenegger, "CUBIC for Fast Long-Distance Networks",
RFC 8312, DOI 10.17487/RFC8312, February 2018,
<https://www.rfc-editor.org/info/rfc8312>.
[Savage-TCP]
Savage, S., Cardwell, N., Wetherall, D., and T. Anderson,
"TCP Congestion Control with a Misbehaving Receiver", ACM
SIGCOMM Computer Communication Review 29(5):71--78,
October 1999.
[sub-mss-prob]
Briscoe, B. and K. De Schepper, "Scaling TCP's Congestion
Window for Small Round Trip Times", BT Technical Report
TR-TUB8-2015-002, May 2015,
<https://arxiv.org/abs/1904.07598>.
[TCP-CA] Jacobson, V. and M. Karels, "Congestion Avoidance and
Control", Laurence Berkeley Labs Technical Report ,
November 1988, <http://ee.lbl.gov/papers/congavoid.pdf>.
[TCPPrague]
Briscoe, B., "Notes: DCTCP evolution 'bar BoF': Tue 21 Jul
2015, 17:40, Prague", tcpprague mailing list archive ,
July 2015, <https://www.ietf.org/mail-
archive/web/tcpprague/current/msg00001.html>.
[VCP] Xia, Y., Subramanian, L., Stoica, I., and S. Kalyanaraman,
"One more bit is enough", Proc. SIGCOMM'05, ACM CCR
35(4)37--48, 2005,
<http://doi.acm.org/10.1145/1080091.1080098>.
Appendix A. The 'Prague L4S Requirements'
This appendix is informative, not normative. It gives a list of
modifications to current scalable congestion controls so that they
can be deployed over the public Internet and coexist safely with
existing traffic. The list complements the normative requirements in
Section 4 that a sender has to comply with before it can set the L4S
identifier in packets it sends into the Internet. As well as
necessary safety improvements (requirements) this appendix also
includes preferable performance improvements (optimizations).
De Schepper & Briscoe Expires August 23, 2020 [Page 27]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
These recommendations have become know as the Prague L4S
Requirements, because they were originally identified at an ad hoc
meeting during IETF-94 in Prague [TCPPrague]. The wording has been
generalized to apply to all scalable congestion controls, not just
TCP congestion control specifically. They were originally called the
'TCP Prague Requirements', but they are not solely applicable to TCP,
so the name has been generalized, and TCP Prague is now used for a
specific implementation of the requirements.
At the time of writing, DCTCP [RFC8257] is the most widely used
scalable transport protocol. In its current form, DCTCP is specified
to be deployable only in controlled environments. Deploying it in
the public Internet would lead to a number of issues, both from the
safety and the performance perspective. The modifications and
additional mechanisms listed in this section will be necessary for
its deployment over the global Internet. Where an example is needed,
DCTCP is used as a base, but it is likely that most of these
requirements equally apply to other scalable congestion controls.
A.1. Requirements for Scalable Transport Protocols
A.1.1. Use of L4S Packet Identifier
Description: A scalable congestion control needs to distinguish the
packets it sends from those sent by Classic congestion controls.
Motivation: It needs to be possible for a network node to classify
L4S packets without flow state into a queue that applies an L4S ECN
marking behaviour and isolates L4S packets from the queuing delay of
Classic packets.
A.1.2. Accurate ECN Feedback
Description: The transport protocol for a scalable congestion control
needs to provide timely, accurate feedback about the extent of ECN
marking experienced by all packets.
Motivation: Classic congestion controls only need feedback about the
existence of a congestion episode within a round trip, not precisely
how many packets were marked with ECN or dropped. Therefore, in
2001, when ECN feedback was added to TCP [RFC3168], it could not
inform the sender of more than one ECN mark per RTT. Since then,
requirements for more accurate ECN feedback in TCP have been defined
in [RFC7560] and [I-D.ietf-tcpm-accurate-ecn] specifies an
experimental change to the TCP wire protocol to satisfy these
requirements. Most other transport protocols already satisfy this
requirement.
De Schepper & Briscoe Expires August 23, 2020 [Page 28]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
A.1.3. Fall back to Reno-friendly congestion control on packet loss
Description: As well as responding to ECN markings in a scalable way,
a scalable congestion control needs to react to packet loss in a way
that will coexist safely with a TCP Reno congestion control
[RFC5681].
Motivation: Part of the safety conditions for deploying a scalable
congestion control on the public Internet is to make sure that it
behaves properly when it builds a queue at a network bottleneck that
has not been upgraded to support L4S. Packet loss can have many
causes, but it usually has to be conservatively assumed that it is a
sign of congestion. Therefore, on detecting packet loss, a scalable
congestion control will need to fall back to Classic congestion
control behaviour. If it does not comply with this requirement it
could starve Classic traffic.
A scalable congestion control can be used for different types of
transport, e.g. for real-time media or for reliable transport like
TCP. Therefore, the particular Classic congestion control behaviour
to fall back on will need to be part of the congestion control
specification of the relevant transport. In the particular case of
DCTCP, the DCTCP specification [RFC8257] states that "It is
RECOMMENDED that an implementation deal with loss episodes in the
same way as conventional TCP." For safe deployment of a scalable
congestion control in the public Internet, the above requirement
would need to be defined as a "MUST".
Even though a bottleneck is L4S capable, it might still become
overloaded and have to drop packets. In this case, the sender may
receive a high proportion of packets marked with the CE bit set and
also experience loss. Current DCTCP implementations react
differently to this situation. At least one implementation reacts
only to the drop signal (e.g. by halving the CWND) and at least
another DCTCP implementation reacts to both signals (e.g. by halving
the CWND due to the drop and also further reducing the CWND based on
the proportion of marked packet). A third approach for the public
Internet has been proposed that adjusts the loss response to result
in a halving when combined with the ECN response. We believe that
further experimentation is needed to understand what is the best
behaviour for the public Internet, which may or not be one of these
existing approaches.
De Schepper & Briscoe Expires August 23, 2020 [Page 29]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
A.1.4. Fall back to Reno-friendly congestion control on classic ECN
bottlenecks
Description: A scalable congestion control needs to react to ECN
marking from a non-L4S, but ECN-capable, bottleneck in a way that
will coexist with a TCP Reno congestion control [RFC5681].
Motivation: Similarly to the requirement in Appendix A.1.3, this
requirement is a safety condition to ensure a scalable congestion
control behaves properly when it builds a queue at a network
bottleneck that has not been upgraded to support L4S. On detecting
Classic ECN marking (see below), a scalable congestion control will
need to fall back to Classic congestion control behaviour. If it
does not comply with this requirement it could starve Classic
traffic.
It would take time for endpoints to distinguish Classic and L4S ECN
marking. An increase in queuing delay or in delay variation would be
a tell-tale sign, but it is not yet clear where a line would be drawn
between the two behaviours. It might be possible to cache what was
learned about the path to help subsequent attempts to detect the type
of marking.
A.1.5. Reduce RTT dependence
Description: A scalable congestion control needs to reduce or
eliminate RTT bias over as wide a range of RTTs as possible, or at
least over the typical range of RTTs that will interact in the
intended deployment scenario.
Motivation: The throughput of Classic congestion controls is known to
be inversely proportional to RTT, so one would expect flows over very
low RTT paths to nearly starve flows over larger RTTs. However,
Classic congestion controls have never allowed a very low RTT path to
exist because they induce a large queue. For instance, consider two
paths with base RTT 1ms and 100ms. If a Classic congestion control
induces a 100ms queue, it turns these RTTs into 101ms and 200ms
leading to a throughput ratio of about 2:1. Whereas if a scalable
congestion control induces only a 1ms queue, the ratio is 2:101,
leading to a throughput ratio of about 50:1.
Therefore, with very small queues, long RTT flows will essentially
starve, unless scalable congestion controls comply with this
requirement.
De Schepper & Briscoe Expires August 23, 2020 [Page 30]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
A.1.6. Scaling down to fractional congestion windows
Description: A scalable congestion control needs to remain responsive
to congestion when RTTs are significantly smaller than in the current
public Internet.
Motivation: As currently specified, the minimum required congestion
window of TCP (and its derivatives) is set to 2 sender maximum
segment sizes (SMSS) (see equation (4) in [RFC5681]). Once the
congestion window reaches this minimum, all known window-based
congestion control algorithms become unresponsive to congestion
signals. No matter how much drop or ECN marking, the congestion
window no longer reduces. Instead, the sender's lack of any further
congestion response forces the queue to grow, overriding any AQM and
increasing queuing delay.
L4S mechanisms significantly reduce queueing delay so, over the same
path, the RTT becomes lower. Then this problem becomes surprisingly
common [sub-mss-prob]. This is because, for the same link capacity,
smaller RTT implies a smaller window. For instance, consider a
residential setting with an upstream broadband Internet access of 8
Mb/s, assuming a max segment size of 1500 B. Two upstream flows will
each have the minimum window of 2 SMSS if the RTT is 6ms or less,
which is quite common when accessing a nearby data centre. So, any
more than two such parallel TCP flows will become unresponsive and
increase queuing delay.
Unless scalable congestion controls are required to comply with this
requirement from the start, they will frequently become unresponsive,
negating the low latency benefit of L4S, for themselves and for
others. One possible sub-MSS window mechanism is described in
[Ahmed19], and other approaches are likely to be feasible.
A.1.7. Measuring Reordering Tolerance in Time Units
Description: A scalable congestion control needs to detect loss by
counting in time-based units, which is scalable, rather than counting
in units of packets, which is not.
Motivation: A primary purpose of L4S is scalable throughput (it's in
the name). Scalability in all dimensions is, of course, also a goal
of all IETF technology. The inverse linear congestion response in
Section 4.3 is necessary, but not sufficient, to solve the congestion
control scalability problem identified in [RFC3649]. As well as
maintaining frequent ECN signals as rate scales, it is also important
to ensure that a potentially false perception of loss does not limit
throughput scaling.
De Schepper & Briscoe Expires August 23, 2020 [Page 31]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
End-systems cannot know whether a missing packet is due to loss or
reordering, except in hindsight - if it appears later. So they can
only deem that there has been a loss if a gap in the sequence space
has not been filled, either after a certain number of subsequent
packets has arrived (e.g. the 3 DupACK rule of standard TCP
congestion control [RFC5681]) or after a certain amount of time (e.g.
the experimental RACK approach [I-D.ietf-tcpm-rack]).
As we attempt to scale packet rate over the years:
o Even if only _some_ sending hosts still deem that loss has
occurred by counting reordered packets, _all_ networks will have
to keep reducing the time over which they keep packets in order.
If some link technologies keep the time within which reordering
occurs roughly unchanged, then loss over these links, as perceived
by these hosts, will appear to continually rise over the years.
o In contrast, if all senders detect loss in units of time, the time
over which the network has to keep packets in order stays roughly
invariant.
Therefore hosts have an incentive to detect loss in time units (so as
not to fool themselves too often into detecting losses when there are
none). And for hosts that are changing their congestion control
implementation to L4S, there is no downside to including time-based
loss detection code in the change (loss recovery implemented in
hardware is an exception, covered later). Therefore requiring L4S
hosts to detect loss in time-based units would not be a burden.
If this requirement is not placed on L4S hosts, even though it would
be no burden on them to do so, all networks will face unnecessary
uncertainty over whether some L4S hosts might be detecting loss by
counting packets. Then _all_ link technologies will have to
unnecessarily keep reducing the time within which reordering occurs.
That is not a problem for some link technologies, but it becomes
increasingly challenging for other link technologies to continue to
scale, particularly those relying on channel bonding for scaling,
such as LTE, 5G and DOCSIS.
Given Internet paths traverse many link technologies, any scaling
limit for these more challenging access link technologies would
become a scaling limit for the Internet as a whole.
It might be asked how it helps to place this loss detection
requirement only on L4S hosts, because networks will still face
uncertainty over whether non-L4S flows are detecting loss by counting
DupACKs. The answer is that those link technologies for which it is
challenging to keep squeezing the reordering time will only need to
De Schepper & Briscoe Expires August 23, 2020 [Page 32]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
do so for non-L4S traffic (which they can do because the L4S
identifier is visible at the IP layer). Therefore, they can focus
their processing and memory resources into scaling non-L4S (Classic)
traffic. Then, the higher the proportion of L4S traffic, the less of
a scaling challenge they will have.
To summarize, there is no reason for L4S hosts not to be part of the
solution instead of part of the problem.
Requirement ("MUST") or recommendation ("SHOULD")? As explained
above, this is a subtle interoperability issue between hosts and
networks, which seems to need a "MUST". Unless networks can be
certain that all L4S hosts follow the time-based approach, they still
have to cater for the worst case - continually squeeze reordering
into a smaller and smaller duration - just for hosts that might be
using the counting approach. However, it was decided to express this
as a recommendation, using "SHOULD". The main justification was that
networks can still be fairly certain that L4S hosts will follow this
recommendation, because following it offers only gain and no pain.
Details:
The speed of loss recovery is much more significant for short flows
than long, therefore a good compromise is to adapt the reordering
window; from a small fraction of the RTT at the start of a flow, to a
larger fraction of the RTT for flows that continue for many round
trips.
This is broadly the approach adopted by TCP RACK (Recent
ACKnowledgements) [I-D.ietf-tcpm-rack]. However, RACK starts with
the 3 DupACK approach, because the RTT estimate is not necessarily
stable. As long as the initial window is paced, such initial use of
3 DupACK counting would amount to time-based loss detection and
therefore would satisfy the time-based loss detection recommendation
of Section 4.3. This is because pacing of the initial window would
ensure that 3 DupACKs early in the connection would be spread over a
small fraction of the round trip.
As mentioned above, hardware implementations of loss recovery using
DupACK counting exist (e.g. some implementations of RoCEv2 for RDMA).
For low latency, these implementations can change their congestion
control to implement L4S, because the congestion control (as distinct
from loss recovery) is implemented in software. But they cannot
easily satisfy this loss recovery requirement. However, it is
believed they do not need to. It is believed that such
implementations solely exist in controlled environments, where the
network technology keeps reordering extremely low anyway. This is
De Schepper & Briscoe Expires August 23, 2020 [Page 33]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
why the scope of the normative recommendation in Section 4.3 is
limited to 'reordering-prone' networks.
Detecting loss in time units also prevents the ACK-splitting attacks
described in [Savage-TCP].
A.2. Scalable Transport Protocol Optimizations
A.2.1. Setting ECT in TCP Control Packets and Retransmissions
Description: This item only concerns TCP and its derivatives (e.g.
SCTP), because the original specification of ECN for TCP precluded
the use of ECN on control packets and retransmissions. To improve
performance, scalable transport protocols ought to enable ECN at the
IP layer in TCP control packets (SYN, SYN-ACK, pure ACKs, etc.) and
in retransmitted packets. The same is true for derivatives of TCP,
e.g. SCTP.
Motivation: RFC 3168 prohibits the use of ECN on these types of TCP
packet, based on a number of arguments. This means these packets are
not protected from congestion loss by ECN, which considerably harms
performance, particularly for short flows.
[I-D.ietf-tcpm-generalized-ecn] counters each argument in RFC 3168 in
turn, showing it was over-cautious. Instead it proposes experimental
use of ECN on all types of TCP packet as long as AccECN feedback
[I-D.ietf-tcpm-accurate-ecn] is available (which is itself a
prerequisite for using a scalable congestion control).
A.2.2. Faster than Additive Increase
Description: It would improve performance if scalable congestion
controls did not limit their congestion window increase to the
standard additive increase of 1 SMSS per round trip [RFC5681] during
congestion avoidance. The same is true for derivatives of TCP
congestion control, including similar approaches used for real-time
media.
Motivation: As currently defined [RFC8257], DCTCP uses the
traditional TCP Reno additive increase in congestion avoidance phase.
When the available capacity suddenly increases (e.g. when another
flow finishes, or if radio capacity increases) it can take very many
round trips to take advantage of the new capacity. TCP Cubic was
designed to solve this problem, but as flow rates have continued to
increase, the delay accelerating into available capacity has become
prohibitive. For instance, with RTT=20 ms, to increase flow rate
from 100Mb/s to 200Mb/s Cubic takes between 50 and 100 round trips.
Every 8x increase in flow rate leads to 2x more acceleration delay.
De Schepper & Briscoe Expires August 23, 2020 [Page 34]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
In the steady state, DCTCP induces about 2 ECN marks per round trip,
so it is possible to quickly detect when these signals have
disappeared and seek available capacity more rapidly, while
minimizing the impact on other flows (Classic and scalable)
[LinuxPacedChirping]. Alternatively, approaches such as Adaptive
Acceleration (A2DTCP [A2DTCP]) have been proposed to address this
problem in data centres, which might be deployable over the public
Internet.
A.2.3. Faster Convergence at Flow Start
Description: Particularly when a flow starts, scalable congestion
controls need to converge (reach their steady-state share of the
capacity) at least as fast as Classic congestion controls and
preferably faster. This affects the flow start behaviour of any L4S
congestion control derived from a Classic transport that uses TCP
slow start, including those for real-time media.
Motivation: As an example, a new DCTCP flow takes longer than a
Classic congestion control to obtain its share of the capacity of the
bottleneck when there are already ongoing flows using the bottleneck
capacity. In a data centre environment DCTCP takes about a factor of
1.5 to 2 longer to converge due to the much higher typical level of
ECN marking that DCTCP background traffic induces, which causes new
flows to exit slow start early [Alizadeh-stability]. In testing for
use over the public Internet the convergence time of DCTCP relative
to a regular loss-based TCP slow start is even less favourable
[Paced-Chirping]) due to the shallow ECN marking threshold needed for
L4S. It is exacerbated by the typically greater mismatch between the
link rate of the sending host and typical Internet access
bottlenecks. This problem is detrimental in general, but would
particularly harm the performance of short flows relative to Classic
congestion controls.
Appendix B. Alternative Identifiers
This appendix is informative, not normative. It records the pros and
cons of various alternative ways to identify L4S packets to record
the rationale for the choice of ECT(1) (Appendix B.1) as the L4S
identifier. At the end, Appendix B.6 summarises the distinguishing
features of the leading alternatives. It is intended to supplement,
not replace the detailed text.
The leading solutions all use the ECN field, sometimes in combination
with the Diffserv field. This is because L4S traffic has to indicate
that it is ECN-capable anyway, because ECN is intrinsic to how L4S
works. Both the ECN and Diffserv fields have the additional
advantage that they are no different in either IPv4 or IPv6. A
De Schepper & Briscoe Expires August 23, 2020 [Page 35]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
couple of alternatives that use other fields are mentioned at the
end, but it is quickly explained why they are not serious contenders.
B.1. ECT(1) and CE codepoints
Definition:
Packets with ECT(1) and conditionally packets with CE would
signify L4S semantics as an alternative to the semantics of
Classic ECN [RFC3168], specifically:
* The ECT(1) codepoint would signify that the packet was sent by
an L4S-capable sender.
* Given shortage of codepoints, both L4S and Classic ECN sides of
an AQM would have to use the same CE codepoint to indicate that
a packet had experienced congestion. If a packet that had
already been marked CE in an upstream buffer arrived at a
subsequent AQM, this AQM would then have to guess whether to
classify CE packets as L4S or Classic ECN. Choosing the L4S
treatment would be a safer choice, because then a few Classic
packets might arrive early, rather than a few L4S packets
arriving late.
* Additional information might be available if the classifier
were transport-aware. Then it could classify a CE packet for
Classic ECN treatment if the most recent ECT packet in the same
flow had been marked ECT(0). However, the L4S service ought
not to need tranport-layer awareness.
Cons:
Consumes the last ECN codepoint: The L4S service is intended to
supersede the service provided by Classic ECN, therefore using
ECT(1) to identify L4S packets could ultimately mean that the
ECT(0) codepoint was 'wasted' purely to distinguish one form of
ECN from its successor.
ECN hard in some lower layers: It is not always possible to support
ECN in an AQM acting in a buffer below the IP layer
[I-D.ietf-tsvwg-ecn-encap-guidelines]. In such cases, the L4S
service would have to drop rather than mark frames even though
they might encapsulate an ECN-capable packet. However, such cases
would be unusual.
Risk of reordering Classic CE packets: Classifying all CE packets
into the L4S queue risks any CE packets that were originally
ECT(0) being incorrectly classified as L4S. If there were delay
De Schepper & Briscoe Expires August 23, 2020 [Page 36]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
in the Classic queue, these incorrectly classified CE packets
would arrive early, which is a form of reordering. Reordering can
cause TCP senders (and senders of similar transports) to
retransmit spuriously. However, the risk of spurious
retransmissions would be extremely low for the following reasons:
1. It is quite unusual to experience queuing at more than one
bottleneck on the same path (the available capacities have to
be identical).
2. In only a subset of these unusual cases would the first
bottleneck support Classic ECN marking while the second
supported L4S ECN marking, which would be the only scenario
where some ECT(0) packets could be CE marked by an AQM
supporting Classic ECN then the remainder experienced further
delay through the Classic side of a subsequent L4S DualQ AQM.
3. Even then, when a few packets are delivered early, it takes
very unusual conditions to cause a spurious retransmission, in
contrast to when some packets are delivered late. The first
bottleneck has to apply CE-marks to at least N contiguous
packets and the second bottleneck has to inject an
uninterrupted sequence of at least N of these packets between
two packets earlier in the stream (where N is the reordering
window that the transport protocol allows before it considers
a packet is lost).
For example consider N=3, and consider the sequence of
packets 100, 101, 102, 103,... and imagine that packets
150,151,152 from later in the flow are injected as follows:
100, 150, 151, 101, 152, 102, 103... If this were late
reordering, even one packet arriving 50 out of sequence
would trigger a spurious retransmission, but there is no
spurious retransmission here, with early reordering,
because packet 101 moves the cumulative ACK counter forward
before 3 packets have arrived out of order. Later, when
packets 148, 149, 153... arrive, even though there is a
3-packet hole, there will be no problem, because the
packets to fill the hole are already in the receive buffer.
4. Even with the current TCP recommendation of N=3 [RFC5681]
spurious retransmissions will be unlikely for all the above
reasons. As RACK [I-D.ietf-tcpm-rack] is becoming widely
deployed, it tends to adapt its reordering window to a larger
value of N, which will make the chance of a contiguous
sequence of N early arrivals vanishingly small.
De Schepper & Briscoe Expires August 23, 2020 [Page 37]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
5. Even a run of 2 CE marks within a Classic ECN flow is
unlikely, given FQ-CoDel is the only known widely deployed AQM
that supports Classic ECN marking and it takes great care to
separate out flows and to space any markings evenly along each
flow.
It is extremely unlikely that the above set of 5 eventualities
that are each unusual in themselves would all happen
simultaneously. But, even if they did, the consequences would
hardly be dire: the odd spurious fast retransmission. Admittedly
TCP (and similar transports) reduce their congestion window when
they deem there has been a loss, but even this can be recovered
once the sender detects that the retransmission was spurious.
Non-L4S service for control packets: The Classic ECN RFCs [RFC3168]
and [RFC5562] require a sender to clear the ECN field to Not-ECT
for retransmissions and certain control packets specifically pure
ACKs, window probes and SYNs. When L4S packets are classified by
the ECN field alone, these control packets would not be classified
into an L4S queue, and could therefore be delayed relative to the
other packets in the flow. This would not cause re-ordering
(because retransmissions are already out of order, and the control
packets carry no data). However, it would make critical control
packets more vulnerable to loss and delay. To address this
problem, [I-D.ietf-tcpm-generalized-ecn] proposes an experiment in
which all TCP control packets and retransmissions are ECN-capable
as long as ECN feedback is available.
Pros:
Should work e2e: The ECN field generally works end-to-end across the
Internet. Unlike the DSCP, the setting of the ECN field is at
least forwarded unchanged by networks that do not support ECN, and
networks rarely clear it to zero.
Should work in tunnels: Unlike Diffserv, ECN is defined to always
work across tunnels. However, tunnels do not always implement ECN
processing as they should do, particularly because IPsec tunnels
were defined differently for a few years.
Could migrate to one codepoint: If all Classic ECN senders
eventually evolve to use the L4S service, the ECT(0) codepoint
could be reused for some future purpose, but only once use of
ECT(0) packets had reduced to zero, or near-zero, which might
never happen.
De Schepper & Briscoe Expires August 23, 2020 [Page 38]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
B.2. ECN Plus a Diffserv Codepoint (DSCP)
Definition:
For packets with a defined DSCP, all codepoints of the ECN field
(except Not-ECT) would signify alternative L4S semantics to those
for Classic ECN [RFC3168], specifically:
* The L4S DSCP would signifiy that the packet came from an L4S-
capable sender.
* ECT(0) and ECT(1) would both signify that the packet was
travelling between transport endpoints that were both ECN-
capable.
* CE would signify that the packet had been marked by an AQM
implementing the L4S service.
Use of a DSCP is the only approach for alternative ECN semantics
given as an example in [RFC4774]. However, it was perhaps considered
more for controlled environments than new end-to-end services.
Cons:
Consumes DSCP pairs: A DSCP is obviously not orthogonal to Diffserv.
Therefore, wherever the L4S service is applied to multiple
Diffserv scheduling behaviours, it would be necessary to replace
each DSCP with a pair of DSCPs.
Uses critical lower-layer header space: The resulting increased
number of DSCPs might be hard to support for some lower layer
technologies, e.g. 802.1p and MPLS both offer only 3-bits for a
maximum of 8 traffic class identifiers. Although L4S should
reduce and possibly remove the need for some DSCPs intended for
differentiated queuing delay, it will not remove the need for
Diffserv entirely, because Diffserv is also used to allocate
bandwidth, e.g. by prioritising some classes of traffic over
others when traffic exceeds available capacity.
Not end-to-end (host-network): Very few networks honour a DSCP set
by a host. Typically a network will zero (bleach) the Diffserv
field from all hosts. Sometimes networks will attempt to identify
applications by some form of packet inspection and, based on
network policy, they will set the DSCP considered appropriate for
the identified application. Network-based application
identification might use some combination of protocol ID, port
numbers(s), application layer protocol headers, IP address(es),
VLAN ID(s) and even packet timing.
De Schepper & Briscoe Expires August 23, 2020 [Page 39]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
Not end-to-end (network-network): Very few networks honour a DSCP
received from a neighbouring network. Typically a network will
zero (bleach) the Diffserv field from all neighbouring networks at
an interconnection point. Sometimes bilateral arrangements are
made between networks, such that the receiving network remarks
some DSCPs to those it uses for roughly equivalent services. The
likelihood that a DSCP will be bleached or ignored depends on the
type of DSCP:
Local-use DSCP: These tend to be used to implement application-
specific network policies, but a bilateral arrangement to
remark certain DSCPs is often applied to DSCPs in the local-use
range simply because it is easier not to change all of a
network's internal configurations when a new arrangement is
made with a neighbour.
Recommended standard DSCP: These do not tend to be honoured
across network interconnections more than local-use DSCPs.
However, if two networks decide to honour certain of each
other's DSCPs, the reconfiguration is a little easier if both
of their globally recognised services are already represented
by the relevant recommended standard DSCPs.
Note that today a recommended standard DSCP gives little more
assurance of end-to-end service than a local-use DSCP. In
future the range recommended as standard might give more
assurance of end-to-end service than local-use, but it is
unlikely that either assurance will be high, particularly given
the hosts are included in the end-to-end path.
Not all tunnels: Diffserv codepoints are often not propagated to the
outer header when a packet is encapsulated by a tunnel header.
DSCPs are propagated to the outer of uniform mode tunnels, but not
pipe mode [RFC2983], and pipe mode is fairly common.
ECN hard in some lower layers:: Because this approach uses both the
Diffserv and ECN fields, an AQM wil only work at a lower layer if
both can be supported. If individual network operators wished to
deploy an AQM at a lower layer, they would usually propagate an IP
Diffserv codepoint to the lower layer, using for example IEEE
802.1p. However, the ECN capability is harder to propagate down
to lower layers because few lower layers support it.
Pros:
Could migrate to e2e: If all usage of Classic ECN migrates to usage
of L4S, the DSCP would become redundant, and the ECN capability
alone could eventually identify L4S packets without the
De Schepper & Briscoe Expires August 23, 2020 [Page 40]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
interconnection problems of Diffserv detailed above, and without
having permanently consumed more than one codepoint in the IP
header. Although the DSCP does not generally function as an end-
to-end identifier (see above), it could be used initially by
individual ISPs to introduce the L4S service for their own locally
generated traffic.
B.3. ECN capability alone
This approach uses ECN capability alone as the L4S identifier. It
would only have been feasible if RFC 3168 ECN had not been widely
deployed. This was the case when the choice of L4S identifier was
being made and this appendix was first written. Since then, RFC 3168
ECN has been widely deployed and L4S did not take this approach
anyway. So this approach is not discussed further, because it is no
longer a feasible option.
B.4. Protocol ID
It has been suggested that a new ID in the IPv4 Protocol field or the
IPv6 Next Header field could identify L4S packets. However this
approach is ruled out by numerous problems:
o A new protocol ID would need to be paired with the old one for
each transport (TCP, SCTP, UDP, etc.).
o In IPv6, there can be a sequence of Next Header fields, and it
would not be obvious which one would be expected to identify a
network service like L4S.
o A new protocol ID would rarely provide an end-to-end service,
because It is well-known that new protocol IDs are often blocked
by numerous types of middlebox.
o The approach is not a solution for AQM methods below the IP layer.
B.5. Source or destination addressing
Locally, a network operator could arrange for L4S service to be
applied based on source or destination addressing, e.g. packets from
its own data centre and/or CDN hosts, packets to its business
customers, etc. It could use addressing at any layer, e.g. IP
addresses, MAC addresses, VLAN IDs, etc. Although addressing might
be a useful tactical approach for a single ISP, it would not be a
feasible approach to identify an end-to-end service like L4S. Even
for a single ISP, it would require packet classifiers in buffers to
be dependent on changing topology and address allocation decisions
De Schepper & Briscoe Expires August 23, 2020 [Page 41]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
elsewhere in the network. Therefore this approach is not a feasible
solution.
B.6. Summary: Merits of Alternative Identifiers
Table 1 provides a very high level summary of the pros and cons
detailed against the schemes described respectively in Appendix B.2
and Appendix B.1, for six issues that set them apart.
+--------------+--------------------+--------------------+
| Issue | DSCP + ECN | ECT(1) + CE |
+--------------+--------------------+--------------------+
| | initial eventual | initial eventual |
| | | |
| end-to-end | N . . . ? . | . . Y . . Y |
| tunnels | . O . . O . | . . ? . . Y |
| lower layers | N . . . ? . | . O . . . ? |
| codepoints | N . . . . ? | N . . . . ? |
| reordering | . . Y . . Y | . O . . . ? |
| ctrl pkts | . . Y . . Y | . O . . . ? |
| | | |
| | | |
+--------------+--------------------+--------------------+
Table 1: Comparison of the Merits of Three Alternative Identifiers
The schemes are scored based on both their capabilities now
('initial') and in the long term ('eventual'). The scores are one of
'N, O, Y', meaning 'Poor', 'Ordinary', 'Good' respectively. The same
scores are aligned vertically to aid the eye. A score of "?" in one
of the positions means that this approach might optimistically become
this good, given sufficient effort. The table summarises the text
and is not meant to be understandable without having read the text.
Appendix C. Potential Competing Uses for the ECT(1) Codepoint
The ECT(1) codepoint of the ECN field has already been assigned once
for the ECN nonce [RFC3540], which has now been categorized as
historic [RFC8311]. ECN is probably the only remaining field in the
Internet Protocol that is common to IPv4 and IPv6 and still has
potential to work end-to-end, with tunnels and with lower layers.
Therefore, ECT(1) should not be reassigned to a different
experimental use (L4S) without carefully assessing competing
potential uses. These fall into the following categories:
De Schepper & Briscoe Expires August 23, 2020 [Page 42]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
C.1. Integrity of Congestion Feedback
Receiving hosts can fool a sender into downloading faster by
suppressing feedback of ECN marks (or of losses if retransmissions
are not necessary or available otherwise).
The historic ECN nonce protocol [RFC3540] proposed that a TCP sender
could set either of ECT(0) or ECT(1) in each packet of a flow and
remember the sequence it had set. If any packet was lost or
congestion marked, the receiver would miss that bit of the sequence.
An ECN Nonce receiver had to feed back the least significant bit of
the sum, so it could not suppress feedback of a loss or mark without
a 50-50 chance of guessing the sum incorrectly.
It is highly unlikely that ECT(1) will be needed for integrity
protection in future. The ECN Nonce RFC [RFC3540] as been
reclassified as historic, partly because other ways have been
developed to protect feedback integrity of TCP and other transports
[RFC8311] that do not consume a codepoint in the IP header. For
instance:
o the sender can test the integrity of the receiver's feedback by
occasionally setting the IP-ECN field to a value normally only set
by the network. Then it can test whether the receiver's feedback
faithfully reports what it expects (see para 2 of Section 20.2 of
[RFC3168]. This works for loss and it will work for the accurate
ECN feedback [RFC7560] intended for L4S.
o A network can enforce a congestion response to its ECN markings
(or packet losses) by auditing congestion exposure (ConEx)
[RFC7713]. Whether the receiver or a downstream network is
suppressing congestion feedback or the sender is unresponsive to
the feedback, or both, ConEx audit can neutralise any advantage
that any of these three parties would otherwise gain.
o The TCP authentication option (TCP-AO [RFC5925]) can be used to
detect any tampering with TCP congestion feedback (whether
malicious or accidental). TCP's congestion feedback fields are
immutable end-to-end, so they are amenable to TCP-AO protection,
which covers the main TCP header and TCP options by default.
However, TCP-AO is often too brittle to use on many end-to-end
paths, where middleboxes can make verification fail in their
attempts to improve performance or security, e.g. by
resegmentation or shifting the sequence space.
De Schepper & Briscoe Expires August 23, 2020 [Page 43]
Internet-Draft ECN Semantics for Low Queuing Delay February 2020
C.2. Notification of Less Severe Congestion than CE
Various researchers have proposed to use ECT(1) as a less severe
congestion notification than CE, particularly to enable flows to fill
available capacity more quickly after an idle period, when another
flow departs or when a flow starts, e.g. VCP [VCP], Queue View (QV)
[QV].
Before assigning ECT(1) as an identifer for L4S, we must carefully
consider whether it might be better to hold ECT(1) in reserve for
future standardisation of rapid flow acceleration, which is an
important and enduring problem [RFC6077].
Pre-Congestion Notification (PCN) is another scheme that assigns
alternative semantics to the ECN field. It uses ECT(1) to signify a
less severe level of pre-congestion notification than CE [RFC6660].
However, the ECN field only takes on the PCN semantics if packets
carry a Diffserv codepoint defined to indicate PCN marking within a
controlled environment. PCN is required to be applied solely to the
outer header of a tunnel across the controlled region in order not to
interfere with any end-to-end use of the ECN field. Therefore a PCN
region on the path would not interfere with any of the L4S service
identifiers proposed in Appendix B.
Authors' Addresses
Koen De Schepper
Nokia Bell Labs
Antwerp
Belgium
Email: koen.de_schepper@nokia.com
URI: https://www.bell-labs.com/usr/koen.de_schepper
Bob Briscoe (editor)
Independent
UK
Email: ietf@bobbriscoe.net
URI: http://bobbriscoe.net/
De Schepper & Briscoe Expires August 23, 2020 [Page 44]