Low Latency, Low Loss, and Scalable Throughput (L4S) Internet Service: Architecture
draft-ietf-tsvwg-l4s-arch-20

# Internet AD comments for {draft-ietf-tsvwg-l4s-arch-19}
CC @ekline

## Comments

### S6.1

* Are there any handy, good references for human visual perception/tolerance
  of delay?

# Security AD comments for {draft-ietf-tsvwg-l4s-arch-19}
CC @paulwouters

This is somewhat out of my area of expertise, but upon reading it, it seems sensible to me. Just a few minor comments that can be taken into account or ignored.

## Comments

### Abstract

This is very very long. Perhaps some can be moved to the Introduction or cut? For example (but not having given it enough thought to actually use this): "This document describes the architecture for the "Low Latency, Low Loss, Scalable Throughput" (L4S) protocol to archive lower latency traffic on the internet. It describes its design decisions to make it work via incremental deployment on the internet and in a worst case won't perform worse than currently deployed latency minimizing techniques."

### Security Considerations

   However, there has
   never been a universal need to police the rate of individual
   application flows - the Internet has generally always relied on self-
   restraint of congestion controls

The first voip clients on the internet in the 90s would blast UDP as fast as possible. ISPs started filtering it. I wouldn't call that self-restraint, but one might argue that they were forced to do so to remain in operation, so perhaps it qualifies a self-restraint :P

   Whether burst policing becomes necessary remains to be seen.  Without
   it, there will be potential for attacks on the low latency of the L4S
   service.

If something can be abused to gain an advantage, it will be. So in that sense burst policing seems inevitable. But I can see how routers near the user could punish abusers to go into a slower queue and this section links to examples for that. So L4S won't really change this. Similarly, it can be used to create a two-tier internet ("premium users") but here too this can already be done without L4S.

** Section 6.4.2.  (Per the telechat conversation, move to COMMENT) My read of Figure 3 is that the suggested architecture can be incrementally deployed.  My other read is that it appears that this staged deployment isn’t safe outside of controlled environments (i.e., for the Internet) until after phase 1.  Assuming this is accurate, please add clear normative language that partial roll-outs of the L4S architecture MUST NOT occur outside of controlled environment until <insert needed capabilities>.

** Section 8.1.
   In the current Internet, scheduling usually enforces separation
   between 'sites'

If would be clearer to explain scheduling of what and by whom.

** Section 8.1

   However, there has
   never been a universal need to police the rate of individual
   application flows - the Internet has generally always relied on self-
   restraint of congestion controls at senders for sharing intra-'site'
   capacity.

I don’t follow the generalization being suggested here.  Is the text suggesting that enterprises do not shape traffic on a per application basis?  “Upstream networks” to provide different treatment of select traffic?  Is there a measurement study that can be cited?

** Section 8.1.  The first paragraph ends with a conclusion that there isn’t a universal need to police individual flow rates.  However, the second paragraph then discusses the use of per-flow-queuing.  One paragraph doesn’t follow from the other.  If no one does per-flow treatment as suggested, why discuss the approach?

** Section 8.2

   It is hoped that self-interest and guidance on dynamic
   behaviour (especially flow start-up, which might need to be
   standardized) will be sufficient to prevent transports from sending
   excessive bursts of L4S traffic, given the application's own latency
   will suffer most from such behaviour. 

Why is “self-interest and guidance on dynamic behaviour” the appropriate threat model?

** Section 8.2

 None of these possible queue protection capabilities are considered a
   necessary part of the L4S architecture, which works without them 

Could this be restated.  The current text suggests to me that various security mitigations are not part of the L4S architecture.  I don’t see how this would be possible if it were to see Internet deployment.

** Section 8.3

   So, in networks that already use rate policers and plan to deploy
   L4S, it will be preferable to redesign these rate policers to be more
   friendly to the L4S service. 

It might be helpful to be more specific on where this traffic shaping is happening as part of the migration consideration.

** Section 8.4.  This section discusses malicious senders, thank you.  Could a statement also be made about an on-path attacker modifying the ECN marks in the absence of integrity mechanisms.  For example, draft-ietf-tsvwg-l4s-ids notes in Section 5.4.1.1:

   If a non-compliant or
   malicious network node did swap ECT(0) to ECT(1), the packet could
   subsequently be ECN-marked by a downstream L4S AQM, but the sender
   would respond to congestion indications thinking it had sent a
   Classic packet.  This could result in the flow yielding excessively
   to other L4S flows sharing the downstream bottleneck.

** Section 8.5. 
   Because L4S can provide low delay for a broad set of applications
   that choose to use it, there is no need for individual applications
   or classes within that broad set to be distinguishable in any way
   while traversing networks.   

From a purely technical view, this might be true.  However, couldn’t local policy require different treatment?

** Section 8.5.
   There may be some types of traffic
   that prefer not to use L4S, but the coarse binary categorization of
   traffic reveals very little that could be exploited to compromise
   privacy.

After L4S is more widely adopted, that specific application could be identified by their _lack_ use of L4S?  In the early days where there is limited L4S use, wouldn’t these early adopter stick out?  

Nits
** Section 2.  Typo. s/two queues is sufficient/two queues are sufficient/

** Section 5.1.  Per “Cubic [RFC8312] was developed to be less unscalable …”, if there a way to rephrase “less unscalable”?

** Section 6.4.1.  Per 

An L4S AQM would often next be needed where the WiFi links in a home
   sometimes become the bottleneck.  
Is this suggesting that L4S AQM isn’t suitable for WiFi links but that is future work?

** Section 4.6.2.  s/Scalable CC/scalable congestion control approach/

# Éric Vyncke, INT AD, comments for draft-ietf-tsvwg-l4s-arch-19
CC @evyncke

Thank you for the work put into this document. Please note that I am far from being an expert on transport issues... so I am afraid that my review is not deep and may have missed obvious points.

 I especially like the incremental deployment approach.

Please find below some non-blocking COMMENT points (but replies would be appreciated even if only for my own education), and some nits.

Special thanks to Wesley Eddy for the shepherd's detailed write-up *including* the rough (?) WG consensus *and* the justification of the intended status. 

Please note that Benno Overeinder is the Internet directorate reviewer (at my request) and you may want to consider this int-dir reviews as well when Benno will complete the review (no need to wait for it though):
https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4s-arch/reviewrequest/16167/

I hope that this review helps to improve the document,

Regards,

-éric

## COMMENTS

### id-nits and obsolete references

[id-nits] (https://author-tools.ietf.org/api/idnits?url=https://www.ietf.org/archive/id/draft-ietf-tsvwg-l4s-arch-19.txt) indicates two references to obsolete RFC (4960 and 7540). Probably worth updating the references.

### Multicast ?

While the answer seems obvious, should there be a mention that L4S technique only applies to unicast traffic ?

### 


### Very long abstract...

This is probably the longest abstract for the last couple of months ;-)

No need to reply or change the document, but the abstract also somehow smells like a marketing text...

### Section 1, on purpose ?

```
   ... Queuing in access
   network bottlenecks is typically configured to cause overall network
   delay to roughly double during a long-running flow, relative to
   expected base (unloaded) path delay [BufferSize].
```

Is this really configured so on purpose ? Probably not, even if the above text seems to indicate that this is on purpose.

### Section 1, TCP only ?

The following text seems to indicate that TCP is the major transport protocol, but (see caveat about my lack of transport expertise) it seems to me that QUIC (and plain UDP) are becoming more important:
```
   The root of the problem is the presence of standard TCP congestion
   control (Reno [RFC5681]) or compatible variants (e.g. TCP
   Cubic [RFC8312]).
```
I.e., if similar mechanisms exist in QUIC, then it is worth mentioning.

### Section 2, incremental deployment ?

```
   ... Because ECN support is
   essential for L4S, senders use the ECN field as the protocol that
   allows the network to identify which packets are L4S and which are
   Classic.
```
seems to contradict the 'incremental deployment' statement as the network needs to be able to differentiate traffic. Or did I miss something ?

### Section 2, which part of the host ?

```
      ... A host needs to distinguish L4S and Classic packets
      with an identifier so that the network can classify them into
      their separate treatments. ...
```
I would assume that a host stack as a local API so that applications can signal L4S vs. classic traffic.

Also, it is unclear (to me) whether it is the sending part of the host or the receiving part ?

Finally, with some transport services (e.g., QUIC) moving to the user space, is the above statement still applicable ?

### Section 4.1

Unsure how `unassignment of an identifier` relates to item a) below.

### Section 6.1

Again, please bear with my lack of knowledge, but I wonder what is the 'load' in the text below. Is the 'load' only interactive traffic (then I wonder how there could be no congestion) or is it a mix with non-interactive traffic (the proverbial SW upgrade, which can be delayed):
```
   With the L4S approach, the following existing applications also
   experience significantly better quality of experience under load:
```

### Section 6.3

This section could probably be extended as Wi-Fi & satellite links have properties going beyond latency (e.g., packet loss).

For a non native English reader, the use of 'pole' is really hard to parse.

### Section 8.1 and others

I wonder whether this section belongs to security considerations or rather to deployment considerations.

### Section 8.4

The text does not convince me that ECN can be trusted (i.e., the network provides ECN integrity). Should there simply be some text about 'best effort' ?

### Privacy considerations

Does the use of visible ECN marking leak some information about its (potentially encrypted) content ?

## NITS

### E.g. comma

AFAIK, all 'e.g.' must be followed by a comma.

### Spec

Suggest to use 'specification' rather than 'spec' ;-)

## Notes

This review is in the ["IETF Comments" Markdown format][ICMF], You can use the
[`ietf-comments` tool][ICT] to automatically convert this review into
individual GitHub issues. 

[ICMF]: https://github.com/mnot/ietf-comments/blob/main/format.md
[ICT]: https://github.com/mnot/ietf-comments


************* 
As noted in https://www.ietf.org/blog/handling-iesg-ballot-positions/, a
DISCUSS ballot is a request to have a discussion; I really think that the
document would be improved with a change here, but can be convinced otherwise.

# GEN AD review of draft-ietf-tsvwg-l4s-arch-19

CC @larseggert

## Comments

### "Abstract", paragraph 0
```
  Abstract
```
Does the abstract really need to be this long?

### Section 3, paragraph 1
```
     Note: The following definitions are copied from the L4S ECN
     spec [I-D.ietf-tsvwg-ecn-l4s-id] for convenience.  If there are
     accidental differences, those in [I-D.ietf-tsvwg-ecn-l4s-id] take
     precedence.
```
Please instruct the RFC Editor to copy in the definitions from
[I-D.ietf-tsvwg-ecn-l4s-id] during production and to remove the note above.

## Nits

All comments below are about very minor potential issues that you may choose to
address in some way - or ignore - as you see fit. Some were flagged by
automated tools (via https://github.com/larseggert/ietf-reviewtool), so there
will likely be some false positives. There is no need to let me know what you
did with these suggestions.

### Outdated references

Reference `[RFC7540]` to `RFC7540`, which was obsoleted by `RFC9113` (this may
be on purpose).

Reference `[RFC4960]` to `RFC4960`, which was obsoleted by `RFC9260` (this may
be on purpose).

### URLs

These URLs in the document did not return content:

 * https://www.bell-labs.com/usr/koen.de_schepper

These URLs in the document can probably be converted to HTTPS:

 * http://doi.acm.org/10.1145/2663716.2663730
 * http://www.cablelabs.com/wp-content/uploads/2013/11/Active_Queue_Management_Algorithms_DOCSIS_3_0.pdf
 * http://bobbriscoe.net/
 * http://dl.acm.org/citation.cfm?doid=2910017.2910633
 * http://ee.lbl.gov/papers/congavoid.pdf
 * http://www.it.uc3m.es

### Grammar/style

#### Section 1, paragraph 8
```
architecture is, rather than why. Finally Section 5 justifies why each eleme
                                  ^^^^^^^
```
A comma may be missing after the conjunctive/linking adverb "Finally".

#### Section 1.1, paragraph 1
```
he document ends with the usual tail pieces, including extensive discussion
                                ^^^^^^^^^^^
```
The word "tailpieces" is spelled as one word.

#### Section 2, paragraph 2
```
es in Section 5.1 and [RFC3649]. Therefore control of queuing and utilizatio
                                 ^^^^^^^^^
```
A comma may be missing after the conjunctive/linking adverb "Therefore".

#### Section 3, paragraph 4
```
(e.g. DNS, VoIP, game sync datagrams, etc). Reno-friendly: The subset of Clas
                                      ^^^
```
A period is needed after the abbreviation "etc.".

#### Section 3, paragraph 9
```
ueue protection in this document. Otherwise the term rate policing is used.
                                  ^^^^^^^^^
```
A comma may be missing after the conjunctive/linking adverb "Otherwise".

#### Section 4.2, paragraph 8
```
o ECT(1) packets [FQ_CoDel_Thresh]. Then if there is a flow of non-ECN or ECT
                                    ^^^^
```
Consider adding a comma here.

#### Section 4.2, paragraph 11
```
 equalize flow-rates (perhaps in a similar way to Approx Fair CoDel [AFCD],
                              ^^^^^^^^^^^^^^^^
```
Consider replacing this phrase with the adverb "similarly" to avoid wordiness.

#### Section 4.2, paragraph 13
```
nd QUIC transports, amongst others. Also an L4S variant of the RMCAT SCReAM c
                                    ^^^^
```
A comma may be missing after the conjunctive/linking adverb "Also".

#### Section 4.3, paragraph 2
```
 more explicit signals can be applied so the queue can be kept short whatever
                                     ^^^
```
Use a comma before "so" if it connects two independent clauses (unless they are
closely connected and short).

#### Section 4.3, paragraph 3
```
doesn't know the round trip times of any of the flows. So if the network is
                                  ^^^^^^^^^
```
Consider simply using "of" instead.

#### Section 4.3, paragraph 3
```
e Classic approach), it has to assume a worst case RTT, otherwise long RTT fl
                                      ^
```
Use "the" before the superlative.

#### Section 5.1, paragraph 5
```
s the recovery time is 24.3 s. In contrast a scalable congestion control like
                                  ^^^^^^^^
```
A comma may be missing after the conjunctive/linking adverb "contrast".

#### Section 5.2, paragraph 6
```
ll treats ECN and drop the same. Therefore ABE exploits any lower queuing de
                                 ^^^^^^^^^
```
A comma may be missing after the conjunctive/linking adverb "Therefore".

#### Section 5.2, paragraph 6
```
rt-up phase. L4S complements BBR. Indeed BBRv2 can use L4S ECN where availab
                                  ^^^^^^
```
A comma may be missing after the conjunctive/linking adverb "Indeed".

#### Section 5.2, paragraph 11
```
displayed a viewport taken from a 360 degree camera in a racing car. The user
                                  ^^^^^^^^^^
```
When "360-degree" is used as a modifier, it is usually spelled with a hyphen.

#### Section 5.2, paragraph 17
```
 packets while building each burst. WiFi, PON and cable all involve such pack
                                    ^^^^
```
Did you mean "Wi-Fi"? (This is the officially approved term by the Wi-Fi
Alliance.).

#### Section 6.1, paragraph 1
```
ifically: * Radio links (cellular, WiFi, satellite) that are distant from th
                                   ^^^^
```
Did you mean "Wi-Fi"? (This is the officially approved term by the Wi-Fi
Alliance.).

#### Section 6.2, paragraph 12
```
ould often next be needed where the WiFi links in a home sometimes become th
                                    ^^^^
```
Did you mean "Wi-Fi"? (This is the officially approved term by the Wi-Fi
Alliance.).

#### Section 6.4.1, paragraph 6
```
stream bottlenecks in one go ---whether or not all the origin servers were up
                                ^^^^^^^^^^^^^^
```
Consider shortening this phrase to just "whether". It is correct though if you
mean "regardless of whether".

#### Section 6.4.2, paragraph 1
```
ll clients that support AccECN (whether or not they support L4S as well). And
                                ^^^^^^^^^^^^^^
```
Consider shortening this phrase to just "whether". It is correct though if you
mean "regardless of whether".

#### Section 6.4.2, paragraph 14
```
 how operators can use an additional local classifiers (see section 5.4 of th
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```
The plural noun "classifiers" cannot be used with the article "an". Did you
mean "an additional local classifier" or "additional local classifiers"?

#### Section 6.4.3, paragraph 8
```
yments of new bursty malware, in a similar way to how traffic from flooding
                              ^^^^^^^^^^^^^^^^
```
Consider replacing this phrase with the adverb "similarly" to avoid wordiness.

#### Section 6.4.4, paragraph 1
```
 distributed across a subnet inter-communicating using lower layer control me
                             ^^^^^^^^^^^^^^^^^^^
```
This word is normally spelled as one.

#### Section 8.1, paragraph 1
```
re, which works without them (in a similar way to how the Internet works wit
                              ^^^^^^^^^^^^^^^^
```
Consider replacing this phrase with the adverb "similarly" to avoid wordiness.

#### Section 8.2, paragraph 7
```
TCP BBR v2 Alpha/Preview Release", github repository; Linux congestion contr
                                   ^^^^^^
```
The official name of this software platform is spelled GitHub.

#### Section 9, paragraph 7
```
net: Understanding and Solutions", Masters Thesis; Karlstad Uni, Dept of Math
                                   ^^^^^^^
```
For an academic degree, use "Master's".

#### Section 9, paragraph 28
```
[SCReAM] Johansson, I., "SCReAM", github repository; , <https://github.com/Er
                                  ^^^^^^
```
The official name of this software platform is spelled GitHub.

#### Section 9, paragraph 29
```
ng for 4G in a network simulator", Masters Thesis, Uni Oslo , June 2019. Ack
                                   ^^^^^^^
```
For an academic degree, use "Master's".

## Notes

This review is in the ["IETF Comments" Markdown format][ICMF], You can use the
[`ietf-comments` tool][ICT] to automatically convert this review into
individual GitHub issues. Review generated by the [`ietf-reviewtool`][IRT].

[ICMF]: https://github.com/mnot/ietf-comments/blob/main/format.md
[ICT]: https://github.com/mnot/ietf-comments
[IRT]: https://github.com/larseggert/ietf-reviewtool

Hi,

Thanks for the spec.  This is outside of my area of expertise, and I've been time constrained this week, but I found this document to be pretty easy to read and certainly the technology sounds interesting and potentially very useful so that you for your work on this.

I guess that my only criticism, and it is not easily/readably actionable, is that some of the sections felt a little bit repetitive from early parts in the document, and perhaps earlier parts of the document could possible benefit from being shortened or made more concise.  One obvious change that has already been suggested by others, getting the abstract down to a single paragraph summary would probably be a fairly easy improvement.  Other than that, I don't really have any other actionable suggestions, sorry.

This paragraph:

       To enable L4S, the standards track Classic ECN spec. [RFC3168]
       has had to be updated to allow L4S packets to depart from the
       'equivalent to drop' constraint.  [RFC8311] is a standards track
       update to relax specific requirements in RFC 3168 (and certain
       other standards track RFCs), which clears the way for the
       experimental changes proposed for L4S.  [RFC8311] also
       reclassifies the original experimental assignment of the ECT(1)
       codepoint as an ECN nonce [RFC3540] as historic.

also stood out as being useful whilst the document was being developed, but perhaps less useful in a published RFC, and hence I did wonder whether this paragraph could be elided or shortened.

Regards,
Rob