Comcast ISP Low Latency Deployment Design Recommendations
draft-livingood-low-latency-deployment-00

Independent Stream                                          J. Livingood
Internet-Draft                                                   Comcast
Intended status: Informational                           24 October 2022
Expires: 27 April 2023

       Comcast ISP Low Latency Deployment Design Recommendations
               draft-livingood-low-latency-deployment-00

Abstract

   The IETF's Transport Area Working Group (TSVWG) has finalized
   experimental RFCs for Low Latency, Low Loss, Scalable Throughput
   (L4S) and new Non-Queue-Building (NQB) per hop behavior.  These
   documents do a good job of describing a new architecture and protocol
   for deploying low latency networking.  But as is normal for many such
   standards, especially those in experimental status, certain design
   decisions are ultimately left to implementers.  This document
   explores the potential implications of key deployment design
   decisions and makes recommendations for those decisions that may help
   drive adoption.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 27 April 2023.

Copyright Notice

   Copyright (c) 2022 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

Livingood                 Expires 27 April 2023                 [Page 1]
Internet-Draft          ISP L4S Deployment Design           October 2022

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  A New Understanding of Application Needs  . . . . . . . . . .   3
   3.  Network Neutrality and Low Latency Networking . . . . . . . .   4
     3.1.  Prioritization: Same for All Traffic  . . . . . . . . . .   5
     3.2.  Thoughput: Shared Across All Traffic  . . . . . . . . . .   5
     3.3.  A New Network Capability  . . . . . . . . . . . . . . . .   5
   4.  Recommended Deployment Design Decisions . . . . . . . . . . .   5
     4.1.  Only Applications Mark Traffic  . . . . . . . . . . . . .   6
     4.2.  All Providers Welcome . . . . . . . . . . . . . . . . . .   7
     4.3.  Cable Modem Device Choice . . . . . . . . . . . . . . . .   7
     4.4.  Opt Out During Experimental Phase . . . . . . . . . . . .   7
   5.  Summary of Recommended Deployment Design Decisions  . . . . .   7
   6.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   8
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   8
   8.  Security Considerations . . . . . . . . . . . . . . . . . . .   8
   9.  Privacy Considerations  . . . . . . . . . . . . . . . . . . .   8
   10. Revision History  . . . . . . . . . . . . . . . . . . . . . .   8
   11. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . .   8
   12. Informative References  . . . . . . . . . . . . . . . . . . .   8
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  10

1.  Introduction

   The IETF's Transport Area Working Group (TSVWG) has finalized
   experimental RFCs for Low Latency, Low Loss, Scalable Throughput
   (L4S) and Non-Queue-Building (NQB) per hop behavior
   [I-D.ietf-tsvwg-l4s-arch] [I-D.ietf-tsvwg-aqm-dualq-coupled]
   [I-D.ietf-tsvwg-ecn-l4s-id] [I-D.ietf-tsvwg-l4sops]
   [I-D.ietf-tsvwg-nqb] [I-D.ietf-tsvwg-dscp-considerations].  These
   documents do a good job of describing a new architecture and protocol
   for deploying low latency networking.  But as is normal for many such
   standards, especially those in experimental status, certain design
   decisions are ultimately left to implementers.

   This document explores the potential implications of key deployment
   design decisions and makes recommendations for those decisions that
   may help drive adoption.  In particular, there are best practices
   based on prior experience as a network operator that should be
   considered and there are network neutrality types of considerations
   as well.  These technologies are benign on their own, but the way

Livingood                 Expires 27 April 2023                 [Page 2]
Internet-Draft          ISP L4S Deployment Design           October 2022

   they are operationally implemented can determine whether they are
   ultimately perceived positively and adopted by the broader Internet
   ecosystem.  That is a key issue for low latency networking, because
   the more applications developers and edge platforms that adopt new
   packet marking for low latency traffic, then the greater the value to
   end users, so ensuring it is received well is key to driving strong
   initial adoption.

   It is worth stating though that these decisions are not embedded in
   or inherent to L4S and NQB per se, but are decisions that can change
   depending upon differing technical, regulatory, business or other
   requirements.  Even two network operators with the same type of
   access technology in the same market area may choose to implement in
   different ways.  Nevertheless, this document suggests that certain
   specific deployment decisions can help maximize the value of low
   latency networking to both users and network operators.

   It is also apparent from the IETF's work that it is clear that nearly
   all modern application types need low latency to some degree and that
   applications are best positioned to express their needs via
   application code and packet marking.  Furthermore, unlike with
   bandwidth priority on a highly/fully utilized link, low latency is
   not a zero sum game - everyone can potentially have lower latency at
   no one else's expense.

   For additional background on latency and why latency matters so much
   to the Internet, please read [BITAG]

2.  A New Understanding of Application Needs

   In the course of working to improve the responsiveness of network
   protocols, the IETF concluded with their L4S and NQB work that there
   were fundamentally two types of Internet traffic and that these two
   major traffic types could benefit from having separate network
   processing queues in order to improve the way the Internet works for
   all applications, and especially for interactive applications.

   One of the two major traffic types is Queue Building (QB) - things
   like file downloads and backups that are designed utilize as much
   network capacity as possible but for which users are not interacting
   with in real-time.  The other was Non-Queue-Building (NQB) - such as
   DNS lookups, voice interaction with artificial intelligence (AI)
   assistants, video conferencing, gaming, and so on.  NQB flows tend to
   be limited in their capacity needs - for example a DNS lookup will
   not need to consume the full capacity of an end user's connection -
   but the end user is highly sensitive to any delays.

Livingood                 Expires 27 April 2023                 [Page 3]
Internet-Draft          ISP L4S Deployment Design           October 2022

   Thus, the IETF created specifications for how two different network
   processing queues can be designed and operated.  Early results, such
   as from the IETF-114 hackathon [IETF-114-Slides], demonstrate that
   L4S and NQB (a.k.a. dual queue networking, and simply "low latency
   networking" hereafter) can work across a variety of access network
   technologies and deliver extraordinary levels of responsiveness for a
   variety of applications.  It seems likely that this new capability
   will enable entirely new classes of applications to become possible,
   driving a wave of new Internet innovation, while also improving the
   applications people use today.

3.  Network Neutrality and Low Latency Networking

   Network Neutrality (a.k.a.  Net Neutrality, and NN hereafter) is a
   concept that can mean a variety of things within a country, as well
   as between different countries, based on different opinions, market
   structures, business practices, laws, and regulations.  Generally
   speaking, at least in the context of the United States' marketplace,
   it has come to mean that Internet Service Providers (ISPs) should not
   block, throttle, or deprioritize lawful application traffic, and
   should not engage in paid prioritization, among other things.  The
   meaning of NN can be complex and ever changing, so the specific
   details are out of scope for this document.  Despite that, NN
   concerns certainly bear on the deployment of new technologies by
   ISPs, at least in the US, and so should be taken into account in
   making deployment design decisions.

   It is also possible that there can be confusion for people who are
   not deep in this highly technical subject between prioritization,
   provisioned end user capacity (throughput), and low latency
   networking.  As it is envisioned in the design of the protocols, the
   addition of a low latency packet processing queue at a network link
   is merely a second packet queue and does not mean that this queue is
   prioritized or that it has different or greater capacity from the
   classic queue.  Thus, a low latency queue does not create a so-called
   "fast lane" (in the way that this term is used in policy-related
   discussions in the U.S. to describe higher than best effort priority
   or greater capacity being assigned to some traffic compared to
   default traffic) - but there are certainly other NN considerations in
   the operational implementation worth exploring.

Livingood                 Expires 27 April 2023                 [Page 4]
Internet-Draft          ISP L4S Deployment Design           October 2022

3.1.  Prioritization: Same for All Traffic

   As noted above, a key aspect of NN goals is that traffic to certain
   Internet destinations or for certain applications should not be
   prioritized over other Internet traffic.  This means in practice that
   all Internet traffic in an ISP network should be carried at the same
   (best effort) priority and that any network management practices
   imposed by the network should be protocol (application) agnostic.
   Low latency networking is fully consistent with this aspect of NN,
   because it is designed so that all traffic is treated on a best
   effort (BE) basis in the ISP network (this may not necessarily be the
   case for a user's in-home Wi-Fi network due to the particulars of how
   the IEEE 802.11 wireless protocol functions at the current time).

   In addition, as noted above, unlike with bandwidth priority on a
   highly/fully utilized link, low latency is not a zero sum game -
   everyone can potentially have lower latency at no one else's expense.

3.2.  Thoughput: Shared Across All Traffic

   Low latency networking is also consistent with the NN goal of not
   creating a fast lane, because the same end user throughput in an ISP
   access network is shared between both classic and low latency (L4S/
   NQB) queues.  Thus, applications do not get access to greater or
   different throughput depending on whether or not the leverage low
   latency networking.

3.3.  A New Network Capability

   Ultimately, the emergence of low latency networking represents a
   fundamental new network capability that applications can choose to
   utilize as their needs dictate.  It reflects a new ground truth about
   two fundamentally different types of application traffic and
   demonstrates that networks continue to evolve in exciting ways.

4.  Recommended Deployment Design Decisions

   Like any network or system, a good deployment design and
   configuration matters and can be the difference between a well-
   functioning and accepted design and one that experiences problems and
   faces opposition.  In the context of deploying low latency networking
   in an ISP network, this document describes some recommended
   deployment design decisions that should help to ensure a deployment
   is resilient, well-accepted, and creates the environment for
   generating strong network effects.  In contrast, creating barriers to
   adoption in the early stages through design and policy decisions will
   presumably reduce the predicted potential network effect, thus
   choking off further investment across the Internet ecosystem, leading

Livingood                 Expires 27 April 2023                 [Page 5]
Internet-Draft          ISP L4S Deployment Design           October 2022

   to a vicious circle of decline - and then the potential value is
   never realized.

4.1.  Only Applications Mark Traffic

   Only applications should mark traffic, not the network.  This is for
   several reasons:

   *  According to the end-to-end principle, this function is best
      delegated to the edge of the network as an architectural best
      practice (the edge being the application in this case).

   *  Application marking maintains the loose coupling between the
      application and network layers, eliminating the need for close
      coordination between networks and application developers.

   *  Application developers know best whether their application is
      compatible with low latency networking and which aspects of their
      traffic flows will or will not benefit.

   *  Application traffic is almost entirely encrypted, which makes it
      very difficult for networks to accurately determine application
      protocols and to further infer which flows will benefit from low
      latency and which flows may be harmed because they need to build a
      queue.

   *  To correctly utilize L4S, the application needs to use a scalable
      congestion control algorithm in order to use the packet marking
      for L4S (DSCP mark).  But only the application (not the network)
      knows what congestion control it is using.  So, with L4S, the
      network cannot properly mark on behalf of the application.

   *  Network operators and equipment vendors attempting to infer
      application type and application need will always make mistakes,
      incorrectly classifying traffic [Lotus], and potentially
      negatively affecting certain flows.

   *  The pace of innovation and iteration is necessarily faster-moving
      in the application edge at layer 7, rather than in the network at
      layer 3 (and below) - where there is greater standards stability
      and a lower rate of major changes.  As a result, the application
      layer is best suited to rapid experimentation and iteration.
      Network operators and equipment vendors trying to infer
      application needs will in comparison always be in a reactive mode,
      one step behind changes made in applications.

Livingood                 Expires 27 April 2023                 [Page 6]
Internet-Draft          ISP L4S Deployment Design           October 2022

4.2.  All Providers Welcome

   Any provider should be able to mark their traffic for the low latency
   queue, with no restrictions other than standards compliance or other
   reasonable and openly documented technical guidelines.  This
   maintains the loose cross-layer coupling that is a key tenet of the
   Internet's architecture by eliminating or greatly reducing any need
   for application providers and networks to coordinate on deployment
   (though such coordination is normal in the early experimental phase
   of any deployment).

   As noted above, this is another example that low latency networking
   will have strong network effects, any barriers to adoption such as
   this should be avoided in order to maximize the value to users and
   the network of a new low latency queue.

4.3.  Cable Modem Device Choice

   Both customer-owned and leased cable modem devices should be
   supported.  This avoids the risk that an ISP can be perceived as
   giving preference to their own network demarcation devices, which may
   carry some monthly recurring fee or other cost.  This also means that
   retail device manufacturers need to make the necessary development
   investment to correctly implement low latency networking, though this
   may not interest or may be outside the capabilities of some
   organizations.  In any case, the more devices that implement then
   adoption is broader, positively driving network effects.

4.4.  Opt Out During Experimental Phase

   In early phases of deployment of low latency networking, ISPs should
   consider making available some mechanism for users to opt out of
   (deactivate) it.  If low latency networking is working correctly, it
   seems extremely unlikely that a user should ever want or need to turn
   it off.  But is is also possible that it may be desirable in some
   troubleshooting situations to turn it off, such as in in cases where
   a particular application has incorrectly implemented low latency
   networking and the developer is working on a bug fix for an extended
   period of time.  As application use of this technology matures, it
   seems likely that there will not be a long term need or practical
   benefit to having an opt out mechanism (and it may be counter
   productive if it insulates developers from having to fix bugs or
   misconfigurations in their software).

5.  Summary of Recommended Deployment Design Decisions

   1  Only Applications Mark Traffic: Not the network

Livingood                 Expires 27 April 2023                 [Page 7]
Internet-Draft          ISP L4S Deployment Design           October 2022

   2  All Providers Welcome: Any provider can mark with no restrictions
      other than standards compliance or other reasonable and openly
      documented technical guidelines

   3  Device Choice: Both customer-owned and leased cable modem devices
      supported

   4  User Opt Out: Customers can opt out during the experimental phase

6.  Acknowledgements

   Thanks to Bob Briscoe, Sebnem Ozer, and Greg White for their review
   and feedback on this document.

7.  IANA Considerations

   RFC Editor: Please remove this section before publication.

   This memo includes no requests to or actions for IANA.

8.  Security Considerations

   RFC Editor: Please remove this section before publication.

   This memo includes no security considerations.

9.  Privacy Considerations

   RFC Editor: Please remove this section before publication.

   This memo includes no security considerations.

10.  Revision History

   RFC Editor: Please remove this section before publication.

   v00: First draft

   v01: ...

11.  Open Issues

   RFC Editor: Please remove this section before publication.

   - Open issues are being tracked in a GitHub repository for this
   document at https://github.com/jlivingood/IETF-L4S-Deployment/issues

12.  Informative References

Livingood                 Expires 27 April 2023                 [Page 8]
Internet-Draft          ISP L4S Deployment Design           October 2022

   [I-D.ietf-tsvwg-l4s-arch]
              Briscoe, B., Schepper, K. D., Bagnulo, M., and G. White,
              "Low Latency, Low Loss, Scalable Throughput (L4S) Internet
              Service: Architecture", Work in Progress, Internet-Draft,
              draft-ietf-tsvwg-l4s-arch-20, 29 August 2022,
              <https://www.ietf.org/archive/id/draft-ietf-tsvwg-l4s-
              arch-20.txt>.

   [I-D.ietf-tsvwg-aqm-dualq-coupled]
              Schepper, K. D., Briscoe, B., and G. White, "DualQ Coupled
              AQMs for Low Latency, Low Loss and Scalable Throughput
              (L4S)", Work in Progress, Internet-Draft, draft-ietf-
              tsvwg-aqm-dualq-coupled-25, 29 August 2022,
              <https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-
              dualq-coupled-25.txt>.

   [I-D.ietf-tsvwg-ecn-l4s-id]
              Schepper, K. D. and B. Briscoe, "Explicit Congestion
              Notification (ECN) Protocol for Very Low Queuing Delay
              (L4S)", Work in Progress, Internet-Draft, draft-ietf-
              tsvwg-ecn-l4s-id-29, 29 August 2022,
              <https://www.ietf.org/archive/id/draft-ietf-tsvwg-ecn-l4s-
              id-29.txt>.

   [I-D.ietf-tsvwg-l4sops]
              White, G., "Operational Guidance for Deployment of L4S in
              the Internet", Work in Progress, Internet-Draft, draft-
              ietf-tsvwg-l4sops-03, 28 April 2022,
              <https://www.ietf.org/archive/id/draft-ietf-tsvwg-l4sops-
              03.txt>.

   [I-D.ietf-tsvwg-nqb]
              White, G. and T. Fossati, "A Non-Queue-Building Per-Hop
              Behavior (NQB PHB) for Differentiated Services", Work in
              Progress, Internet-Draft, draft-ietf-tsvwg-nqb-13, 21
              October 2022, <https://www.ietf.org/archive/id/draft-ietf-
              tsvwg-nqb-13.txt>.

   [I-D.ietf-tsvwg-dscp-considerations]
              Custura, A., Fairhurst, G., and R. Secchi, "Considerations
              for Assigning a new Recommended DiffServ Codepoint
              (DSCP)", Work in Progress, Internet-Draft, draft-ietf-
              tsvwg-dscp-considerations-05, 10 August 2022,
              <https://www.ietf.org/archive/id/draft-ietf-tsvwg-dscp-
              considerations-05.txt>.

Livingood                 Expires 27 April 2023                 [Page 9]
Internet-Draft          ISP L4S Deployment Design           October 2022

   [BITAG]    Broadband Internet Technical Advisory Group, "Latency
              Explained", 10 January 2022,
              <https://bitag.org/documents/BITAG_latency_explained.pdf>.

   [Lotus]    Eckerseley, P., "Packet Forgery By ISPs: A Report on the
              Comcast Affair", 28 November 2007,
              <https://www.eff.org/wp/packet-forgery-isps-report-
              comcast-affair>.

   [IETF-114-Slides]
              White, G., "First L4S Interop Event @ IETF Hackathon", 25
              July 2022,
              <https://datatracker.ietf.org/meeting/114/materials/
              slides-114-tsvwg-update-on-l4s-work-in-ietf-114-hackathon-
              00.pdf>.

Author's Address

   Jason Livingood
   Comcast
   Philadelphia, PA
   United States of America
   Email: jason_livingood@comcast.com

Livingood                 Expires 27 April 2023                [Page 10]