[Search] [txt|html|xml|pdf|bibtex] [Tracker] [Email] [Nits]

Versions: 00                                                            
Network Working Group                                           K. Corre
Internet-Draft                                                      EXFO
Intended status: Informational                         17 September 2021
Expires: 21 March 2022


                 Framework for QUIC Throughput Testing
                 draft-corre-quic-throughput-testing-00

Abstract

   This document adapts the [RFC6349] Framework for TCP Throughput
   Testing to the [RFC9000] QUIC protocol.  The adapted framework
   describes a practical methodology for measuring end-to-end QUIC
   Throughput in a managed IP network.  The goal of the methodology is
   to provide a better indication in regard to user experience.  In this
   framework, QUIC, UDP, and IP parameters are specified to optimize
   QUIC Throughput.

Discussion Venues

   This note is to be removed before publishing as an RFC.

   Discussion of this document takes place on the QUIC Working Group
   mailing list (quic@ietf.org), which is archived at
   https://mailarchive.ietf.org/arch/browse/quic/.

   Source for this draft and an issue tracker can be found at
   https://github.com/Sparika/draft-corre-quic-throughput-testing.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 21 March 2022.





Corre                     Expires 21 March 2022                 [Page 1]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


Copyright Notice

   Copyright (c) 2021 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Simplified BSD License text
   as described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.1.  Impacting changes between TCP and QUIC  . . . . . . . . .   3
   2.  Conventions and Definitions . . . . . . . . . . . . . . . . .   5
   3.  Methodology . . . . . . . . . . . . . . . . . . . . . . . . .   5
     3.1.  Path MTU  . . . . . . . . . . . . . . . . . . . . . . . .   6
     3.2.  Round-Trip Time (RTT) and Bottleneck Bandwidth (BB) . . .   6
     3.3.  Measuring RTT . . . . . . . . . . . . . . . . . . . . . .   6
     3.4.  Measuring BB  . . . . . . . . . . . . . . . . . . . . . .   6
     3.5.  Measuring QUIC Throughput . . . . . . . . . . . . . . . .   6
     3.6.  Minimum QUIC Congestion Control Credit  . . . . . . . . .   7
   4.  QUIC Metrics  . . . . . . . . . . . . . . . . . . . . . . . .   8
     4.1.  Transfert Time Ratio  . . . . . . . . . . . . . . . . . .   8
       4.1.1.  Maximum Achievable QUIC Throughput Calculation  . . .   8
       4.1.2.  QUIC Transfer Time and Transfer Time Ratio
               Calculation . . . . . . . . . . . . . . . . . . . . .  11
     4.2.  QUIC Efficiency . . . . . . . . . . . . . . . . . . . . .  12
       4.2.1.  QUIC Efficiency Percentage Calculation  . . . . . . .  13
     4.3.  Buffer Delay  . . . . . . . . . . . . . . . . . . . . . .  13
   5.  Conducting QUIC Throughput Tests  . . . . . . . . . . . . . .  14
     5.1.  Using Multiple QUIC Streams . . . . . . . . . . . . . . .  14
     5.2.  1-RTT and 0-RTT . . . . . . . . . . . . . . . . . . . . .  15
     5.3.  Results Interpretation  . . . . . . . . . . . . . . . . .  15
   6.  Security Considerations . . . . . . . . . . . . . . . . . . .  16
     6.1.  0-RTT attack {#0RTTattack}  . . . . . . . . . . . . . . .  17
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  17
   Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . .  17
   References  . . . . . . . . . . . . . . . . . . . . . . . . . . .  17
     Normative References  . . . . . . . . . . . . . . . . . . . . .  17
     Informative References  . . . . . . . . . . . . . . . . . . . .  18
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  18





Corre                     Expires 21 March 2022                 [Page 2]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


1.  Introduction

   This document adapts the [RFC6349] Framework for TCP Throughput
   Testing to the QUIC protocol [RFC9000].  RFC6349 defines a
   methodology for testing sustained TCP Layer performance.  Like TCP
   [RFC793], QUIC is a connection-oriented transport protocol.  However,
   there are multiple differences between both protocols and some of
   these differences impact the throughput testing methodology.

   For easier reference, this document follows the same organization as
   [RFC6349] and each section describes changes relevant to the
   equivalent [RFC6349] section.  The scope and goals section is omitted
   as the objective of this document is precisely to remain within the
   same scope and goals of [RFC6349].  Section 3 presents changes to the
   individual steps of the [RFC6349] methodology.  Section 4 describes
   the three main metrics used to measure QUIC Throughput.  Section 5
   covers conducting QUIC Throughput testing.  In particular, the
   section presents QUIC streams usage in the context of QUIC Throughput
   testing and discusses possible results interpretation.  Finally,
   Section 6 presents additional security considerations.

1.1.  Impacting changes between TCP and QUIC

   This methodology proposes QUIC Throughput performance testing
   focusing on:

   *  Bottleneck Bandwidth (BB)

   *  Round-Trip Time (RTT)

   *  Send and Receive Socket Buffers

   *  Available Control-Flow Credits and QUIC CWND

   *  Path Maximum Transmission Unit (MTU)

   There are multiple changes between TCP and QUIC impacting the
   throughput testing methodology.  Firstly, the multiple QUIC headers
   and frames result in overheads variable in size which in turn impact
   the computation for the maximum achievable QUIC throughput, in
   particular the Transfert Time Ration metric presented in Section 4.1.
   Secondly, QUIC provides streams that can be used for throughput
   testing but which may also result in variable overheads.

   TODO:






Corre                     Expires 21 March 2022                 [Page 3]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


   Thirdly, QUIC provides congestion control genericity and receiver-
   controlled control flow credits instead of the TCP sliding receiver
   window.  While QUIC Loss Detection and Congestion Control [RFC9002]
   exmplifies with a congestion controller similar to TCP
   NewReno~[RFC6582], the signals QUIC provides for congestion control
   are generic and are designed to support different sender-side
   algorithms.  In this document, the achievable QUIC Throughput is the
   amount of data per unit of time that QUIC transports when in the QUIC
   Equilibrium state.  Derived from Round-Trip Time (RTT) and network
   Bottleneck Bandwidth (BB), the Bandwidth-Delay Product (BDP)
   determines the Send and Received Socket buffer sizes required to
   achieve the maximum QUIC Throughput.  Throughout this document,
   "maximum achievable throughput" refers to the theoretical achievable
   throughput when QUIC is in the Equilibrium state.  In this document,
   we assume that QUIC uses a transmitting side congestion controller
   with a congestion window (QUIC CWND) and congestion controller states
   similar to TCP NewReno.

                    New path or      +------------+
               persistent congestion |   Slow     |
           (O)---------------------->|   Start    |
                                     +------------+
                                           |
                                   Loss or |
                           ECN-CE increase |
                                           v
    +------------+     Loss or       +------------+
    | Congestion |  ECN-CE increase  |  Recovery  |
    | Avoidance  |------------------>|   Period   |
    +------------+                   +------------+
              ^                            |
              |                            |
              +----------------------------+
                 Acknowledgment of packet
                   sent during recovery

      Figure 1: QUIC NewReno Congestion Control States and Transitions

   On the receiving side, QUIC does not use a sliding receiver window
   and instead uses a control flow credits mechanism to inform the
   transmitting end on the maximum total amount of data the receiver is
   willing to receive for each stream and connection.  The amount of
   available control flow credits does not directly control the QUIC
   Throughput but insufficient credits may lead to a stream or a whole
   connection being rate-limited.  This is discussed in Section 3.6.






Corre                     Expires 21 March 2022                 [Page 4]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


2.  Conventions and Definitions

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

3.  Methodology

   Overall, the testing methodology remains the same as the one
   described in [RFC6349] with a few minor differences.  The following
   represents the sequential order of steps for this testing
   methodology:

   1.  Identify the Path MTU.  As QUIC relies on UDP, Packetization
       Layer Path MTU Discovery for Datagram Transports (Datagram PMTUD)
       [RFC8899] SHOULD be conducted instead of Packetization Layer Path
       MTU Discovery (PLPMTUD).  It is important to identify the path
       MTU so that the QUIC Throughput Test Device (TTD) is configured
       properly to avoid fragmentation.

   2.  Baseline Round-Trip Time and Bandwidth.  This step establishes
       the inherent, non-congested Round-Trip Time (RTT) and the
       Bottleneck Bandwidth (BB) of the end-to-end network path.  These
       measurements are used to provide estimates of the QUIC minimum
       congestion control credit and Send Socket Buffer sizes that
       SHOULD be used during subsequent test steps.

   3.  QUIC Connection Throughput Tests.  With baseline measurements of
       Round-Trip Time and Bottleneck Bandwidth, single- and multiple-
       QUIC-connection throughput tests SHOULD be conducted to baseline
       network performance.

   These three (3) steps are detailed in Section 3.1 to Section 3.5.

   Regarding, the QUIC TTD, the same key characteristics and criteria
   SHOULD be considered than with the [RFC6349] TCP TTD.  The test host
   MAY be a standard computer or a dedicated communications test
   instrument and in both cases, it MUST be capable of emulating both a
   client and a server.  One major difference is that contrary to TCP,
   QUIC may not be provided by OSs usually providing transport protocol
   implementations and may instead be sourced by an application from a
   third-party library.  These different implementations may present "a
   large behavioural heterogeneity" [marx2020same] potentially impacting
   the QUIC throughput measurements.  Thus, in addition to the OS
   version (e.g. a specific LINUX OS kernel version), the QUIC
   implementation used by the test hosts MUST be considered, including



Corre                     Expires 21 March 2022                 [Page 5]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


   used congestion control algorithms available and supported QUIC
   options.  The QUIC test implementation and hosts MUST be capable of
   generating and receiving stateful QUIC test traffic at the full BB of
   the NUT.

   QUIC includes precise RTT statistics and MAY be directly used to
   gather RTT statistics if the QUIC implementation exposes them.  As
   QUIC packets are mostly encrypted, packet capture tools do not allow
   to measure QUIC RTT and retransmissions and SHOULD not be used for
   this methodology.

3.1.  Path MTU

   QUIC implementations SHOULD use Datagram Path MTU Discovery
   techniques (Datagram PMTUD).

3.2.  Round-Trip Time (RTT) and Bottleneck Bandwidth (BB)

   Before stateful QUIC testing can begin, it is important to determine
   the baseline RTT (i.e. non-congested inherent delay) and BB of the
   end-to-end network to be tested.  These measurements are used to
   calculate the BDP and to provide estimates for the congestion control
   credit and Send Socket Buffer sizes that SHOULD be used in subsequent
   test steps.

3.3.  Measuring RTT

   In addition to the solutions proposed in [RFC6349] for measuring RTT,
   short QUIC sessions MAY be employed to baseline RTT values off-peak
   hours and outside of test sessions.  This solution requires that the
   QUIC implementation expose RTT measurements [RFC9002]. latest_rtt
   SHOULD be used to sample baseline RTT using the minimum observed
   latest_RTT during off-peak hours and outside of test sessions.
   instead of latest_rtt, min_rtt MAY be used to sample baseline RTT
   during off-peak hours and outside of test sessions. smoothed_rtt MUST
   not be used to baseline RTT.

3.4.  Measuring BB

   This section is unchanged.

3.5.  Measuring QUIC Throughput

   This methodology specifically defines QUIC Throughput measurement
   techniques to verify maximum achievable QUIC performance in a managed
   business-class IP network.





Corre                     Expires 21 March 2022                 [Page 6]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


   With baseline measurements of RTT and BB from Section 3.2, a series
   of single- and/or multiple-QUIC-connection throughput tests SHOULD be
   conducted.

   Like with TCP, the number of trials and the choice between single or
   multiple QUIC connections will be based on the intention of the test.
   In all circumstances, it is RECOMMENDED to run the tests in each
   direction independently first and then to run them in both directions
   simultaneously.  It is also RECOMMENDED to run the tests at different
   times of the day.

   The metrics measured are the QUIC Transfer Time Ratio, the QUIC
   Efficiency Percentage, and the Buffer Delay Percentage.  These 3
   metrics are defined in Section 4 and MUST be measured in each
   direction.

3.6.  Minimum QUIC Congestion Control Credit

   As for [RFC6349] TCP throughput testing, the QUIC throughput test
   MUST ensure that the QUIC performance is never rate-limited.  Whereas
   TCP uses a sliding receiver window (TCP RWND), QUIC relies on
   congestion control credits that are not automatically sliding.
   Instead, available control-flow credits are increased by sending
   MAX_DATA and MAX_STREAM_DATA frames.  The algorithm used to send
   these frames depends on the QUIC implementation or may be left for
   the application to control at each receiving side.

   In addition to setting Send Socket Buffer size higher than the BDP,
   the QUIC TTD MUST ensure that connections and streams always have
   Control-Flow Credits (CFC) in excess of the BDP so that any QUIC
   stream or connection is never restricted below the BDP before
   MAX_DATA and MAX_STREAM_DATA frames have a chance to arrive.  If a
   QUIC connection or stream has CFC below the minimum required CFC,
   then there is a risk of the CFC reaching 0 before the limits can be
   increased.  If a QUIC stream has CFC at 0, then that stream is rate-
   limited and there is a risk of the QUIC Throughput to not be optimal.
   Note that other streams may still manage to fill the BDP of the NUT.
   If a QUIC connection has CFC at 0, then that connection is rate-
   limited and the QUIC Throughput cannot be optimal.  The QUIC TTD MUST
   report every events and events' duration where a QUIC stream or
   connection is rate-limited.

   A QUIC implementation could implement a sliding receive window
   similar to TCP RWND by sending MAX_DATA and MAX_STREAM_DATA frames
   with each ACK frame.  In this case, the minimum CFC (i.e. initial
   data limit) is similar to the minimum TCP RWND and example numbers
   proposed in [RFC6349].  However, such a solution imposes a heavy
   overhead and it is more likely that sending of MAX_DATA and



Corre                     Expires 21 March 2022                 [Page 7]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


   MAX_STREAM_DATA frames are delayed by the QUIC implementation.  Thus,
   the minimum CFC should be set sufficiently large so that the QUIC
   implementation can send MAX_DATA and MAX_STREAM_DATA frames with new
   limits before control-flow credits are exhausted.

4.  QUIC Metrics

   The proposed metrics for measuring QUIC Throughput remain the same as
   for measuring TCP Throughput with some minor differences.

4.1.  Transfert Time Ratio

   The first metric is the QUIC Transfer Time Ratio, which is the ratio
   between the Actual QUIC Transfer Time versus the Ideal QUIC Transfer
   Time.  The Actual QUIC Transfer Time is simply the time it takes to
   transfer a block of data across QUIC connection(s).  The Ideal QUIC
   Transfer Time is the predicted time for which a block of data SHOULD
   transfer across QUIC connection(s), considering the BB of the NUT.

         QUIC          Actual QUIC Transfer Time
     Transfer Time =  ---------------------------
         Ratio          Ideal QUIC Transfer Time

   The Ideal QUIC Transfer Time is derived from the Maximum Achievable
   QUIC Throughput, which is related to the BB and Layer 1/2/3/4
   overheads associated with the network path.  The following sections
   provide derivations for the Maximum Achievable QUIC Throughput and
   example calculations for the QUIC Transfer Time Ratio.

4.1.1.  Maximum Achievable QUIC Throughput Calculation

   This section provides formulas to calculate the Maximum Achievable
   QUIC Throughput, with examples for T3 (44.21 Mbps) and Ethernet.

   On the contrary to TCP, the QUIC overhead is variable in size, in
   particular, due to variable-length fields and streams multiplexing.
   Furthermore, QUIC is a versioned protocol and the overheads may
   change from version to version.  Other underlying protocol changes
   such as IP version may also increase overheads or make them variable
   in size.  The following calculations are based on IP version 4 with
   IP headers of 20 Bytes, UDP headers of 8 Bytes, QUIC version 1, and
   within an MTU of 1500 Bytes.









Corre                     Expires 21 March 2022                 [Page 8]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


   We are interested in estimating the maximal QUIC Throughput at
   equilibrium, so we first simplify the problem by considering an
   optimistic scenario with data exchanged over 1-RTT packets with only
   a single stream frame per packet.  For the content of the STREAM
   frame, we consider that the offset field is used but not the length
   field since there is only one frame per packet.

   Here are the 1-RTT packet and STREAM frame headers as defined in
   [RFC9000].

      1-RTT Packet {
        Header Form (1) = 0,
        Fixed Bit (1) = 1,
        Spin Bit (1),
        Reserved Bits (2),
        Key Phase (1),
        Packet Number Length (2),
        Destination Connection ID (0..160),
        Packet Number (8..32),
        Packet Payload (8..),
      }

      STREAM Frame {
        Type (i) = 0x08..0x0f,
        Stream ID (i),
        [Offset (i)],
        [Length (i)],
        Stream Data (..),
      }

   Supposing all variable-length fields are encoded as the minimum size
   possible, the overhead would be 3 Bytes for QUIC 1-RTT packet and 2
   Bytes for QUIC STREAM frame.  This gives us a minimal IP/UDP/QUIC
   overhead of 20 + 8 + 3 + 2 = 33 Bytes.  Note that alternatively,
   considering all variable length fields encoded as the largest size
   possible, the overhead would be 25 Bytes for QUIC 1-RTT packet and 17
   Bytes for QUIC STREAM frame.  In this worst-case scenario, the
   overhead could mount up to 20 + 8 + 25 + 17 = 70 Bytes for one STREAM
   frame per packet.

   To estimate the Maximum Achievable QUIC Throughput, we first need to
   determine the maximum quantity of Frames Per Second (FPS) permitted
   by the actual physical layer (Layer 1) speed.

   For a T3 link, the calculation formula is:






Corre                     Expires 21 March 2022                 [Page 9]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


     FPS = T3 Physical Speed /
           ((MTU + PPP + Flags + CRC16) X 8)

     FPS = (44.21 Mbps /
           ((1500 Bytes + 4 Bytes
             + 2 Bytes + 2 Bytes) X 8 )))
     FPS = (44.21 Mbps / (1508 Bytes X 8))
     FPS = 44.21 Mbps / 12064 bits
     FPS = 3664

   Then, to obtain the Maximum Achievable QUIC Throughput (Layer 4), we
   simply use:

     (MTU - 33) in Bytes X 8 bits X max FPS

   For a T3, the maximum QUIC Throughput =

     1467 Bytes X 8 bits X 3664 FPS

     Max QUIC Throughput = 11736 bits X 3664 FPS
     Max QUIC Throughput = 43.0 Mbps

   On Ethernet, the maximum achievable Layer 2 throughput is limited by
   the maximum Frames Per Second permitted by the IEEE802.3 standard.

   The maximum FPS for 100-Mbps Ethernet is 8127, and the calculation
   formula is:

     FPS = (100 Mbps / (1538 Bytes X 8 bits))

   The maximum FPS for GigE is 81274, and the calculation formula is:

     FPS = (1 Gbps / (1538 Bytes X 8 bits))

   The maximum FPS for 10GigE is 812743, and the calculation formula is:

     FPS = (10 Gbps / (1538 Bytes X 8 bits))

   The 1538 Bytes equates to:

     MTU + Ethernet + CRC32 + IFG + Preamble + SFD
      IFG = Inter-Frame Gap
      SFD = Start of Frame Delimiter

   where MTU is 1500 Bytes, Ethernet is 14 Bytes, CRC32 is 4 Bytes, IFG
   is 12 Bytes, Preamble is 7 Bytes, and SFD is 1 Byte.





Corre                     Expires 21 March 2022                [Page 10]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


   Then, to obtain the Maximum Achievable QUIC Throughput (Layer 4), we
   simply use:

     (MTU - 33) in Bytes X 8 bits X max FPS

   For 100-Mbps Ethernet, the maximum QUIC Throughput =

     1467 Bytes X 8 bits X 8127 FPS

     Max QUIC Throughput = 11736 bits X 8127 FPS
     Max QUIC Throughput = 95.4 Mbps

4.1.2.  QUIC Transfer Time and Transfer Time Ratio Calculation

   The following table illustrates the Ideal QUIC Transfer Time of a
   single QUIC connection when its Send Socket Buffer size equals or
   exceeds the BDP and the sender is never control-flow limited.

   +============+=======+=======+================+====================+
   | Link Speed |   RTT |   BDP | Max. Ach. QUIC |         Ideal QUIC |
   |     (Mbps) |  (ms) | (KBs) |    Thr. (Mbps) | Transfer Time (s)* |
   +============+=======+=======+================+====================+
   |      1.536 | 50.00 |   9.6 |            1.4 |              571.0 |
   +------------+-------+-------+----------------+--------------------+
   |     44.210 | 25.00 | 138.2 |           43.0 |               18.6 |
   +------------+-------+-------+----------------+--------------------+
   |    100.000 |  2.00 |  25.0 |           95.4 |                8.4 |
   +------------+-------+-------+----------------+--------------------+
   |  1,000.000 |  1.00 | 125.0 |          953.7 |                0.8 |
   +------------+-------+-------+----------------+--------------------+
   | 10,000.000 |  0.05 |  62.5 |        9,537.5 |                0.1 |
   +------------+-------+-------+----------------+--------------------+

       Table 1: Link Speed, RTT, BDP, Maximum QUIC Throughput, and
                Ideal QUIC Transfer Time for a 100-MB File

      |  Note (*): Transfer times are rounded for simplicity.

   For a 100-MB file (100 X 8 = 800 Mbits), the Ideal QUIC Transfer Time
   is derived as follows:

                               800 Mbits
       Ideal QUIC    = -------------------------
     Transfer Time         Maximum Achievable
                            QUIC Throughput






Corre                     Expires 21 March 2022                [Page 11]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


   To illustrate the QUIC Transfer Time Ratio, an example would be the
   bulk transfer of 100 MB over 5 simultaneous QUIC connections (each
   connection transferring 100 MB).  In this example, the Ethernet
   service provides a Committed Access Rate (CAR) of 500 Mbps.  Each
   connection may achieve different throughputs during a test, and the
   overall throughput rate is not always easy to determine (especially
   as the number of connections increases).

   The Ideal QUIC Transfer Time would be ~8.4 seconds, but in this
   example, the Actual QUIC Transfer Time was 12 seconds.  The QUIC
   Transfer Time Ratio would then be 12/8.4 = 1.43, which indicates that
   the transfer across all connections took 1.43 times longer than the
   ideal.  Note that our estimation of the Maximum Achievable QUIC
   Throughput is overestimated due to the optimistic scenario initially
   described.  The actual QUIC Transfer Time is thus expected to be
   higher than the Ideal QUIC Transfer Time due to eventual network
   inefficiencies but also by overheads caused by the selected QUIC
   implementation as well as the number of concurrent streams being used
   for the test.

4.2.  QUIC Efficiency

   In [RFC6349], the TCP efficiency metric represents the percentage of
   Bytes that were not retransmitted.  However, when packet loss occurs
   in QUIC, Bytes are not retransmitted per se, instead, new packets are
   generated.  As a result, a packet may contain retransmitted frames as
   well as new frames, and measuring the amount of retransmitted Bytes
   is not straightforward.

   Instead, the QUIC efficiency MUST be measured by calculating the
   ratio between QUIC Payload Bytes and UDP Bytes.  QUIC Payload Bytes
   are the total number of payload Bytes sent or received over QUIC
   while UDP Bytes are the total number of bytes sent or received on the
   UDP socket, i.e. including QUIC protocol overheads.  This method
   allows to capture the impact of retransmission as retransmission of
   frames will impact the UDP Bytes without impacting the QUIC Bytes.
   Retransmitted data will effectively lower the measured QUIC
   Efficiency.

                  QUIC payload Bytes
       QUIC =  -----------------------  X 100
     Efficiency         UDP Bytes









Corre                     Expires 21 March 2022                [Page 12]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


   In comparison with [RFC6349], the sender side QUIC Efficiency metric
   also measures the inherent overheads of QUIC caused by headers,
   frames, multiple streams, and implementation choices.  As the
   receiver side UDP Bytes also measures these overheads but does not
   measure retransmission overheads, the receiver side QUIC Efficiency
   MAY be used to normalize the sender side QUIC Efficiency.

    Normalized Sender QUIC Efficiency
       QUIC =  --------------------------
    Efficiency  Receiver QUIC Efficiency

4.2.1.  QUIC Efficiency Percentage Calculation

   As an example, considering a scenario where the MTU is 1500 bits and
   the QUIC overhead is 50bits per packet on average.  An application
   sending 100,000 Bytes over QUIC would emit approximately 552 UDP
   packets of 1500 bits resulting in a total of 103,448 UDP Bytes.  The
   QUIC Efficiency Percentage would be calculated as:

                           100,000
     QUIC Efficiency % =  ---------  X 100 = 96,6%
                           103,448

   Considering a similar example with 1% of packet loss, the UDP Bytes
   emitted would be approximately 104,500 Bytes.  The resulting QUIC
   Efficiency Percentage would thus be calculated as:

                           100,000
     QUIC Efficiency % =  ---------  X 100 = 95,6%
                           104,500

   On the receiver side, the measured UDP Bytes would be approximately
   103,448 UDP Bytes in both scenarios since no loss overheads would be
   measured resulting in Receiver QUIC Efficiency = 96,6%. Normalizing
   the previously measured QUIC Efficiency metrics (sender side) would
   thus result in Normalised QUIC Efficiency = 100% in the first example
   and Normalised QUIC Efficiency = 99% in the example with 1% of packet
   loss.

4.3.  Buffer Delay

   The third metric is the Buffer Delay Percentage, which represents the
   increase in RTT during a QUIC Throughput test versus the inherent or
   baseline RTT.  This metric remains identical to the one described in
   [RFC6349].






Corre                     Expires 21 March 2022                [Page 13]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


     Average RTT     Total RTTs during transfer
   during transfer = ----------------------------
                     Transfer duration in seconds


                           Average RTT
                         during transfer
                          - Baseline RTT
     Buffer Delay % = --------------------- X 100
                           Baseline RTT

   Note that while [RFC9002] specifies QUIC RTT measurements in the form
   of latest_rtt, smoothed_rtt, and min_rtt, the average RTT used in the
   Buffer Delay Percentage metric is derived from the total of all
   measured RTTs during the actual test at every second divided by the
   test duration in seconds.  RTT metrics smoothed_rtt and min_rtt
   described in [RFC9002] MUST NOT be used to compute the Average RTT
   during transfer.  RTT metric latest_rtt MAY be used for computing
   Average RTT during transfer by sampling it at every second.

5.  Conducting QUIC Throughput Tests

   The methodology for QUIC Throughput testing follows the same
   considerations as described in [RFC6349].

   In the end, a QUIC Throughput Test Device (QUIC TTD) SHOULD generate
   a report with the calculated BDP and QUIC Throughput results.  The
   report SHOULD also include the results for the 3 metrics defined in
   Section 4.  As QUIC does not use a receive window, a QUIC Throughput
   test is constituted of multiple Send and Receive Socket Buffer
   experiments.  The report SHOULD include QUIC Throughput results for
   each Socket Buffer size tested.  The goal is to provide achievable
   versus actual QUIC Throughput results for the Socket Buffer size
   tested when no fragmentation occurs and to provide a clear
   relationship between these 3 metrics and user experience.  As an
   example, for the same Transfer Time Ratio, a better QUIC Efficiency
   could be obtained at the cost of higher Buffer Delays.

5.1.  Using Multiple QUIC Streams

   In addition to opening multiple connections, QUIC allows data to be
   exchanged over fully multiplexed streams.  The primary goal of the
   methodology relates to ensuring the capacity of a managed network to
   deliver a particular SLA using QUIC.  As the NUT cannot observe the
   number of streams opened for a QUIC connection, we do not expect the
   NUT to apply specific policies based on the number of streams.  In
   the basic test scenario, A QUIC TTD MAY use two streams over a single
   connection where one stream is a test stream filling the BDP of the



Corre                     Expires 21 March 2022                [Page 14]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


   NUT and the other is a control stream used for controlling the test
   and exchanging test results.

   Implementers of a QUIC Throughput test may however want to use
   multiple streams in parallel.  On one hand, using multiple streams
   over a single QUIC connection may result in a variable number of
   STREAM frames per packet, with the actual numbers depending on the
   number of streams, the amount of data per stream, and the QUIC
   implementation [marx2020same].  This overhead variability may lower
   the QUIC Efficiency, increase the QUIC Transfer Time Ratio, and more
   generally reduce the reproducibility of the test with respect to the
   capacity of the NUT to provide an SLA.

   On the other hand, using multiple streams over a single connection is
   a typical scenario for using QUIC and one of the selling-point of the
   protocol compared to TCP.  Unstable networks in particular those with
   a high packet loss are expected to benefit from stream multiplexing.
   Implementing more complex scenarios in a QUIC Throughput test to
   better represent the use-cases of the NUT is possible.  A QUIC TTD
   MAY use multiple test streams to fill the BDP of the NUT.  In such a
   case, the QUIC TTD SHOULD also run the test using a single test
   stream to facilitate root-cause analysis.  The specific parameters of
   a scenario using multiple test streams depend however on the use-case
   being considered.  For instance, emulating multiple HTTP/3 browsing
   sessions may involve a large number of short-lived small streams,
   while emulating real-time conversational streaming sessions may
   involve long-lived streams with large chunks of data.  Details for
   such scenarios and relevant QoE influence factors are out-of-scope of
   the proposed methodology.

5.2.  1-RTT and 0-RTT

   Another new feature of the QUIC protocol is the reduced number of RTT
   required to establish a connection enabling so-called 1-RTT
   connection establishment and 0-RTT connection re-establishment.  It
   may be interesting to get a measure of how these QUIC features result
   in gain over a particular network.  However, the methodology
   described in this document is interested in throughput measurements
   at an equilibrium state.  While the present methodology baselines
   minimum RTT and measures average RTT during throughput testing,
   measuring connection re-establishment speed is out-of-scope for the
   proposed methodology.

5.3.  Results Interpretation

   For cases where the test results are not equal to the ideal values,
   some possible causes are as follows:




Corre                     Expires 21 March 2022                [Page 15]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


   *  Network congestion causing packet loss, which may be inferred from
      a poor Normalised QUIC Efficiency % (i.e. higher Normalised QUIC
      Efficiency % = less packet loss).

   *  Too many QUIC streams in parallel or an inefficient QUIC
      implementation with regards to protocol overheads may be inferred
      from a poor QUIC Efficiency at both the sender and receiver sides
      (i.e. low QUIC Efficiency but high Normalised QUIC Efficiency).

   *  Rate limiting by traffic policing may cause packet loss (i.e. low
      Normalised QUIC Efficiency).

   *  Network congestion causing an increase in RTT, which may be
      inferred from the Buffer Delay Percentage (i.e. 0% = no increase
      in RTT over baseline).

   *  Rate limiting by traffic shaping may also cause an increase in
      RTT.

   *  Maximum UDP Buffer Space.  All operating systems have a global
      mechanism to limit the quantity of system memory to be used by UDP
      sockets.  On some systems, each socket is subject to a memory
      limit that is applied to the total memory used for input data,
      output data, and controls.  On other systems, there are separate
      limits for input and output buffer spaces per socket.  Client/
      server IP hosts might be configured with Maximum UDP Buffer Space
      limits that are far too small for high-performance networks.
      These socket buffers MUST be large enough to hold a full BDP of
      UDP Bytes plus some overhead.

   *  Path MTU.  The client/server IP host system SHOULD use the largest
      possible MTU for the path.  This may require enabling Path MTU
      Discovery [RFC1191] and [RFC4821].  Since [RFC1191] is flawed,
      Path MTU Discovery is sometimes not enabled by default and may
      need to be explicitly enabled by the system administrator.
      [RFC4821] describes a new, more robust algorithm for MTU discovery
      and ICMP black hole recovery.

6.  Security Considerations

   Measuring QUIC network performance raises security concerns.  Metrics
   produced within this framework may create security issues.

   Security considerations mentionned in [RFC6349] remain valid for QUIC
   Throughput testing.  In particular, as QUIC encrypts traffic over
   UDP, QUIC Throughput testing may appear as a denial-of-service
   attack.  Cooperation between the end-user customer and the network
   provider is thus required.



Corre                     Expires 21 March 2022                [Page 16]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


6.1.  0-RTT attack {#0RTTattack}

   In practice, a QUIC Throughput test probably implements a control
   channel to control the test and one or more throughput channels to
   send data.  The control channel would serve to authenticate the TTD,
   request a test session through some throughput channels, and exchange
   test results.  As 0-RTT packets are vulnerable to replay attacks
   [RFC9001], 0-RTT packets MUST not be used by a TTD client initiating
   a control channel to a TTD server.

7.  IANA Considerations

   This document has no IANA actions.

Acknowledgments

   Thanks to Sylvain Nadeau for his valuable inputs.

References

Normative References

   [RFC1191]  Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191,
              DOI 10.17487/RFC1191, November 1990,
              <https://www.rfc-editor.org/rfc/rfc1191>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/rfc/rfc2119>.

   [RFC4821]  Mathis, M. and J. Heffner, "Packetization Layer Path MTU
              Discovery", RFC 4821, DOI 10.17487/RFC4821, March 2007,
              <https://www.rfc-editor.org/rfc/rfc4821>.

   [RFC6349]  Constantine, B., Forget, G., Geib, R., and R. Schrage,
              "Framework for TCP Throughput Testing", RFC 6349,
              DOI 10.17487/RFC6349, August 2011,
              <https://www.rfc-editor.org/rfc/rfc6349>.

   [RFC6582]  Henderson, T., Floyd, S., Gurtov, A., and Y. Nishida, "The
              NewReno Modification to TCP's Fast Recovery Algorithm",
              RFC 6582, DOI 10.17487/RFC6582, April 2012,
              <https://www.rfc-editor.org/rfc/rfc6582>.

   [RFC793]   Postel, J., "Transmission Control Protocol", STD 7,
              RFC 793, DOI 10.17487/RFC0793, September 1981,
              <https://www.rfc-editor.org/rfc/rfc793>.



Corre                     Expires 21 March 2022                [Page 17]


Internet-Draft    Framework for QUIC Throughput Testing   September 2021


   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

   [RFC8899]  Fairhurst, G., Jones, T., Tüxen, M., Rüngeler, I., and T.
              Völker, "Packetization Layer Path MTU Discovery for
              Datagram Transports", RFC 8899, DOI 10.17487/RFC8899,
              September 2020, <https://www.rfc-editor.org/rfc/rfc8899>.

   [RFC9000]  Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based
              Multiplexed and Secure Transport", RFC 9000,
              DOI 10.17487/RFC9000, May 2021,
              <https://www.rfc-editor.org/rfc/rfc9000>.

   [RFC9001]  Thomson, M., Ed. and S. Turner, Ed., "Using TLS to Secure
              QUIC", RFC 9001, DOI 10.17487/RFC9001, May 2021,
              <https://www.rfc-editor.org/rfc/rfc9001>.

   [RFC9002]  Iyengar, J., Ed. and I. Swett, Ed., "QUIC Loss Detection
              and Congestion Control", RFC 9002, DOI 10.17487/RFC9002,
              May 2021, <https://www.rfc-editor.org/rfc/rfc9002>.

Informative References

   [marx2020same]
              Marx, R., Herbots, J., Lamotte, W., and P. Quax, "Same
              standards, different decisions: A study of quic and http/3
              implementation diversity", Proceedings of the Workshop on
              the Evolution, Performance, and Interoperability of QUIC,
              2020.

Author's Address

   Kevin Corre
   EXFO

   Email: kevin.corre@exfo.com














Corre                     Expires 21 March 2022                [Page 18]