Internet-Draft quic-te November 2023
Zheng, et al. Expires 11 May 2024 [Page]
Intended Status:
Z. Zheng
Alibaba Inc.
Y. Ma
Uber Technologies Inc.
Y. Liu
Alibaba Inc.
M. Kühlewind

QUIC-enabled Service Differentiation for Traffic Engineering


This document defines a method for supporting QUIC-enabled service differentiation for traffic engineering through multipath and QUIC connection identifier (CID) encoding. This approach enables end-host networking stacks and applications to select packet routing paths in a wide area network (WAN), potentially improving end-to-end performance, cost, and reliability. The proposed method can be used in conjunction with segment-routing traffic engineering technologies, such as SRv6 TE.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 11 May 2024.

1. Introduction

This draft outlines a method that enables a WAN to manage packets from a single QUIC connection with differentiated traffic engineering policies at packet-level granularity. For instance, a WAN can offer two priority routing paths for packets from the same QUIC connection: (1) it directs first-time transmitted packets on a default-priority, low-cost path, and (2) it routes retransmitted packets and handshake packets on a high-priority (low-latency) but more expensive path. Consequently, this approach achieves cost-efficiency and reduces tail latency simultaneously.

While there are existing solutions to support Differentiated Service (DiffServ), such as using Differentiated Services Code Point (DSCP) or opening multiple ports to enable QoS control, they have several limitations. For example, DSCP values may be modified by an ISP or a network service provider, and opening multiple ports raises security concerns (more details as discussed in Section 2).

The solution proposed in this draft can overcome these limitations, as it is designed with the following properties: (1) The service differentiation encoding is immune to modifications by middleboxes. (2) It provides traffic engineering control at packet-level granularity within a connection. (3) A user-space application can directly control each packet's priority without the kernel's special support. (4) It does not change the deployment method, as current QUIC applications have.

In a nutshell, this proposal establishes multiple routing paths at different priority levels inside a single QUIC connection and encodes a packet's routing priority in its CID. Specifically, this proposal reuses mechanisms from Multipath QUIC [Multipath-QUIC], but makes two major changes: (1) Multiple QUIC paths are allowed to be created between a single-homed client and server (e.g., the client and server can both have only one (IP, port) address). (2) This proposal leverages different CIDs to describe various priority levels for QUIC packets that belong to the same connection.

In this draft, QUIC paths can share the same perceived 4-tuple, but as long as their underlying physical routing paths are not the same, they are treated as different paths. This proposal shares many key similarities to Multipath QUIC: (1) Senders manage per-path congestion status; (2) RTT measurements are per-path. (3) Each path uses a separate packet number space (PNS); (4) A path ID is the sequence number of the CID used to send packets. (5) Once multipath is negotiated, ACK_MP frame is used to acknowledge packets in multiple packet number spaces.

The proposed solution is intended to work with edge routers that have segment routing capabilities, such as SRv6 [RFC8986]. Note that the QUIC CID is not intended to be directly used for routing in Cloud WAN, but SHOULD be translated to SRv6 SID or an MPLS label by an edge router for traffic engineering purposes.

1.1. Conventions and Definitions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

This document uses the following terms:

  • Cloud WAN: The backbone network provided and controlled by a Cloud provider, or the backbone network operated by an application service provider.

  • Edge router: Edge router of a backbone network.

  • QUIC path: The path described in Multipath QUIC protocol, which is a logical end-to-end transmission path created by the client.

  • Routing path: The network path that a packet physically traverses, also specified as the Cloud WAN path in this document.

2. Alternatives to support service differentiation in traffic engineering

This section describes existing solutions that offer service differentiation in traffic engineering and their limitations when applied to the internet.

The most commonly used solution is to use the DSCP packet marking [RFC8837] for traffic engineering. However, such a solution faces several limitations: (1) A DSCP marker is subject to modifications by an ISP or a network service provider. (2) Marking DSCP on mobile devices may require kernel changes. (3) DSCP is marked at the granularity of connection or session, which means packets with the same 4-tuple are treated with the same priority, and hence, it cannot support packet-level QoS control.

The second solution is to use multiple ports, either source or destination ports, to differentiate QoS between two endpoints. When this solution is used, packets with different priorities are sent to sockets bound to different ports. However, such a solution also faces several drawbacks. Source ports may be modified by a NAT in the network, which changes the priority. Using destination ports to support QoS raises security concerns, as opening many ports increases the potential attack surface for cyber threats. Destination ports other than some well-known ports (e.g., 80, 433) may also be subject to blocking by firewalls.

3. Using Multipath QUIC

3.1. New Transport Parameter

This draft defines a new transport parameter to claim the adoption of priority paths. It is negotiated during QUIC handshake as described in [RFC9000]. The new parameter is defined as PRIORITY_PATHS. This parameter is used to indicate the number of priority levels:

  • 0, 1: Only one path is available, i.e., multiple priority levels are not considered.

  • n (n > 1): n paths are available, i.e., n priority levels are available for use.

When the client supports PRIORITY_PATHS, it sends a value greater than 1 to the server side. After receiving it, the server returns the configured value of PRIORITY_PATHS to the client. The PRIORITY_PATHS value that a client uses MUST be consistent with the returned value from the server, i.e., the server determines how many priority paths can be used.

The server and CID parser share the same PRIORITY_PATHS values, which are configured by the control plane. After negotiation, the client obtains the same value as the server.

3.2. QUIC Connection Setup

When a QUIC client starts a new connection, it uses a CID generated using the default priority (defined as 0). If PRIORITY_PATHS is supported, a server SHOULD respond with a CID that represents the default priority and indicate to the client that it supports PRIORITY_PATHS.

3.3. Paths Setup with More Priorities

After QUIC connection negotiation is completed, endpoints need to exchange CIDs with each other. These CIDs are generated using the method described in Section 4. Figure 1 illustrates an example of this process. The client sends CIDs with priority 1 and priority 2 (C-CID1 and C-CID2), and the server also sends CIDs representing these two priority levels (S-CID1 and S-CID2).

Client                                                  Server

  (CID exchange)
  NEW_CONNECTION_ID[C-CID2]        --->
                          <---       NEW_CONNECTION_ID[S-CID2]

  (New path init with priority 1)
  PATH_CHALLENGE[S-CID1]           --->
                   <---   PATH_RESPONSE,PATH_CHALLENGE[C-CID1]

  PATH_RESPONSE[S_CID1]            --->
Figure 1: An example of CID exchange and path setup

After the CID exchange is completed, the client selects S-CID1 to establish a new path of priority 1 via PATH_CHALLENGE frame. Even though the packet that uses S-CID1 comes from the same 4-tuple as the default path (i.e., priority 0), the server SHOULD treat it as a new path establishment request and respond with PATH_RESPONSE and PATH_CHALLENGE frames using a CID of the same path priority, i.e., C-CID1.

3.4. Effect of CID rotation

When an endpoint rotates to a new CID on an existing path, it will not send PATH_CHALLENGE frame and the new CID does not change the path priority, so the peer will not treat this new CID as an attempt to create a new path.

4. CID Generation and Parsing

4.1. Overview of End-to-end Network

  | Server |
       | Cloud DCN
| CID parser2 |
       | Cloud WAN
| CID parser2 |
       | Internet access
       | network
  | Client |

Figure 2: End-to-end network

QUIC packets traverse the end-to-end network as described in Figure 2. This network includes three parts: internet access network, Cloud WAN, and Cloud DCN.

The Cloud WAN has various routing paths with different performances, thus priority scheduling is applied. The client and server assign different priorities to the CIDs they generate and send. At the Cloud WAN edge, two CID parsers are responsible for parsing the priority of incoming packets and forwarding them to the appropriate routing path.

4.2. CID Generation on the Server Side

There are two purposes for encoding the CID: 1. The client and server can understand the priority represented by the CID; 2. There MUST NOT be a correlation between the CID and its priority, which is important to prevent privacy leaks if an attacker observes.

Their CIDs are generated on the server side for packets from the client to the server. The format of an encoded CID on the server side is defined below:

  Random Bits (32 .. 152),
  Priority Bits (8),

To prevent the priority value from being observed by an attacker, the server uses encryption (e.g., AES-128-ECS cipher) to obscure the CID, which is similar to [QUIC-LB]. The encryption key and length are pre-configured on the server.

Note that when the generated CID is sent to the client through NEW_CONNECTION_ID frame, the Sequence Number in this frame needs to satisfy priority = sequence_number % n (n is described in Section 3.1). The client will use Sequence Number to calculate priority.

4.3. CID Generation on the Client Side

Their CIDs are generated on the client side for packets from the server to the client. Its format is the same as the definition in [RFC9000], which is completely random.

When the server sends QUIC packets with these CIDs, an additional byte is appended before the first byte of a CID to indicate the priority. The CID parser 2 will remove this byte after receiving these packets from the server. Therefore, CIDs within these packets remain completely random as they were generated on the client side. As they traverse through the Cloud WAN and internet access network, the priority cannot be observed. Moreover, the DCN operates as a private network that attackers typically cannot access.

4.4. CID Parsing

CID parser 1 parses QUIC packets sent from the client to the server. These packets contain encrypted CIDs. Thus, parser 1 initially decrypts the CIDs (using the same encryption key as the server, which is also pre-configured). For each CID, it only needs to be decrypted once. It then reads the last byte of the CID, which indicates the priority and forwards the packets.

CID Parser 2 parses QUIC packets from the server to the client. It extracts the first byte of the CID, which represents the priority, then removes it and forwards the packets.

When forwarding a packet, its priority is translated to an SRv6 SID or an MPLS label to map it to the appropriate routing path. Thus, both parsers record the mapping of priority levels to specific traffic engineering policies. The configuration file is issued by the control plane and can be updated during runtime.

In the case of SRv6 TE, such a priority level is mapped to SRv6 SIDs as shown below:

(priority 0) ==> (SRv6 SID 0)
(priority 1) ==> (SRv6 SID 1)
(priority 2) ==> (SRv6 SID 2)

Based on the parsed priority, the parser adds the corresponding SRv6 SID to the packet, so that the packet can be routed on the desired priority path. As these operations only include matching, removing bytes, and inserting new headers, they can be efficiently offloaded to hardware.

5. Examples

5.1. Packet Flow from Server to Client

Figure 3 shows a typical example that how the packets from a server to a client are routed on different Cloud WAN routing paths according to different CIDs (priorities).

                | Cloud VM           |
                | +----------------+ |API: priority description
                | |   APP ser^er   | |(retrans_pkts: high_priority)
                | +----------------+ |(other_pkts: default_priority)
                | | Multipath QUIC | |
                | +----------------+ |
                |                    |
                        |    |
         Path1[C+DCID0] |    | Path2[C+DCID1]
         (other_pkts)   |    | (retrans_pkts)
|                       |    |              Cloud WAN  |
|                   +---v----v----+                    |
|                 +-+ CID parser2 +-+                  |
|                 | +-------------+ |                  |
|         C+DCID0 |                 |C+DCID1           |
|                 |                 |                  |
|              +--v-+            +--v-+                |
|              | R1 |            | R2 |                |
|              +--+-+            +--+-+                |
|    Regular link |                 | Premium link     |
|                 |                 |                  |
|                 | +-------------+ |                  |
|                 +-> CID parser1 <-+                  |
|                   +-----+-------+                    |
|                         |                            |
                     | ISP edge |
                  C+DCID0 || C|DCID1
                |                     |
                | +-----------------+ |
                | | Multipath QUIC  | |
                | +-----------------+ |
                | |   APP client    | |
                | +-----------------+ |
                | Mobile de^ice       |

Figure 3: An example of packet flow from server to client

The app first describes its priority requirements through the API to the QUIC layer. As shown in this figure, it describes that retransmitted packets should be routed on a high-priority path, while other packets are routed on a default-priority path.

When these QUIC packets arrive at the edge of Cloud WAN, the CID parser parses each packet and extracts the priority encoded in the CIDs as described in Section 4.4. Based on the priority, packets are sent on different routing paths via added SRv6 SIDs. For example, low-priority packets will be routed on a default low-cost path, while high-priority packets will routed on a preimum low-latency path.

Since packets using different CIDs belong to the same QUIC connection, they will be combined into a complete and ordered stream by QUIC and then submitted to the upper-layer application.

5.2. Deployment of CID Parser

This section discusses some experiences regarding the deployment of CID parsers in a Cloud WAN. Two issues should be considered in the actual deployment:

  • Where to deploy?

  • Translating QUIC CID to SRv6 segment identifiers or MPLS labelling?

For the first one, the parser is usually deployed on the edge of Cloud WAN. For uplink traffic, it is deployed at the intersection of the ISP network; for downlink traffic, it is deployed at the edge routers of a datacenter region.

For the second one, this document recommends using SRv6 because it has better scalability and flexibility.

7. IANA Considerations

The following entry in Table Table 1 should be added to the "QUIC Transport Parameters" registry under the "QUIC Protocol" heading:

Table 1: New Transport Parameter
Value Parameter Name. Specification
TBD (experiments use 0x5052494f52495459) PRIORITY_PATHS Section 3.1

8. Normative References

Liu, Y., Ma, Y., De Coninck, Q., Bonaventure, O., Huitema, C., and M. Kühlewind, "Multipath Extension for QUIC", Work in Progress, Internet-Draft, draft-ietf-quic-multipath-04, , <>.
Duke, M., Banks, N., and C. Huitema, "QUIC-LB: Generating Routable QUIC Connection IDs", Work in Progress, Internet-Draft, draft-ietf-quic-load-balancers-15, , <>.
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, , <>.
Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, , <>.
Jones, P., Dhesikan, S., Jennings, C., and D. Druta, "Differentiated Services Code Point (DSCP) Packet Markings for WebRTC QoS", RFC 8837, DOI 10.17487/RFC8837, , <>.
Filsfils, C., Ed., Camarillo, P., Ed., Leddy, J., Voyer, D., Matsushima, S., and Z. Li, "Segment Routing over IPv6 (SRv6) Network Programming", RFC 8986, DOI 10.17487/RFC8986, , <>.
Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based Multiplexed and Secure Transport", RFC 9000, DOI 10.17487/RFC9000, , <>.

Authors' Addresses

Zhilong Zheng
Alibaba Inc.
Yunfei Ma
Uber Technologies Inc.
Yanmei Liu
Alibaba Inc.
Mirja Kühlewind