[Search] [txt|pdf|bibtex] [Tracker] [Email] [Diff1] [Diff2] [Nits]

Versions: 00 01 02                                                      
Network working group                              L. Dunbar
Internet Draft
Category: Informational                              Huawei
Expires: November 2017


                                              March 27, 2017


              Architectural View of E2E Latency and Gaps

          draft-dunbar-e2e-latency-arch-view-and-gaps-01.txt

Abstract

   Ultra-Low Latency is a highly desired property for many types of
   services, such as 5G MTC (Machine Type Communication) requiring
   E2E connection for V2V to be less than 2ms, AR/VR requiring delay
   less than 5ms, V2X less than 20ms, etc.

   This draft examines the E2E latency from architectural
   perspective, from studying how different OSI layers contribute to
   E2E latency, how different domains, which can be different
   operators' domains or administrative domains, contribute to E2E
   latency, to analyzing the gaps of recent technology advancement
   in reducing latency.

   By studying the contributing factors to E2E latency from various
   angles, the draft identifies some gaps of recent technology
   advancement for E2E services traversing multiple domains and
   involving multiple layers, which can be the basis for IAB to
   organize more in-depth discussion, like workshop or cross Area
   (or industry wide) BOF, as the scope of the discussion will be
   too wide for one IETF WG. The discussion might touch upon
   multiple IETF areas.

   The goal of those further "deep-dive" workshop or cross area BOF
   is to establish more comprehensive foundation to IESG and the
   IETF community in identifying potential new work initiatives for
   reducing E2E latency of services over the Internet.









Dunbar, et al         Expires November 2017           [Page 1]


Internet-Draft    E2E Over Internet Latency Taxonomy





Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79. This document may not be
   modified, and derivative works of it may not be created, except
   to publish it as an RFC and to translate it into languages other
   than English.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other
   documents at any time.  It is inappropriate to use Internet-
   Drafts as reference material or to cite them other than as "work
   in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html

   This Internet-Draft will expire on September 27, 2017.


Copyright Notice

   Copyright (c) 2017 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with
   respect to this document. Code Components extracted from this
   document must include Simplified BSD License text as described in



Dunbar, et al                                           [Page 2]


Internet-Draft    E2E Over Internet Latency Taxonomy



   Section 4.e of the Trust Legal Provisions and are provided
   without warranty as described in the Simplified BSD License.


Table of Contents


   1. Introduction................................................. 4
   2. Terminology.................................................. 5
   3. AR/VR Use Case............................................... 5
   4. Contributing Factors to E2E Latency.......................... 6
   5. Application Layer Initiative in reducing E2E latency......... 6
      5.1. Content Placement mechanisms need visibility to Network. 7
   6. Transport Layer Initiatives in reducing Latency and gaps..... 7
      6.1. TCP Layer Latency Improvement Alone is not enough....... 7
      6.2. LTE Latency Impact on TCP Performance................... 8
      6.3. Low Latency via Multipath TCP Extension................. 9
   7. Network and Link Layer Initiatives in reducing E2E Latency... 9
   8. Radio Channel Quality Impact to flows with High QoS......... 10
   9. E2E Latency Contributed by multiple domains................. 10
   10. Conclusion................................................. 11
   11. Security Considerations.................................... 11
   12. IANA Considerations........................................ 11
   13. Acknowledgements........................................... 11
   14. References................................................. 11
      14.1. Normative References.................................. 11
      14.2. Informative References................................ 12
   15. Appendix:.................................................. 12
      15.1. Example: multi-Segments Latency for services via
      Cellular Access............................................. 12
      15.2. Latency contributed by multiple nodes................. 14
      15.3. Latency through the Data Center that hosts S-GW & P-GW 14
   Authors' Addresses............................................. 15
















Dunbar, et al                                           [Page 3]


Internet-Draft    E2E Over Internet Latency Taxonomy



1. Introduction

   Ultra-Low Latency is a highly desired property for many types of
   services, such as 5G MTC (Machine Type Communication) requiring
   E2E connection for V2V to be less than 2ms, AR/VR requiring delay
   less than 5ms, V2X less than 20ms, etc.

   This draft is to examine E2E latency from architectural
   perspective, from studying how different OSI layers contribute to
   E2E latency, how different domains, which can be different
   operators' domains or administrative domains, contribute to E2E
   latency, to analyzing the gaps of recent technology advancement
   in reducing latency.

   The primary purpose of studying E2E Latency from architectural
   perspective is to help the IETF community identify potential work
   areas for reducing E2E latency of services over the Internet.

   In recent years, the internet industry has been exploring
   technologies and innovations at all layers of the OSI stack to
   reduce latency. At the upper (application) layer, more contents
   are distributed to the edges closer to end points and more
   progress in Mobile Edge Computing (MEC) has been made. At the
   Transport layer, there are QUIC/L4S initiatives. At the network
   layer, there are IP/MPLS Hardened pipe (RFC 7625), latency
   optimized router design, and BBF's Broadband Assured Services
   (BAS). At the link layer, there are IETF DETNET, IEEE 802.1 TSN
   (Time Sensitive Networking), and Flex Ethernet (OIF).

   By studying the contributing factors to E2E latency from various
   angles, the draft identifies some gaps of recent technology
   advancement for E2E services traversing multiple domains and
   involving multiple layers, which can be the basis for IAB to
   organize more in-depth discussion, like a workshop or cross Area
   (or industry wide) BOF, as the scope of the discussion will be
   too wide for one IETF WG. The discussion might touch upon
   multiple IETF areas.

   The goal of the further "deep-dive" workshop or cross area BOF is
   to establish more comprehensive foundation to IESG and the IETF
   community in identifying potential work initiatives for reducing
   E2E latency of services over the Internet.








Dunbar, et al                                           [Page 4]


Internet-Draft    E2E Over Internet Latency Taxonomy



2. Terminology

   DA:      Destination Address

   DC:      Data Center

   E2E:     End To End

   GTP:     GPRS Tunneling Protocol (GTP) is a group of IP-based
            communications protocols used to carry general packet
            radio service (GPRS) within GSM, UMTS and LTE networks.
            In 3GPP architectures, GTP can be decomposed into
            separate protocols, GTP-C, GTP-U and GTP'. GTP-C is
            used for signaling. GTP-U is used for carrying user
            data.

   LTE:     Long Term Evolution

   TS:      Tenant System

   VM:      Virtual Machines

   VN:      Virtual Network



  3. AR/VR Use Case

   The E-2-E delays of AR/VR system come from delay of multiple
   systems:

   - Tracking delay
   - Application delay
   - Rendering delay
   - Display delay


   For human beings not to feel dizzy viewing AR/VR images, the
   oculus delay should be less than 19.3ms, which includes display
   delay, computing delay, transport delay, and sensoring delay.
   That means the "Network Delay" budget is only 5ms at the most.








Dunbar, et al                                           [Page 5]


Internet-Draft    E2E Over Internet Latency Taxonomy



  4. Contributing Factors to E2E Latency

   Internet data is packaged and transported in small pieces of
   data. The flow of these small pieces of data directly affects a
   user's internet experience. When data packets arrive in a smooth
   and timely manner, the user sees a continuous flow of data; if
   data packets arrive with large and variable delays between
   packets, the user's experience is degraded.

   Key contributing factors to E2E latency:

   - Generation: delay between physical event and availability of
     data
   - Transmission: signal propagation, initial signal encoding
   - Processing: Forwarding, encap/decap, NAT, encryption,
     authentication, compress, error coding, signal translation
   - Multiplexing: Delays needed to support sharing; Shared channel
     acquisition, output queuing, connection establishment
   - Grouping: Reduces frequency of control information and
     processing; Packetization, message aggregation


   The 2013 ISOC Workshop [Latency-ISOC] on Internet Latency
   concluded that:

        o Bandwidth alone is not enough in reducing latency
        o Bufferbloat is one of the main causes for high latency in
          the Internet.


   Figure 1 of the 2013 ISOC workshop report showed that the timing
   of download of an apparently uncluttered example Web page
   (ieeexplore.ieee.org), actually comprised of over one hundred
   objects, transferred over 23 connections needing 10 different DNS
   look-ups. This phenomenon just further proves that reducing E2E
   latency will need multiple layers coordination and interaction.



  5. Application Layer Initiative in reducing E2E latency

   More and more End to End services over internet are from end
   users/devices to applications hosted in data centers.

   As most content today is distributed, E2E services usually do not
   traverse the globe but rather more often than not, the network


Dunbar, et al                                           [Page 6]


Internet-Draft    E2E Over Internet Latency Taxonomy



   segments that the E2E service traverses are from end users to
   regional data centers. The practice of content distribution to
   the edge has transformed reaching low latency goals from fighting
   against the speed of light to optimizing communication between
   end users and their desired content.

   However, without awareness of latency characteristics of network
   segments, the content distribution mechanisms & algorithms might
   not achieve their intended optimal result.

5.1. Content Placement mechanisms need visibility to Network

   To be added.



  6. Transport Layer Initiatives in reducing Latency and gaps

   IETF QUIC, L4S are some of the initiatives in reducing E2E
   latency at the Transport Layer.

   IETF QUIC focus on the improvement from end points. It doesn't
   take into consideration of the network latency that the data
   packets traverse.

   The IETF L4S uses AQM for network nodes to purposely drop packets
   or send indication to end points when their queues are above
   certain thresholds. The goal is for the end nodes to reduce
   transmission rate when intermediate nodes buffers are almost
   full. It has following issues:

     As network aggregates many flows from many different end points
     and most flows have variable data rate, an intermediate network
     node+port's buffer being almost full at one specific time
     doesn't mean that the same amount of traffic will traverse the
     same port a few microseconds later. If all end (source) points
     reduce transmission rate upon receiving the AQM indication (or
     experiencing packets drop), traffic through the network can be
     greatly reduced (i.e. leaving no queue in the buffer). Then all
     end points can increase their rate, causing traffic pattern
     oscillation and buffer congestion again.

6.1. TCP Layer Latency Improvement Alone is not enough

   The following example shows why simply optimizing transport layer
   alone is not enough. More details can be found at
   https://www.w3.org/Protocols/HTTP/Performance/Pipeline.html.



Dunbar, et al                                           [Page 7]


Internet-Draft    E2E Over Internet Latency Taxonomy



     Typical web pages today contain a HyperText Markup Language
     (HTML) document and many embedded images. Twenty or more
     embedded images are quite common. Each of these images is an
     independent object in the Web, retrieved (or validated for
     change) separately. The common behavior for a web client,
     therefore, is to fetch the base HTML document, and then
     immediately fetch the embedded objects, which are typically
     located on the same server.

     The large number of embedded objects represents a change from
     the environment in which the Web transfer protocol, the
     Hypertext Transfer Protocol (HTTP), was designed. As a result,
     HTTP/1.0 handles multiple requests from the same server
     inefficiently, creating a separate TCP connection for each
     object.

6.2. LTE Latency Impact on TCP Performance

   HTTP/TCP is the dominating application and transport layer
   protocol suite used on the internet today. According to HTTP
   Archive (http://httparchive.org/trends.php), the typical size of
   HTTP based transactions over the internet are in the range of a
   few 10's of Kbytes up to 1 Mbyte. In this size range, the TCP
   slow start period is a significant part of the total transport
   period of the packet stream.

   During TCP slow start, TCP exponentially increases its congestion
   window, i.e. the number of segments it brings into flight, until
   it fully utilizes the throughput that LTE (Radio + EPC) can
   offer. The incremental increases are based on TCP ACKs which are
   received after one round trip delay in the LTE system. Thus, as
   it turns out, during TCP slow start the performance is latency
   limited in Radio Network (LTE). Hence, improved latency in LTE
   can improve the perceived data rate for TCP based data
   transactions, which in its turn reduces the time it takes to
   complete a data down-load or upload.

   Despite rather small (in terms of milliseconds) improvements that
   can be achieved over the radio round trip time, the total
   increase in the perceived throughput and delay savings of
   downloading an item below 1MB is significant due to the additive
   effect of LTE latency improvements in the TCP slow start[LTE-
   Research].







Dunbar, et al                                           [Page 8]


Internet-Draft    E2E Over Internet Latency Taxonomy



6.3. Low Latency via Multipath TCP Extension

   There are some research work on how to use multi-path TCP to
   reduce E2E latency, such as
   http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7510787. The
   paper proposes an MPTCP extension that sends data redundantly
   over multiple paths in the network, which basically exchanges
   bandwidth for latency. The integration into the MPTCP protocol
   provides benefits such as transparent end-to-end connection
   establishment, multipath-enabled congestion control, and the
   prevention of head of line blocking. The research paper claims
   that their proposed Multipath TCP extension can halve the average
   round-trip time and reduce its standard deviation by a factor of
   19 for a real world mobile scenario in a stressed environment.
   Those kind of researchers should be invited to the "Reducing
   latency over Internet Deep-Dive" workshop or cross-area BOF (to
   be organized by IAB).



  7. Network and Link Layer Initiatives in reducing E2E Latency

   Several industry initiatives already exist for improving latency
   at the Link and Network layers:

     - Link Layer: IEEE 802.1 TSN (Time Sensitive Networking), and
        Flex Ethernet (OIF).
     - The network layer: IETF DETNET, IP/MPLS Hardened pipe (RFC
        7625).


   Gaps:

   IEEE 802.1 TSN (Time Sensitive Networking) requires stringent
   synchronous timing among all the nodes, which is suitable for
   small scoped network, but not suitable for the internet because
   most routers/switches in the network don't support synchronous
   timing.

   IP/MPLS hardened pipe can guarantee no congestion and no
   buffering on all nodes along the path, therefore, ensure the
   lowest latency along the path. The hardened pipe is ideal for
   flows with steady bandwidth requirement.




Dunbar, et al                                           [Page 9]


Internet-Draft    E2E Over Internet Latency Taxonomy



   But for applications that don't have steady flow size, the
   hardened pipe requires reserving the peak rate dedicated
   channels, which, like TDM, will incur bandwidth waste when
   application traffic goes below peak rate.

   Traffic Engineering is one of the most commonly used methods to
   reduce congestion at the network layer. However, it doesn't
   completely prevent transient congestion. Depending on the tunnel
   sizing, there could be momentary traffic bursts that exceed the
   tunnel size, thus causing congestion if there isn't adequate
   headroom on the trunk carrying the tunnel to absorb the burst. Or
   a link or node outage, that reroutes the tunnel onto a secondary
   path that becomes overloaded, could cause congestion.


  8. Radio Channel Quality Impact to flows with High QoS.

   QoS is one of the key methods employed by fixed IP network to
   reduce latency for some flows. However, in Radio network, if a
   UE's channel condition is poor, the eNB may schedule more frames
   to other UEs whose flow are marked with much lower QoS.

   There are many studies showing how Radio quality negatively
   impact to the TCP performance.

   It is beneficial to the whole industry if there is a workshop to
   get people or SDOs working on different layers of Internet
   service together to showcase their work or their pain points.

   IESG can make much more informed decision on creating useful
   initiatives when the community is aware of other work and
   obstacles.

  9. E2E Latency Contributed by multiple domains

   All of the latency improvement initiatives in the link layer have
   been within a single domain, such as IETF DETNET, IEEE 802.1 TSN
   (Time Sensitive Networking), and Flex Ethernet (OIF). The network
   layer latency improvement, such as IP/MPLS Hardened pipe (RFC
   7625) is also within a single domain.

   But E2E services usually traverse more than one domain, which can
   be administrative domains or multiple operators' networks.

   Yet today, there is no interface between domains to:





Dunbar, et al                                           [Page 10]


Internet-Draft    E2E Over Internet Latency Taxonomy



   - Inquire about the latency characteristics or capabilities from
     another domain
   - Negotiate or reserve latency capabilities from another domain.
   - Have a standardized method to characterize latency


   IETF/IAB is an ideal organization to tackle those issues because
   IETF has the expertise.





10. Conclusion

   As end to end services traverse multiple types of network
   segments and domains, and involve multiple layers, more informed
   decision in each layer technological improvement is important.

   - Need across domain coordination
   - Need across layer coordination

11. Security Considerations

   As the trend is going more encryption, it is getting more
   difficult for various network segments to detect applications
   sessions. Therefore, it is more important to create ways for
   better coordination among different layers, for improved latency,
   trouble shooting, restoration, etc.



12. IANA Considerations

   This section gives IANA allocation and registry considerations.

13. Acknowledgements

   Special thanks to Jari Arkko for encouraging writing this draft.
   And many thanks to Andy Malis, Jim Guichard, Spenser Dawkins, and
   Donald Eastlake for suggestions and comments to this draft.

14. References

   14.1. Normative References




Dunbar, et al                                           [Page 11]


Internet-Draft    E2E Over Internet Latency Taxonomy



   14.2. Informative References

   [LTE-latency]  https://www.ericsson.com/research-blog/lte/lte-
             latency-improvement-gains/

   [Latency-ISOC] 2013 ISOC organized Latency over Internet workshop
             report



15. Appendix:

15.1. Example: multi-Segments Latency for services via Cellular
   Access

   Via Cellular network, there are User Plane Latency and Control
   Plane Latency. Control plane deals with signaling and control
   functions, while user plane deals with actual user data
   transmission.

   The User Plane latency can be measured by the time it takes for a
   small IP packet to travel from the terminal through the network
   to the internet server, and back. The Control Plane latency is
   measured as the time required for the UE (User Equipment) to
   transit from idle state to active state.

   User Plane latency is relevant for the performance of many
   applications. This document mainly focuses on the User Plane
   Latency. The following diagram depicts a logical path from an end
   user (smart phone) application to the application controller
   hosted in a data center via 4G Mobile network, which utilize the
   Evolved Packet Core (EPC).

       +------+             +---------+
       |DC    |             | EPC     |           +----+
       |Apps  |<----------->|P-GW/S-GW|< -------> | eNB|<---> UE
       |      |             +---------+  Mobile   +----+ Radio
       +------+  Internet               Backhaul         Access


   Mobility Management Entity (MME) is responsible for
   authentication of the mobile device. MME retains location
   information for each user and then selects the Serving Gateway
   (S-GW) for a UE at the initial attach and at time of intra-LTE
   handover involving Core Network (CN) node relocation.





Dunbar, et al                                           [Page 12]


Internet-Draft    E2E Over Internet Latency Taxonomy



   The Serving Gateway (S-GW) resides in the user plane where it
   forwards and routes packets to and from the eNodeB (eNB)
   and packet data network gateway (P-GW). The S-GW also serves as
   the local mobility anchor for inter-eNodeB handover and mobility
   between 3GPP networks.

   P-GW (Packet Data Network Gateway) provides connectivity from the
   UE to external packet data networks by being the point of exit
   and entry of traffic for the UE. A UE may have simultaneous
   connectivity with more than one P-GW for accessing multiple
   Packet Data Networks. The P-GW performs policy enforcement,
   packet filtering for each user, charging support, lawful
   interception and packet screening. Another key role of the P-GW
   is to act as the anchor for mobility between 3GPP and non-3GPP
   technologies such as WiMAX and 3GPP2 (CDMA 1X and EvDO).

   Very often P-GW and S-GW are co-located. The data traffic between
   eNB and S-GW is encapsulated by GTP-U.

   The figure above shows that the end to end services from/to UE
   consists of the following network segments:

   - Radio Access network - RAN
   - Mobile Backhaul network that connect eNB to S-GW.
   - Network within the DC that hosts S-GW & P-GW
   - Packet Data Network, which can dedicated VPN, internet, or
     other data network.
   - Network within the DC that hosts the App.


   The RAN (Radio Access Network) is between UE (e.g. smart phone)
   and eNB. 3GPP has a group TSG RAN working on improving
   performance (including latency) of the Radio Access network.
   There are many factors impacting the latency through RAN.

   The Mobile Backhaul Network connects eNBs to S-GW/P-GW, with data
   traffic being encapsulated in GTP protocol. The number of UEs
   that one eNB can handle are in 100s. The number of UEs that one
   S-GW/P-GW can handle are in millions. Therefore, the mobile
   backhaul network connects 10s of thousands of eNBs to S-GW/P-GW.
   Therefore, the number of network nodes in the Mobile Backhaul
   network can be very large. Therefore, any new protocol
   improvement in reducing latency can play a big part in reducing
   the overall latency for the end to end services.





Dunbar, et al                                           [Page 13]


Internet-Draft    E2E Over Internet Latency Taxonomy



15.2. Latency contributed by multiple nodes

   The variant of delay for data packets through network is caused
   by network nodes along the path as the transmission delay on
   physical link is fixed. When there is no congestion, the latency
   across most routers and switches are very small, in the magnitude
   of ~20us (worst case in ~40us). When congestion occurs within a
   node, i.e. with buffer/queues being used to avoid dropping
   packets, latency across a node can be in the magnitude of micro-
   seconds. The recent improvements made within router architecture
   have greatly improved latency through a node. However, there is
   no standard methods for routers to characterize and expose
   various latency characteristics through a network node.

   Data packets also traverse through network functions, such as FW,
   DPI, OPS, whose latency vary depending on the depth of the
   processing and the equipment performance.


15.3. Latency through the Data Center that hosts S-GW & P-GW

   S-GW and P-GW are hosted in Data center. There are typically 2~3
   tiers of switches connecting the servers that hosts S-GW & P-GW
   to the external network, as depicted in the following:

                            +---------+
                            | Gateway |
                            +---------+

          \              +-------+         +------+             /
           \           +/------+ |       +/-----+ |            /
            \          | Aggr11| + ----- |AggrN1| +           /
             \         +---+---+/        +------+/           /
              \         /     \            /      \         /
               \       /       \          /        \       /
                \   +---+    +---+      +---+     +---+   /
                 \- |T11|... |T1x|      |T21| ... |T2y|---
                    +---+    +---+      +---+     +---+
                      |        |          |         |
                    +-|-+    +-|-+      +-|-+     +-|-+   Servers
                    |   |... |SGW|      | S |     | S |<-
                    +---+    +---+      +---+     +---+
                    |   |... |PGW|      | S | ... | S |
                    +---+    +---+      +---+     +---+
                    |   |... | S |      | S | ... | S |
                    +---+    +---+      +---+     +---+




Dunbar, et al                                           [Page 14]


Internet-Draft    E2E Over Internet Latency Taxonomy



   As the distance within data center can be small, the transmission
   delay within data center can be negligent. The majority of
   latency within data center is caused by the switching within the
   gateway routers, traffic traversing through middleware boxes such
   as FW, DPI, IPS, value added services, the top of the rack
   switches, and aggregation switches.

   If the S-GW and P-GW are hosted in large data center, there could
             be latency contributed by the
             encapsulation/decapsulation such as work specified by
             NVO3.

 Authors' Addresses

   Linda Dunbar
   Huawei Technologies
   5430 Legacy Drive, Suite #175
   Plano, TX 75024, USA
   Phone: (469) 277 5840
   Email: linda.dunbar@huawei.com






























Dunbar, et al                                           [Page 15]