Skip to main content

Customer Experience Index for Evaluating Network Quality for Cloud Applications
draft-hz-ippm-cei-01

Document Type Active Internet-Draft (individual)
Authors Sifa Liu , Yaojing Wang , Wei Sun , Xiang Huang , Shuai Zhou , Hongyi Huang , Tianran Zhou , Jian Ge , MA KE
Last updated 2024-05-14
RFC stream (None)
Intended RFC status (None)
Formats
Stream Stream state (No stream defined)
Consensus boilerplate Unknown
RFC Editor Note (None)
IESG IESG state I-D Exists
Telechat date (None)
Responsible AD (None)
Send notices to (None)
draft-hz-ippm-cei-01
IP Performance Measurement                                        S. Liu
Internet-Draft                                              Huawei Cloud
Intended status: Standards Track                                 Y. Wang
Expires: 15 November 2024                                         Huawei
                                                                  W. Sun
                                                                X. Huang
                                                                 S. Zhou
                                                            Huawei Cloud
                                                           H. Huang, Ed.
                                                            T. Zhou, Ed.
                                                                  Huawei
                                                                   J. Ge
                                                                   K. Ma
                                                                   CAICT
                                                             14 May 2024

   Customer Experience Index for Evaluating Network Quality for Cloud
                              Applications
                          draft-hz-ippm-cei-01

Abstract

   This document outlines a unified Customer Experience Index (CEI)
   designed to assist cloud vendors in assessing network quality,
   reflecting the customer experience with cloud applications when
   accessed via the public network.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 15 November 2024.

Liu, et al.             Expires 15 November 2024                [Page 1]
Internet-Draft  Customer Experience Index for Evaluating        May 2024

Copyright Notice

   Copyright (c) 2024 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   3
     1.2.  Terminology . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Motivation  . . . . . . . . . . . . . . . . . . . . . . . . .   4
   3.  Customer Experience Index . . . . . . . . . . . . . . . . . .   4
     3.1.  Observation . . . . . . . . . . . . . . . . . . . . . . .   4
     3.2.  Unified Index . . . . . . . . . . . . . . . . . . . . . .   5
     3.3.  Parameter Tuning  . . . . . . . . . . . . . . . . . . . .   6
       3.3.1.  Weight Proportion . . . . . . . . . . . . . . . . . .   6
       3.3.2.  Parameter a, b  . . . . . . . . . . . . . . . . . . .   6
   4.  Security Considerations . . . . . . . . . . . . . . . . . . .   6
   5.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   6
   6.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   6
     6.1.  Normative References  . . . . . . . . . . . . . . . . . .   7
     6.2.  Informative References  . . . . . . . . . . . . . . . . .   7
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   7

1.  Introduction

   This document introduces a unified Customer Experience Index (CEI)
   designed to assist cloud vendors in assessing the network quality
   that mirrors the customer experience of cloud applications when
   accessed via the public network.  The CEI, once quantified, empowers
   cloud vendors to proactively enhance network services, aiding in
   network planning and construction.  Furthermore, it enables cloud
   customers to distinguish the service quality of various cloud
   vendors, allowing them to select cost-effective services tailored to
   their applications.

   Cloud vendors and cloud enterprises focus on different network
   indicators (Key Performance Index) used to anticipate the quality of
   customer experience regarding various applications(e.g., gaming,

Liu, et al.             Expires 15 November 2024                [Page 2]
Internet-Draft  Customer Experience Index for Evaluating        May 2024

   audio and video, online stores).  However, KPIs only provide implicit
   information and cannot directly reflect the customers' perceived
   experience.  Moreover, there is no unified evaluation method of
   customer experience based on common network KPIs in the industry.  On
   the other hand, it is difficult for cloud vendors to directly access
   application-level Key Quality Index (KQI) data though it may
   explicitly imply customer experience.  As the number of enterprises
   who deploy the service in the cloud gradually increases, there is
   growing demand for deriving authentic customer experience from basic
   network metrics to facilitate network optimizations.

   A significant gap persists between network KPIs and customer
   experience.  The primary network KPIs accessible to most cloud
   vendors—network latency, packet-loss rate, and jitter—encompass three
   categories.  Considering multiple dimensions of network quality
   proves beneficial for end-users.
   [I-D.teigen-ippm-app-quality-metric-reqs] Customers' demands for
   experience quality vary across different cloud services and are
   linked to specific KPIs.  For instance, those accessing real-time
   interactive games prioritize network latency; those utilizing video-
   on-demand services are more concerned with packet-loss rate than
   latency; and those engaging with cloud storage services consider both
   latency and packet-loss rate.  No single KPI can provide an accurate
   reflection of the experience for diverse services.  Both cloud
   vendors and customers seek unified evaluation standards for
   experience quality when accessing cloud services.

   This document accounts for a range of key network-observable
   indicators, offering a unified, objective, and comprehensive CEI to
   help enterprises evaluate customer experience through measurable
   network KPIs in a reasonable and fair manner.  Predominantly based on
   three network KPIs—network latency, packet-loss rate, and jitter—the
   CEI aims to thoroughly assess network quality.  The allocation of
   weights to these KPIs within the CEI can be customized to suit
   different application scenarios.

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

1.2.  Terminology

   CEI: Customer Experience Index

Liu, et al.             Expires 15 November 2024                [Page 3]
Internet-Draft  Customer Experience Index for Evaluating        May 2024

   KPI: Key Performance Indicator

   KQI: Key Quality Indicator

2.  Motivation

   Cloud service providers aim to precisely evaluate the network
   quality, crucial to the customer experience of their cloud services,
   and implement targeted improvements to their network infrastructure.
   Similarly, cloud customers seek a unified and fair scoring standard
   to guide their selection of superior cloud services.  But they
   currently face some challenges:

   *  Cloud service providers often find it challenging to gather direct
      feedback on customer experiences.

   *  Although obtaining network monitoring data, such as Key
      Performance Indicators (KPIs), is relatively straightforward in
      real-time, a clear correlation between the customer experience's
      Key Quality Indicators (KQI) and network KPIs remains elusive.

   *  Network quality is primarily assessed using three KPIs: network
      latency, packet-loss rate, and jitter.  Cloud service providers
      support a wide array of applications, including those sensitive to
      latency (like gaming applications) and packet loss (such as audio
      and video applications).  This diversity necessitates different
      KPIs to gauge the experience quality accurately, leading to a
      fragmented evaluation approach.

3.  Customer Experience Index

   This document introduces the Customer Experience Index (CEI), a
   measure reflecting customer experience with cloud services.  It
   enables cloud service providers to swiftly evaluate their service
   quality through a synthesis of key network metrics.

3.1.  Observation

   Customer experience often exhibits distinct zones—sensitive and
   smooth—based on their response to changes in specific indicators.
   For instance, in scenarios sensitive to latency, such as cloud
   gaming, customer satisfaction remains high within an acceptable
   latency range (smooth zone).  However, exceeding a certain latency
   threshold leads to a sharp decline in experience (sensitive zone).

   Accordingly, the CEI employs an S-Curve for its calculation, a method
   prevalent in biostatistics and sociology for modeling ecosystems and
   urbanization trends.  The S-Curve, particularly through the Sigmoid

Liu, et al.             Expires 15 November 2024                [Page 4]
Internet-Draft  Customer Experience Index for Evaluating        May 2024

   function, effectively maps values to a (0,1) interval, delineating
   two smooth zones and a sensitive zone, mirroring the nuanced nature
   of customer experience.

        ^
      1-|  '''---...           ┓
        |           ''--..     ┃- smooth zone
        |                 '.   ┛
        |                   -       ┓
        |                    .      ┃- sensitive zone
        |                     .     ┃
        |                      -    ┛
        |                   ┏   '.
        |      smooth zone -┃     ''--..
      0-|                   ┗           '''---...
        +-------------------------------------------->

                         Figure 1: S-curve Example

   Therefore, this document designs the following formula to evaluate
   customer experience for each network KPI:

   f(x) = (1+e^b)/(1+e^(a*x+b))

   *  x is the value measured by network KPI, a and b are tunable
      parameters, and f(x) represents S-curve for certain KPI.

   *  Parameter a represents the overall slope of the curve, mainly
      affecting the range of the central sensitive area.

   *  Parameter b represents the offset and scaling of the curve.  The
      initial smoothing area can be shielded via tuning b, which can
      express KPIs that immediately enter the sensitive area from the
      very beginning.

3.2.  Unified Index

   Each KPI is represented by a distinct S-curve to ensure independence
   among the indicators.  Specifically, unique S-curves for network
   latency, packet-loss rate, and jitter are created by assigning
   specific parameters (a and b), offering tailored indexes for
   applications sensitive to these different metrics.  The comprehensive
   CEI score is then derived by aggregating these three S-curves, each
   weighted appropriately:

   CEI(x, y, z) = w1 * f1(x) + w2 * f2(y) + w3 * f3(z)

Liu, et al.             Expires 15 November 2024                [Page 5]
Internet-Draft  Customer Experience Index for Evaluating        May 2024

   *  x, y, and z respectively indicate values of the three major
      network KPIs: network latency, packet-loss rate, and jitter.

   *  f1, f2, f3 represent the three individual S-curves.

   *  w1, w2, w3 represent the empirical weights.

3.3.  Parameter Tuning

3.3.1.  Weight Proportion

   The CEI's flexibility allows for fine-tuning to meet specific
   application needs by adjusting its weight values (w1, w2, w3),
   enabling precise adaptation for various application categories.
   Typically, cloud customers engage in scenarios that are either
   sensitive to latency—like gaming applications—or to packet loss, such
   as audio and video streaming.  For instance, in latency-sensitive
   scenarios, the weights for latency, packet-loss rate, and jitter
   could be adjusted to a ratio of 7:2:1 (w1:w2:w3); whereas for packet-
   loss-sensitive scenarios, a ratio of 2:7:1 (w1:w2:w3) might be more
   appropriate.  This tailored approach allows the CEI to accurately
   assess network quality for different types of applications from a
   specific viewpoint (e.g., a fixed test point) across various cloud
   vendors.

3.3.2.  Parameter a, b

   The parameters a and b of the CEI formula can be fine-tuned via:

   *  Determine the initial values of parameters a and b by fitting each
      KPI CEI curve based on a large amount of operational data.

   *  Parameters a and b can be further tuned based on preferences of
      certain application class.  For example, when the packet-loss rate
      is no higher than 𝑘, it is desired that CEI goes up as the network
      latency lowers.  CEI can set tuning goals according to such
      preferences and fine-tune parameters a and b.

4.  Security Considerations

   TBD.

5.  IANA Considerations

   This document has no IANA actions.

6.  References

Liu, et al.             Expires 15 November 2024                [Page 6]
Internet-Draft  Customer Experience Index for Evaluating        May 2024

6.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/rfc/rfc2119>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

6.2.  Informative References

   [I-D.teigen-ippm-app-quality-metric-reqs]
              Teigen, B. I. and M. Olden, "Requirements for a Network
              Quality Framework Useful for Applications, Users, and
              Operators", Work in Progress, Internet-Draft, draft-
              teigen-ippm-app-quality-metric-reqs-02, 18 October 2023,
              <https://datatracker.ietf.org/doc/html/draft-teigen-ippm-
              app-quality-metric-reqs-02>.

Authors' Addresses

   Sifa Liu
   Huawei Cloud
   China
   Email: liusifa@huawei.com

   Yaojing Wang
   Huawei
   China
   Email: wangyaojing1@huawei.com

   Wei Sun
   Huawei Cloud
   China
   Email: james.sunwei@huawei.com

   Xiang Huang
   Huawei Cloud
   China
   Email: terran.huang@huawei.com

Liu, et al.             Expires 15 November 2024                [Page 7]
Internet-Draft  Customer Experience Index for Evaluating        May 2024

   Shuai Zhou
   Huawei Cloud
   China
   Email: zhoushuai@huawei.com

   Hongyi Huang (editor)
   Huawei
   China
   Email: hongyi.huang@huawei.com

   Tianran Zhou (editor)
   Huawei
   China
   Email: zhoutianran@huawei.com

   Jian Ge
   CAICT
   China
   Email: gejian@caict.ac.cn

   Ke Ma
   CAICT
   China
   Email: make@caict.ac.cn

Liu, et al.             Expires 15 November 2024                [Page 8]