Internet Architecture Board                                  F. Bronzino
Internet-Draft                              Universite Savoie Mont Blanc
Intended status: Informational                                 E. Culley
Expires: 1 May 2021                                              Comcast
                                                             N. Feamster
                                                                  S. Liu
                                                   University of Chicago
                                                            J. Livingood
                                                              P. Schmitt
                                                    Princeton University
                                                         28 October 2020

  IAB COVID-19 Workshop: Interconnection Changes in the United States


   During the early weeks and months of the COVID-19 pandemic,
   significant changes to Internet usage occurred as a result of a
   sudden global shift to people working, studying and quarantining at
   home.  One aspect that this affected was interconnection between
   networks, which this paper studies.  This paper explores some of the
   effects of these changes on Internet interconnection points, in terms
   of utilization, traffic ratios, and other performance characteristics
   such as latency.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 1 May 2021.

Bronzino, et al.           Expires 1 May 2021                   [Page 1]

Internet-Draft  COVID-19 Interconnection Traffic Effects    October 2020

Copyright Notice

   Copyright (c) 2020 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Long-Term Interconnection Data  . . . . . . . . . . . . . . .   3
   3.  Comcast's COVID-19-Related Experiences  . . . . . . . . . . .   4
   4.  Detailed Statistical Observations . . . . . . . . . . . . . .   6
   5.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   6
   6.  Security Considerations . . . . . . . . . . . . . . . . . . .   6
   7.  Normative References  . . . . . . . . . . . . . . . . . . . .   6
   8.  Informative References  . . . . . . . . . . . . . . . . . . .   6
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   7

1.  Introduction

   During the early weeks and months of the COVID-19 pandemic
   [WHO-Declaration], significant changes to Internet usage occurred as
   a result of a sudden global shift to people working, studying and
   quarantining at home.  One aspect that this affected was
   interconnection between networks, which this paper studies.

   In 2016, the Interconnection Measurement Project [Tinker-Blog] was
   launched.  The IMP platform initially collected interconnection-
   related data from seven U.S.-based cable-based Internet Service
   Providers (ISPs) and an [ArXiv-Paper] shared some of the details and
   findings.  The initial focus of the project was to explore
   utilization characteristics of interconnect links during a period of
   time when video traffic was steadily increasing.  The project
   concluded that there was ample aggregate capacity on interconnect
   links between ISPs and peers (including content providers), and that
   ISPs continually added capacity to their interconnects to keep pace
   with the growth in traffic.

Bronzino, et al.           Expires 1 May 2021                   [Page 2]

Internet-Draft  COVID-19 Interconnection Traffic Effects    October 2020

   This IAB paper looks specifically at the long-term interconnection
   data from one of those ISPs, Comcast.  We examine the long-term pre-
   COVID-19 trend as well as what occurred as COVID-19 impacted the
   Internet from February 2020 through August 2020.  We also include
   observations from Comcast concerning interconnection changes during
   this timeframe.

   We hope that this information will be useful to the IAB workshop and
   the Internet community more broadly.  It may serve as an interesting
   and useful historical reference in the future.

2.  Long-Term Interconnection Data

   The IMP platform collected interconnection data starting in 2016,
   roughly four years prior to the COVID-19-driven shift in Internet
   usage.  This provides an interesting capability for a before and
   after view of interconnection.  A full explanation of the data can be
   found in Section 3 of the [ArXiv-Paper].  At a high level, the
   maindata collected encompasses:

   *  Timestamp (representing a five-minute interval)

   *  Region (representing an aggregated link group)

   *  Anonymized partner network

   *  Access ISP

   *  Total ingress bytes

   *  Total egress bytes

   *  Capacity

   Utilization is captured based on sampled IPFIX records, with a packet
   sampling rate of 1/1,000.  SNMP polling data yields information about
   the capacity on each link.  The IMP platform does not have direct
   access to partner network identities---this dataset only includes an
   anonymous identifier corresponding to that particular partner.
   However, in collaboration with Comcast, IMP has worked to identify
   specific partner networks in the dataset for specific peers in the
   interest of detail COVID-19-related study.

   Focusing on Comcast within this dataset reveals several trends, in
   both aggregate capacity and utilization, as well as how utilization
   and capacity changed during the period of the COVID-19 pandemic.  The
   data also reveals how traffic volumes changed for specific peers
   during this time period.

Bronzino, et al.           Expires 1 May 2021                   [Page 3]

Internet-Draft  COVID-19 Interconnection Traffic Effects    October 2020

   The IMP data that tracks capacity shows Comcast steadily adding
   capacity from mid-2018 to present, with a significant increase in the
   rate of additional capacity from the beginning of 2020, increasing
   further in the second and third quarters of 2020.  Specifically, we
   analyzed the rate at which capacity was added during these periods,
   on a month-by-month basis.  We found that Comcast was adding
   aggregate capacity on its interconnects at *nearly twice* the rate as
   it was being added during 2019.

   Over a long timeframe, Comcast's daily peak hour interconnection port
   utilization has remained consistently around 90%. During the period
   at the end of March and the beginning of April 2020, daily peak
   utilization briefly increased to about 97% but steadily returned to
   normal levels in a matter of weeks by the end of April 2020, as
   Comcast quickly increased the rate at which it added capacity to keep
   pace with growing traffic demands.

   We also explored traffic volumes associated with each Comcast peer,
   comparing the ranks and volumes of each individual peer as measured
   on September 1, 2020 as compared to January 1, 2020, in both the
   upstream and downstream directions.  Doing so allowed us to
   understand both the magnitude of changes in traffic volumes and
   ratios, as well as how specific individual peers deviated from normal
   baseline behavior, in terms of both upstream and downstream traffic
   rates and ratios.  The data suggests that both the upstream and
   downstream directions saw some changes, although the deviations in
   upstream traffic patterns were greater: the Pearson coefficient for
   downstream traffic was 0.977, whereas for upstream traffic the
   Pearson correlation coefficient was only 0.935, indicating a more
   significant shift in upstream traffic ratios during this timeframe.

3.  Comcast's COVID-19-Related Experiences

   Comcast observed a wide range of significant changes in Internet
   usage as residential users remained at home and shifted to working
   and studying from home.  Changes in usage patterns observed in the
   access network in sum led to changes in the traffic flowing to
   interconnected networks.  As the pandemic developed, there was a wide
   variety of changes in traffic volumes.  At some locations in the
   network little change was detected while other locations saw a huge
   growth in the volume of traffic.

   At the peak of the surge, in March and April 2020, the average amount
   of growth observed across locations and types was roughly 33%. Voice
   & video conferencing (conferencing hereafter) jumped as much as 285%
   and Wi-Fi use increased 36% among our Xfinity Mobile (MVNO) customers
   [June-Blog] [July-Blog].  As this continued into May 2020 we observed
   conferencing remained up roughly 210-285%, VPN traffic up 30-40% and

Bronzino, et al.           Expires 1 May 2021                   [Page 4]

Internet-Draft  COVID-19 Interconnection Traffic Effects    October 2020

   gaming downloads up 20-80%, and web-based streaming video consumption
   up 20-40% [May-Blog].  In this several week period, traffic
   essentially grew at or more than it had in the prior year, which was
   significant growth in a short period of time.

   In the months following the onset of the pandemic Comcast observed:

   *  Overall average downstream peak growth is up 13% (up as much as
      20% at times).

   *  Overall average upstream peak growth is up 36%.

   *  In the access network, an average of 771 network augments per week
      were performed, peaking at over 1,800 in a single week and over
      with over 7 weeks with more than 1,000 per week.  For comparison,
      the average earlier in the year was roughly 350 per week.

   *  In the core network, over 500 augments were made in order to add
      146 Tbps in capacity.

   *  On a daily basis roughly 700,000 automated speed tests from
      customer homes were conducted in order to gauge the customer
      experience during this time.  Average speeds to customers (both
      downstream and upstream) have generally remained at or above 105%
      of advertised speeds since March 1, 2020 in all regions.  National
      average speeds have remained between 110% - 115% of advertised
      speeds over the same period.  [NF-Paper-1] and [NF-Paper-2]

   *  The share of streaming video as a percentage of total traffic is
      declined slightly from 67% to 63%.  Despite strong growth,
      conferencing occupies a small share of total traffic and grew from
      1% share to 4%. But gaming software released have driven
      significant download spikes since late April 2020.

   *  For interconnection, peering coordinators across operators worked
      cooperatively and quickly to cut through any red tape and add new
      capacity as quickly as possible.

   *  In 2019, settlement free interconnection capacity [SFI-Policy] - a
      subset of overall interconnect types - grew by 15%. Between
      January and August 2020, driven by COVID-19 changes, there was an
      overall 37% increase in capacity from that prior 2019 level.  And
      between March and October 2020 one Settlement Free Peer alone
      increased 115%.

   *  Between March and October 2020 from Comcast observed other notable
      per-peer traffic increases of 245% and 3,900%.

Bronzino, et al.           Expires 1 May 2021                   [Page 5]

Internet-Draft  COVID-19 Interconnection Traffic Effects    October 2020

4.  Detailed Statistical Observations

   As briefly mentioned in previous sections, downstream traffic rates
   from many partners remained stable---the Pearson correlation
   coefficient for peak download rates between January 1, 2020 and
   September 2, 2020 is 0.977, indicating that the peak download rates
   to most peers was similar between these two time periods.  On the
   other hand, certain peers experienced either a significant increase
   or decrease in peak download rates---often by two or three orders of
   magnitude.  Similarly, other peers experienced a decrease in peak
   downstream rates by several orders of magnitude.

   On the other hand, upstream traffic rates were far less stable: In
   contrast, the Pearson coefficient for upload rates between January 1,
   2020 and September 2, 2020 was only 0.935, suggesting more a more
   significant deviation in peak upstream rates.  As with peak download
   rates, some peers experienced significant decreases, as well: in one
   outlier case, peak rate decreased by almost five orders of magnitude.
   A small handful of peers saw similar decreases.  Yet, a far greater
   number of peers saw increases in peak upload rates by two to three
   orders of magnitude.

5.  IANA Considerations

   This document includes no request to IANA.

6.  Security Considerations

   This document includes no security considerations.

7.  Normative References

8.  Informative References

              Feamster, NF., "Revealing Utilization at Internet
              Interconnection Points", 5 September 2016,

              Nafshi, EN., "COVID-19 Network Report: How A Smart Network
              Delivered Speed and Stability When it Mattered", 13 July
              2020, <

Bronzino, et al.           Expires 1 May 2021                   [Page 6]

Internet-Draft  COVID-19 Interconnection Traffic Effects    October 2020

              Werner, TW., "Cresting the Wave: The Factors that Powered
              our Network Through the COVID-19 Surge", 15 June 2020,

   [May-Blog] Comcast, "COVID-19 Network Update", 20 May 2020,

              Jones, AJ., Sevcik, PS., and AL. Lacy, "NetForecast Design
              Audit Report of Comcast's Network Performance Measurement
              System", April 2020, <

              Jones, AJ., Sevcik, PS., and AL. Lacy, "NetForecast's
              Report on Comcast's Network Performance Measurement System
              Results Data", May 2020, <

              Comcast Cable Communications Management, LLC, "Comcast
              Settlement-Free Interconnection (SFI) Policy", October
              2013, <>.

              Feamster, NF., "Interconnection Measurement Project
              Website", 9 May 2016, <https://freedom-to-

              Adhanom Ghebreyesus, TAG., "WHO Director-General's opening
              remarks at the media briefing on COVID-19 - 11 March
              2020", 11 March 2020,

Authors' Addresses

Bronzino, et al.           Expires 1 May 2021                   [Page 7]

Internet-Draft  COVID-19 Interconnection Traffic Effects    October 2020

   Francesco Bronzino
   Universite Savoie Mont Blanc


   Elizabeth Culley
   Mount Laurel, NJ
   United States of America


   Nick Feamster
   University of Chicago
   Chicago, IL
   United States of America


   Shinan Liu
   University of Chicago
   Chicago, IL
   United States of America


   Jason Livingood
   Philadelphia, PA
   United States of America


   Paul Schmitt
   Princeton University
   Princeton, NJ
   United States of America


Bronzino, et al.           Expires 1 May 2021                   [Page 8]