Internet Engineering Task Force C. Bastian
Internet-Draft T. Klieber
Intended status: Informational J. Livingood, Ed.
Expires: December 28, 2009 J. Mills
R. Woundy
Comcast
June 26, 2009
Comcast's Protocol-Agnostic Congestion Management System
draft-livingood-woundy-congestion-mgmt-00
Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on December 28, 2009.
Copyright Notice
Copyright (c) 2009 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents in effect on the date of
publication of this document (http://trustee.ietf.org/license-info).
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document.
Bastian, et al. Expires December 28, 2009 [Page 1]
Internet-Draft An ISP Congestion Management System June 2009
Abstract
This document describes the congestion management system of Comcast
Cable, a large cable broadband Internet Service Provider (ISP) in the
U.S. Comcast completed deployment of this congestion management
system on December 31, 2008.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Key Terminology . . . . . . . . . . . . . . . . . . . . . . . 3
3. Historical Overview . . . . . . . . . . . . . . . . . . . . . 7
4. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
5. Implemetation and Configuration . . . . . . . . . . . . . . . 9
5.1. Thresholds For Determining When a CMTS Port Is in a
Near Congestion State . . . . . . . . . . . . . . . . . . 13
5.2. Thresholds For Determining When a User Is in an
Extended High Consumption State and for Release from
that Classification . . . . . . . . . . . . . . . . . . . 14
5.3. Effect of BE Quality of Service on Users&apos
Broadband Experience . . . . . . . . . . . . . . . . . . . 17
5.4. Equipment/Software Used and Location . . . . . . . . . . . 19
6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 21
7. Exceptional Network Utilization Considerations . . . . . . . . 21
8. Security Considerations . . . . . . . . . . . . . . . . . . . 21
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 22
11. Change Log . . . . . . . . . . . . . . . . . . . . . . . . . . 22
12. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 22
13. Informative References . . . . . . . . . . . . . . . . . . . . 22
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 24
Bastian, et al. Expires December 28, 2009 [Page 2]
Internet-Draft An ISP Congestion Management System June 2009
1. Introduction
Comcast Cable is a large broadband Internet Service Provider (ISP),
based in the U.S., serving the majority of its customers via cable
modem technology. During the late part of 2008, and completing on
December 31, 2008, Comcast deployed a new congestion management
system across its entire network. This new system was developed in
response to dissatisfaction in the Internet community as well as
complaints to the U.S. Federal Communications Commission (FCC)
regarding Comcast's old system, which targeted specific peer-to-peer
(P2P) applications. This new congestion management system is
protocol-agnostic, meaning that it does not examine or impact
specific user applications or network protocols, which is perceived
as a more fair system for managing network resources at limited times
when congestion may occur.
The purpose of this document is to describe how this example of a
large scale congestion management system functions. This is
primarily in response to questions from other ISPs as well as
solution developers, who are interested in learning from and/or
deploying similar systems in other networks.
2. Key Terminology
This section defines the key terms used in this document. Some terms
below refer to elements of the Comcast network. As a result, it may
be helpful to refer to Figure 1 when reviewing some of these terms.
2.1. Cable Modem
A device located at the customer premise used to access the Comcast
High Speed Internet (HSI) network. In some cases, the cable modem is
owned by the customer, and in other cases it is owned by the cable
operator. This device has an interface (i.e., someplace to plug in a
cable) for connecting the coaxial cable provided by the cable company
to the modem, as well as one or more interfaces for connecting the
modem to a customer's PC or home gateway device (e.g., home gateway,
router, firewall, access point, etc.). In some cases, the cable
modem function, i.e., the ability to access the Internet, is
integrated into a home gateway device or Embedded Multimedia Terminal
Adapter (eMTA). Once connected, the cable modem links the customer
to the HSI network and ultimately the broader Internet.
2.2. Cable Modem Termination System (CMTS)
A piece of hardware located in a cable operator's local network
(generally in a "headend", Section 2.10) that acts as the gateway to
Bastian, et al. Expires December 28, 2009 [Page 3]
Internet-Draft An ISP Congestion Management System June 2009
the Internet for cable modems in a particular geographic area. A
simple way to think of the CMTS is as a router with interfaces on one
side leading to the Internet and interfaces on the other connecting
to Optical Nodes and then customers, in a so-called "last mile""
network.
2.3. Cable Modem Termination System (CMTS) Port
Also referred to simply as a "port". A port is a physical interface
on a device used to connect cables in order to connect with other
devices for transferring information/data. An example of a physical
port is a CMTS port. A CMTS has both upstream and downstream network
interfaces to serve the local access network, which are referred to
as upstream or downstream ports. A port generally serves a
neighborhood of hundreds of homes. Over time, CMTS ports tend to
server fewer and fewer homes, as the network is segmented for
capacity growth purposes. Prior to DOCSIS version 3, a single CMTS
physical port was used for either transmitting or receiving data
downstream or upstream to a given neighborhood. With DOCSIS version
3, and the channel bonding feature, multiple CMTS physical ports can
be combined to create a virtual port. A CMTS is also briefly defined
in Section 2.6 of [RFC3083].
2.4. Channel Bonding
A technique for combining multiple downstream and/or upstream
channels to increase customers' download and/or upload speeds,
respectively. Multiple channels from the Hybrid Fiber Coax (HFC,
Section 2.11) network can be bonded into a single virtual port
(called a bonded group), which acts as a large single channel or port
to provide increased speeds for customers. Channel bonding is a
feature of Data Over Cable Service Interface Specification (DOCSIS)
version 3.
[EDITORIAL NOTE: Any other good DOCSIS 3.0 channel bonding references
to include here?]
2.5. Coaxial Cable (Coax)
A type of cable used by a cable operator to connect customer premise
equipment (CPE) -- such as TVs, cable modems (including eMTAs), and
Set Top Boxes -- to the HFC network. This cable may be used both
within the home as well as in segments of the last mile network
running to a home or customer premise location. There are many
grades of coaxial cable that are used for different purposes.
Different types of coaxial cable are used for different purposes on
the network.
Bastian, et al. Expires December 28, 2009 [Page 4]
Internet-Draft An ISP Congestion Management System June 2009
2.6. Comcast High Speed Internet (HSI)
A service/product offered by Comcast for delivering Internet service
over a broadband connection.
2.7. Customer Premise Equipment (CPE)
Any device that resides at the customer's residence.
2.8. Data Over Cable Service Interface Specification (DOCSIS)
A reference standard developed by CableLabs that specifies how
components on cable networks need to be built to enable HSI service
over an HFC network. These standards define the specifications for
the cable modem and the CMTS such that any DOCSIS certified cable
modem will work on any DOCSIS-certified CMTS, independent of the
selected vendor. The interoperability of cable modems and CMTSs
allows customers to purchase a DOCSIS-certified modem from a retail
outlet and use it on their cable-networked home. These standards are
available to the public at the CableLabs website, at
http://www.cablelabs.com.
[EDITORIAL NOTE: Any other good DOCSIS references to include here?]
2.9. Downstream
Description of the direction in which a signal travels, in this case
from the network to a user. Downstream traffic occurs when users are
downloading something from the Internet, such as watching a web-based
video, reading web pages, or downloading software updates.
2.10. Headend
A cable facility responsible for receiving TV signals for
distribution over the HFC network to the end customers. This
facility typically also houses one or more CMTSs. This is sometimes
also called a "hub".
2.11. Hybrid Fiber Coax (HFC)
A network architecture used primarily by cable companies, comprised
of fiber optic and coaxial cables that currently deliver Voice,
Video, and Internet services to customers.
2.12. Internet Protocol Detail Record (IPDR)
Standardized technology for monitoring and/or recording subscribers'
upstream and downstream Internet usage data based on their cable
Bastian, et al. Expires December 28, 2009 [Page 5]
Internet-Draft An ISP Congestion Management System June 2009
modem. The data is collected from the CMTS and sent to a server for
further processing. Additional information is available at
http://www.ipdr.org, as well as [IPDR Standard] and [CableLabs DOCSIS
IPDR Support].
2.13. Optical Node
A component of the HFC network generally located in customers' local
neighborhoods which is used to convert the optical signals sent over
fiber-optic cables to electrical signals that can be sent over
coaxial cable to customers' cable modems, or vice versa. A fiber
optic cable connects the Optical Node, through distribution hubs, to
the CMTS and coaxial cable connects the Optical Node to customers'
cable modems.
2.14. Provisioned Bandwidth
The peak speed associated with a tier of service purchased by a
customer. For example, a customer with a 50 Mbps downstream and 10
Mbps upstream speed tier would be said to be provisioned with 50 Mbps
of downstream bandwidth and 10 Mbps of upstream bandwidth. This is
often referred to as 50/10 service in industry parlance.
The Provisioned Bandwidth is the speed that a customer's modem is
configured (and the network is engineered) to deliver on a regular
basis (which is not the same as a "Committed Information Rate" or a
guaranteed rate). Internet speeds are generally a best efforts
service that are dependent on a number of variables, many of which
are outside the control of an Internet Service Provide (ISP). In
general, speeds do not typically exceed a customer's provisioned
speed. Comcast, however, invented a technology called "PowerBoost"
[EDITORIAL NOTE: Include external technical references to
PowerBoost?] that, for example, enables users to experience brief
boosts above their provisioned speeds while they transfer large files
over the Internet, by utilizing excess capacity which may be
available in the network at that time.
2.15. Quality of Service (QoS)
Set of techniques to manage network resources to ensure a level of
performance to specific data flows. One method for providing QoS to
a network is by differentiating the type of traffic by class or flow
and assigning priorities to each type. When the network becomes
congested, the data packets that are marked as having higher priority
will have higher likelihood of being serviced.
[EDITORIAL NOTE: Any good QoS references to include here?]
Bastian, et al. Expires December 28, 2009 [Page 6]
Internet-Draft An ISP Congestion Management System June 2009
2.16. Upstream
Description of the direction in which a signal travels, in this case
from the user to the network. Upstream traffic occurs when users are
uploading something to the network, such as sending email, sending
files to another computer, or uploading photos to a digital photo
website.
3. Historical Overview
Comcast began the engineering project to develop a new congestion
management system in March 2008, the same month that Comcast hosted
the 71st meeting of the IETF in Philadelphia, PA, USA. On May 28,
2008, Comcast participated in an IETF Peer-to-Peer Infrastructure
Workshop [I-D.p2pi-cooper-workshop-report], hosted by the
Massachusetts Institute of Technology (MIT) in Cambridge, MA, USA.
In order to participate in this workshop, interested attendees were
asked to submit a paper to a technical review team, which Comcast did
on May 9, 2008, in the [Comcast P2Pi Position Paper]. Comcast
subsequently attended and participated in this valuable workshop.
During the workshop, Comcast outlined the high-level design for a new
congestion management system [Comcast IETF P2P Infrastructure
Workshop Presentation] and solicitied comments and other feedback
from attendees and other members of the Internet community
(presentations were also posted to the IETF's P2Pi mailing list).
The congestion management system outlined in that May 2008 workshop
was later tested in trial markets and is in essence what was then
deployed by Comcast later in 2008.
Following an August 2008 [FCC Memorandum and Opinion] regarding how
Comcast managed congestion on its High-Speed Internet ("HSI")
network, Comcast disclosed to the FCC and the public additional
technical details of the congestion management system that it
intended to and did implement by the end of 2008, including the
thresholds involved in this new system. That system is detailed in
this document. While the description of how this system is deployed
in the Comcast network is necessarily specific to the various
technologies and designs specific to that network, a similar system
could be deployed on virtually any large scale ISP network or other
IP network.
4. Summary
Comcast's HSI network has elements which are shared across many
subscribers. This means that Comcast's HSI customers share upstream
Bastian, et al. Expires December 28, 2009 [Page 7]
Internet-Draft An ISP Congestion Management System June 2009
and downstream bandwidth with their neighbors. Although the
available bandwidth is substantial, so, too, is the demand. Thus,
when a relatively small number of customers in a neighborhood place
disproportionate demands on network resources, this can cause
congestion that degrades their neighbors' Internet experience. The
goal of Comcast's new congestion management system is to enable all
users of our network resources to access a "fair share" of that
bandwidth, in the interest of ensuring a high-quality online
experience for all of Comcast's HSI customers.
Importantly, the new approach is protocol-agnostic; that is, it does
not manage congestion by focusing on the use of the specific
protocols that place a disproportionate burden on network resources,
or any other protocols. Rather, the new approach focuses on managing
the traffic of those individuals who are using the most bandwidth at
times when network congestion threatens to degrade subscribers'
broadband experience and who are contributing disproportionately to
such congestion at those points in time.
Specific details about these practices, including relevant threshold
information, the type of equipment used, and other particulars, are
discussed at some length later in this document. At the outset,
however, we present a very high-level, simplified overview of how
these practices work. Despite all the detail provided further below,
the fundamentals of this approach can be summarized succinctly:
1. Software installed in the Comcast network continuously examines
aggregate traffic usage data for individual segments of Comcast's
HSI network. If overall upstream or downstream usage on a
particular segment of Comcast's HSI network reaches a pre-
determined level, the software moves on to step two.
2. At step two, the software examines bandwidth usage data for
subscribers in the affected network segment to determine which
subscribers are using a disproportionate share of the bandwidth.
If the software determines that a particular subscriber or
subscribers have been the source of high volumes of network
traffic during a recent period of minutes, traffic originating
from that subscriber or those subscribers temporarily will be
assigned a lower priority status.
3. During the time that a subscriber's traffic is assigned the lower
priority status, such traffic will not be delayed so long as the
network segment is not actually congested. If, however, the
network segment becomes congested, such traffic could be delayed.
4. The subscriber's traffic returns to normal priority status once
his or her bandwidth usage drops below a set threshold over a
Bastian, et al. Expires December 28, 2009 [Page 8]
Internet-Draft An ISP Congestion Management System June 2009
particular time interval.
Comcast undertook considerable effort, over the course of many
months, to formulate our plans for this congestion management
approach, adjusting them, and subjecting them to real-world trials.
Market trials were conducted in Chambersburg, PA; Warrenton, VA; Lake
City, FL; East Orange, FL; and Colorado Springs, CO, between June and
September, 2008. This enabled us to validate the utility of the
general approach and collect substantial trial data to test multiple
variations and alternative formulations.
5. Implemetation and Configuration
To understand exactly how these new congestion management practices
work, it is helpful to have a general understanding of how Comcast's
HSI network is designed. Comcast's HSI network is what is commonly
referred to as a hybrid fiber-coax network, with coaxial cable
connecting each subscriber's cable modem to an Optical Node, and
fiber optic cables connecting the Optical Node, through distribution
hubs, to the Cable Modem Termination System (CMTS), which is also
known as a "data node". The CMTSes are then connected to higher-
level routers, which in turn are connected to Comcast's Internet
backbone facilities. Today, Comcast has over 3,200 CMTSes deployed
throughout our network, serving over 15 million HSI subscribers.
Each CMTS has multiple "ports" that handle traffic coming into and
leaving the CMTS. In particular, each cable modem deployed on the
Comcast HSI network is connected to the CMTS through the ports on the
CMTS. These ports can be either "downstream" ports or "upstream"
ports, depending on whether they send information to cable modems
(downstream) or receive information from cable modems (upstream)
attached to the port. (Note that the term "port" as used here
generally contemplates single channels on a CMTS, but these
statements will apply to virtual channels, also known as "bonded
groups", in a DOCSIS 3.0 environment.) Currently, on average,
approximately 275 cable modems share the same downstream port and
about 100 cable modems share the same upstream port, however this is
constantly changing (both numbers generally become smaller over
time). Both types of ports can experience congestion that could
degrade the broadband experience of our subscribers and, unlike with
the previous congestion management practices, both upstream and
downstream traffic are be subject to management in this new
congestion management system.
To implement Comcast's new protocol-agnostic congestion management
practices, Comcast purchased new hardware and software that was
deployed near the Regional Network Routers ("RNRs") that are further
Bastian, et al. Expires December 28, 2009 [Page 9]
Internet-Draft An ISP Congestion Management System June 2009
upstream in Comcast's network. This new hardware consists of
Internet Protocol Detail Record ("IPDR") servers, Congestion
Management servers, and PacketCable Multimedia ("PCMM") servers.
Further details about each of these pieces of equipment can be found
below, in Section 5.4. It is important to note here, however, that
even though the physical location of these servers is at the RNR, the
servers communicate with -- and manage individually -- multiple ports
on multiple CMTSes to effectuate the practices described in this
document. That is to say, bandwidth usage on one CMTS port will have
no effect on whether the congestion management practices described
herein are applied to a subscriber on a different CMTS port.
Figure 1 provides a simplified graphical depiction of the network
architecture just described:
Figure 1: Simplified Network Diagram Showing High-Level Comcast
Network and Servers Relevant to Congestion Management
-----------------
///// \\\\\
| Comcast Internet Backbone |
\\\\\ ------
----------------// \\
+------------+ / \
| Congestion | | Internet |
| Management |<+++GigE++++ +---->| Backbone |
| Server | + | | Router |
+------------+ + | \ /
+ Fiber \\ //
+------------+ + | -----
| QoS | + |
| Server |<+++GigE++++ \/
| | + -----
+------------+ + // \\
+ / \
+------------+ + | Regional |
| Statistics | +++++++>| Network |
| Collection |<+++GigE++++ | Router |
| Server | \ /
+------------+ +---Fiber------>\\ //<------Fiber----+
| ----- |
| |
\/ \/
----- -----
// \\ // \\
/ \ / \
| Local | | Local |
| Market | | Market |
Bastian, et al. Expires December 28, 2009 [Page 10]
Internet-Draft An ISP Congestion Management System June 2009
| Router | | Router |
\ / \ /
+--------->\\ //<------------+ \\ //
| ----- | -----
| /\ | /\
Fiber | Fiber |
| Fiber | Fiber
| | | |
\/ \/ \/ \/
//----\\ //----\\ //----\\ //----\\
| CMTS | | CMTS | | CMTS | | CMTS |
\\----// \\----// \\----// \\----//
/\ /\ /\ /\
| | | |
Fiber Fiber Fiber Fiber
| | | |
\/ \/ \/ \/
+---------+ +---------+ +---------+ +---------+
| Optical | | Optical | | Optical | | Optical |
| Node | | Node | | Node | | Node |
+---------+ +---------+ +---------+ +---------+
/\ /\ /\ /\ /\ /\
|| || ||______ || _____|| ||
Coax Coax |__Coax| Coax |Coax__| Coax
|| || || || || ||
\/ \/ \/ \/ \/ \/
+=======+ +=======+ +=======+ +=======+ +=======+ +=======+
= Cable = = Cable = = Cable = = Cable = = Cable = = Cable =
= Modem = = Modem = = Modem = = Modem = = Modem = = Modem =
+=======+ +=======+ +=======+ +=======+ +=======+ +=======+
================================================================
+ Note: This diagram is a simplification of the actual network +
+ and servers, which in actuality includes redundant network +
+ links, redundant network devices, redundant servers, and +
+ other details too complex to represent here. +
================================================================
Figure 1
Each Comcast HSI subscriber's cable modem has a "bootfile", which is
essentially a configuration file that contains certain pieces of
information about the subscriber's service to ensure that the service
functions properly. (Note: No personal information is included in
the bootfile; it only includes information about the service that the
subscriber has purchased.) For example, the bootfile contains
information about the maximum speed (what we refer to in this
Bastian, et al. Expires December 28, 2009 [Page 11]
Internet-Draft An ISP Congestion Management System June 2009
document as the "provisioned bandwidth") that a particular modem can
achieve based on the tier (personal/residential, commercial, etc.)
the customer has purchased. Bootfiles are generally reset from time
to time to account for changes in the network and other updates, and
this is usually done through a command sent from the network and
without any effect on the subscriber. In preparation for the
transition to this new congestion management system, Comcast sent new
bootfiles to our HSI customers' cable modems that created two Quality
of Service ("QoS") levels for Internet traffic going to and from the
cable modem: (1) "Priority Best-Effort" traffic ("PBE"); and (2)
"Best-Effort" traffic ("BE"). As with previous changes to cable
modem bootfiles, the replacement of the old bootfile with the new
bootfile requires no active participation by Comcast customers.
Thereafter, all traffic going to or coming from cable modems on the
Comcast HSI network is designated as either PBE or BE. PBE is the
default status for all Internet traffic coming from or going to a
particular cable modem. Traffic is designated BE for a particular
cable modem only when both of two conditions are met:
o First, the usage level of a particular upstream or downstream port
of a CMTS, as measured over a particular period of time, must be
nearing the point where congestion could degrade users'
experience. We refer to this as the "Near Congestion State" and,
based on the technical trials we have conducted, we have
established a threshold, described in more detail below, for when
a particular CMTS port enters that state.
o Second, a particular subscriber must be making a significant
contribution to the bandwidth usage on the particular port, as
measured over a particular period of time. We refer to this as
the "Extended High Consumption State" and, based on the technical
trials we have conducted, we have established a threshold,
described in more detail below, for when a particular user enters
that state.
When, and only when, both conditions are met, a user's upstream or
downstream traffic (depending on which type of port is in the Near
Congestion State) is designated as BE. Then, to the extent that
actual congestion occurs, any delay resulting from the congestion
will affect BE traffic before it affects PBE traffic.
We now explain the foregoing in greater detail in the following
sections.
Bastian, et al. Expires December 28, 2009 [Page 12]
Internet-Draft An ISP Congestion Management System June 2009
5.1. Thresholds For Determining When a CMTS Port Is in a Near
Congestion State
For a CMTS port to enter the Near Congestion State, traffic flowing
to or from that CMTS port must exceed a specified level (the "Port
Utilization Threshold") for a specific period of time (the "Port
Utilization Duration"). The Port Utilization Threshold on a CMTS
port is measured as a percentage of the total aggregate upstream or
downstream bandwidth for the particular port during the relevant
timeframe. The Port Utilization Duration on the CMTS is measured in
minutes.
Values for each of the thresholds to be used as part of this new
management technique have been tentatively established after an
extensive process of lab tests, simulations, technical trials, vendor
evaluations, customer feedback, and a third-party consulting
analysis. In the same way that specific anti-spam or other network
management practices are adjusted to address new issues that arise,
it is a near certainty that these values will change over time, as
Comcast gathers more data and performs additional analysis resulting
from wide-scale use of the new technique. Moreover, as with any
large network or software system, software bugs and/or unexpected
errors may arise, requiring software patches or other corrective
actions. As always, Comcast's decisions on these matters are driven
by the marketplace imperative that we deliver the best possible
experience to our HSI subscribers.
Given our experience so far, we have determined that a starting point
for the upstream Port Utilization Threshold should be 70 percent and
the downstream Port Utilization Threshold should be 80 percent. For
the Port Utilization Duration, we have determined that the starting
point should be approximately 15 minutes (although some technical
limitations in some newer CMTSes deployed on Comcast's network may
make this time period vary slightly). Thus, over any 15-minute
period, if an average of more than 70 percent of a port's upstream
bandwidth capacity or more than 80 percent of a port's downstream
bandwidth capacity is utilized, that port is determined to be in a
Near Congestion State.
Based on the trials conducted and operational experience to date, a
typical CMTS port on our HSI network is in a Near Congestion State
only for relatively small portions of the day, if at all, though
there is no way to forecast what will be the busiest time on a
particular port on a particular day. Moreover, the trial data and
operational experience indicate that, even when a particular port is
in a Near Congestion State, the instances where the network actually
becomes congested during the Port Utilization Duration are few, and
managed users whose traffic is delayed during those congested periods
Bastian, et al. Expires December 28, 2009 [Page 13]
Internet-Draft An ISP Congestion Management System June 2009
perceive little, if any, effect, as discussed below.
5.2. Thresholds For Determining When a User Is in an Extended High
Consumption State and for Release from that Classification
Once a particular CMTS port is in a Near Congestion State, the
software examines whether any cable modems are consuming bandwidth
disproportionately. (Note: Although each cable modem is typically
assigned to a particular household, the software does not and cannot
actually identify individual users or analyze particular users'
traffic.) For purposes of this document, we use "cable modem",
"user", and "subscriber" interchangeably to mean a subscriber account
or user account and not an individual person.). For a user to enter
an Extended High Consumption State, he or she must consume greater
than a certain percentage of his or her provisioned upstream or
downstream bandwidth (the "User Consumption Threshold") for a
specific length of time (the "User Consumption Duration"). The User
Consumption Threshold is measured as a user's consumption of a
particular percentage of his or her total provisioned upstream or
downstream bandwidth. That bandwidth is the maximum speed that a
particular modem can achieve based on the tier (personal/residential,
commercial, etc.) the customer has purchased. For example, if a user
buys a service with speeds of 8 Mbps downstream and 1 Mbps upstream,
then his or her provisioned downstream speed is 8 Mbps and
provisioned upstream speed is 1 Mbps. It is also important to note
that because the User Consumption Threshold is a percentage of
provisioned bandwidth for a particular user account, and not a static
value, users of higher speed tiers have correspondingly higher User
Consumption Thresholds. Lastly, the User Consumption Duration is
measured in minutes.
Following lab tests, simulations, technical trials, customer
feedback, vendor evaluations, and a third-party consulting analysis,
we have determined that the appropriate starting point for the User
Consumption Threshold is 70 percent of a subscriber's provisioned
upstream or downstream bandwidth, and that the appropriate starting
point for the User Consumption Duration is 15 minutes. That is, when
a subscriber uses an average of 70 percent or more of his or her
provisioned upstream or downstream bandwidth over a particular 15-
minute period, that user is then in an Extended High Consumption
State. The User Consumption Thresholds have been set sufficiently
high that using the HSI connection for VoIP or most streaming video
cannot alone cause subscribers to our standard-level HSI service to
exceed the User Consumption Threshold. For example, while Comcast's
standard-level HSI service provisions downstream bandwidth at 12
Mbps, today, streaming video (even some HD video) from Hulu uses less
than 2.5 Mbps, a Vonage or Skype VoIP call uses less than 131 Kbps,
and streaming music uses less than 128 kbps (in this example, 70
Bastian, et al. Expires December 28, 2009 [Page 14]
Internet-Draft An ISP Congestion Management System June 2009
percent of 12 Mbps is 8.4 Mbps). As noted above, these values are
subject to change as necessary in the same way that specific anti-
spam or other network management practices are adjusted to address
new issues that arise, or should unexpected software bugs or other
problems arise.
Based on data collected from the trial markets where the new
management practices are being tested, on average less than one-third
of one percent of subscribers have had their traffic priority status
changed to the BE state on any given day. For example, in Colorado
Springs, CO, the largest test market, on any given day in August
2008, an average of 22 users out of 6,016 total subscribers in the
trial had their traffic priority status changed to BE at some point
during the day.
A user's traffic is released from a BE state when the user's
bandwidth consumption drops below 50 percent of his or her
provisioned upstream or downstream bandwidth for a period of
approximately 15 minutes. These release criteria are intended to
minimize (and hopefully prevent) user QoS oscillation (hysteresis),
i.e., a situation in which a particular user could cycle repeatedly
between BE and PBE. NetForecast, Inc., an independent consultant
retained to provide analysis and recommendations regarding Comcast's
trials and related congestion management work, suggested this
approach, which has worked well in our ongoing trials and lab
testing. In trials, we have observed that user traffic rarely
remains in a managed state longer than the initial 15-minute period.
Simply put, there are four steps to determining whether the traffic
associated with a particular cable modem is designated as PBE or BE:
1. Determine if the CMTS port is in a Near Congestion State.
2. If yes, determine whether any users are in an Extended High
Consumption State.
3. If yes, change those users' traffic to BE from PBE. If the
answer at either step one or step two is no, no action is taken.
4. If a user's traffic has been designated BE, check user
consumption at next interval. If user consumption has declined
below predetermined threshold, reassign the user's traffic as
PBE. If not, recheck at next interval.
Figure 2 graphically depicts how this congestion management process
works, using an example of a situation where upstream port
utilization may be reaching a Near Congestion State (the same
diagram, with different values in the appropriate places, could be
Bastian, et al. Expires December 28, 2009 [Page 15]
Internet-Draft An ISP Congestion Management System June 2009
used to depict the management process for downstream ports, as well):
Figure 2: Upstream Congestion Management Decision Flowchart
/\
/ \
+------------+ / \ +---------+ +---------+
| Start | / \ | | / /
| Congestion | / \ | | / /
| Management +-->+ Question +--YES-->| Result |--THEN-->/ ACTION /
| Process | \ #1 / | #1 | / #1 /
| | \ / | | / /
+------------+ \ / +---------+ +---------+
\ / |
\/ |
| THEN
| |
NO |
| \/
| /\
\/ / \
+---------+ / \
| | / \
| | / \
| Result |<-------------NO------------+ Question +
| #2 | \ #2 /
| | \ /
+---------+ \ /
\ /
\/
|
|
YES
|
/\ |
/ \ \/
+---------+ / \ +---------+
| | / \ | |
| | / \ THEN, AT | |
| Result |<--YES--+ Question + <---NEXT ANALYSIS------+ Result |
| #4 | \ #3 / POINT /\ | #3 |
| | \ / | | |
+---------+ \ / | +---------+
\ / |
\/ |
| |
| |
+------------NO-------------+
Bastian, et al. Expires December 28, 2009 [Page 16]
Internet-Draft An ISP Congestion Management System June 2009
KEY TO FIGURES ABOVE:
Question #1: Is the CMTS Upstream Port Utilization at an average
of OVER 70% for OVER 15 minutes?
Result #1: CMTS marked in a Near Congestion State, indicating
congestion MAY occur soon.
Action #1: Search most recent analysis timeframe (approx. 15 mins.)
of IPDR usage data.
Question #2: Are any users consuming an average of OVER 70% of
provisioned upstream bandwidth for OVER 15 minutes?
Result #2: No action taken.
Result #3: Change user's upstream traffic from Priority Best Effort
(PBE) to Best Effort (BE).
Question #3: Is the user in Best Effort (BE) consuming an average
of LESS THAN 50% of provisioned upstream bandwidth?
Result #4: Change user's upstream traffic back to Priority Best
Effort (PBE) from Best Effort (BE).
Figure 2
5.3. Effect of BE Quality of Service on Users&apos Broadband Experience
When a CMTS port is in a Near Congestion State and a cable modem
connected to that port is in an Extended High Consumption State, that
cable modem's traffic is designated as BE. Depending upon the level
of congestion in the CMTS port, this designation may or may not
result in the user's traffic being delayed or, in extreme cases,
dropped before PBE traffic is dropped. This is because of the way
that the CMTS handles traffic. Specifically, CMTS ports have what is
commonly called a "scheduler" that puts all the packets coming from
or going to cable modems on that particular port in a queue and then
handles them in turn. A certain number of packets can be processed
by the scheduler in any given moment; for each time slot, PBE traffic
is given priority access to the available capacity, and BE traffic is
processed on a space-available basis. (It is important to note that
congestion can occur in any IP network, and, when it does, packets
can be delayed or dropped. As a result, applications and protocols
have been designed to deal with this reality. Our congestion
management systems attempts to ensure that, in those rare cases where
packets may be dropped, BE packets are dropped before PBE packets are
dropped.)
A rough analogy would be to busses that empty and fill up at
incredibly fast speeds. As empty busses arrive at the figurative
"bus stop" -- every two milliseconds in this case -- they fill up
with as many packets as are waiting for "seats" on the bus, to the
limits of the bus' capacity. During non-congested periods, the bus
will usually have several empty seats, but, during congested periods,
the bus will fill up and packets will have to wait for the next bus.
Bastian, et al. Expires December 28, 2009 [Page 17]
Internet-Draft An ISP Congestion Management System June 2009
It is in the congested periods that BE packets will be affected. If
there is no congestion, packets from a user in a BE state should have
little trouble getting on the bus when they arrive at the bus stop.
If, on the other hand, there is congestion in a particular instance,
the bus may become filled by packets in a PBE state before any BE
packets can get on. In that situation, the BE packets would have to
wait for the next bus that is not filled by PBE packets. In reality,
this all takes place in two-millisecond increments, so even if the
packets miss 50 "busses", the delay only will be about one-tenth of a
second.
During times of actual network congestion, when BE traffic might be
delayed, there are a variety of effects that could be experienced by
a user whose traffic is delayed, depending upon what applications he
or she is using. Typically, a user whose traffic is in a BE state
during actual congestion may find that a webpage loads sluggishly, a
peer-to-peer upload takes somewhat longer to complete, or a VoIP call
sounds choppy. Of course, the same thing could happen to the
customers on a port that is congested in the absence of any
congestion management; the difference here is that the effects of any
such delays are shifted toward those who have been placing the
greatest burden on the network, instead of being distributed randomly
among the users of that port without regard to their consumption
levels.
NetForecast, Inc. explored the potential risk of a worst-case
scenario for users whose traffic is in a BE state: the possibility of
"bandwidth starvation" in the theoretical case where 100 percent of
the CMTS bandwidth is taken up by PBE traffic for an extended period
of time. In theory, such a condition could mean that a given user
whose traffic is designated BE would be unable to effectuate an
upload or download (as noted above, both are managed separately) for
some period of time. However, when these management techniques were
tested, first in company testbeds and then in our real-world trials
conducted in the five markets, such a theoretical condition did not
occur. In addition, trial results demonstrated that these management
practices have very modest real-world impacts. In addition, Comcast
did not receive a single customer complaint in any of the trial
markets that could be traced to the new congestion management system,
despite having broadly publicized these trials.
Comcast continues to monitor how user traffic is affected by these
new congestion management techniques and will make the adjustments
necessary to ensure that all Comcast HSI customers have a high-
quality Internet experience.
Bastian, et al. Expires December 28, 2009 [Page 18]
Internet-Draft An ISP Congestion Management System June 2009
5.4. Equipment/Software Used and Location
The above-mentioned functions are carried out using three different
types of application servers, supplied by three different vendors.
As mentioned above, these servers are installed near Comcast's
regional network routers. The exact locations of various servers is
not particularly relevant to this document, as this does not change
the fact that they manage individual CMTS ports.
The first application server is an IPDR server, which collects
relevant cable modem volume usage information from the CMTS, such as
how many aggregate upstream or downstream bytes a subscriber uses
over a particular period of time. IPDR has been adopted as a
standard by many industry organizations and initiatives, such as
CableLabs, ATIS, ITU, and 3GPP, among others.
The second application server is the Congestion Management server,
which uses Simple Network Management Protocol ("SNMP") [RFC3410] to
measure CMTS port utilization and detect when a port is in a Near
Congestion State. When this happens, the Congestion Management
server then queries the relevant IPDR data for a list of cable modems
meeting the criteria set forth above for being in an Extended High
Consumption State.
If one or more users meet the criteria to be managed, then the
Congestion Management server notifies a third application server, the
PCMM application server, as to which users have been in an Extended
High Consumption State and whose traffic should be treated as BE.
The PCMM servers are responsible for signaling a given CMTS to set
the traffic for specific cable modems with a BE QoS, and for tracking
and managing the state of such CMTS actions. If no users meet the
criteria to be managed, no users will have their traffic managed.
Figure 3 graphically depicts the high-level management flows among
the congestion management components on Comcast's network, as
described above:
Figure 3: Simplified Diagram Showing High-Level Management Flows
Relevant to the System
+---------------+ +---------------+
| Congestion | Instruct QoS Server | QoS |
| Management |******to Change QoS for****>| Server |
| Server | a Device | |
+----+---+------+ +-------+-------+
/\ /\ *
| | *
X | Relay Selected *
Bastian, et al. Expires December 28, 2009 [Page 19]
Internet-Draft An ISP Congestion Management System June 2009
| +---Statistics: Bytes---+ *
X Up/Down by Device | QoS Action:
| | Change from PBE
X +-------+-------+ to BE, or from
| | Statistics | BE to PBE
X | Collection | *
Periodic SNMP | Server | *
Requests to +---------------+ *
Check CMTS Port /\ *
Utilization | *
Levels Statistics Sent *
X Periodically From CMTS *
| | *
X +-------+-------+ *
| | CMTS | *
+-X-X-X-X-X-X-X-X->| in |<************
| Headend |
+---------------+
H /\ /\ H
H Internet H
H Traffic H
H to/from H
H User H
H \/ \/ H
/-------------------\
/+-------------------+\
/ | User's Home | \
/ | | \
| +---------+ |
| | Cable | |
| | Modem | |
| +---------+ |
| |
+-------------------+
=================================================================
= Notes: =
= 1-Statistics Collection Servers use IP Detail Records (IPDR). =
= 2-QoS Servers use PacketCable Multimedia (PCMM) =
= to set QoS gates on the CMTS. =
= 3-This figure is a simplificiation of the actual network and =
= servers, which included redundancies and other complexities =
= not necessary to depict the functional design. =
=================================================================
Figure 3
Bastian, et al. Expires December 28, 2009 [Page 20]
Internet-Draft An ISP Congestion Management System June 2009
6. Conclusion
Comcast's transition to this new protocol-agnostic congestion
management system began in October 2008, and Comcast completed the
transition on December 31, 2008. As described above, the new
approach does not manage congestion by focusing on managing the use
of specific protocols. Nor will this approach use TCP "reset
packets" [RFC3360]. Rather, the new system acts such that (1) during
periods when a CMTS port is in a Near Congestion State, (2) it
identifies the subscribers on that port who have consumed a
disproportionate amount of bandwidth over the preceding 15 minutes,
(3) it lowers the priority status of those subscribers' traffic to BE
status until those subscribers meet the release criteria, and (4)
during periods of congestion, delay BE traffic before PBE traffic is
delayed. Our trials and our network-wide deployment indicate that
this new congestion management system ensures a quality online
experience for all of our HSI customers.
7. Exceptional Network Utilization Considerations
This system was developed to cope with somewhat "normal" occurrences
of congestion that could occur on virtually any IP network. It
should also be noted, however, that such a system could also prove
particularly useful in the case of "exceptional network utilization"
events which existing network usage models do not or cannot
accurately predict. For example, in the case of a severe global
pandemic, it may be expected that large swaths of the population may
need to work remotely, via their Internet connection. In such a
case, a largely unprecedented level of utilization may occur. In
such cases, it may be helpful to have a flexible congestion
management system that could adapt to this situation and help
allocate network resources while additional capacity is being brought
online or while a temporary condition persists.
8. Security Considerations
There are no security considerations in this document.
NOTE TO RFC EDITOR: PLEASE REMOVE THIS NULL SECTION PRIOR TO
PUBLICATION.
9. IANA Considerations
There are no IANA considerations in this document.
Bastian, et al. Expires December 28, 2009 [Page 21]
Internet-Draft An ISP Congestion Management System June 2009
NOTE TO RFC EDITOR: PLEASE REMOVE THIS NULL SECTION PRIOR TO
PUBLICATION.
10. Acknowledgements
The authors wish to acknowledge the hard work of the many people who
helped to develop and/or review this document, as well as the people
who helped deploy the system in such a short period of time.
11. Change Log
NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION.
o v00 2009-06-26 first version published
12. Open Issues
NOTE TO RFC EDITOR: PLEASE REMOVE THIS SECTION PRIOR TO PUBLICATION.
1. Review the Terminology section, add external references as needed
2. Need to add more stable URL references for FCC docs in
Informative Refs - change from Comcast file locations to FCC file
locations.
3. Change reference in text to draft-p2pi-cooper-workshop-report-01
to a normal xref once an RFC is issued
4. Make the ASCII art a bit smaller (vertically)
5. Double-check IPDR references - ensure most recent version of
standard is referenced (incl. possibly DOCSIS 3.0 reference w/r/t
IPDR)
6. May need to add references to the DOSCIS standard in Section 2.8
7. Close out any editorial notes in the document
13. Informative References
[CableLabs DOCSIS IPDR Support]
Yassini, R., "Data-Over-Cable Service Interface
Specifications - DOCSIS 2.0 - Operations Support System
Interface Specification", DOCSIS 2.0 CM-SP-OSSIv2.0-C01-
Bastian, et al. Expires December 28, 2009 [Page 22]
Internet-Draft An ISP Congestion Management System June 2009
081104, November 2008, <http://www.cablelabs.com/
specifications/CM-SP-OSSIv2.0-C01-081104.pdf>.
[Comcast IETF P2P Infrastructure Workshop Presentation]
Livingood, J. and R. Woundy, "Comcast's IETF P2P
Infrastructure Workshop Presentation on May 28, 2008", FCC
Filings Comcast Network Management Proceedings, May 2008,
<http://trac.tools.ietf.org/area/rai/trac/raw-attachment/
wiki/PeerToPeerInfrastructure/02-Comcast-IETF-P2Pi.pdf>.
[Comcast P2Pi Position Paper]
Livingood, J. and R. Woundy, "Comcast's IETF P2P
Infrastructure Workshop Position Paper", FCC
Filings Comcast Network Management Proceedings, May 2008,
<http://trac.tools.ietf.org/area/rai/trac/raw-attachment/
wiki/PeerToPeerInfrastructure/
16%20ietf-p2pi-comcast-20080509.pdf>.
[FCC Filing - Attachment A: Old Congestion Management System]
Zachem, K., "Attachment A: Comcast Corporation Description
of Current Network Management Practices", FCC
Filings Comcast Network Management Proceedings,
September 2008, <http://downloads.comcast.net/docs/
Attachment_A_Current_Practices.pdf>.
[FCC Filing - Attachment B: New Congestion Management System]
Zachem, K., "Attachment B: Comcast Corporation Description
of Planned Network Management Practices to be Deployed
Following the Termination of Current Practices", FCC
Filings Comcast Network Management Proceedings,
September 2008, <http://downloads.comcast.net/docs/
Attachment_B_Future_Practices.pdf>.
[FCC Filing - Attachment C: Implementation and Compliance Plan]
Zachem, K., "Attachment C: Comcast Corporation Network
Management Transition Compliance Plan", FCC
Filings Comcast Network Management Proceedings,
September 2008, <http://downloads.comcast.net/docs/
Attachment_C_Compliance_Plan.pdf>.
[FCC Filing - Congestion Management Deployment Completion Letter]
Zachem, K., "Letter to the FCC Advising of Successful
Deployment of Comcast's New Congestion Management System",
FCC Filings Comcast Network Management Proceedings,
January 2009, <http://downloads.comcast.net/docs/
comcast-nm-transition-notification.pdf>.
[FCC Filing - Cover Letter]
Bastian, et al. Expires December 28, 2009 [Page 23]
Internet-Draft An ISP Congestion Management System June 2009
Zachem, K., "Cover Letter to the FCC Regarding Comcast's
Network Management Techniques", FCC Filings Comcast
Network Management Proceedings, September 2008,
<http://downloads.comcast.net/docs/Cover_Letter.pdf>.
[FCC Memorandum and Opinion]
Martin, K., Copps, M., Adelstein, J., Tate, D., and R.
McDowell, "FCC Memorandum and Opinion Regarding Reasonable
Network Management", File No. EB-08-IH-1518 WC Docket No.
07-52, August 2008, <http://hraunfoss.fcc.gov/
edocs_public/attachmatch/FCC-08-183A1.pdf>.
[I-D.p2pi-cooper-workshop-report]
Peterson, J. and A. Cooper, "Report from the IETF workshop
on P2P Infrastructure, May 28, 2008",
draft-p2pi-cooper-workshop-report-01 (work in progress),
February 2009.
[IPDR Standard]
Cotton, S., Cockrell, B., Walls, P., and T. Givoly,
"Network Data Management - Usage (NDM-U) For IP-Based
Services Service Specification - Cable Labs DOCSIS 2.0
SAMIS", IPDR Service Specifications NDM-U, November 2004,
<http://www.ipdr.org/public/Service_Specifications/3.X/
DOCSIS(R)3.5-A.0.pdf>.
[RFC3083] Woundy, R., "Baseline Privacy Interface Management
Information Base for DOCSIS Compliant Cable Modems and
Cable Modem Termination Systems", RFC 3083, March 2001.
[RFC3360] Floyd, S., "Inappropriate TCP Resets Considered Harmful",
BCP 60, RFC 3360, August 2002.
[RFC3410] Case, J., Mundy, R., Partain, D., and B. Stewart,
"Introduction and Applicability Statements for Internet-
Standard Management Framework", RFC 3410, December 2002.
Bastian, et al. Expires December 28, 2009 [Page 24]
Internet-Draft An ISP Congestion Management System June 2009
Authors' Addresses
Chris Bastian
Comcast Cable Communications
One Comcast Center
1701 John F. Kennedy Boulevard
Philadelphia, PA 19103
US
Email: chris_bastian@cable.comcast.com
URI: http://www.comcast.com
Tom Klieber
Comcast Cable Communications
One Comcast Center
1800 Bishops Gate Drive
Mount Laurel, NJ 08054
US
Email: tom_klieber@cable.comcast.com
URI: http://www.comcast.com
Jason Livingood (editor)
Comcast Cable Communications
One Comcast Center
1701 John F. Kennedy Boulevard
Philadelphia, PA 19103
US
Email: jason_livingood@cable.comcast.com
URI: http://www.comcast.com
Jim Mills
Comcast Cable Communications
One Comcast Center
1800 Bishops Gate Drive
Mount Laurel, NJ 08054
US
Email: jim_mills@cable.comcast.com
URI: http://www.comcast.com
Bastian, et al. Expires December 28, 2009 [Page 25]
Internet-Draft An ISP Congestion Management System June 2009
Richard Woundy
Comcast Cable Communications
27 Industrial Avenue
Chelmsford, MA 01824
US
Email: richard_woundy@cable.comcast.com
URI: http://www.comcast.com
Bastian, et al. Expires December 28, 2009 [Page 26]
