Internet Engineering Task Force N. Kuhn, Ed.
Internet-Draft Telecom Bretagne
Intended status: Informational P. Natarajan, Ed.
Expires: August 11, 2014 Cisco Systems
D. Ros
Simula Research Laboratory
N. Khademi
University of Oslo
February 7, 2014
AQM Evaluation Guidelines
draft-kuhn-aqm-eval-guidelines-00
Abstract
Unmanaged large buffers in today's networks have given rise to a slew
of performance issues. These performance issues can be addressed by
some form of Active Queue Management (AQM), optionally in combination
with a packet scheduling scheme such as fair queuing. The IETF AQM
working group was recently formed to standardize AQM schemes that are
robust, easily implemented, and successfully deployed in today's
networks. This document describes various criteria for performing
precautionary evaluations of AQM proposals. This document also helps
in ascertaining whether any given AQM proposal should be taken up for
standardization by the AQM WG.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 11, 2014.
Kuhn, et al. Expires August 11, 2014 [Page 1]
Internet-Draft AQM Evaluation Guidelines February 2014
Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Guidelines for AQM designers . . . . . . . . . . . . . . 4
1.2. Reducing the latency and maximizing the goodput . . . . . 5
1.3. Organization of the document . . . . . . . . . . . . . . 5
1.4. Requirements Language . . . . . . . . . . . . . . . . . . 6
2. Metrics of interest . . . . . . . . . . . . . . . . . . . . . 6
2.1. Queue-related metrics . . . . . . . . . . . . . . . . . . 6
2.1.1. Link utilization . . . . . . . . . . . . . . . . . . 6
2.1.2. Queuing delay and queue size . . . . . . . . . . . . 7
2.1.3. Packet loss . . . . . . . . . . . . . . . . . . . . . 7
2.2. End-to-end Metrics . . . . . . . . . . . . . . . . . . . 8
2.2.1. Flow Completion time . . . . . . . . . . . . . . . . 8
2.2.2. Packet loss . . . . . . . . . . . . . . . . . . . . . 8
2.2.3. Goodput . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.4. Latency and jitter . . . . . . . . . . . . . . . . . 9
2.2.5. QoE metrics . . . . . . . . . . . . . . . . . . . . . 9
2.3. Discussion on the trade-off between latency and goodput . 10
3. Evaluation scenarios . . . . . . . . . . . . . . . . . . . . 10
3.1. Topology and notations . . . . . . . . . . . . . . . . . 11
3.2. Generic scenarios . . . . . . . . . . . . . . . . . . . . 13
3.2.1. Traffic Profiles . . . . . . . . . . . . . . . . . . 13
3.2.1.1. Topology Description . . . . . . . . . . . . . . 13
3.2.1.2. TCP-friendly Sender . . . . . . . . . . . . . . 13
3.2.1.3. Aggressive Transport Sender . . . . . . . . . . . 14
3.2.1.4. Unresponsive Transport Sender . . . . . . . . . . 14
3.2.1.5. Traffic Mix . . . . . . . . . . . . . . . . . . . 14
3.2.2. Burst absorption . . . . . . . . . . . . . . . . . . 15
3.2.2.1. Topology Description . . . . . . . . . . . . . . 16
3.2.2.2. Generic bursty traffic . . . . . . . . . . . . . 16
3.2.2.3. Realistic bursty traffic . . . . . . . . . . . . 16
3.2.3. Inter-RTT and intra-protocol fairness . . . . . . . . 17
Kuhn, et al. Expires August 11, 2014 [Page 2]
Internet-Draft AQM Evaluation Guidelines February 2014
3.2.4. Fluctuating network conditions . . . . . . . . . . . 18
3.2.4.1. Topology Description . . . . . . . . . . . . . . 18
3.2.4.2. Mild Congestion . . . . . . . . . . . . . . . . . 18
3.2.4.3. Medium Congestion . . . . . . . . . . . . . . . . 19
3.2.4.4. Heavy Congestion . . . . . . . . . . . . . . . . 19
3.2.4.5. Varying Available Bandwidth . . . . . . . . . . . 19
3.3. Diverse Network Environments . . . . . . . . . . . . . . 19
3.3.1. Medium bandwidth, medium delay: Wi-Fi . . . . . . . . 20
3.3.2. Low bandwidth, high delay: Rural broadband networks
and satellite links . . . . . . . . . . . . . . . . . 20
3.3.3. High bandwidth, low delay: data centers . . . . . . . 21
3.3.4. Low and high buffers . . . . . . . . . . . . . . . . 22
4. Deployment . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.1. Operator control knobs and auto-tuning . . . . . . . . . 22
4.2. Parameter sensitivity and stability analysis . . . . . . 23
4.3. Implementation cost . . . . . . . . . . . . . . . . . . . 24
4.4. Interaction with packet scheduling . . . . . . . . . . . 24
4.5. ECN behavior . . . . . . . . . . . . . . . . . . . . . . 25
4.6. Packet sizes and congestion notification . . . . . . . . 25
5. Comparing AQMs . . . . . . . . . . . . . . . . . . . . . . . 25
5.1. Performance comparison . . . . . . . . . . . . . . . . . 26
5.2. Deployment comparison . . . . . . . . . . . . . . . . . . 27
6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 27
7. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 27
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 27
9. Security Considerations . . . . . . . . . . . . . . . . . . . 27
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 27
10.1. Normative References . . . . . . . . . . . . . . . . . . 27
10.2. Informative References . . . . . . . . . . . . . . . . . 28
Appendix A. Additional Stuff . . . . . . . . . . . . . . . . . . 28
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 28
1. Introduction
Active Queue Management (AQM) addresses the concerns arising from
using unnecessarily large and unmanaged buffers in order to improve
network and application performance. Several AQM algorithms have
been proposed in the past years, most notable being Random Early
Detection (RED), BLUE, and Proportional Integral controller (PI). In
general, these algorithms actively interact with Transmission Control
Protocol (TCP) and other transport protocol that deploys a congestion
control scheme to manage the amount of data they keep in the network.
While the available buffer space in the routers and switches is
sufficiently enough to accommodate the short-term buffering
requirements, this has the effect of reducing mean buffer occupancy,
and therefore both end-to-end delay and jitter. Some of these
algorithms, notably RED, have also been widely implemented on network
devices. However, we haven't realized the benefits of RED AQM scheme
Kuhn, et al. Expires August 11, 2014 [Page 3]
Internet-Draft AQM Evaluation Guidelines February 2014
since it is reported to be usually turned off. The main reason of
this reluctance to use RED is that its parameters' sensitiveness to
the operating conditions in the network and the difficulty of tuning
them to realize some benefits in today's deployment.
In order to meet mostly throughput-based SLA requirements and to
avoid packet drops, many network providers resort to increasing the
available buffer space. This increase is also referred to as
Bufferbloat [BB2012]. Deploying large unmanaged buffers on the
Internet lead to the increase in end-to-end delay, resulting in poor
performance for latency sensitive applications such as real-time
multimedia (e.g., voice, video, gaming, etc.). The degree to which
this affects modern networking equipment, especially consumer-grade
equipment, produces problems even with commonly used web services.
Active queue management is thus essential to control queuing delay
and decrease network latency.
The AQM working group was recently formed within the TSV area to
address the problems with large unmanaged buffers in the Internet.
Specifically, the AQM WG is tasked with standardizing AQM schemes
that not only address concerns with such buffers, but also are robust
under wide variety of operating conditions. In order to ascertain
whether the WG should undertake standardizing an AQM proposal, the WG
requires guidelines for evaluating AQM proposals. This document
provides the necessary guidelines.
1.1. Guidelines for AQM designers
One of the key objectives behind formulating the guidelines is to
help ascertain whether a specific AQM is not only better than drop-
tail but also safe to deploy. Thus, the evaluation of AQM
performance can be divided into two categories: (1) the guidelines to
quantify AQM schemes' performance in terms of latency reduction,
goodput maximization and the trade-off between the two and (2) the
guidelines for safe deployment, including self adaptation, stability
analysis, fairness, design/implementation complexity and robustness
to different operating conditions.
This memo recognizes that an AQM scheme MAY NOT be suitable for all
possible network environments relevant to the IETF such as home
networks, data centers, enterprise edge etc. Therefore, this
document considers two different categories of evaluation scenarios:
(1) generic scenarios that any AQM proposal SHOULD be evaluated
against, and (2) evaluation scenarios specific to a network
environment. Irrespective of whether or not an AQM is standardized
by the WG, we recommend the relevant scenarios and metrics discussed
in this document to be considered. Since a specific AQM scheme MAY
NOT be applicable to all network environments, the specific
Kuhn, et al. Expires August 11, 2014 [Page 4]
Internet-Draft AQM Evaluation Guidelines February 2014
evaluation scenarios enable to establish the environments where the
AQM is applicable. These guidelines do not present every possible
scenario and cannot cover every possible aspect of a particular
algorithm. In addition, it is worth noting that the proposed
criteria are not bound to a particular evaluation toolset.
This document details how an AQM designer can rate the feasibility of
their proposal in different types of network devices, given the
various architecture possibilities (switches, routers, firewalls,
hosts, drivers, etc.) where an AQM may be implemented. To this end,
these guidelines state that an AQM's resource requirements SHOULD be
measured, considering which parts of the AQM run in real-time on the
data versus the components than run at higher levels or larger time-
scales.
1.2. Reducing the latency and maximizing the goodput
The trade-off between reducing the latency and maximizing the goodput
is intrinsically linked to each AQM scheme and is a central key to
evaluating its performance. This trade-off MUST be considered in
various scenarios to ensure the safety of an AQM deployment.
Whenever possible, solutions should aim at both maximizing goodput
and minimizing latency. This document proposes guidelines that
enable the reader to quantify (1) reduction of latency and (2)
maximization of goodput and (3) the trade-off between the two.
The tester SHOULD discuss the performance of its proposal in terms of
performance and deployment in comparison with those of drop-tail:
basically, these guidelines provide the tools to understand the cost
(in terms of deployment) versus the potential gain in performance of
the introduction of the proposed scheme.
1.3. Organization of the document
This memo is organized as follows:
o Section 2 defines the set of metrics that SHOULD be measured to
better evaluate the performance of an AQM scheme. All of them
SHOULD be considered regardless the topology, the traffic and the
goal.
o Section 3 presents a set of scenarios that COULD be considered to
evaluate the performance of an AQM scheme. One AQM algorithm may
not perform well for all network environments: this section helps
in determining environments where a specific AQM scheme is
applicable. The AQM performance for the whole set of scenarios
MAY not be evaluated, but for each selected scenario, the metrics
presented in Section 2 MUST be considered.
Kuhn, et al. Expires August 11, 2014 [Page 5]
Internet-Draft AQM Evaluation Guidelines February 2014
o Section 4 details deployment issues that MUST be discussed, such
as stability, implementation cost, implementation feasibility,
control knobs, etc.
o Section 5 presents how these guidelines can be used to fairly
compare various AQM schemes.
1.4. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
2. Metrics of interest
End-to-end delay is the result of propagation delay, serialization
delay, service delay in a switch, and queuing delay, summed over the
network elements in the path. Among those, only the queuing delay is
variable for a certain network path. AQM or scheduling algorithms
may reduce this delay by providing signals to the sender on the
emergence of congestion, but any impact on the goodput must be
carefully considered. This section presents the metrics that MUST be
used to better quantify (1) the reduction of latency, (2)
maximization of goodput and (3) the trade-off between the two. These
metrics MUST be considered to better assess the performance of an AQM
scheme.
2.1. Queue-related metrics
The queue-related metrics enable a better understanding of the AQM
behavior under tests and the impact of its internal parameters. This
section provides details on (1) the metrics that SHOULD be evaluated
and (2) how to represent them.
The metrics presented are the link utilization, the queuing delay and
the queue size in order to quantify the trade-off between goodput and
delay. Considering the fact that AQM schemes may drop packets, the
AQM tester SHOULD look carefully at the drops that the scheme
provokes.
2.1.1. Link utilization
The definition of the link utilization is given in the section 2.3.5
of RFC5136 [RFC5136]: "the utilization now represents the fraction of
the capacity that is being used and is a value between zero (meaning
nothing is used) and one (meaning the link is fully saturated)."
Kuhn, et al. Expires August 11, 2014 [Page 6]
Internet-Draft AQM Evaluation Guidelines February 2014
The link utilization is a metric that MUST be measured at the output
of the sending device and illustrates the link between queuing delay
and packet-dropping rates, which are key elements to understand the
internal behavior of the algorithm. The goodput metric for end-to-
end performance evaluation will be discussed in Section 2.2.3.
The guidelines advise that the tester SHOULD determine the minimum,
average and maximum measurements of the link utilization and the
coefficient of variation for the average value as well.
2.1.2. Queuing delay and queue size
The queuing delay is the time a packet waits in a queue until it can
be transmitted to the lower layers. The queue size is the number of
bytes which are occupying the queue.
Both queue size and queuing delay are needed because of fluctuating
link speeds. Moreover, AQM algorithm may be based on the length of
the queue (such as RED) or the queuing delay (such as CoDel or PIE).
The guidelines advice that the tester SHOULD determine the minimum,
average and maximum measurements of these metrics and the coefficient
of variation for the average values as well.
2.1.3. Packet loss
Losses can occur for various reasons. The losses under
considerations in this section are the losses that crop up in the
queue where AQM schemes take place.
Two classes of loss can be distinguished:
o AQM-induced losses: voluntary drops that are provoked by the AQM
scheme;
o buffer overflow: losses that occur when the buffer is full (so
called "tail drop").
For each of these cases, these guidelines advise one measure of:
o the long-term packet loss probability;
o the time interval between consecutive losses: for this metric, the
distribution and average value SHOULD be measured.
Kuhn, et al. Expires August 11, 2014 [Page 7]
Internet-Draft AQM Evaluation Guidelines February 2014
2.2. End-to-end Metrics
The impact of the introduction of AQM schemes on the end-to-end
performance MUST be evaluated: this section presents the metrics that
enable to evaluate the benefits provided by the tested AQM.
The metrics presented are the latency, the goodput, the packet loss
synchronization, the Quality of Experience (QoE) related metrics and
the fairness. The objective of these metrics is to quantify how much
the introduction of an AQM:
o reduces the latency;
o impacts on the goodput;
o impacts on the fairness between the flows.
2.2.1. Flow Completion time
The flow completion time is an important performance metric for the
end user. Considering the fact that an AQM scheme may drop packets,
the flow completion time is directly linked to its algorithm and this
is all the more true when the flows are short.
An AQM evaluation SHOULD measure the distribution of the flow
completion time.
2.2.2. Packet loss
The packet losses, that crop up in the queue where AQM schemes take
place, impact on the end-to-end performance at the receiver's side.
This metric may be already included in Section 2.1.3, however its
end-to-end aspect may ease the understanding of each proposal.
The tester MUST evaluate, at the receiver:
o the long term packet loss probability;
o the packet loss inter-arrival time;
o the packet loss pattern.
The guidelines advice that the tester SHOULD determine the minimum,
average and maximum measurements of these metrics and the coefficient
of variation for the average value as well.
Kuhn, et al. Expires August 11, 2014 [Page 8]
Internet-Draft AQM Evaluation Guidelines February 2014
2.2.3. Goodput
The goodput may be already included with latency measurements, but
measuring the goodput enable an end-to-end appreciation of how well
the AQM improves transport and application performance. The measured
end-to-end goodput is inversely proportional to the AQM scheme's
packet drops -- the smaller the packet drops, fewer packets need
retransmission, minimizing AQM's impact on transport and application
performance. Additionally, AQM scheme may resort to Explicit
Congestion Notification (ECN) marking as an initial means to control
delay. Again, marking packets instead of dropping them reduces
number of packet retransmissions and increases goodput. Overall,
end-to-end goodput values help evaluate the AQM scheme's
effectiveness in minimizing packet drops that impact application
performance and estimate how well the AQM scheme works with ECN.
If scheduling comes into play, a measure of how individual queues are
serviced may be necessary: the scheduling introduced on top of the
AQM may starve some flows and boost others. The utilization of the
link does not cover this, as the utilization would be the same,
whereas the goodput let the tester see if some flows are starved or
not.
The guidelines advice that the tester SHOULD determine the minimum,
average and maximum measurements of the goodput and the coefficient
of variation for the average value as well.
2.2.4. Latency and jitter
The end-to-end latency differs from the queuing delay: it is linked
to the network topology and the paths characteristics. Moreover, the
jitter strongly depends on the traffic and the topology as well. The
introduction of an AQM scheme would impact on these metrics and the
end-to-end evaluation of performance MUST consider them for a better
understanding.
The guidelines advice that the tester SHOULD determine the minimum,
average and maximum measurements for these metrics and the
coefficient of variation for their average values as well.
2.2.5. QoE metrics
An AQM evaluation study COULD measure the quality of experience of
end users for selected applications which are sensitive to latency,
such as video-streaming or VoIP. For these specific applications,
one SHOULD estimate the average Mean Opinion Score (MOS). Many AQM
proposals attempt to reduce the latency and these QoE metrics can
Kuhn, et al. Expires August 11, 2014 [Page 9]
Internet-Draft AQM Evaluation Guidelines February 2014
provide strong arguments for the developments of various AQM
solutions.
The evaluation of QoE SHOULD consider the end-to-end latency and
jitter detailed in Section 2.2.4.
2.3. Discussion on the trade-off between latency and goodput
The metrics presented in this section MUST be considered, in order to
discuss and quantify the trade-off between latency and medium
utilization wherever present.
This trade-off can also be illustrated with figures following the
recommendations of the section IV-B of [TCPEVAL2008]. For each
scenarios, the output SHOULD be four graphs:
o Queue related trade-off: queuing delay vs. link utilization: the
x-axis shows the average queuing delay and the y-axis the average
link utilization;
o Queue related trade-off: drop rate vs. queuing delay: the x-axis
shows the queuing delay and the y-axis the drop rate;
o End-to-end trade-off: end-to-end delay vs. goodput: the x-axis
shows the average end-to-end delay and the y-axis the average
goodput;
o End-to-end trade-off: drop rate vs. end-to-end delay: the x-axis
shows the end-to-end delay and the y-axis the drop rate.
Concerning the drop rate, the AQM tester can distinguish two classes
of drops, AQM-induced losses and buffer overflow, resulting in two
graphs for the 'drop rate vs. queuing delay' graph. Each of this
pair of graphs provide part of a better understanding (1) of the
delay/goodput/drop-rate trade-off for a given congestion control
mechanism, and (2) of how the goodput and average queue size vary as
a function of the traffic load.
3. Evaluation scenarios
This section presents the set of scenarios that COULD be considered
to evaluate the performance of AQM scheme: some scenarios MUST be
considered, whereas others MAY NOT. One AQM algorithm may not work
in all networking environments: this section helps in determining
environments where an AQM proposal is applicable. The performance
for the whole set of scenarios MAY not be evaluated, but for each
selected scenario, the metrics presented in Section 2 MUST be
considered. Each following subsections can be seen as a potential
Kuhn, et al. Expires August 11, 2014 [Page 10]
Internet-Draft AQM Evaluation Guidelines February 2014
working area for the tested AQM algorithm. The output of these
guidelines would be a list of competencies for each AQM, which will
let the AQM WG have clear criteria to compare the AQMs.
While presenting the performance of an AQM algorithm for the selected
scenarios, the tester MUST provide any parameter that had to be set
beforehand. Moreover, the values for these parameters MUST be
explained and justified as detailed in Section 4.2.
The tester SHOULD compare its proposal's performance and deployment
with those of drop-tail: basically, these guidelines provide the
tools to understand the cost (in terms of deployment) versus the
potential gain in performance of the introduction of the proposed
scheme.
This section is organized as follows:
o Section 3.1 presents the topology which is common to all the
scenarios.
o Section 3.2 presents the scenarios that MUST be considered in the
evaluation of one AQM proposal. This section will refer to
Section 3.1 in order to detail the content of each scenario.
o Section 3.3 explains scenarios that MAY be exploited to determine
the potential working area of an AQM proposal in specific
contexts, by referring to Section 3.1 to detail the topology
considered.
3.1. Topology and notations
This section presents the topology that can be used for each of the
following scenario and corresponding notations.
Kuhn, et al. Expires August 11, 2014 [Page 11]
Internet-Draft AQM Evaluation Guidelines February 2014
+--------------+ +--------------+
|senders A| | | |receivers B|
|---------+ | | +-----------|
| | | |
|--------------| |--------------|
|traffic class1|RTTA1.1, RTTR1.1,|traffic class1|
|--------------|CA1.1 CR1.1|--------------|
| SEN.Flow1.1 +---------+ +-----------+ REC.Flow1.1 |
| + | | | | + |
| + |RTTA1.X,| | RTTR1.X,| + |
| + |CA1.X | | CR1.X| + |
| SEN.Flow1.X +-----+ | | +--------+ REC.Flow1.X |
|--------------| | | | | |--------------|
| + | +-+---+---+ +--+--+---+ | + |
| | | |Central L| |Central R| | | |
| | | |---------|RTTLR|---------| | | |
| | | | AQM |CLR | | | | |
| | | | BuffSize+-----+ | | | |
| + | | (Bsize) | | | | + |
|--------------| +-----+--++ ++-+------+ |--------------|
|traffic classN|RTTAN.1,| | | | RTTRN.1,|traffic classN|
|--------------|CAN.1 | | | | CRN.1|--------------|
| SEN.FlowN.1 +---------+ | | +-----------+ REC.FlowN.1 |
| + | | | | + |
| + |RTTAN.Y, | | RTTRN.Y,| + |
| + |CAN.Y | | CRN.Y| + |
| SEN.FlowN.Y +------------+ +-------------+ REC.FlowN.Y |
+--------------+ +--------------+
Figure 1: Topology and notations
Figure 1 is a generic topology where:
o various class of traffic can be introduced;
o each class of traffic can consider a various number of flows;
o each link is characterized by a unique couple (RTT,capacity);
o Flows are generated between A and B, sharing a bottleneck (nodes L
and R);
o RTTAB.C (resp. CAB.C) is the RTT (resp. capacity) for a flow of
traffic class B, with the ID of C, from node A to L;
o The links are supposed to be symmetric, but notations can be
adapted if it is not the case.
Kuhn, et al. Expires August 11, 2014 [Page 12]
Internet-Draft AQM Evaluation Guidelines February 2014
The size of the buffers MUST be carefully, set considering the
bandwith-delay product.
3.2. Generic scenarios
The following scenarios are generic and MUST be considered whatever
the context is.
3.2.1. Traffic Profiles
Network and end devices need to be configured with reasonable amount
of buffers in order to absorb transient bursts. In some situations,
network providers configure devices with large buffers to avoid
packet drops and increase goodput. Transmission Control Protocol
(TCP) fills up these unmanaged buffers until the TCP sender receives
a signal (packet drop) to cut down the sending rate. The larger the
buffer, the higher the buffer occupancy, and therefore the queuing
delay. On the other hand, an efficient AQM scheme sends out early
congestion signals to TCP senders so that the queuing delay is
brought under control.
Not all applications run over the same flavor of TCP. Variety of
senders generate different traffic profiles at the networking device.
For example, there could be senders that either do not respond to
congestion signals (aka unresponsive flows) or do not cut down their
sending rate as expected (aka aggressive flows). An AQM scheme
should ensure queuing delay is under control irrespective of these
traffic profiles.
This document will evaluate an AQM proposal based on the metrics
presented in Section 2 irrespective of traffic profiles involved --
different senders (TCP variants, unresponsive, aggressive), traffic
mix with different applications, etc. Additionally, the AQM scheme
MUST NOT require operator tuning to work with varying traffic
profiles.
3.2.1.1. Topology Description
The topology is presented in Figure 1. For this scenario, the
capacities of the links MUST be set to 10Mbps and the RTTs to 100ms.
3.2.1.2. TCP-friendly Sender
This scenario helps evaluate how AQM scheme adapts to a TCP-friendly
transport sender. Single TCP New Reno flow between sender A and
receiver B, that transfers a large file for a period of 50s. Other
TCP friendly congestion control schemes such as TCP-friendly rate
control [RFC5348] etc MAY also be considered.
Kuhn, et al. Expires August 11, 2014 [Page 13]
Internet-Draft AQM Evaluation Guidelines February 2014
For each TCP-friendly transport considered, the graphs described in
Section 2.3 MUST be generated. We expect that an AQM proposal
exhibit similar behavior for all the TCP-friendly transports
considered.
3.2.1.3. Aggressive Transport Sender
This scenario helps evaluate how AQM scheme adapts to a transport
sender whose sending rate is more aggressive than a single TCP-
friendly sender. Single TCP Cubic flow between sender A and receiver
B, that transfers a large file for a period of 50s. Other congestion
control schemes such as (ref) MAY also be considered in-order to help
understand how the AQM scheme adapts to that particular aggressive
transport.
For each flavor of aggressive transport, the graphs described in
Section 2.3 MUST be generated.
3.2.1.4. Unresponsive Transport Sender
This scenario helps evaluate how AQM scheme adapts to a transport
sender who is not responsive to congestion signals (ECN marks and/or
packet drops) from the AQM scheme. In order to create a test
environment that results in queue build up, we consider unresponsive
flow(s) whose sending rate is greater than the bottleneck link
capacity between nodes L and R. Note that faulty transport
implementations on end hosts and/or faulty network elements en-route
that "hide" congestion signals in packet headers
[I-D.ietf-aqm-recommendation] may also lead to a similar situation,
such that the AQM scheme needs to adapt to unresponsive traffic.
This scenario consists of UDP flow(s) with an aggregate rate of
12Mbps between sender A and receiver B, that transfers a large file
for a period of 50s. Graphs described in Section 2.3 MUST be
generated.
3.2.1.5. Traffic Mix
This scenario helps evaluate how AQM scheme adapts to a traffic mix
consisting of different applications such as FTP, web, voice, video
traffic. These testing cases presented in this subsection have been
inspired by the table 2 of [DOCSIS2013]:
o Bulk TCP transfer: (continuous file transmission, or repeating 5MB
file transmission);
o Realistic HTTP web traffic (repeated download of 700kB);
Kuhn, et al. Expires August 11, 2014 [Page 14]
Internet-Draft AQM Evaluation Guidelines February 2014
o VoIP, Gaming (each of them 87kbps UDP stream);
o Constant bit rate UDP traffic (1Mbps UDP flow).
Figure 2 present the various cases for the traffic that MUST be
generated between sender A and receiver B.
+----+-------------------+--------+
|Case| Number of traffic |Comments|
+ +----+----+----+----+ +
| |VoIP|Webs|CBR |FTP | on FTP |
+----+----+----+----+----+--------+
| A | 1 | 1 | 0 | 0 | |
| | | | | | |
| B | 1 | 1 | 0 | 1 | cont. |
| | | | | | |
| C | 1 | 1 | 0 | 5 | repeat |
| | | | | | (5MB) |
| D | 1 | 1 | 1 | 5 | repeat |
| | | | | | (5Mb) |
+----+----+----+----+-------------+
Figure 2: Traffic Mix scenarios
For each of these scenarios, the graphs described in Section 2.3 MUST
be generated. In addition, other metrics such as end-to-end latency,
jitter, flow completion time, QoE MUST be generated.
3.2.2. Burst absorption
Packet arrivals can be bursty due to various reasons. Dropping one
or more packets from a burst may result in performance penalties for
the corresponding flows since the dropped packets have to be
retransmitted. Performance penalties may turn into unmet SLAs and be
disincentives to AQM adoption. Therefore, an AQM scheme SHOULD be
designed to accommodate transient bursts. AQM mechanisms do not
present the same tolerance to bursts of packets arriving in the
buffer: this tolerance MUST be quantified.
Note that accommodating bursts translates to higher queue length and
queuing delay. Naturally, it is important that the AQM scheme brings
bursty traffic under control quickly. On the other hand, spiking
packet drops inorder to bring packet bursts quickly under control
could result in multiple drops per flow and severely impact transport
and application performance. Therefore, an AQM scheme SHOULD bring
bursts under control by balancing both aspects -- (1) queuing delay
spikes are minimized and (2) performance penalties for ongoing flows
in terms of packet drops are minimized.
Kuhn, et al. Expires August 11, 2014 [Page 15]
Internet-Draft AQM Evaluation Guidelines February 2014
AQM maintain short queues to allow the remaining space in the queue
for bursts of packets. The tolerance to burst of packets depends on
the number of packets in the queue, which is directly linked to the
AQM policy. Moreover, one AQM scheme may implement a feature
controlling the maximum size of accepted bursts, which may be set by
the number of packets in the buffer, or the currently estimated
queuing delay. Also, the impact of the buffer size on the burst
allowance MAY be evaluated, detailed in Section 3.3.4.
3.2.2.1. Topology Description
The topology is presented in Figure 1. For this scenario, the
capacities of the links MUST be set to 10Mbps and the RTTs to 100ms.
3.2.2.2. Generic bursty traffic
The following traffic should be considered from sender A to receiver
B:
o One Constant bit rate UDP traffic: 1Mbps UDP flow;
o One TCP transfer: repeating 5MB file transmission;
o Burst of packets: size of the burst from 5 to MAX_BUFFER_SIZE
packets.
For each of these scenarios, the graphs described in Section 2.3 MUST
be generated. In addition, other metrics such as end-to-end latency,
jitter, flow completion time, QoE MUST be generated. Moreover, the
tester MUST provide the flow completion time, detailed in
Section 2.2.1, for each burst size considered.
3.2.2.3. Realistic bursty traffic
The following bursty traffic SHOULD be considered:
o IW10: TCP transfer with initial congestion window set to 10
(repeating 5MB file transmission);
o Bursty video frames (H.264/AVC) (60fps);
o HTTP web traffic (repeated download of 700kB);
o Constant bit rate UDP traffic (1Mbps UDP flow).
Figure 3 present the various cases for the traffic that MUST be
generated between sender A and receiver B.
Kuhn, et al. Expires August 11, 2014 [Page 16]
Internet-Draft AQM Evaluation Guidelines February 2014
+--------------------------------+
|Case| Number of traffic |
| +-----+----+----+-----------+
| |Video|Webs| CBR| FTP (IW10)|
+----|-----|----|----|-----------|
| A | 0 | 1 | 1 | 0 |
|----|-----|----|----|-----------|
| B | 0 | 1 | 1 | 1 |
|----|-----|----|----|-----------|
| C | 1 | 1 | 1 | 0 |
+----|-----|----|----|-----------|
| D | 1 | 1 | 1 | 0 |
+----|-----|----|----|-----------|
| E | 1 | 1 | 1 | 1 |
+----+-----+----+----+-----------+
Figure 3: Bursty traffic scenarios
For each of these scenarios, the graphs described in Section 2.3 MUST
be generated. In addition, other metrics such as end-to-end latency,
jitter, flow completion time, QoE MUST be generated.
3.2.3. Inter-RTT and intra-protocol fairness
TCP dynamics are a driving force for AQM design. It is therefore
important to evaluate against a set of RTT (e.g., from 5 ms to 200
ms). Also, asymmetry in terms of RTT between various paths SHOULD be
considered so that the fairness between the flows can be discussed as
one may react faster to congestion than another. The impact of the
scheduling and the AQM introduced on this lack of fairness SHOULD be
evaluated.
Moreover, introducing an AQM and/or scheduling schemes may result in
the absence of fairness between the flows, even when the RTTs are
identical. This potential lack of fairness SHOULD be evaluated.
The topology that SHOUD be exploited is the one of Figure 1:
o to evaluate the inter-RTT fairness, for each run, two flows
(Flow1.1 and Flow1.2) SHOULD be introduced and the set of RTT
SHOULD be: RTTA1.1 in [5ms;100ms] and RTTA1.2 in [100ms;200ms].
o to evaluate the impact of the RTT value on the AQM performance and
the intra-protocol fairness, for each run, two flows (Flow1.1 and
Flow1.2) SHOULD be introduced and the set of RTT SHOULD be:
RTTA1.1 in [5ms;200ms] and RTTA1.2 in [5ms;200ms], with
(RTTA1.1)=(RTTA1.2).
Kuhn, et al. Expires August 11, 2014 [Page 17]
Internet-Draft AQM Evaluation Guidelines February 2014
These flows MUST have the same congestion control algorithm.
The output that MUST be measured is the ratio between the average
goodput values of the two flows (Section 2.2.3) and the packet drop
rate for each flow (Section 2.2.2).
3.2.4. Fluctuating network conditions
Network devices experience varying operating conditions depending on
factors such as time of day, deployment scenario etc. For example:
o Traffic and congestion levels are higher during peak hours than
off-peak hours.
o A queue's draining rate could vary depending on other queues. A
low load on high priority queue implies higher draining rate for
lower priority queues.
If the target context is a stable environment, the tester MUST
illustrate their stability over time.
In context where the network conditions can vary over time, an AQM
scheme MUST be robust enough to control network latencies under
fluctuating network conditions, without the need for operator tuning
of AQM parameters. This document will evaluate AQM proposals under
varying congestion levels and varying draining rates.
3.2.4.1. Topology Description
The topology is presented in Figure 1. For this scenario, the
capacities of the links MUST be set to 10Mbps and the RTTs to 100ms.
3.2.4.2. Mild Congestion
This scenario helps evaluate how an AQM scheme adapts to a light load
of incoming traffic resulting in mild congestion -- packet drop rates
less than 1%. The scenario consists of 4-5 TCP New Reno flows between
sender A and receiver B. All TCP flows start at random times during
the initial second. Each TCP flow transfers a large file for a
period of 50s.
For this scenario, the graphs described in Section 2.3 MUST be
generated.
Kuhn, et al. Expires August 11, 2014 [Page 18]
Internet-Draft AQM Evaluation Guidelines February 2014
3.2.4.3. Medium Congestion
This scenario helps evaluate how an AQM scheme adapts to incoming
traffic resulting in medium congestion -- packet drop rates between
1%-3%. The scenario consists of 10-20 TCP New Reno flows between
sender A and receiver B. All TCP flows start at random times during
the initial second. Each TCP flow transfers a large file for a
period of 50s.
For this scenario, the graphs described in Section 2.3 MUST be
generated.
3.2.4.4. Heavy Congestion
This scenario helps evaluate how an AQM scheme adapts to incoming
traffic resulting in heavy congestion -- packet drop rates between
5%-10%. The scenario consists of 30-40 TCP New Reno flows between
sender A and receiver B. All TCP flows start at random times during
the initial second. Each TCP flow transfers a large file for a
period of 50s.
For this scenario, the graphs described in Section 2.3 MUST be
generated.
3.2.4.5. Varying Available Bandwidth
This scenario helps evaluate how an AQM scheme adapts to varying
available bandwidth on the outgoing link. To simulate varying
draining rates, the bottleneck bandwidth between nodes 'Central L'
and Central R' vary over the course of the experiment as follows --
100Mbps during 0-50s, 10Mbps during 50-100s, 100Mbps during 100-150s.
The scenario consists of 50 TCP New Reno flows between sender A and
receiver B. All TCP flows start at random times during the initial
second. Each TCP flow transfers a large file for a period of 150s.
In order to better assess the impact of draining rates on the AQM
behavior, the tester MUST compare its performance with those of tail-
drop.
For this scenario, the graphs described in Section 2.3 MUST be
generated. Moreover, one graph MUST be generated for each of the
three phases previously detailed.
3.3. Diverse Network Environments
This section presents scenarios with are related to specific network
environments. These classical network environments which COULD be
considered to evaluate the performance of the AQM under tests.
Kuhn, et al. Expires August 11, 2014 [Page 19]
Internet-Draft AQM Evaluation Guidelines February 2014
Each subsection presents a generic scenario and a use case. The
scenarios are classified according to a relation between the delay
(high, medium or low) and the capacity (high, medium, low). One
scenario details as well that the impact of the sizes of the buffers
should be evaluated. The guidelines selected those which are of
interest for the evaluation of the performance of AQM proposals. On
top of these abstracted scenario, these guidelines present use cases
for each selected scenario, by proposing a carefully dimensioned
topology.
3.3.1. Medium bandwidth, medium delay: Wi-Fi
This scenario is introduced to carefully evaluate AQM proposals in a
generic context, where the link between the delay and the bandwidth
is not specific and assess how AQM proposals can control latency in
this context.
We refer to Figure 1 to detail the topology:
o Sender A to Central L: capacity=100Mbps, RTT=10ms;
o Central L to Central R: capacity=20Mbps, RTT=10ms;
o Central R to Receiver B: capacity=100Mbps, RTT=10ms;
o The tester MAY include a packet loss rate of 1 to 3% on the Wi-Fi
link (between Central L and Central R).
The traffic that MUST generated between the sender A and the receiver
B is:
o Five repeating TCP transfers: repeating 5MB file transmission;
o One continuous TCP transfer: continuous file transmission;
o Four HTTP web traffic (repeated download of 700kB);
For this scenario, the graphs described in Section 2.3 MUST be
generated.
3.3.2. Low bandwidth, high delay: Rural broadband networks and
satellite links
In the context of low bandwith and high delay, the burst absorption
capacity of an AQM is seriously challenged. Indeed, due to the
important bandwith-delay product, the sending buffer should be large,
resulting in potentially large congestion windows and large bursts
arrivals to the gateways. The tolerance to large incoming bursts is
Kuhn, et al. Expires August 11, 2014 [Page 20]
Internet-Draft AQM Evaluation Guidelines February 2014
a key feature of an AQM introduced in this context: this is the
reason why this challenging context is detailed in these guidelines.
We refer to Figure 1 to detail the topology:
o Sender A to Central L: capacity=10Mbps, RTT=10ms;
o Central L to Central R: capacity=1Mbps, RTT=200ms;
o Central R to Receiver B: capacity=10Mbps, RTT=10ms;
The traffic that MUST generated between the sender A and the receiver
B is:
o Five repeating TCP transfers: repeating 5MB file transmission;
o One continuous TCP transfer: continuous file transmission;
o Four HTTP web traffic (repeated download of 700kB);
For this scenario, the graphs described in Section 2.3 MUST be
generated.
3.3.3. High bandwidth, low delay: data centers
In the context of high bandwith and low delay, the specific
characteristics require updated thresholds, which determine the
behavior of an AQM. As a result, the auto-tuning of an AQM is
seriously challenged. This is the reason why this challenging
context is detailed in these guidelines.
We refer to Figure 1 to detail the topology:
o Sender A to Central L: capacity=1Gbps, RTT=0.1ms;
o Central L to Central R: capacity=1Gbps, RTT=0.1ms;
o Central R to Receiver B: capacity=1Gbps, RTT=0.1ms;
The traffic that MUST generated between the sender A and the receiver
B is:
o Four repeating TCP transfers: repeating 5MB file transmission;
For this scenario, the graphs described in Section 2.3 MUST be
generated.
Kuhn, et al. Expires August 11, 2014 [Page 21]
Internet-Draft AQM Evaluation Guidelines February 2014
3.3.4. Low and high buffers
The size of the buffers impacts on AQMs performance, whether its
algorithm is based on the queue length or the queueing delay. The
tester MAY consider cases where the buffer is low (i.e. 1/10 BDP) and
when the buffer is large (i.e. 10 BDP).
We refer to Figure 1 to detail the topology:
o Sender A to Central L: capacity=100Mbps, RTT=10ms;
o Central L to Central R: capacity=20Mbps, RTT=10ms;
o Central R to Receiver B: capacity=100Mbps, RTT=10ms;
The traffic that MUST generated between the sender A and the receiver
B is:
o Five repeating TCP transfers: repeating 5MB file transmission;
o One continuous TCP transfer: continuous file transmission;
o Four HTTP web traffic (repeated download of 700kB);
For this scenario, the graphs described in Section 2.3 MUST be
generated. Moreover, these guidelines advise to plot the
characteristics of the queue (such as queue length or queuing delay)
as explained in Section 2.1.2.
4. Deployment
This section details deployment issues that MUST be discussed, such
as stability, implementation cost, implementation feasibility,
control knobs, etc.
4.1. Operator control knobs and auto-tuning
One of the biggest hurdles for RED deployment was/is its parameter
sensitivity to operating conditions -- how difficult it is to tune
important RED parameters for a deployment in order to get maximum
benefit from the RED implementation. Fluctuating congestion levels
and network conditions add to the complexity. Incorrect parameter
values lead to poor performance. Naturally, various network
operators felt it unwise to turn on the AQM (RED) implementation.
Any AQM scheme is likely to have parameters whose values affect the
AQM's control law and behavior. Exposing all these parameters as
control knobs to a network operator (or user) can easily result in an
Kuhn, et al. Expires August 11, 2014 [Page 22]
Internet-Draft AQM Evaluation Guidelines February 2014
unsafe AQM deployment. Unexpected AQM behavior ensues when parameter
values are not set properly. A minimal number of control knobs
minimizes the number of ways a, possible naive, user can break the
AQM system. Fewer control knobs make the AQM scheme more user-
friendly and easier to deploy and debug.
We recommend that an AQM scheme SHOULD minimize the control knobs
exposed for operator tuning. An AQM scheme SHOULD expose only those
knobs that control the "larger" AQM behavior such as queue delay
threshold, queue length threshold, etc.
Additionally, an AQM scheme's safety is directly related to its
stability under varying operating conditions such as varying traffic
profiles and fluctuating network conditions, as described in
Section 3.2.4 and in Section 3.2.1. Operating conditions vary often
and hence it is necessary that the AQM MUST remain stable under these
conditions without the need for additional external tuning. If AQM
parameters require tuning under these conditions, then the AQM MUST
self-adapt necessary parameter values by employing auto-tuning
techniques.
4.2. Parameter sensitivity and stability analysis
An AQM scheme's control law is the primary means by which the AQM
controls queuing delay. Hence understanding the AQM control law is
critical to understanding AQM behavior. The AQM's control law may
include several input parameters whose values affect the AQM output
behavior and stability. Additionally, AQM schemes may auto-tune
parameter values in-order to maintain stability under different
network conditions (such as different congestion levels, draining
rates or network environments). The stability of these auto-tuning
techniques is also important to understand.
AQM proposals SHOULD provide background material showing control
theoretic analysis of the AQM control law and the input parameter
space within which the control law operates as expected. For
parameters that are auto-tuned, the material SHOULD include stability
analysis of the auto-tuning mechanism(s) as well. Such analysis
helps the WG understand AQM control law better and the network
conditions/deployments under which the AQM is stable.
The impact of every externally tuned parameter MUST be discussed. As
an example, if an AQM proposal needs various external tuning to work
on different network environments presented in Section 3, these
external modifications MUST be clear for deployment issues. Also,
the frequency at which some parameters are re-configured MUST be
evaluated, as it may impact the capacity of the AQM to absorb
incoming bursts of packets.
Kuhn, et al. Expires August 11, 2014 [Page 23]
Internet-Draft AQM Evaluation Guidelines February 2014
4.3. Implementation cost
An AQM's successful deployment is directly related to its ease of
implementation. Network platforms may need hardware or software
implementations of the AQM. Depending on a platform's capabilities
and limitations, the platform may or may not be able to implement
some or all parts of the AQM logic.
AQM proposals SHOULD provide pseudo-code for the complete AQM scheme,
highlighting generic implementation-specific aspects of the scheme
such as "drop-tail" vs. "drop-head", inputs from platform (current
queueing delay, queue length), computations involved, need for timers
etc. This helps identify costs associated with implementing the AQM
on a particular hardware or software platform. Also, it helps WG
understand what kind of platforms can easily support the AQM and
which cannot.
AQM proposals SHOULD highlight parts of AQM logic that are platform
dependent and discuss if and how AQM behavior could be impacted by
the platform. For example, a queue-delay based AQM scheme requires
current queuing delay as input from the platform. If the platform
already maintains this value, then it is trivial to implement the AQM
logic on the platform. On the other hand, if the platform provides
indirect means to estimate queuing delay (ex: timestamps, deque rate
etc.), then the AQM behavior is sensitive to how good the queuing
delay estimate turns out on that platform. Highlighting the AQM's
sensitivity to queuing delay estimate helps implementers identify
optimal means of implementing the AQM on a platform.
4.4. Interaction with packet scheduling
On top of the introduction of an AQM scheme, a router may schedule
the transmission of packets in a specific manner by introducing a
scheduling scheme. This algorithm may create sub-queues and
integrate an AQM scheme on each of these sub-queues. Another
scheduling policy may modify the way packets are sequenced, modifying
the timestamp of each packet.
Both schedulers and AQMs can be introduced when packet are either
enqued or dequed. If both schedulers and AQM are implemented when
packet are enqued, their interaction should not be a major issue.
However, if one is introduced when packets are enqued and the others
when they are dequed, there may be destructive interactions.
The scheduling and the AQM schemes conjointly impact on the end-to-
end performance. During the evaluation process of an AQM proposal,
the tester MUST discuss the feasibility to add scheduling on top of
Kuhn, et al. Expires August 11, 2014 [Page 24]
Internet-Draft AQM Evaluation Guidelines February 2014
its algorithm. This discussion MAY detail if AQM is placed while
packets are enqued and dequed.
4.5. ECN behavior
Apart from packet drops, Explicit Congestion Notification (ECN) is an
alternative means to signal data senders about network congestion. A
network device explicitly marks specific bit(s) in packet headers to
convey congestion information to data senders. A data sender
implementing ECN treats the marked packet as if it were a packet drop
and reacts the same way as it would to a packet drop. Note that ECN
minimizes performance penalties, since packets do not have to be
retransmitted.
An AQM scheme SHOULD support ECN and SHOULD leverage ECN as an
initial means to control queuing delay before resorting to packet
drops. An AQM scheme SHOULD self-adapt and remain stable even with
faulty and/or unresponsive ECN implementations en-route.
4.6. Packet sizes and congestion notification
An AQM scheme may be considering packet sizes while generating
congestion signals. [I-D.ietf-tsvwg-byte-pkt-congest] discusses the
motivations behind the same. For example, control packets such as
DNS requests/responses, TCP SYNs/ACKs are small, and their loss can
severely impact application performance. An AQM scheme may therefore
be biased towards small packets by dropping them with smaller
probability compared to larger packets. However, such an AQM scheme
is unfair to data senders generating larger packets. Data senders,
malicious or otherwise, are motivated to take advantage of the AQM
scheme by transmitting smaller packets, and could result in unsafe
deployments and unhealthy transport and/or application designs.
An AQM scheme SHOULD adhere to recommendations outlined in
[I-D.ietf-tsvwg-byte-pkt-congest], and SHOULD NOT provide undue
advantage to flows with smaller packets.
5. Comparing AQMs
This memo recognizes that the guidelines mentioned above may be used
for comparing AQMs. This memo recommends that AQM schemes MUST be
compared against both performance (Section 3) and deployment
(Section 4) categories. In addition, this section details how best
to achieve a fair comparison of AQM schemes by avoiding certain
pitfalls.
Kuhn, et al. Expires August 11, 2014 [Page 25]
Internet-Draft AQM Evaluation Guidelines February 2014
5.1. Performance comparison
AQM schemes MUST be compared against all the generic scenarios
discussed in Section 3.2. AQM schemes MAY be compared for specific
network environments such as data center, home networks etc. For a
particular network environment, AQM schemes MUST be compared against
all the scenarios listed for that network environment Section 3.3.
For each evaluation scenario, the schemes MUST be compared against
the metrics discussed under that scenario. Moreover, if an AQM
scheme's parameter(s) were externally tuned for optimization or other
purposes, these values MUST be disclosed.
Note that AQM schemes belong to different varieties such as queue-
length based scheme (ex: RED) or queue-delay based scheme (ex: CoDel,
PIE). Also, AQM schemes expose different control knobs associated
with different semantics. For example, while both PIE and CoDel are
queue-delay based schemes and each expose a knob to control the
queueing delay -- PIE's "queueing delay reference" vs. CoDel's
"queueing delay target", the two schemes' knobs have different
semantics resulting in different control points. Such differences in
AQM schemes can be easily overlooked while making comparisons.
This document recommends the following procedures for a fair
performance comparison of two AQM schemes:
1. Comparable control parameters and comparable input values:
Carefully identify the set of parameters that control similar
behavior between the two AQM schemes and ensure these parameters
have comparable input values. For example, while comparing how
well a queue-length based AQM X controls queueing delay vs.
queue-delay based AQM Y, identify the two schemes' parameters
that control queue delay and ensure that their input values are
comparable. Similarly, to compare two AQMs on how well they
accommodate bursts, identify burst-related control parameters and
ensure they are configured with similar values.
2. Compare over a range of input configurations: There could be
situations when the set of control parameters that affect a
specific behavior have different semantics between the two AQM
schemes. As mentioned above, PIE's knob to control queue delay
has different semantics from CoDel's. In such situations, the
schemes MUST be compared over a range of input configurations.
For example, compare PIE vs. CoDel over the range of delay input
configurations -- 5ms, 10ms, 15ms etc.
Kuhn, et al. Expires August 11, 2014 [Page 26]
Internet-Draft AQM Evaluation Guidelines February 2014
5.2. Deployment comparison
AQMs MUST be compared against the deployment criteria discussed in
Section 4.
6. Acknowledgements
This work has been partially supported by the European Community
under its Seventh Framework Programme through the Reducing Internet
Transport Latency (RITE) project (ICT-317700).
7. Contributors
Many thanks to Gorry Fairhurst, Amadou B. Bagayoko, Chamil Kulatunga
Michael Welzl, Fred Baker, Rong Pan and David Collier-Brown for
detailed and wise feedback on this document.
8. IANA Considerations
This memo includes no request to IANA.
9. Security Considerations
All drafts are required to have a security considerations section.
10. References
10.1. Normative References
[I-D.ietf-aqm-recommendation]
Baker, F. and G. Fairhurst, "IETF Recommendations
Regarding Active Queue Management", draft-ietf-aqm-
recommendation-01 (work in progress), January 2014.
[I-D.ietf-tsvwg-byte-pkt-congest]
Briscoe, B. and J. Manner, "Byte and Packet Congestion
Notification", draft-ietf-tsvwg-byte-pkt-congest-12 (work
in progress), November 2013.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", RFC 2119, 1997.
[RFC5136] Chimento, P. and J. Ishac, "Defining Network Capacity",
RFC 5136, 2008.
Kuhn, et al. Expires August 11, 2014 [Page 27]
Internet-Draft AQM Evaluation Guidelines February 2014
10.2. Informative References
[BB2012] CACM Staff, , "BufferBloat: what's wrong with the
internet?", Commun. ACM vol. 55, 2008.
[DOCSIS2013]
White, G. and D. Rice, "Active Queue Management Algorithms
for DOCSIS 3.0", Technical repport - Cable Television
Laboratories , 2013.
[RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP
Friendly Rate Control (TFRC): Protocol Specification", RFC
5348, September 2008.
[TCPEVAL2008]
Andrew, L., Marcondes, C., Floyd, S., Dunn, L., Guillier,
R., Gang, W., Eggert, L., Ha, S., and I. Rhee, "Towards a
common TCP evaluation suite", PFLDnet 6th, 2008.
Appendix A. Additional Stuff
This becomes an Appendix.
Authors' Addresses
Nicolas Kuhn (editor)
Telecom Bretagne
2 rue de la Chataigneraie
Cesson-Sevigne 35510
France
Phone: +33 2 99 12 70 46
Email: nicolas.kuhn@telecom-bretagne.eu
Preethi Natarajan (editor)
Cisco Systems
510 McCarthy Blvd
Milpitas, California
United States
Email: prenatar@cisco.com
David Ros
Simula Research Laboratory
Email: dros@simula.no
Kuhn, et al. Expires August 11, 2014 [Page 28]
Internet-Draft AQM Evaluation Guidelines February 2014
Naeem Khademi
University of Oslo
Department of Informatics, PO Box 1080 Blindern
N-0316 Oslo
Norway
Phone: +47 2285 24 93
Email: naeemk@ifi.uio.no
Kuhn, et al. Expires August 11, 2014 [Page 29]