Test Cases for Evaluating RMCAT Proposals
draft-sarker-rmcat-eval-test-00
The information below is for an old version of the document.
| Document | Type |
This is an older version of an Internet-Draft whose latest revision state is "Replaced".
|
|
|---|---|---|---|
| Authors | Zaheduzzaman Sarker , Varun Singh , Xiaoqing Zhu , Michael A. Ramalho | ||
| Last updated | 2014-02-14 | ||
| Replaced by | draft-ietf-rmcat-eval-test, RFC 8867 | ||
| RFC stream | (None) | ||
| Formats | |||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | I-D Exists | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-sarker-rmcat-eval-test-00
Network Working Group Z. Sarker
Internet-Draft Ericsson AB
Intended status: Informational V. Singh
Expires: August 18, 2014 Aalto University
X. Zhu
M. A. Ramalho
Cisco Systems
February 14, 2014
Test Cases for Evaluating RMCAT Proposals
draft-sarker-rmcat-eval-test-00
Abstract
The Real-time Transport Protocol (RTP) is used to transmit media in
multimedia telephony applications, these applications are typically
required to implement congestion control. The RMCAT working group is
currently working on candidate algorithms for such interactive real-
time multimedia applications. This document describes the test cases
to be used in the evaluation of the performance of those candidate
algorithms.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 18, 2014.
Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
Sarker, et al. Expires August 18, 2014 [Page 1]
Internet-Draft Test Scenarios for RMCAT February 2014
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
3. Basic Structure of Test cases . . . . . . . . . . . . . . . . 3
4. Basic Test Cases . . . . . . . . . . . . . . . . . . . . . . 7
4.1. Variable Available Capacity . . . . . . . . . . . . . . . 7
4.2. Maximum Media Bit Rate is Greater than Link Capacity . . 10
4.3. Competing Flows with same RMCAT Algorithm . . . . . . . . 13
4.4. RMCAT Flow competing with a long TCP Flow . . . . . . . . 16
4.5. RMCAT Flow competing with short TCP Flows . . . . . . . . 19
4.6. Congested Feedback Link . . . . . . . . . . . . . . . . . 22
4.7. Round Trip Time Fairness . . . . . . . . . . . . . . . . 27
4.8. Media Pause and Resume . . . . . . . . . . . . . . . . . 29
4.9. Explicit Congestion Notification Usage . . . . . . . . . 31
5. Wireless Access Links . . . . . . . . . . . . . . . . . . . . 31
5.1. Cellular Network Specific Test Cases . . . . . . . . . . 31
5.2. Wi-Fi Network Specific Test Cases . . . . . . . . . . . . 31
6. Security Considerations . . . . . . . . . . . . . . . . . . . 31
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31
8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 31
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 32
9.1. Normative References . . . . . . . . . . . . . . . . . . 32
9.2. Informative References . . . . . . . . . . . . . . . . . 33
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 33
1. Introduction
This memo describes a set of test cases for evaluating candidate
RMCAT congestion control algorithm proposals, it is based on the
guidelines enumerated in [I-D.ietf-rmcat-eval-criteria] and
requirements discussed in [I-D.ietf-rmcat-cc-requirements]. The test
cases cover basic usage scenarios and are described using a common
structure, which allows for additional test cases to be added to
those described herein to accommodate other topologies and/or the
modeling of different link characteristics. Each test case
incorporates the metrics, evaluation guidelines and parameters
described in [I-D.ietf-rmcat-eval-criteria]. It is the intention of
this work to capture the consensus of the RMCAT working group
participants regarding the test cases upon which the performance of
the candidate RMCAT proposals should be evaluated.
Sarker, et al. Expires August 18, 2014 [Page 2]
Internet-Draft Test Scenarios for RMCAT February 2014
2. Terminology
The terminology defined in RTP [RFC3550], RTP Profile for Audio and
Video Conferences with Minimal Control [RFC3551], RTCP Extended
Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback
(RTP/AVPF) [RFC4585], Support for Reduced-Size RTCP [RFC5506], and
RTP Circuit Breaker algorithm [I-D.ietf-avtcore-rtp-circuit-breakers]
apply.
3. Basic Structure of Test cases
All test cases in this document follow a basic structure allowing
implementers to describe a new test scenarios without repeatedly
explaining common attributes. The structure includes a general
description section that describes the test case and motivations,
additionally it defines a set of attributes that characterize the
testbed, i.e., the network path between communicating peers and the
diverse traffic sources.
o Define the test case:
* General description: describes the motivation and the goals of
the test case.
* Additionally, describe the desired rate adaptation behaviour.
* Define a checklist to evaluate the desired behaviour: this
indicates the minimum set of metrics (e.g., link utiilization,
media sending rate) that a proposed algorithm needs to measure
to validate the expected rate adaptation behaviour. It should
also indicate the time granularity (e.g., averaged over 10ms,
100ms, or 1s) for measuring certain metrics. Typical
measurement interval is 200ms.
o Define testbed topology: every test case needs to define an
evaluation testbed topology. Figure 1 shows such an evaluation
topology. In this evaluation topology, S1..Sn are traffic
sources, these sources either generate media traffic and use an
RMCAT candidate congestion control algorithm or use a different
congestion control algorithm than that one used in the media
source. R1..Rn are the corresponding receivers. A test case can
have one or more such traffic source (S) and corresponding
receiver (R). The path from the source to destination is denoted
as upstream and the path from a destination to a source is denoted
as downstream. It is possible for a testbed to have different
path characteristics in either directions. The following basic
structure of test case has been described from the perspective of
endpoints attached on the left-hand side of Figure 1.
Sarker, et al. Expires August 18, 2014 [Page 3]
Internet-Draft Test Scenarios for RMCAT February 2014
o
+---+ +---+
|S1 |====== \ UPSTREAM --> / =======|R1 |
+---+ \\ // +---+
\\ //
+---+ +-----+ +-----+ +---+
|S2 |=======| A |------------------------------>| B |=======|R2 |
+---+ | |<------------------------------| | +---+
+-----+ +-----+
(...) // \\ (...)
// <-- DOWNSTREAM \\
+---+ // \\ +---+
|Sn |====== / \ ======|Rn |
+---+ +---+
Figure 1: Example of A Testbed Topology
o In a laboratory testbed environment there exists large amount of
traffic on the network path between the endpoints, which may be
contributed by other devices connected to such a testbed. It is
recommended to not route non-test traffic through the testbed.
o Define testbed attributes:
* Duration: defines the duration of the test.
* Path characteristics: defines the end-to-end transport level
path characteristics of the testbed in a particular test case.
Two sets of attributes describes the path characteristics, one
for the upstream path and the other for the downstream path.
The path characteristics for a particular path direction is
applicable to all the Sources "S" sending traffic on that path.
If only one attribute is specified, it is used for both path
directions.
+ Path direction: upstream or downstream.
+ Bottleneck-link capacity: defines minimum capacity of the
end-to-end path
+ One-way propagation delay: describes the end-to-end latency
along the path when network queues are empty i.e., the time
it takes for a packet to go from the sender to the receiver,
it does not include any queuing delay.
+ Maximum end-to-end jitter: defines maximum jitter can be
observed along the path.
Sarker, et al. Expires August 18, 2014 [Page 4]
Internet-Draft Test Scenarios for RMCAT February 2014
+ Bottleneck queue type: for example, Droptail, FQ-CoDel, or
PIE.
+ Bottleneck queue size: defines size of queue in terms of
queuing time when the queue is full (in milliseconds).
+ Path loss ratio: characterize the non-congested losses
observed on the end-to-end path. MUST describe the loss
pattern or loss model.
* Application-related: defines the media-related behaviour for
implementing the test case
+ Media Source: defines the characteristics of the media
sources present. When using more than one media source, the
different attributes are enumerated separately for each
different media source.
- Media type: Video/Voice/Application/Text
- Media flow direction: upstream, downstream or both.
- Number of media sources: defines the total number of
media sources
- Media codec: Constant Bit Rate (CBR) or Variable Bit Rate
(VBR)
- Media source behaviour: describes the media encoder
behavior. It MUST define the main parameters than affect
the adaptation behaviour. This may include but not
limited to:
o Adaptability: describes the adaptation options. For
example, in case of video it defines the range of
bitrate adaptation, frame rate adaption, video
resolution adaptation. In case of voice, it defines
the range of bitrate adaptation, sampling rate, frame
size.
o Output variation : for VBR encoder, it defines the
encoder output variation from the average target rate.
For example on average the encoder output may vary
between 5% to 15% above or below the average target
bit rate.
o Encoder's responsiveness to a new bit rate request:
value typically between 10ms to 1000ms.
Sarker, et al. Expires August 18, 2014 [Page 5]
Internet-Draft Test Scenarios for RMCAT February 2014
- Media content: describes the chosen media sequences; For
example, test sequences are available at: [xiph-seq]
[HEVC-seq].
- Media timeline: describes the point when the media source
is introduced and removed from the testbed. For example,
the media source may begin transmitting when the test
case begins or a few seconds after, etc.
- Startup behaviour: the media starts at a defined bit
rate, which may be the minimum, maximum bit rate, or a
value in between (in Kbps).
+ Competing traffic source: describes the characteristics of
the competing traffic source, the different types of
competing flows are enumerated in
[I-D.ietf-rmcat-eval-criteria].
- Traffic direction: Upstream, downstream or both.
- Type of sources: defines the types of competing traffic
sources. Types of competing traffic flows are listed in
[I-D.ietf-rmcat-eval-criteria]. For example, the number
of TCP flows connected to a web browser, the mean size
and distribution of the content downloaded.
- Number of sources: defines the total number of competing
sources of each type.
- Congestion control: enumerate the congestion control used
by each type of competing traffic.
- Traffic timeline: describes when the competing traffic is
added and removed from the test case.
* Additional attributes: describes attributes essential for
implementing a test case which are not included in the above
structure. These attributes MUST be well defined, so that
other implementers are able to implement it.
Any attribute can have a set of values (enclosed within "[]"). Each
member value of such a set MUST be treated as different value for the
same attribute. It is desired to run separate tests for each such
attribute value.
The test cases described in this document follow the above structure.
Sarker, et al. Expires August 18, 2014 [Page 6]
Internet-Draft Test Scenarios for RMCAT February 2014
4. Basic Test Cases
4.1. Variable Available Capacity
In this test case the bottleneck-link capacity between the two
endpoints varies over time. This test is designed to measure the
responsiveness of the candidate algorithm. This test tries to
address the requirement 1(a) in [I-D.ietf-rmcat-cc-requirements],
which requires the algorithm to adapt the flow(s) and provide lower
end-to-end latency when there exists:
o an intermediate bottleneck
o change in available capacity due to interface change and/or
routing change.
o persistent network load due to competing traffic
It should be noted that the exact variation in available capacity due
to any of the above depends on the under-lying technologies. Hence,
we describe a set of known factors, which may be extended to devise a
more specific test case targeting certain behaviour in a certain
network environment.
Expected behavior: the candidate algorithms is expected to detect the
variation in available capacity and adapt the media stream(s)
accordingly. The flows stabilize around their maximum bitrate as the
as the maximum link capacity is large enough to accommodate the
flows. When the available capacity drops, the flow(s) adapts by
decreasing its sending bit rate, and when congestion disappears, the
flow(s) are again expected to ramp up.
To evaluate the performance of the candidate algorithms it is
expected to log enough information to visualize the following metrics
at a fine enough time granularity:
1. Flow level:
a. End-to-end delay
b. Losses observed at the receiving endpoint
c. Feedback message overhead
2. Transport level:
a. Bandwidth utilization
Sarker, et al. Expires August 18, 2014 [Page 7]
Internet-Draft Test Scenarios for RMCAT February 2014
b. Queue length (ms):
+ average over the length of the session
+ 5 and 95 percentile
+ median, maximum, minimum
Testbed Topology: Two (2) media sources S1 and S2 are connected to
their corresponding R1 and R2. The media traffic is transported over
the upstream path and corresponding feedback/control traffic is
transported over the downstream path.
+---+ +---+
|S1 |===== \ UPSTREAM --> / =======|R1 |
+---+ \\ // +---+
\\ //
+-----+ +-----+
| A |------------------------------>| B |
| |<------------------------------| |
+-----+ +-----+
// \\
// <-- DOWNSTREAM \\
+---+ // \\ +---+
|S2 |====== / \ ======|R2 |
+---+ +---+
Figure 2: Testbed Topology for Variable Available Capacity
Testbed attributes:
o Test duration: 120s
o Path characteristics:
* Path direction: Upstream and downstream.
* Bottleneck-link capacity: 4Mbps.
* One-Way propagation delay: [10ms, 100ms].
* Maximum end-to-end jitter: 30ms.
* Bottleneck queue type: Drop tail.
* Bottleneck queue size: 300ms.
* path loss ratio: 0%.
Sarker, et al. Expires August 18, 2014 [Page 8]
Internet-Draft Test Scenarios for RMCAT February 2014
o Application-related:
* Media Source:
+ Media type: Video
+ Media direction: Upstream.
+ Number of media sources: Two (2).
+ Media codec: VBR
+ Media source behaviour:
- Adaptability:
o Bit rate range: 150 Kbps - 1.5 Mbps
o Frame Resolution: 144p - 720p (or 1080p)
o Frame rate: 10fps - 30fps
- Variation from target bitrate: +/-5%
- Responsiveness to new bit rate request: 100ms
+ Media content: Foreman media sequence.
+ Media timeline:
- Start time: 0s.
- End time: 119s.
* Media startup behaviour: [200Kbps, 1500Kbps].
* Competing traffic:
+ Number of sources : Zero (0)
o Test specific setup:
* Number of bandwidth variations: Three (4)
* Variation pattern:
+ Sequence number: 1
Sarker, et al. Expires August 18, 2014 [Page 9]
Internet-Draft Test Scenarios for RMCAT February 2014
+ Path direction: Upstream
+ Bottleneck Capacity: 2Mbps.
+ Start time: 30s
* Variation pattern:
+ Sequence number: 2
+ Path direction: Upstream
+ Bottleneck Capacity: 3.5Mbps
+ Start time: 50s
* Variation pattern:
+ Sequence number: 3
+ Path direction: Upstream
+ Bottleneck Capacity: 1Mbps
+ Start time: 70s
* Variation pattern:
+ Sequence number: 4
+ Path direction: Upstream
+ Bottleneck Capacity: 2Mbps
+ Start time: 100s
4.2. Maximum Media Bit Rate is Greater than Link Capacity
In this case, the application will attempt to ramp up to its maximum
bit rate, since the link capacity is limited to a value lower, the
congestion control scheme is expected to stabilize the sending bit
rate close to the available bottleneck capacity. This situation can
occur when the endpoints are connected via thin long networks even
though the advertised capacity of the access network may be higher.
The test case addresses the requirement 1 and 10 of the
[I-D.ietf-rmcat-cc-requirements].
Sarker, et al. Expires August 18, 2014 [Page 10]
Internet-Draft Test Scenarios for RMCAT February 2014
Expected behavior: the candidate algorithm is expected to detect the
path capacity constraint, converges to bottleneck link's capacity and
adapt the flow to avoid unwanted oscillation when the sending bit
rate is approaching the bottleneck link's capacity. The oscillations
occur when the media flow(s) attempts to reach its maximum bit rate,
overshoots the usage of the available bottleneck capacity, to rectify
it reduces the bit rate and starts to ramp up again.
Testbed topology: One media source S1 is connected to corresponding
R1. The media traffic is transported over the upstream path and
corresponding feedback/control traffic is transported over the
downstream path.
UPSTREAM -->
+---+ +-----+ +-----+ +---+
|S1 |=======| A |------------------------------>| B |=======|R1 |
+---+ | |<------------------------------| | +---+
+-----+ +-----+
<-- DOWNSTREAM
Figure 3: Testbed Topology for Limited Link Capacity
To evaluate the performance of the candidate algorithms it is
expected to log enough information to visualize the following metrics
at a fine enough time granularity:
1. Flow level:
a. End-to-end delay.
b. RTP packet losses observed at the receiving endpoint.
c. Variation in sending bit rate and goodput. Mainly observing
the frequency and magnitude of oscillations.
d. Convergence time.
e. Feedback message overhead.
2. Transport level:
a. Bandwidth utilization.
b. Queue length (ms):
+ average over the length of the session
+ 5 and 95 percentile
Sarker, et al. Expires August 18, 2014 [Page 11]
Internet-Draft Test Scenarios for RMCAT February 2014
+ median
Testbed attributes:
o Test duration: 60s
o Path characteristics:
* Path direction: Upstream and downstream.
* Bottleneck-link capacity: 1Mbps
* One-Way propagation delay: [10ms,100 ms]
* Maximum end-to-end jitter: 30ms.
* Bottleneck queue type: Droptail. Additional tests with other
AQM schemes are recommended: FQ-CoDel, PIE
* Bottleneck size: 300ms
* Path loss ratio: 0%
o Application-related:
* Media Source:
+ Media type: Video
+ Media direction: Upstream.
+ Number of media sources: One (1)
+ Media codec: VBR
+ Media source behaviour:
- Adaptability:
o Bit rate range: 150 Kbps - 1.5 Mbps
o Frame Resolution: 144p - 720p (or 1080p)
o Frame rate: 10fps - 30fps
- Variation from target bitrate: +/-5%
- Responsiveness to new bit rate request: 100ms
Sarker, et al. Expires August 18, 2014 [Page 12]
Internet-Draft Test Scenarios for RMCAT February 2014
+ Media content: Foreman video sequence
+ Media timeline:
- Start time: 0s.
- End time: 59s.
* Media startup behaviour: [200Kbps, 1500Kbps].
* Competing traffic:
+ Number of sources : Zero (0)
o Test specific setup: None
4.3. Competing Flows with same RMCAT Algorithm
In this test case, more than one RMCAT media flow shares the
bottleneck link and each of them uses the same congestion control
algorithm. This is a typical scenario wherein a real-time
interactive application sends more than one media flows to the same
destination and these flows are multiplexed over the same port. In
such a scenario it is likely that the flows will be routed via the
same path and need to share the available bandwidth amongst
themselves. For the sake of simplicity it is assumed that there are
no other competing traffic sources in the bottleneck link and that
there is sufficient capacity to accommodate all the flows
individually. While this appears to be a variant of the test case
defined in Section 4.1 , however it tests the capacity sharing
distribution of the candidate algorithm. Whereas, the previous test
case measures the stability and responsiveness of the candidate
algorithm. This test case particularly addresses the requirements
2,3 and 10 in [I-D.ietf-rmcat-cc-requirements].
Expected behavior: It is expected that the competing flows will
converge to an optimum bit rate to accommodate all the flows with
minimum possible latency and loss. Specifically, the test introduces
three media flows at different time instances, when the second flow
appears there should still be room to accommodate another flow on the
bottleneck link. Lastly, when the third flow appears the bottleneck
link should be saturated.
To evaluate the performance of the candidate algorithms it is
expected to log enough information to visualize the following metrics
metrics at a fine enough time granularity:
1. Flow level:
Sarker, et al. Expires August 18, 2014 [Page 13]
Internet-Draft Test Scenarios for RMCAT February 2014
a. End-to-end delay.
b. RTP packet losses observed at the receiving endpoint.
c. Variation in sending bit rate and goodput. Mainly observing
the frequency and magnitude of oscillations.
d. Convergence time.
e. Feedback message overhead.
2. Transport level:
a. Bandwidth utilization.
b. Queue length (ms):
+ average over the length of the session
+ 5 and 95 percentile
+ median
Testbed topology: Three media sources S1, S2, S3 are connected to
respective R1, R2, R3. The media traffic is transported over the
upstream path and corresponding feedback/control traffic is
transported over the downstream path.
+---+ +---+
|S1 |===== \ UPSTREAM --> / =======|R1 |
+---+ \\ // +---+
\\ //
+---+ +-----+ +-----+ +---+
|S2 |=======| A |------------------------------>| B |=======|R2 |
+---+ | |<------------------------------| | +---+
+-----+ +-----+
// \\
// <-- DOWNSTREAM \\
+---+ // \\ +---+
|S3 |====== / \ ======|R3 |
+---+ +---+
Figure 4: Testbed Topology for Multiple RMCAT Flows
Testbed attributes:
o Test duration: 60s
Sarker, et al. Expires August 18, 2014 [Page 14]
Internet-Draft Test Scenarios for RMCAT February 2014
o Path characteristics:
* Path direction: Upstream, Downstream
* Bottleneck-link capacity: 3.5Mbps
* One-Way propagation delay: [10ms, 50ms]
* Maximum end to end jitter: 30ms
* Bottleneck queue type: Droptail
* Bottleneck queue size: 300ms
* Link loss ratio: 0.0%
o Application-related:
* Media Source:
+ Media type: Video
+ Media direction: Upstream
+ Number of media sources: Three (3)
+ Media codec: VBR
+ Media source behaviour:
- Adaptability:
o Bit rate range: 150 Kbps - 1.5 Mbps
o Frame Resolution: 144p - 720p (or 1080p)
o Frame rate: 10fps - 30fps
- Variation from target bitrate: +/-5%
- Responsiveness to new bit rate request: 100ms
+ Media content: Foreman video sequence
+ Media timeline: New media flows are added sequentially, at
short time intervals. See test specific setup below.
+ Media startup behaviour: 200Kbps.
Sarker, et al. Expires August 18, 2014 [Page 15]
Internet-Draft Test Scenarios for RMCAT February 2014
* Competing traffic:
+ Number of sources : Zero (0)
o Test specific setup:
* Media flow timeline:
+ Flow no: One (1)
+ Start time: 0s
+ End time: 59s
* Media flow appearance:
+ Flow no: Two (2)
+ Start time: 20s
+ End time: 59s
* Media flow appearance:
+ Flow no: Three (3)
+ Start time: 40s
+ End time: 59s
4.4. RMCAT Flow competing with a long TCP Flow
In this test case, one or more RMCAT media flow shares the bottleneck
link with at least one long lived TCP flows. Long lived TCP flows
download data throughout the session and are expected to have
infinite amount of data to send and receive. This is a scenario
wherein a multimedia application co-exists with a large file
download. The test case measures the adaptivity of the candidate
algorithm to competing traffic, it addresses the requirement 8 in
[I-D.ietf-rmcat-cc-requirements].
Expected behavior: depending on the convergence observed in test case
4.1 and 4.2, the candidate algorithm may be able to avoid congestion
collapse. In the worst case, the media stream will fall to the
minimum media bit rate.
Sarker, et al. Expires August 18, 2014 [Page 16]
Internet-Draft Test Scenarios for RMCAT February 2014
To evaluate the performance of the candidate algorithms it is
expected to log enough information to visualize the following metrics
at a fine enough time granularity:
1. Flow level:
a. End-to-end delay for the RMCAT flow.
b. RTP packet losses observed at the receiving endpoint.
c. Variation in sending bit rate and goodput. Mainly observing
the frequency and magnitude of oscillations.
d. Variation in the sending rate of the TCP flow
e. TCP throughput.
f. Convergence time.
g. Feedback message overhead.
2. Transport level:
a. Bandwidth utilization.
b. Queue length (ms):
+ average over the length of the session
+ 5 and 95 percentile
Testbed topology: One (1) media source S1 is connected to
corresponding R1, but both endpoints are additionally receiving and
sending data, respectively. The media traffic (S1->S2) is
transported over the upstream path and corresponding feedback/control
traffic is transported over the downstream path. Likewise media
traffic (S2->S1) is transported over the downstram path and
corresponding feedback/control traffic is transported over the
upstream path. The TCP uploads traffic over upstream path and
downloads over downstream path. Hence, TCP traffic exits in both
path directions.
+--+ +--+
|S1|===== \ UPSTREAM --> / =======|R1|
+--+ \\ // +--+
\\ //
+-----+ +-----+ +-----+ +-----+
|R_tcp|======| A |---------------------------->| B |=====|R_tcp|
Sarker, et al. Expires August 18, 2014 [Page 17]
Internet-Draft Test Scenarios for RMCAT February 2014
+-----+ | |<----------------------------| | +-----+
+-----+ +-----+
// \\
// <-- DOWNSTREAM \\
+--+ // \\ +--+
|R2|====== / \ ======|S2|
+--+ +--+
Figure 5: Testbed Topology for TCP vs RMCAT Flows
Testbed attributes:
o Test duration: 120s
o Path characteristics:
* Path direction: Upstream, Downstream
* Bottleneck-link capacity: 2Mbps
* One-Way propagation delay: [10ms, 150ms]
* Maximum end to end jitter: 30ms
* Bottleneck queue type: Droptail, but would benefit from running
the same test with different AQM schemes: FQ-Codel, or PIE.
* Bottleneck queue size: [20ms, 250ms, 1000ms]
* Path loss ratio: 0.0%
o Application-related:
* Media Source:
+ Media type: Video
+ Media direction: Upstream and Downstream
+ Number of media sources: One (1)
+ Media codec: VBR
+ Media source behaviour:
- Adaptability:
o Bit rate range: 150 Kbps - 1.5 Mbps
Sarker, et al. Expires August 18, 2014 [Page 18]
Internet-Draft Test Scenarios for RMCAT February 2014
o Frame Resolution: 144p - 720p (or 1080p)
o Frame rate: 10fps - 30fps
- Variation from target bitrate: +/-5%
- Responsiveness to new bit rate request: 100ms
+ Media content: Foreman video sequence
+ Media timeline:
- Start time: 5s.
- End time: 119s.
+ Media startup behaviour: [200Kbps, 1500Kbps].
* Competing traffic:
+ Number and Types of sources : one (1), long-lived TCP
+ Traffic direction : Downstream
+ Congestion control: Default TCP congestion control.
+ Traffic timeline:
- Start time: 0s.
- End time: 119s.
o Test specific setup: None
4.5. RMCAT Flow competing with short TCP Flows
In this test case, one or more RMCAT media flow shares the bottleneck
link with at multiple short-lived TCP flows. Short-lived TCP flows
resemble the on/off pattern observed in the web traffic, wherein
clients (browsers) connect to a server and download a resource
(typically a webpage, few images, text files, etc.) using several
TCP connections (up to 4). This scenario shows the performance of
the multimedia application when several browser windows are active.
The test case measures the adaptivity of the candidate algorithm to
competing web traffic, it addresses the requirements 2 in
[I-D.ietf-rmcat-cc-requirements].
Sarker, et al. Expires August 18, 2014 [Page 19]
Internet-Draft Test Scenarios for RMCAT February 2014
Depending on the number of short TCP flows, the cross-traffic either
appears as a short burst flow or resembles a long TCP flow. The
intention of this test is to observe the impact of short-term burst
on the behaviour of the candidate algorithm.
To evaluate the performance of the candidate algorithms it is
expected to log enough information to visualize the following metrics
at a fine enough time granularity:
1. Flow level:
a. End-to-end delay for the RMCAT flow.
b. RTP packet losses observed at the receiving endpoint.
c. Variation in sending bit rate and goodput. Mainly observing
the frequency and magnitude of oscillations.
d. Variation in the sending rate of the TCP flow.
e. TCP throughput.
f. Convergence time.
g. Feedback message overhead.
2. Transport level:
a. Bandwidth utilization.
b. Queue length (ms):
+ average over the length of the session
+ 5 and 95 percentile
Testbed topology: One (1) media source S1 is connected to
corresponding R1, but both endpoints are additionally receiving and
sending data, respectively. The media traffic (S1->S2) is
transported over the upstream path and corresponding feedback/control
traffic is transported over the downstream path. Likewise media
traffic (S2->S1) is transported over the downstram path and
corresponding feedback/control traffic is transported over the
upstream path. The TCP uploads traffic over upstream path and
downloads over downstream path. Hence, TCP traffic exits in both
path directions. The topology described here is similar to the one
described in Figure 5.
Sarker, et al. Expires August 18, 2014 [Page 20]
Internet-Draft Test Scenarios for RMCAT February 2014
Testbed attributes:
o Test duration: 300s
o Path characteristics:
* Path direction: Upstream, Downstream
* Bottleneck-link capacity: 2.0Mbps
* One-Way propagation delay: [10ms, 150ms]
* Maximum end to end jitter: 30ms
* Bottleneck queue type: Droptail, but would benefit from running
the same test with different AQM schemes: FQ-Codel, or PIE.
* Bottleneck queue size: 300ms
* Path loss ratio: 0.0%
o Application-related:
* Media Source:
+ Media type: Video
+ Media direction: Upstream and Downstream
+ Number of media sources: One (1)
+ Media codec: VBR
+ Media source behaviour:
- Adaptability:
o Bit rate range: 150 Kbps - 1.5 Mbps
o Frame Resolution: 144p - 720p (or 1080p)
o Frame rate: 10fps - 30fps
- Variation from target bitrate: +/-5%
- Responsiveness to new bit rate request: 100ms
+ Media content: Foreman video sequence
Sarker, et al. Expires August 18, 2014 [Page 21]
Internet-Draft Test Scenarios for RMCAT February 2014
+ Media timeline:
- Start time: 0s.
- End time: 299s.
+ Media startup behaviour: [200Kbps, 1500Kbps].
* Competing traffic:
+ Number and Types of sources : Ten (10), short-lived TCP
flows.
+ Traffic direction : Downstream
+ Congestion algorithm: Default TCP Congestion control.
+ Traffic timeline: Each short TCP flow is modeled as a
sequence of file downloads interleaved with idle periods.
See test specific setup. Not all short TCPs start at the
same time, 2 start in the ON state while 8 start in an OFF
stats. The model for the idle times for the OFF state is
discussed in the Short-TCP model.
o Test specific setup:
* Short-TCP traffic model:
+ File sizes: uniform distribution between 100KB to 1MB
+ Idle period: the duration of the OFF state is derived from
an exponential distribution with the mean value of 10
seconds.
4.6. Congested Feedback Link
RMCAT WG has been chartered to define algorithms for RTP hence it is
assumed that RTCP, RTP header extension or such would be used as
signalling means for the adaptation algorithm in the backchannel.
Due to asymmetry nature of the link between communicating peers it is
possible to observer lack such backchannel information due to
impaired backchannel link (even when forward channel might be
unimpaired). This test case is designed to observer candidate
congestion control behaviour in such an event. This test case
addresses requirement number 5 and in particular, requirement number
7.
Sarker, et al. Expires August 18, 2014 [Page 22]
Internet-Draft Test Scenarios for RMCAT February 2014
It is expected that the candidate algorithms should cope with the
lack of backchannel information and adapt to minimize the performance
degradation of media flows in the forward channel.
It SHOULD be noted that for this test case log is needed for the
reference case where the downstream channel has no impairments
To evaluate the performance of the candidate algorithms it is
expected to log enough information to visualize the following metrics
at a fine enough time granularity:
1. Flow level:
a. End-to-end delay
b. Losses observed at the receiving endpoint
c. Feedback message overhead
2. Transport level:
a. Bandwidth utilization
b. Queue length (ms):
+ average over the length of the session
+ 5 and 95 percentile
.
Testbed topology: One (1) media source S1 is connected to
corresponding R1, but both endpoints are additionally receiving and
sending data, respectively. The media traffic (S1->S2) is
transported over the upstream path and corresponding feedback/control
traffic is transported over the downstream path. Likewise media
traffic (S2->S1) is transported over the downstram path and
corresponding feedback/control traffic is transported over the
upstream path.
+---+ +---+
|S1 |===== \ UPSTREAM --> / =======|R1 |
+---+ \\ // +---+
\\ //
+-----+ +-----+
| A |------------------------------>| B |
| |<------------------------------| |
+-----+ +-----+
Sarker, et al. Expires August 18, 2014 [Page 23]
Internet-Draft Test Scenarios for RMCAT February 2014
// \\
// <-- DOWNSTREAM \\
+---+ // \\ +---+
|R2 |===== / \ ======|S2 |
+---+ +---+
Figure 6: Testbed Topology for Congested Feedback Link
Testbed attributes:
o Test duration: 60s
o Path characteristics:
* Path direction: Upstream and downstream.
* Bottleneck-link capacity: 2Mbps.
* One-Way propagation delay: [10ms, 100ms].
* Maximum end-to-end jitter: 30ms.
* Bottleneck queue type: Drop tail.
* Bottleneck queue size: 300ms.
* path loss ratio: 0%.
o Application-related:
* Media Source:
+ Media type: Video
+ Media direction: Upstream
+ Number of media sources: One (1).
+ Media codec: VBR
+ Media source behaviour:
- Adaptability:
o Bit rate range: 150 Kbps - 1.5 Mbps
o Frame Resolution: 144p - 720p (or 1080p)
Sarker, et al. Expires August 18, 2014 [Page 24]
Internet-Draft Test Scenarios for RMCAT February 2014
o Frame rate: 10fps - 30fps
- Variation from target bitrate: +/-5%
- Responsiveness to new bit rate request: 100ms
+ Media content: Foreman media sequence
+ Media timeline:
- Start time: 0s.
- End time: 59s.
* Media Source:
+ Media type: Video
+ Media direction: Downstream
+ Number of media sources: One (1).
+ Media codec: VBR
+ Media source behaviour:
- Adaptability:
o Bit rate range: 150 Kbps - 1.5 Mbps
o Frame Resolution: 144p - 720p (or 1080p)
o Frame rate: 10fps - 30fps
- Variation from target bitrate: +/-5%
- Responsiveness to new bit rate request: 100ms
+ Media content: Foreman media sequence
+ Media timeline:
- Start time: 0s.
- End time: 59s.
* Media startup behaviour: 200Kbps.
Sarker, et al. Expires August 18, 2014 [Page 25]
Internet-Draft Test Scenarios for RMCAT February 2014
* Competing traffic:
+ Number of sources : Zero (0)
o Test specific setup:
* Number of bandwidth variation: Two (2)
* Variation pattern:
+ Sequence number: 1
+ Path direction: Upstream
+ Bottleneck Capacity: 1Mbps
+ Start time: 20s
* Variation pattern:
+ Sequence number: 2
+ Path direction: Upstream
+ Bottleneck Capacity: 600Kbps
+ Start time: 30s
* Variation pattern:
+ Sequence number: 3
+ Path direction: Upstream
+ Bottleneck Capacity: 2Mbps
+ Start time: 50s
* Variation pattern:
+ Sequence number: 1
+ Path direction: Downstream
+ Bottleneck Capacity: 800kbps
+ Start time: 30s
Sarker, et al. Expires August 18, 2014 [Page 26]
Internet-Draft Test Scenarios for RMCAT February 2014
* Variation pattern:
+ Sequence number: 2
+ Path direction: Downstream
+ Bottleneck Capacity: 2Mbps
+ Start time: 50s
4.7. Round Trip Time Fairness
In this test case, more than one RMCAT media flow shares the
bottleneck link, but the end-to-end path latency for each RMCAT flow
is different. For the sake of simplicity it is assumed that there
are no other competing traffic sources in the bottleneck link and
that there is sufficient capacity to accommodate all the flows.
While this appears to be a variant of the test case 4.2, it tests the
capacity sharing distribution of the candidate algorithm under
different RTTs. This test case particularly addresses the
requirements 2 [I-D.ietf-rmcat-cc-requirements].
It is expected that the competing flows will converge to an optimum
bit rate to accommodate all the flows with minimum possible latency
and loss. Specifically, the test introduces five media flows at the
same time instance.
To evaluate the performance of the candidate algorithms it is
expected to log enough information to visualize the following metrics
at a fine enough time granularity:
1. Flow level:
a. End-to-end delay.
b. RTP packet losses observed at the receiving endpoint.
c. Variation in sending bit rate and goodput. Mainly observing
the frequency and magnitude of oscillations.
d. Convergence time.
2. Transport level:
a. Bandwidth utilization.
b. Queue length (ms):
Sarker, et al. Expires August 18, 2014 [Page 27]
Internet-Draft Test Scenarios for RMCAT February 2014
+ average over the length of the session
+ 5 an 95 percentile
Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to
their corresponding media sinks R1,R2,..,R5. The media traffic is
transported over the upstream path and corresponding feedback/control
traffic is transported over the downstream path. The topology is the
same as test case 4.3 defined in Section 4.3. The end-to-end path
delay for S1-R1 is 10ms, S2-R2 is 25ms, S3-R3 is 50ms, S4-R4 is
100ms, S5-R5 is 150ms.
Testbed attributes:
o Test duration: 60s
o Path characteristics:
* Path direction: Upstream, Downstream
* Number of bottlenecks: One (1)
* Bottleneck link capacity: 3.0Mbps
* One-Way propagation delay for each path is: 10ms, 25ms, 50ms,
100ms, 150ms.
* Maximum end to end jitter: 30ms
* Bottleneck queue type: Droptail
* Bottleneck queue size: 300ms
* Link loss ratio: 0.0%
o Application-related:
* Media Source:
+ Media direction: Upstream
+ Number of media sources: Five (5)
+ Encoder configuration:
- Bit rate generation: VBR
- Bit rate range: 150 Kbps - 1.5 Mbps
Sarker, et al. Expires August 18, 2014 [Page 28]
Internet-Draft Test Scenarios for RMCAT February 2014
- Frame Resolution: 144p-720p (or 1080p)
- Frame rate: 10fps-30fps
- Variation from target bit rate: +/-5%
- Responsiveness to new bit rate request: 60ms
+ Media content: Foreman video sequence
+ Media timeline:
- Start time: 0s.
- End time: 59s.
+ Media startup behaviour: [200 Kbps, 1500 Kbps].
* Competing traffic:
+ Number of sources : Zero (0)
o Test specific setup: None
4.8. Media Pause and Resume
In this test case, more than one real-time interactive media flows
share the link bandwidth and all flows reach to a steady state by
utilizing the link capacity in an optimum way. At these stage one of
the media flow is paused for a moment. This event will result in
more available bandwidth for the rest of the flows and as they are on
a shared link. When the paused media flow will resume it would no
longer have the same bandwidth share on the link. It has to make it
way through the other existing flows in the link to achieve a fair
share of the link capacity. This test case is important specially
for real-time interactive media which consists of more than one media
flows and can pause/resume media flow at any point of time during the
session. This test case directly addresses the requirement number
1.B in [I-D.ietf-rmcat-cc-requirements]. One can think it as a
variation of test case 4.3. However, it is different as the
candidate algorithms can use different strategies to increase it s
efficiency, for example the fairness, convergence time, reduce
oscillation etc, by capitalizing the fact that they have previous
information of the link.
To evaluate the performance of the candidate algorithms it is
expected to log enough information to visualize the following metrics
at a fine enough time granularity:
Sarker, et al. Expires August 18, 2014 [Page 29]
Internet-Draft Test Scenarios for RMCAT February 2014
1. Flow level:
a. End-to-end delay.
b. RTP packet losses observed at the receiving endpoint.
c. Variation in sending bit rate and goodput. Mainly observing
the frequency and magnitude of oscillations.
d. Convergence time.
e. Feedback message overhead.
2. Transport level:
a. Bandwidth utilization.
b. Queue length (ms):
+ average over the length of the session
+ 5 and 95 percentile
Testbed Topology: Same as test case 4.3 defined in Section 4.3
Testbed attributes: The general description of the testbed parameters
are same as test case 4.3 with changes in the test specific setup as
below-
o Other test specific setup:
* Media flow timeline:
+ Flow no: One (1)
+ Start time: 0s
+ Flow duration: 59s
+ Pause time: not required
+ Resume time: not required
* Media flow appearance:
+ Flow no: Two (2)
+ Start time: 0s
Sarker, et al. Expires August 18, 2014 [Page 30]
Internet-Draft Test Scenarios for RMCAT February 2014
+ Flow duration: 59s
+ Pause time: 20s
+ Resume time: 30s
* Media flow appearance:
+ Flow no: Three (3)
+ Start time: 0s
+ Flow duration:59s
+ Pause time: not required
+ Resume time: not required
4.9. Explicit Congestion Notification Usage
TBD
5. Wireless Access Links
5.1. Cellular Network Specific Test Cases
Additional cellular network specific test cases are define in
[I-D.draft-sarker-rmcat-cellular-eval-test-cases]
5.2. Wi-Fi Network Specific Test Cases
TBD
[Editor's Note: We should encourage people to come up with possible
WiFi Network specific test cases]
6. Security Considerations
Security issues have not been discussed in this memo.
7. IANA Considerations
There are no IANA impacts in this memo.
8. Acknowledgements
Much of this document is derived from previous work on congestion
control at the IETF.
Sarker, et al. Expires August 18, 2014 [Page 31]
Internet-Draft Test Scenarios for RMCAT February 2014
The content and concepts within this document are a product of the
discussion carried out in the Design Team.
9. References
9.1. Normative References
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, July 2003.
[RFC3551] Schulzrinne, H. and S. Casner, "RTP Profile for Audio and
Video Conferences with Minimal Control", STD 65, RFC 3551,
July 2003.
[RFC3611] Friedman, T., Caceres, R., and A. Clark, "RTP Control
Protocol Extended Reports (RTCP XR)", RFC 3611, November
2003.
[RFC4585] Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey,
"Extended RTP Profile for Real-time Transport Control
Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, July
2006.
[RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size
Real-Time Transport Control Protocol (RTCP): Opportunities
and Consequences", RFC 5506, April 2009.
[RFC5506] Johansson, I. and M. Westerlund, "Support for Reduced-Size
Real-Time Transport Control Protocol (RTCP): Opportunities
and Consequences", RFC 5506, April 2009.
[I-D.ietf-avtcore-rtp-circuit-breakers]
Perkins, C. and V. Singh, "RTP Congestion Control: Circuit
Breakers for Unicast Sessions", draft-ietf-avtcore-rtp-
circuit-breakers-01 (work in progress), October 2012.
[I-D.ietf-rmcat-eval-criteria]
Singh, V. and J. Ott, "Evaluating Congestion Control for
Interactive Real-time Media", draft-ietf-rmcat-eval-
criteria-00 (work in progress), January 2014.
[I-D.ietf-rmcat-cc-requirements]
Jesup, R., "Congestion Control Requirements For RMCAT",
draft-ietf-rmcat-cc-requirements-00 (work in progress),
July 2013.
[I-D.draft-sarker-rmcat-cellular-eval-test-cases]
Sarker, et al. Expires August 18, 2014 [Page 32]
Internet-Draft Test Scenarios for RMCAT February 2014
Sarker, Z., "Evaluation Test Cases for Interactive Real-
Time Media over Cellular Networks", , <http://www.ietf.org
/id/draft-sarker-rmcat-cellular-eval-test-cases-00.txt>.
9.2. Informative References
[I-D.ietf-rtcweb-use-cases-and-requirements]
Holmberg, C., Hakansson, S., and G. Eriksson, "Web Real-
Time Communication Use-cases and Requirements", draft-
ietf-rtcweb-use-cases-and-requirements-10 (work in
progress), December 2012.
[RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion
Control Algorithms", BCP 133, RFC 5033, August 2007.
[RFC5166] Floyd, S., "Metrics for the Evaluation of Congestion
Control Mechanisms", RFC 5166, March 2008.
[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion
Control", RFC 5681, September 2009.
[SA4-EVAL]
R1-081955, 3GPP., "LTE Link Level Throughput Data for SA4
Evaluation Framework", 3GPP R1-081955, 5 2008.
[SA4-LR] S4-050560, 3GPP., "Error Patterns for MBMS Streaming over
UTRAN and GERAN", 3GPP S4-050560, 5 2008.
[xiph-seq]
Xiph.org, , "Video Test Media",
http://media.xiph.org/video/derf/ , .
[HEVC-seq]
HEVC, , "Test Sequences",
http://www.netlab.tkk.fi/~varun/test_sequences/ , .
[TCP-eval-suite]
Lachlan, A., Marcondes, C., Floyd, S., Dunn, L., Guillier,
R., Gang, W., Eggert, L., Ha, S., and I. Rhee, "Towards a
Common TCP Evaluation Suite", Proc. PFLDnet. 2008, August
2008.
Authors' Addresses
Sarker, et al. Expires August 18, 2014 [Page 33]
Internet-Draft Test Scenarios for RMCAT February 2014
Zaheduzzaman Sarker
Ericsson AB
Luleae, SE 977 53
Sweden
Phone: +46 10 717 37 43
Email: zaheduzzaman.sarker@ericsson.com
Varun Singh
Aalto University
School of Electrical Engineering
Otakaari 5 A
Espoo, FIN 02150
Finland
Email: varun@comnet.tkk.fi
URI: http://www.netlab.tkk.fi/~varun/
Xiaoqing Zhu
Cisco Systems
510 McCarthy Blvd
Milpitas, CA 95134
USA
Email: xiaoqzhu@cisco.com
Michael A. Ramalho
Cisco Systems, Inc.
8000 Hawkins Road
Sarasota, FL 34241
USA
Phone: +1 919 476 2038
Email: mramalho@cisco.com
Sarker, et al. Expires August 18, 2014 [Page 34]