Network Working Group K. Dubray
INTERNET-DRAFT Bay Networks
Expiration Date: May 1997 November 1996
Terminology for IP Multicast Benchmarking
<draft-ietf-bmwg-mcast-00.txt>
Status of this Memo
This document is an Internet-Draft. Internet-Drafts are working
documents of the Internet Engineering Task Force (IETF), its areas,
and its working groups. Note that other groups may also distribute
working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet- Drafts as reference
material or to cite them other than as ``work in progress.''
To learn the current status of any Internet-Draft, please check the
``1id-abstracts.txt'' listing contained in the Internet- Drafts
Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe),
munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or
ftp.isi.edu (US West Coast).
Abstract
The purpose of this draft is to add terminology specific to the
benchmarking of multicast IP forwarding devices. It builds upon the
tenets set forth in RFC 1242, RFC 1944, and other IETF
Benchmark Methodology Working Group (BMWG) effort and extends
them to the multicast paradigm.
[While primarily directed towards intermediate IP multicast
forwarding devices on LANs, elements of this text may or may not be
applicable to other media as well.]
1. Introduction
Network forwarding devices are being required to take a single
frame and support delivery to a number of destinations having
membership to a particular group. As such, multicast support may
place a different burden on the resources of these network
forwarding devices than with unicast or broadcast traffic types.
By clearly identifying benchmarks and related terminology in this
document, it is hoped that detailed methodologies can be generated
in subsequent documents. Taken in tandem, these two efforts
endeavor to assist the clinical, empirical, and consistent
characterization of certain aspects of multicast technologies and
their individual implementations.
Dubray, K. Expires May 1997 [Page 1]
INTERNET-DRAFT Multicast Benchmarking Terminology November 1996
2. Definition Format
This section cites the template suggested by RFC 1242 in the
specification of a term to be defined.
Term to be defined.
Definition:
The specific definition for the term.
Discussion:
A brief discussion of the term, its application and any
restrictions on measurement procedures.
Measurement units:
Units used to record measurements of this term, if applicable.
[Issues:]
List of issues or conditions that effect this term. This
field is optional in this draft.
[See Also:]
List of other terms that are relevant to the discussion
of this term. This field is optional in this draft.
2.1 Existing Terminology
This document draws on existing terminology defined in other
BMWG work. Examples include, but are not limited to:
Throughput (RFC 1242, section 3.17)
Latency (RFC 1242, section 3.8)
Constant Load (RFC 1242, section 3.4)
Frame Loss Rate (RFC 1242, section 3.6)
Overhead behavior (RFC 1242, section 3.11)
3. Term Definitions
This section will present the terminology to be defined in
this document.
3.1 Device Under Test (DUT).
Definition:
The network forwarding device being tested.
Dubray, K. Expires May 1997 [Page 2]
INTERNET-DRAFT Multicast Benchmarking Terminology November 1996
Discussion:
Measurement units:
Not applicable.
3.2 System Under Test (SUT).
Definition:
The collective set of network devices being tested as a singular
entity.
Discussion: A system under test may be comprised of a variety
of networking devices. Some devices may be active in the
forwarding decision making process, such as routers or switches;
other devices may be passive such as CSU/DSUs. Regardless
of constituent components, the system is treated as a "black box"
to which stimuli is offered and response measured.
Measurement units:
Not applicable.
3.3 Target Rate.
Definition:
The requested rate at which the test device attempts to offer the
DUT or SUT test traffic.
Discussion:
There are networks events (e.g., collisions) that may preclude the
test device from delivering the requested rate to the SUT. In this
case, differentiation is made between target rate and offered rate.
[May need to be reconciled with terminology of BMWG works-in-
progress.]
Measurement units:
Frames per second.
3.4 Offered Rate.
Definition:
The actual resultant rate at which the test device is successful
in offering test traffic to the SUT.
Dubray, K. Expires May 1997 [Page 3]
INTERNET-DRAFT Multicast Benchmarking Terminology November 1996
Discussion:
Contrast with Target Rate. Note relationship to Forwarding Rate.
[May need to be reconciled with terminology of BMWG works-in-
progress.]
Measurement units:
Frames per second.
3.5 Forwarding Rate.
Definition:
The rate at which the SUT has been observed to successfully
forward test traffic to the traffic's correct destination(s) in
response to a particular offered rate.
Discussion:
Note the specification of "correct destination(s)" in the
definition. The reporting of a forwarding rate MUST
correspond to an associated Offered Rate. Frame loss is not
a constraint when reporting Forwarding Rate.
[May need to be reconciled with terminology of BMWG works-in-
progress.]
Measurement units:
Frames per second.
3.6 Maximum Forwarding Rate (MFR).
Definition:
The rate at which the SUT has been observed to successfully
forward test traffic to the traffic's correct destination(s) in
response to the test device's maximum offered rate.
Discussion:
Because a DUT's maximum forwarding rate does not always equal
the largest forwarding rate of the DUT, this metric can sometimes
indicate oversubscription or congestion internal to the DUT/SUT.
For example, consider the following table:
Test Device DUT
Offered Rate Forwarding Rate
------------- ---------------
1. 14,880 fps 7,400 fps
2. 13,880 fps 8,472 fps
3. 12,880 fps 12,880 fps
Dubray, K. Expires May 1997 [Page 4]
INTERNET-DRAFT Multicast Benchmarking Terminology November 1996
The tester's maximum offered rate is 14,880 frames per second,
as indicated in line 1. Per the definition, the corresponding
MFR for the DUT is 7,440 fps - not the 12,880 fps indicated in
line 3.
When reporting the MFR, the corresponding test device's maximum
offered load MUST be cited. This is due to the fact that not
all test devices deliver the maximum usable bandwidth. In the
case when the test device is able to exceed the maximum, legal
bandwidth, the test results SHOULD reflect that the test was
conducted in a overload condition.
Measurement units:
Frames per second.
3.7 Flow.
Definition:
An equivalence class of packets comprising one or more data
streams.
Discussion:
In the scope of this document, Flow will be considered a logical
identifier used to discriminate between a set or sets of packets
offered the DUT.
For example, one flow may identify a set of unicast packets
offered to the DUT. Another flow may differentiate the
multicast packets destined to multicast group X. Yet another
flow may distinguish the set of multicast packets destined to
multicast group Y.
Measurement units:
Not applicable.
3.8 Group Flow.
Definition:
A specific type of flow where the packets comprising the flow
are destined to a particular multicast group.
Discussion:
Measurement units:
Not applicable.
Dubray, K. Expires May 1997 [Page 5]
INTERNET-DRAFT Multicast Benchmarking Terminology November 1996
3.9 Service Flow.
Definition:
A specific type of flow where the packets comprising the flow
require particular treatment or treatments by the network
forwarding devices along the path to the packets' destination(s).
Discussion:
Measurement units:
Not applicable.
3.10 Mixed Throughput (MT).
Definition:
The maximum rate at which none of the offered frames, comprised
from a unicast flow and a multicast flow, to be forwarded are
dropped by the device.
Discussion:
Often times, throughput is collected on a homogenous traffic
type - though the packets' destinations may vary, the packets
follow the same packet forwarding path through the DUT.
Based on the RFC 1242 definition for throughput, the Mixed
Thoughput benchmark attempts to characterize the DUT's
ability to process both unicast and multicast frames in the
same aggregated traffic stream.
Measurement units:
Frames per second
3.11 Scaled Group Throughput (SGT).
Definition:
The maximum number of multicast groups that a DUT/SUT can
support and still yield the same throughput as supporting a
single multicast group.
Discussion:
A desirable attribute of many Internet mechanisms is the ability
to "scale." This benchmark seeks to demonstrate the ability
of a SUT to scale the number of multicast groups upwards while
holding it to the RFC 1242 definition of throughput for a single
multicast group.
Measurement units:
Number of multicast groups.
Dubray, K. Expires May 1997 [Page 6]
INTERNET-DRAFT Multicast Benchmarking Terminology November 1996
3.12 Extraction Throughput (ET)
Definition:
The maximum rate at which none of the frames offered in an
transitional format to the SUT are dropped in the process of
converting those frames to their appropriate, final format and
subsequent correct delivery.
Discussion:
A popular technique in presenting frames to devices that may
not support a protocol feature is to encapsulate, or tunnel,
the packet containing the unsupported feature in a format that
is supported by that device. This benchmark attempts to
characterize the overhead behavior associated with that
transitional process.
Consideration may need to be given with respect to the impact
of different frame formats on usable bandwidth.
Measurement units:
Frames per second.
3.13 Fairness.
Definition:
The ability of a SUT to fulfill the requirements of a flow
without compromising the requirements, if any, of other flows.
Discussion:
Measurement units:
Not applicable.
3.14 Multicast Latency.
Definition:
The set of individual latencies from a single input port on
the DUT or SUT to all tested ports belonging to the destination
multicast group.
Discussion:
This benchmark is based on the RFC 1242 definition of latency.
While it is useful to collect latency between a pair of source
and destination multicast ports, it may be insightful to collect
the same type of measurements across a range of ports supporting
that group flow.
A variety of statistical exercises can be applied to the set of
latencies measurements.
Measurement units:
Time units with enough precision to reflect measurement.
Dubray, K. Expires May 1997 [Page 7]
INTERNET-DRAFT Multicast Benchmarking Terminology November 1996
3.15 Min/Max Latency.
Definition:
The difference between the maximum latency measurement and the
minimum latency measurement from the set of latencies produced by
the Multicast Latency benchmark.
Discussion:
This statistic may yield some insight into how a particular
implementation handles its multicast traffic. This may be useful
to users of multicast synchronization types of applications.
Measurement units:
Time units with enough precision to reflect measurement.
3.16 Prune Delay.
Definition:
The time duration it takes a DUT/SUT to cease forwarding multicast
packets after a corresponding "prune" or similar event has been
successfully generated.
Discussion:
While it is important to understand how quickly a system can
process multicast frames; it may be beneficial to understand
how quickly that same system can stop the process as well.
Measurement units:
Microseconds.
3.16 Group Membership Delay.
Definition:
The time duration it takes a DUT/SUT to start forwarding multicast
packets from the time a successful IGMP group membership report has
been generated.
Discussion:
Many different factors can contribute to different results, such as
the number or type of multicast-related protocols configured
on the system under test.
A consideration for the related methodology: possible need to
differentiate a specifically-forwarded multicast frame from those
sprayed by protocols implementing a flooding tactic to solicit prune
feedback.
Measurement units:
Microseconds.
Dubray, K. Expires May 1997 [Page 8]
INTERNET-DRAFT Multicast Benchmarking Terminology November 1996
3.17 Multicast Group Capacity.
Definition:
The maximum number of multicast groups a SUT/DUT can support
while maintaining the ability to forward multicast frames
to all multicast groups registered to that SUT/DUT.
Discussion:
Measurement units:
Multicast groups.
3.18 Aggregated Multicast Throughput (AMT)
Definition:
The maximum rate at which none of the offered frames to be
forwarded through N destination interfaces of the same multicast
group are dropped.
Discussion:
Another "scaling" type of exercise, designed to identify the
DUT/SUT's ability to handle traffic as a function of the
multicast destination ports it is required to support.
Measurement units:
The ordered pair (N,t) where,
N = the number of destination ports of the multicast group.
t = the throughput, in frames per second, relative to the
source stream.
4. Security Considerations
Security issues are not addressed in this memo.
5. References
[1] Bradner, S. Benchmarking Terminology for Network
Interconnection Devices. RFC 1242. July, 1991.
[2] Bradner, S., McQuaid, J. Benchmarking Methodology for Network
Interconnect Devices. RFC 1944. May, 1996.
[3] Craig, R. Terminology for Cell/Call Benchmarking. <draft-ietf-
bmwg-call-00.txt> November, 1996. Work in progress.
[4] Mandeville, R. Benchmarking Terminology for LAN Switching
Devices. <draft-ietf-bmwg-lanswitch-01.txt> November, 1996.
Work in progress.
5. Author's Address
Kevin Dubray
Bay Networks, Inc.
2 Federal Street
Billerica, MA 01984
(508) 436-3862
kdubray@baynetworks.com
or direct discussion to the Benchmarking Methodology Working Group:
bmwg@harvard.edu
Dubray, K. Expires May 1997 [Page 9]
