Network Working Group Jerry Perser
INTERNET-DRAFT Spirent
Expires in: August 2001 David Newman
Network Test
Sumit Khurana
Telcordia
Shobha Erramilli
Telcordia
Scott Poretsky
Avici
February 2001
Terminology for Benchmarking Network-layer
Traffic Control Mechanisms
<draft-ietf-bmwg-dsmterm-00.txt>
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other documents
at any time. It is inappropriate to use Internet- Drafts as
reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Table of Contents
1. Introduction ............................................... 2
2. Existing definitions ....................................... 3
3. Term definitions .............................................3
3.1 Channel Capacity ..........................................3
3.2 Classification ............................................4
3.3 Code Point Forwarding Rate ................................4
3.4 Conforming ................................................5
3.5 Congestion ................................................5
3.6 Congestion Management .....................................6
3.7 Delay .....................................................6
3.8 Expected Output Vector ....................................7
Perser, Newman, & Poretsky [Page 1]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
3.9 Flow ......................................................8
3.10 Jitter ...................................................8
3.11 Nonconforming ............................................9
3.12 Offered Load Vector .....................................10
3.13 Policy ..................................................10
3.14 Provision ...............................................11
3.15 Sequence number .........................................11
3.16 Shaping .................................................12
3.17 Stream ..................................................12
3.18 Tail dropping ...........................................13
3.19 Unburdened Response .....................................14
4. Security Considerations .....................................14
5. References ..................................................14
6. Author's Address ............................................15
7. Full Copyright Statement ....................................16
1. Introduction
Driven by Internet economics, service providers and enterprises
alike have shown strong interest in adding traffic-control
capabilities to network devices. These capabilities would enable
network operators to define and deliver minimum or maximum levels of
bandwidth, delay, and jitter for multiple classes of traffic.
Perhaps more importantly, network operators would be able to set
pricing according to the level of service delivered.
Networking device manufacturers have responded with a bewildering
array of mechanisms for controlling network traffic. While there are
numerous _policy management_ and _quality of service_ frameworks,
many of them rely on one of two network-layer mechanisms for the
actual control of forwarding rate, delay, and jitter. These two
mechanisms are the IP precedence setting in the IP header and the
diff-serve code point (DSCP) defined in [3].
This document describes the various terms to be used in benchmarking
devices that implement traffic control based on IP precedence or
DSCP criteria. This document is narrowly focused, in that it
describes only terms for measuring behavior of a device or a group
of devices using one of these two mechanisms. End-to-end and
service-level measurements are beyond the scope of this document.
This document introduces several new terms that will allow
measurements to be taken during periods of congestion. New
terminology is needed because most existing benchmarking terms
assume the absence of congestion. For example, throughput is one of
the most widely used measurements _- yet RFC 1242 defines throughput
as a rate in the absence of loss. As a result, throughput is not a
meaningful measurement where congestion exists.
Another key difference from existing benchmarking terminology is the
definition of measurements as observed on egress as well as ingress
of a device/system under test. Again, the existence of congestion
Perser, Newman, Khurana, Erramilli, Poretsky [Page 2]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
requires the addition of egress measurements as well as those taken
on ingress; without observing traffic leaving a device it is not
possible to say whether traffic-control mechanisms effectively dealt
with congestion.
The principal measurements introduced in this document are
forwarding rate, latency, and jitter, all of which can be observed
with or without congestion of the DUT/SUT.
2. Existing definitions
RFC 1242 "Benchmarking Terminology for Network Interconnect Devices"
and RFC 2285 " Benchmarking Terminology for LAN Switching Devices"
should be consulted before attempting to make use of this document.
RFC 2474 " Definition of the Differentiated Services Field (DS
Field) in the IPv4 and IPv6 Headers" section 2, contains discussions
of a number of terms relevant to network-layer traffic control
mechanisms and should also be consulted.
For the sake of clarity and continuity this RFC adopts the template
for definitions set out in Section 2 of RFC 1242. Definitions are
indexed and grouped together in sections for ease of reference.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
this document are to be interpreted as described in RFC 2119.
3. Term definitions
3.1 Channel Capacity
Definition:
The maximum forwarding rate of a link or set of aggregated
links at which none of the offered packets are dropped by the
DUT/SUT.
Discussion:
Channel capacity measures the data rate at the egress
interface(s) of the DUT/SUT. In contrast, throughput as defined
in RFC 1242 measures the data rate is based on the ingress
interface(s) of the DUT/SUT.
Ingress-based measurements do not account for congestion of the
DUT/SUT. Channel capacity, as an egress measurement, does take
congestion into account.
Understanding channel capacity is a necessary precursor to any
measurement involving congestion. Throughput numbers can be
higher than channel capacity because of queueing.
Perser, Newman, Khurana, Erramilli, Poretsky [Page 3]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
Measurement units:
N-octet packets per second
Issues:
See Also:
Throughput [1]
3.2 Classification
Definition:
Local mapping of Policies to the IP Precedence or DSCP value in
each frame.
Discussion:
Could be based on the DS field or IP Precedence in the packet
header. Could be based on MF criteria such as IP Source and
destination addresses, protocol and port number.
Covers reclassification. A Hop does not know if a previous Hop
classified the packets. Since the scope of the document is a
per-hop-behavior, we avoid end-to-end discussions.
Classification should cover all cases defined in policy. This
may include rules mandating packet drops, not just forwarded
packets.
Measurement units:
n/a
Issues:
See Also:
3.3 Code Point Forwarding Rate
Definition:
The number of packets per second in for all packets containing
a single DSCP (or IP precedence) that a device can be observed
to successfully transmit to the correct destination interface
in response to a specified offered load.
Discussion:
- CP based, not port nor IP stream based.
- Per-hop measurement. A DUT may change the DSCP for a
multiple-hop measurement.
Perser, Newman, Khurana, Erramilli, Poretsky [Page 4]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
Measurement units:
N-octet packets per second
Issues:
See Also:
3.4 Conforming
Definition:
A packet meeting a certain policy.
Discussion:
Consider a policy that allows a given application to consume
only X bit/s of channel capacity and no more.
In a congestion scenario, some individual packets will be
conforming and others will not.
It cannot be said that an entire stream or flow is conforming,
since an offered load that exceeds policy boundaries will
necessarily carry some nonconforming traffic.
Measurement units:
n/a
Issues:
See Also:
3.5 Congestion
Definition:
A condition in which one or more egress interfaces are offered
more packets than can be forwarded at any given instant.
Discussion:
This condition is a superset of the overload definition [2].
That definition assumes the overload is introduced by strictly
by the tester on ingress of a DUT/SUT. That may or may not be
the case here.
Another difference is that with multiple-DUT measurements,
congestion may occur at multiple points. For example, multiple
edge devices collectively may congest a core device. In
contrast, overload [1] deals only with overload on ingress.
Perser, Newman, Khurana, Erramilli, Poretsky [Page 5]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
Ingress observations alone are not sufficient to cover all
cases in which congestion may occur. A device with an infinite
amount of memory could buffer an infinite amount of packets,
and eventually forward all of them. However, these packets may
or may not be forwarded during the test duration. Even though
ingress interfaces accept all packets without loss, this
hypothetical device may still be congested.
The definition presented here explicitly defines congestion as
an event observable on egress interfaces. Regardless of
internal architecture, any device that cannot forward packets
on one or more egress interfaces is congested.
Measurement units:
n/a
Issues:
See Also:
3.6 Congestion Management
Definition:
An implementation of one or more per-hop-behaviors to avoid or
minimize the condition of congestion.
Discussion:
Congestion management may seek either to control congestion or
avoid it altogether. Such mechanisms classify packets based
upon IP Precedence or DSCP settings in a packet's IP header.
Congestion avoidance mechanisms seek to prevent congestion
before it actually occurs.
Congestion control mechanisms gives one or more service classes
preferential treatment over other classes during periods of
congestion.
Measurement units:
n/a
Issues:
See Also:
3.7 Delay
Definition:
Perser, Newman, Khurana, Erramilli, Poretsky [Page 6]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
The time interval starting when the last bit of the input frame
reaches the input port of the DUT/SUT and ending when the last
bit of the output frame is seen on the output port of the
DUT/SUT.
Discussion:
Delay measurement is measured the same regardless of the type
of DUT/SUT. Latency [1] require some knowledge of whether the
DUT/SUT is a "store and forward" or a "bit forwarding" device.
The fact that a DUT/SUT's technology has a lower delay than
another technology should be visible in the measurement.
The measurement point of the last bit is more like the way an
IP datagram is processed. A datagram is not passed up or down
the stack unless it is complete. Competition happens at the
last bit of the datagram.
Delay can be run at any offered load. Recommend at or below
the channel capacity for non-congested delay. For congested
delay, run the offered load above the channel capacity.
Measurement units:
Seconds.
Issues:
See Also:
3.8 Expected Output Vector
Definition:
A vector describing the expected output rate of packets having
a specific code-point for each code-point in the code-point set
depending on the offered load vector and device PHB
configurations of the DUT/SUT.
Discussion:
The DUT is configured in a certain way in order that service
discrimination happens for behavior aggregates when a specific
traffic mix consisting of multiple behavior aggregates is
applied. This term attempts to capture the expected behavior
for which the device is configured given a certain load.
The actual algorithms or mechanism, that the DUT uses to
achieve service discrimination, is not important in describing
the expected output vector.
Measurement units:
bits per second
N-octets per second
Perser, Newman, Khurana, Erramilli, Poretsky [Page 7]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
See Also:
Offered Load Vector
3.9 Flow
Definition:
A flow is a one or more of packets sharing a common intended
pair of source and destination interfaces.
Discussion:
Packets are grouped by which interface they ingress and egress
the DUT/SUT.
A flow can contain multiple source IP addresses and/or
destination IP addresses. All packets in a flow must enter on
the same ingress interface and exit on the same egress
interface, and have some common network layer content.
Microflows [3] are a subset of flows. As defined in [3],
microflows require application-to-application measurement. In
contrast, flows use lower-layer classification criteria. Since
this document focuses on network-layer classification criteria,
we concentrate here on the use of network-layer identifiers in
describing a flow. Flow identifiers also may reside at the
data-link, transport, or application layers of the ISO model.
However, identifiers other than those at the network layer are
out of scope for this document.
A flow may contain a single code point/IP precedence or may
contain multiple values destined for a single egress interface.
This is determined by the test methodology.
Measurement units:
n/a
Issues:
See Also:
Microflow [3]
Streams
3.10 Jitter
Definition:
Variation in delay.
Discussion:
Perser, Newman, Khurana, Erramilli, Poretsky [Page 8]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
Jitter as the absolute value of the difference between the
delay measurement of two adjacent packets |D(i) - D(i+1)|. The
jitter between two packets is reported as the "instantaneous
jitter". Packets lost are not counted in the jitter
measurement.
Average Jitter is the average of the instantaneous jitter
measured during the test duration.
Peak-to-peak jitter is the maximum delay minus the minimum
delay of the packets forwarded by the DUT/SUT.
Measurement units:
Seconds (instantaneous)
Seconds P-P (peak to peak)
Seconds avg (average)
Issues:
See Also:
3.11 Nonconforming
Definition:
A packet in violation of a given policy.
Discussion:
A packet can be said to be nonconforming when it violates the
terms of a given policy. For example, a policy may set an upper
bound on bandwidth for a given class of traffic. When a DUT/SUT
is offered packets of that class in excess of the terms allowed
by the policy, the DUT/SUT must discard some packets of that
class. The discarded packets are nonconforming to policy.
Note that the decision to drop or forward packets is
essentially time-based. Even though two packets may share a
common DSCP or IP precedence value, one may be forwarded and
one may be dropped depending on when the packets are offered to
the DUT. It is the combination of classification and policy
that determines whether packets are conforming or
nonconforming.
Measurement units:
n/a
Issues:
Perser, Newman, Khurana, Erramilli, Poretsky [Page 9]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
See Also:
3.12 Offered Load Vector
Definition:
A vector describing the rate at which packets having a specific
code-point are offered to the DUT/SUT, for each code-point in a
code-point set.
Discussion:
Offered loads across the different code-point classes,
constituting a code-point set, determine the metrics associated
with a specific code-point traffic class.
Measurement Units:
bits per second
N-octets per second
Issues:
Packet size.
Traffic descriptors such as peak rate, average rate etc.
See Also:
Expected output vector
3.13 Policy
Definition:
A rule or set of rules used to classify packets.
Discussion:
In the context of data networking, policies consist of one or
(usually) more rules to administer, manage, and control access
to network resources. These resources include applications,
devices, bandwidth, and any other elements attached to a common
network.
In the context of network devices, PHBs are the mechanisms by
which a policy may be enforced. Packets offered to a DUT/SUT
may be conforming or nonconforming to a given policy. A PHB may
cause some packets to be dropped or forwarded based on the
rules laid down by a given policy.
Measurement units:
n/a
Issues:
See Also:
Perser, Newman, Khurana, Erramilli, Poretsky [Page 10]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
Conforming
Nonconforming
PHB
3.14 Provision
Definition:
Configuration of the DUT/SUT to match a specified rule or rules
in the policy.
Discussion:
A DUT/SUT must be configured in accordance with the rules of a
policy for that policy's rules to be enforced. Provision may be
done on a dynamic or a static basis.
Measurement units:
n/a
Issues:
See Also:
Conforming
Nonconforming
PHB
Policy
3.15 Sequence number
Definition:
A field in the IP payload portion of the packet that is used to
verify the order of the packets on the egress of the DUT/SUT.
Discussion:
The traffic generator sets the sequence number value and the
traffic receiver checks the value upon receipt of the packet.
The traffic generator changes the value on each packet
transmitted based on an algorithm agreed to by the traffic
receiver.
The traffic receiver keeps track of the sequence numbers on a
per stream basis. In addition to number of received packets,
the traffic receiver may also report number of in-sequence
packets, number of out-sequence packets and number of duplicate
packets.
The recommended algorithm to use to change the sequence number
on sequential packets is an incrementing value.
Measurement units:
Perser, Newman, Khurana, Erramilli, Poretsky [Page 11]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
n/a
Issues:
See Also:
Stream
3.16 Shaping
Definition:
DUT/SUT control of code point forwarding rate so that a flow or
stream does not exceed or drop below a specified rate set in
the policy.
Discussion:
[3] defines a behavior aggregate in which a DUT/SUT performs a
specific forwarding treatment on a group of packets. Shaping is
one such treatment: A device may drop or forward packets within
a stream or flow depending on whether the packets are in or out
of the terms of the policy.
Measurement units:
n/a
Issues:
See Also:
Conforming
Nonconforming
Policy
3.17 Stream
Definition:
A group of packets tracked as a single entity by the traffic
receiver. A stream shares a common content such as type (IP,
UDP), frame size, or payload.
Discussion:
Streams are tracked by "sequence number" or "unique signature
field" (RFC 2889). Streams define how individual packet's
statistics are grouped together to form an intelligible
summary.
Common stream groupings would be by egress interface,
destination address, source address, DCSP, or IP precedence. A
stream using sequence numbers can track the ordering of packets
as they transverse the DUT/SUT.
Streams are not restricted to a pair of source and destination
interfaces as long as all packets are tracked as a single
Perser, Newman, Khurana, Erramilli, Poretsky [Page 12]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
entity. A mulitcast stream can be forward to multiple
destination interfaces.
Measurement units:
n/a
Issues:
See Also:
Flow
MicroFlow [3]
Sequence number
3.18 Tail dropping
Definition:
The condition in which a DUT/SUT discards newly arriving
packets.
Discussion:
Every DUT/SUT has a finite amount of traffic it can forward,
beyond which congestion occurs. Once the offered load crosses
the congestion threshold, the device may discard any additional
traffic that arrives until congestion clears.
Tail dropping is typically a function of offered load exceeding
a DUT/SUT's buffer capacity, but other factors internal to the
DUT/SUT may also come into play. In terms of what is externally
observable, tail dropping can be said to occur only when
offered load exceeds channel capacity. Since a DUT/SUT may
buffer traffic on ingress, the actual threshold for tail
dropping may be higher than channel capacity.
Measurement units:
n/a
Issues:
Some congestion management mechanisms seek to avoid tail dropping by
discarding packets before offered load exceeds channel capacity. In
the presence of such mechanisms, neither congestion nor tail
dropping should occur.
See Also:
Channel capacity
Congestion
Perser, Newman, Khurana, Erramilli, Poretsky [Page 13]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
3.19 Unburdened Response
Definition:
A performance measure obtained when mechanisms used to support
DiffServ are disabled.
Discussion:
Enabling Diffserv mechanisms such as scheduling algorithms may
impose an additional processing overhead for packets, which may
cause the aggregate response to suffer even when traffic
belonging to only one class, the best effort class, is offered
to the device. Comparisons with "unburdened performance" may
thus be in order when obtaining metrics to ensure that enabling
Diffserv mechanisms doesn't impose an excessive performance
penalty.
Measurement units:
TBD.
4. Security Considerations
Documents of this type do not directly effect the security of
the Internet or of corporate networks as long as benchmarking
is not performed on devices or systems connected to operating
networks.
5. References
[1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, July 1991.
[2] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, February 1998.
[3] K. Nichols, S. Blake, F. Baker, D. Black,"Definition of the
Differentiated Services Field (DS Field) in the IPv4 and
IPv6 Headers", RFC 2474, December 1998.
[4] S. Blake, D. Black, M. Carlson, E. Davies, Z. Wang, W.
Weiss, "An Architecture for Differentiated Services", RFC
2475, December 1998.
Perser, Newman, Khurana, Erramilli, Poretsky [Page 14]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
6. Author's Address
Jerry Perser
Spirent Communications
26750 Agoura Road
Calabasas, CA 91302
USA
Phone: + 1 818 676 2300
EMail: jerry.perser@spirentcom.com
David Newman
Network Test
321 Newark St., Suite 301
Hoboken, NJ 07030
USA
Phone: + 1 201 798 9999
EMail: dnewman@networktest.com
Sumit Khurana
Telcordia Technologies
445 South Street
Morristown, NJ 07960
USA
Phone: + 1 973 829 3170
EMail: sumit@research.telcordia.com
Shobha Erramilli
Telcordia Technologies
331 Newman Springs Road,
Navesink, NJ 07701
USA
Phone: + 1 732 758 5508
EMail: shobha@research.telcordia.com
Scott Poretsky
Avici Systems
101 Billerica Ave_Building #6
N. Billerica, MA 01862
USA
Phone: + 1 978 964 2287
EMail: sporetsky@avici.com
Perser, Newman, Khurana, Erramilli, Poretsky [Page 15]
INTERNET-DRAFT Differentiated Services Termology Feb. 2001
7. Full Copyright Statement
Copyright (C) The Internet Society (1998). All Rights
Reserved.
This document and translations of it may be copied and
furnished to others, and derivative works that comment on or
otherwise explain it or assist in its implementation may be
prepared, copied, published and distributed, in whole or in
part, without restriction of any kind, provided that the above
copyright notice and this paragraph are included on all such
copies and derivative works. However, this document itself may
not be modified in any way, such as by removing the copyright
notice or references to the Internet Society or other Internet
organizations, except as needed for the purpose of developing
Internet standards in which case the procedures for copyrights
defined in the Internet Standards process must be followed, or
as required to translate it into languages other than English.
The limited permissions granted above are perpetual and will
not be revoked by the Internet Society or its successors or
assigns. This document and the information contained herein is
provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE
INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY
THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY
RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
FOR A PARTICULAR PURPOSE.
Perser, Newman, Khurana, Erramilli, Poretsky [Page 16]