Network Working Group R. Craig
INTERNET-DRAFT Cisco Systems
Expires in six months March 1997
Terminology for Cell/Call Benchmarking
<draft-ietf-bmwg-call-01.txt>
Status of this Memo
This document is an Internet-Draft. Internet-Drafts are working
documents of the Internet Engineering Task Force (IETF), its areas,
and its working groups. Note that other groups may also distribute
working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet- Drafts as reference
material or to cite them other than as ``work in progress.''
To learn the current status of any Internet-Draft, please check the
``1id-abstracts.txt'' listing contained in the Internet- Drafts
Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe),
munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or
ftp.isi.edu (US West Coast).
Abstract
The purpose of this draft is to add terminology specific to the cell
and call-based switch environment to that defined by the Benchmarking
Methodology Working Group (BMWG) of the Internet Engineering Task
Force (IETF) in RFC1242.
While primarily directed towards wide area switches, portions of the
document may be useful for benchmarking other devices such as ADSU's.
1. Introduction
In light of the increasing use of cell-based and/or circuit-switched
transport layers in building networks, it would be useful to develop
a set of benchmarks with which to compare technologies,
implementation strategies, and products.
1.1 Terminology Brought Forward
The terminology defined in RFC 1242 applies equally well to this
memo. There is also a certain amount of overlap with terms
defined in draft-ietf-bmwg-lanswitch-00.txt.
Benchmarking Methodology Working Group [Page 1]
INTERNET-DRAFT Cell/Call Benchmarking Terminology March 1997
2. Definition Format (from RFC1242)
Term to be defined.
Definition:
The specific definition for the term.
Discussion:
A brief discussion of the term, its application and any
restrictions on measurement procedures.
Measurement units:
Units used to record measurements of this term, if applicable.
3. Term Definitions
3.1 Virtual Circuit
This group applies to those switches that are connection-oriented.
3.1.1 Call setup time
Definition: the length of time for the virtual circuit to be
established.
Discussion: as measured from the initiation of the signalling to
circuit establishment.
Measurement units: fractional seconds
Issues:
See also:
3.1.2 Call setup rate (sustained)
Definition: the maximum sustained rate of successful connection
establishment.
Discussion: without loss of existing calls.
Measurement units: calls per second
Issues:
See also:
3.1.3 Call maintenance overhead
Benchmarking Methodology Working Group [Page 2]
INTERNET-DRAFT Cell/Call Benchmarking Terminology March 1997
Definition: the amount of work required to maintain the calls
that have been established.
Discussion: a method to obtain the desired result would be to
benchmark with PVC's in place, then with SVC's. The difference in
results would be the overhead.
Measurement units:
Issues:
See also:
3.1.4 Call teardown time
Definition: the length of time for the virtual circuit to be torn
down.
Discussion: measured from the start of the signalling to the
freeing of the resources associated with that call (end to end, if
applicable).
Measurement units: fractional seconds
Issues:
See also:
3.1.5 Call teardown rate (sustained)
Definition: the maximum rate at which calls can be successfully
torn down.
Discussion: without loss of existing calls, and without failure
to tear down any calls that have been signalled to be destroyed.
Measurement units: teardowns per second
Issues:
See also:
3.1.6 Impact of Signalling on Forwarding
Definition: cells per second versus calls per second
Discussion: some devices use the same engine for cell forwarding
and call maintenance. In this case, interaction between the two
Benchmarking Methodology Working Group [Page 3]
INTERNET-DRAFT Cell/Call Benchmarking Terminology March 1997
functions will be inevitable. More interesting, however, would be
the case where the two processing functions are clearly separate,
yet still interact.
Measurement units: cells per second versus calls per second
Issues:
See also:
3.2 Cell/Packet Interaction
This group applies to cell-based switches, connection-oriented or
not.
3.2.1 Packet disassembly/reassembly time (peak)
Definition: the length of time to disassemble a layer 3 packet
into layer 2 cells, or reassemble cells into a packet.
Discussion: with no packet or cell loss or corruption. To arrive
at a baseline, one could measure the switching rate for cells
derived from ~1440 byte frames which are flowing across the switch
as cells, then forward those same frames into the switch from an
interface which would require them to be disassembled. For
example, the baseline measurement is taken while switching cells
OC3-OC3. Then switch FDDI or POS-OC3 and take the delta in
performance as the SAR overhead.
Measurement units: the appropriate fraction of a second
Issues:
See also:
3.2.2 Packet disassembly/reassembly rate (sustained)
Definition: the maximum sustained rate at which packets can be
disassembled/reassembled into/from cells.
Discussion: without loss or corruption.
Measurement units: packets per second
Issues:
See also:
Benchmarking Methodology Working Group [Page 4]
INTERNET-DRAFT Cell/Call Benchmarking Terminology March 1997
3.2.3 Full packet drop rate (on cell loss)
Definition: the rate at which cell loss triggering full packet
drop can be detected/sustained.
Discussion: When a packet is disassembled into cells, typically
many cells result. When these cells are transmitted, they are
subject to loss or corruption. The device should recognize at the
cell/packet boundary that a cell or cells belonging to a given
packet has been lost and should drop that packet, immediately
freeing those resources. A couple of things are of interest here:
whether the switch is able to detect very small amounts of cell
loss and correctly drop the associated packets and whether large
amounts of cell loss perturb this ability in any way.
Measurement units: (dropped) packets per second
Issues:
See also:
3.2.4 End to end data integrity
Definition: the percentage of packets (post-reassembly) that
actually contain undetected data link layer corruption.
Discussion: some network devices have been known to regenerate
CRC's over the re-assembled packet (i.e., the CRC is not carried
end to end), resulting in undetected data link layer corruption or
re-ordering of cells in a packet.
Measurement units: percentage
Issues: production of a stream of traffic containing internal
checksums sufficiently strong to detect cell re-ordering (the IP
checksum is not). The ISIS LSP checksum is.
See also:
3.3 Switch Fabric
This group applies to all switches.
3.3.2 Topology Table Size
Definition: number of network elements supported.
Discussion: switches may support a limited topology due to static
Benchmarking Methodology Working Group [Page 5]
INTERNET-DRAFT Cell/Call Benchmarking Terminology March 1997
table sizes or processing limitations. This is true whether it's
a "LAN" switch running spanning tree or a "WAN" switch running
OSPF. The effect of a limited topology table on a switch in a
real-world environment can be disastrous.
A similar metric (2.14 Address handling) is mentioned in "draft-
ietf-bmwg-lanswitch-00.txt". Here, a more general metric is
intended.
Measurement units: number
Issues: Measuring the effects of an overflow is probably
meaningless, since in the multi-switch case, there is no longer
any network to speak of, hence, nothing to measure.
If a device handles table overflow gracefully, this should be
noted. Similarly, if a device crashes and burns on table
overflow, this should be noted.
See also:
3.3.3 Topology Table Learning Rate
Definition: the rate at which the topology table can be filled or
updated.
Discussion: a single switch in isolation learning MAC addresses
will flood frames when the rate exceeds its learning capability.
This metric is covered in "2.15 Address learning speed" of
"draft-ietf-bmwg-lanswitch-00.txt". We generalize the metric here
to include the topological databases of routing protocols used in
switched networks (among the switches themselves) as well as the
spanning tree recalculation among multiple LAN switches.
Measurement units: frames per second 1) with maximum diversity of
addresses, 2) with routing instability introduced.
Issues:
See also:
3.3.5 Throughput (from RFC1242) (Cell forwarding rate)
Definition: The maximum rate at which none of the offered frames
are dropped by the device.
Discussion: This metric probably overlaps work being done in the
ATM Forum.
Benchmarking Methodology Working Group [Page 6]
INTERNET-DRAFT Cell/Call Benchmarking Terminology March 1997
Measurement units: cells per second
Issues:
See also:
3.3.6 Blocking Probability
Definition: likelihood of successful simultaneous communication
amongst multiple ports.
Discussion: a switch is termed "non-blocking" if multiple ports
are able to communicate across the switch fabric at the same time.
We are interested in the probability of blocking occurring in the
1:1, N:1, and N:M scenarios.
One may calculate the "ideal" throughput in the absence of
blocking, then take the delta with the experimental case and treat
that as an empirical measurement of blocking probability, if
enough samples are taken.
Measurement units: percentage likelihood of blocking.
Issues:
See also:
3.5 Congestion Control
This group applies to all switches.
3.5.1 Congestion avoidance
Definition: effectiveness of measures taken by the switch to
avoid congestion.
Discussion: connections that are bursting above their committed
rate may have cells buffered at the ingress, in order to avoid
congestion in the trunks and impact on other connections, or they
may simply be marked "discard-eligible" and forwarded into the
network, hoping for the best.
Distinguishing between these two approaches should be relatively
simple. In the first case, latency for the bursting session
increases, but there is no cell loss. Other sessions are
unaffected. In the second case, there may be cell loss across any
of the sessions, and latency may increase across all.
Benchmarking Methodology Working Group [Page 7]
INTERNET-DRAFT Cell/Call Benchmarking Terminology March 1997
Measurement units: dropped cells, latency
Issues:
See also:
3.5.2 Congestion management
Definition: effectiveness of measures taken by the switch to deal
with congestion.
Discussion: in the face of sustained traffic above committed rate
on multiple sessions, a switch has little choice but to begin
discarding cells, since buffering cannot be infinite. This case
might arise if one were wildly profligate in over-subscribing
trunk bandwidth, or if one had neglected to analyze the network
applications to be run over the network and they were found to be
network-hostile (UDP, IPX, AT, NetBIOS, for example).
The switch has some discretion in deciding which cells to drop.
Presumably, the strategy should involve something resembling
"fairness".
The basic idea is that ill-behaved connections should not starve
others for resources.
Measurement units: latency, cell drops
Issues:
See also:
3.6 Inter-switch protocols
This group applies to all switches.
3.6.1 Impact of Routing on Forwarding
Definition: interaction between routing protocol and data
forwarding operations.
Discussion: No amount of routing fluctuation should have an
impact on data forwarding for unaffected destinations. Similarly,
no amount of data forwarding should cause the routing to become
unstable.
Measurement units: route flaps per second versus cells per
second, cells per second versus route stability (table fluctuation
Benchmarking Methodology Working Group [Page 8]
INTERNET-DRAFT Cell/Call Benchmarking Terminology March 1997
or peer loss).
Issues:
See also:
3.6.2 Impact of Congestion Control
Definition: interaction between congestion control and data
forwarding operations.
Discussion: switches may share views of congestion in-band
through the network. Should these feedback messages be delayed or
lost, the potential exists for an incorrect picture of current
network conditions, which may exacerbate congestion and lead to
cell loss. Worse, it is possible to enter a stable oscillation
state, where ever-increasing waves of congestion overwhelm the
switches.
Measurement units:
Issues:
See also:
3.7 Quality of Service
This group applies to all switches.
3.7.1 Traffic Management Policing
Definition: impact of misbehaving class on others, for example
data forwarding on voice or video frames and vice versa.
Discussion: we wish to quantify the potential interaction amongst
the various classes of service. Constant bit rate (CBR), variable
bit rate (VBR) (real and non-real time?), and available bit rate
(ABR) streams are established, within their respective service
levels, but sufficient to subscribe the trunk to 90%. The bit
rate of each is increased until it has exceeded its allocation by
a degree which should cause loss or delay in the other streams.
Measurement units: cells (lost) per second, latency
Issues: some switches perform compression and silence
suppression. Should these features be disabled?
See also:
Benchmarking Methodology Working Group [Page 9]
INTERNET-DRAFT Cell/Call Benchmarking Terminology March 1997
3.8 Multicast
3.8.1 Cell replication
Definition: the device's ability to forward a cell to multiple
ports simultaneously (multicast).
Discussion:
Measurement units: replication factor 1:N and cells per second
measured at ingress versus cells per second measured at the
egresses.
Issues:
See also:
3.8.2 Impact of multicast on unicast
Definition: switch's ability to insulate unicast traffic from the
effects of multicast.
Discussion: a poorly-designed replication scheme could easily
swamp unicast traffic. Yet, multicast traffic often has QoS
needs. How does one reconcile the competing requirements?
Measurement units: cell loss, delay
Issues:
See also:
Security Considerations
Security issues are not addressed in this memo.
Editor's Address
Robert Craig
Cisco Systems
7025 Kit Creek Road
PO Box 14987
Research Triangle Park, NC 27709
(919) 472-2886
rcraig@cisco.com
Benchmarking Methodology Working Group [Page 10]
