Network Working Group
INTERNET-DRAFT
Expires in: April 2005
Scott Poretsky
Quarry Technologies
Shankar Rao
Qwest Communications
October 2004
Terminology for Accelerated Stress Benchmarking
<draft-ietf-bmwg-acc-bench-term-04.txt>
Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, I certify that any applicable
patent or other IPR claims of which I am aware have been disclosed, or
will be disclosed, and any of which I become aware will be disclosed,
in accordance with RFC 3668.
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in
progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
ABSTRACT
This document provides the Terminology for performing Stress
Benchmarking of networking devices. The three phases of the Stress
Test: Startup, Instability and Recovery are defined along with the
benchmarks and configuration terms associated with the each phase.
Also defined are the Benchmark Planes fundamental to stress testing
configuration, setup and measurement. The terminology is to be
used with the companion framework and methodology documents.
Table of Contents
1. Introduction ............................................... 3
2. Existing definitions ....................................... 3
3. Term definitions............................................ 3
3.1 General Terms............................................. 3
Poretsky and Rao [Page 1]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
3.1.1 Benchmark Planes...................................... 3
3.1.2 Configuration Sets.................................... 4
3.1.3 Startup Conditions.................................... 4
3.1.4 Instability Conditions................................ 5
3.1.5 Aggregate Forwarding Rate............................. 6
3.1.6 Controlled Session Loss............................... 6
3.1.7 Uncontrolled Session Loss............................. 6
3.2 Benchmark Planes.......................................... 7
3.2.1 Control Plane......................................... 7
3.2.2 Data Plane............................................ 7
3.2.3 Management Plane...................................... 8
3.2.4 Security Plane........................................ 8
3.3 Startup................................................... 9
3.3.1 Startup Phase......................................... 9
3.3.2 Benchmarks............................................10
3.3.2.1 Stable Aggregate Forwarding Rate..................10
3.3.2.2 Stable Latency....................................10
3.3.2.3 Stable Session Count..............................11
3.3.3 Control Plane.........................................11
3.3.3.1 Control Plane Configuration Set...................11
3.3.3.2 Control Plane Startup Conditions..................12
3.3.4 Data Plane............................................12
3.3.4.1 Data Plane Configuration Set......................12
3.3.4.2 Traffic Profile...................................13
3.3.5 Management Plane......................................13
3.3.5.1 Management Plane Configuration Set................13
3.3.6 Security Plane........................................14
3.3.6.1 Security Plane Configuration Set..................14
3.3.6.2 Security Plane Startup Conditions.................15
3.4 Instability...............................................15
3.4.1 Instability Phase.....................................15
3.4.2 Benchmarks............................................16
3.4.2.1 Unstable Aggregate Forwarding Rate................16
3.4.2.2 Degraded Aggregate Forwarding Rate................17
3.4.2.3 Average Degraded Aggregate Forwarding Rate........17
3.4.2.4 Unstable Latency..................................17
3.4.2.5 Unstable Uncontrolled Sessions Lost...............18
3.4.3 Instability Conditions................................18
3.4.3.1 Control Plane Instability Conditions..............18
3.4.3.2 Data Plane Instability Conditions.................19
3.4.3.3 Management Plane Instability Conditions...........19
3.4.3.4 Security Plane Instability Conditions.............20
3.5 Recovery..................................................20
3.5.1 Recovery Phase........................................20
3.5.2 Benchmarks............................................21
3.5.2.1 Recovered Aggregate Forwarding Rate...............21
3.5.2.2 Recovered Latency.................................21
3.5.2.3 Recovery Time.....................................22
3.5.2.4 Recovered Uncontrolled Sessions Lost..............22
3.5.2.5 Variability Benchmarks............................23
4. Security Considerations.....................................23
5. References..................................................23
6. Author's Address............................................24
Appendix 1 - White Box Benchmarks..............................24
Poretsky and Rao [Page 2]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
1. Introduction
Routers in an operational network are simultaneously configured with
multiple protocols and security policies while forwarding traffic and
being managed. To accurately benchmark a router for deployment it is
necessary to test that router in operational conditions by
simultaneously configuring and scaling network protocols and security
policies, forwarding traffic, and managing the device. It is helpful
to accelerate these network operational conditions so that the
router under test can be benchmarked with faster test duration.
Testing a router in accelerated network conditions is known as
Accelerated Stress Testing.
This document provides the Terminology for performing Stress
Benchmarking of networking devices. The three phases of the Stress
Test: Startup, Instability and Recovery are defined along with the
benchmark and configuration terms associated with the each phase.
Benchmarks for stress testing are defined using the Aggregate
Forwarding Rate and control plane Session Count during each phase
of the test. Also defined are the Benchmark Planes fundamental to
stress testing configuration, setup and measurement. These are
the Control Plane, Data Plane, Management Plane and Security Plane
For each plane, the Configuration Set, Startup Conditions, and
Instability Conditions are defined. White Box benchmarks are
provided in Appendix 1 for additional DUT behavior measurements.
The terminology is to be used with the companion methodology
document [6].
2. Existing definitions
RFC 1242 "Benchmarking Terminology for Network Interconnect
Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching
Devices" should be consulted before attempting to make use of this
document.
For the sake of clarity and continuity this RFC adopts the template
for definitions set out in Section 2 of RFC 1242. Definitions are
indexed and grouped together in sections for ease of reference.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
this document are to be interpreted as described in RFC 2119.
3. Term definitions
3.1 General Terms
3.1.1 Benchmark Planes
Definition:
The features, conditions, and behavior for the Accelerated Stress
Benchmarking.
Poretsky and Rao [Page 3]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
Discussion:
There are four Benchmark Planes: Control Plane, Data Plane,
Management Plane, and Security Plane as shown in Figure 1. The
Benchmark Planes define the Configuration, Startup Conditions,
Instability Conditions, and Failure Conditions used for the test.
___________ ___________
| Control | | Management|
| Plane |___ ___| Plane |
| | | | | |
----------- | | -----------
\/ \/ ___________
___________ | Security |
| |<-----------| Plane |
| DUT | | |
|--->| |<---| -----------
| ----------- |
| |
| ___________ |
| | Data | |
|--->| Plane |<---|
| |
-----------
Figure 1. Router Accelerated Stress Benchmarking Planes
Measurement units:
N/A
Issues:
None
See Also:
Control Plane
Data Plane
Management Plane
Security Plane
3.1.2 Configuration Sets
Definition:
The features and scaling limits used during the Accelerated Stress
Benchmarking.
Discussion:
There are four Configuration Sets: Control Plane Configuration Set,
Data Plane Configuration Set, Management Plane Configuration Set,
and Security Plane Configuration Set.
Measurement units:
N/A
Poretsky and Rao [Page 4]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
Issues:
None
See Also:
Control Plane Configuration Set
Data Plane Configuration Set
Management Plane Configuration Set
Security Plane Configuration Set
3.1.3 Startup Conditions
Definition:
Test conditions that occur at the start of the Accelerated
Life Benchmark to establish conditions for the remainder of
the test.
Discussion:
Startup Conditions may cause stress on the DUT and produce
failure. Startup Conditions are defined for the Control
Plane and Security Plane.
Measurement units:
N/A
Issues:
None
See Also:
Control Plane Startup Conditions
Data Plane Startup Conditions
Management Plane Startup Conditions
Security Plane Startup Conditions
3.1.4 Instability Conditions
Definition:
Test conditions that occur during the Accelerated Stress
Benchmark to produce instability and stress the DUT.
Discussion:
Instability Conditions are applied to the DUT after the
Startup Conditions have completed. Instability Conditions
occur for the Control Plane, Data Plane, Management Plane,
and Security Plane.
Measurement units:
N/A
Issues:
None
Poretsky and Rao [Page 5]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
See Also:
Control Plane Instability Conditions
Data Plane Instability Conditions
Management Plane Instability Conditions
Security Plane Instability Conditions
3.1.5 Aggregate Forwarding Rate
Definition:
Sum of forwarding rates for all interfaces on the
DUT during the Startup Phase.
Discussion:
Each interface of the DUT forwards traffic at some
measured rate. The Aggregate Forwarding Rate is the
sum of forwarding rates for all interfaces on the DUT.
Measurement units:
pps
Issues:
None
See Also:
Startup Phase
3.1.6 Controlled Session Loss
Definition:
Control Plane sessions that are intentionally brought
down during the Stress test.
Discussion:
The test equipment is able to control protocol
session state with the DUT.
Measurement units:
None
Issues:
None
See Also:
Uncontrolled Session Loss
3.1.7 Uncontrolled Session Loss
Definition:
Control Plane sessions that are in the down state
but were not intentionally brought down during the
Stress test.
Poretsky and Rao [Page 6]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
Discussion:
The test equipment is able to control protocol
session state with the DUT. The test equipment
is also to monitor for sessions lost with the
DUT which the test equipment itself did not
intentionally bring down.
Measurement units:
N/A
Issues:
None
See Also:
Controlled Session Loss
3.2 Benchmark Planes
3.2.1 Control Plane
Definition:
The Description of the control protocols enabled for
the Accelerated Stress Benchmarking.
Discussion:
The Control Plane defines the Configuration, Startup
Conditions, and Instability Conditions of the control
protocols. Control Plane protocols may include routing
protocols, multicast protocols, and MPLS protocols.
These can be enabled or disabled for a benchmark test.
Measurement units:
N/A
Issues:
None
See Also:
Benchmark Planes
Control Plane Configuration Set
Control Plane Startup Conditions
Control Plane Instability Conditions
3.2.2 Data Plane
Definition:
The data traffic profile used for the Accelerated Stress
Benchmarking.
Discussion:
The Data Plane defines the Configuration, Startup
Conditions, and Instability Conditions of the data
traffic. The Data Plane includes the traffic and
interface profile.
Poretsky and Rao [Page 7]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
Measurement Units:
N/A
See Also:
Benchmark Planes
Data Plane Configuration Set
Data Plane Startup Conditions
Data Plane Instability Conditions
3.2.3 Management Plane
Definition:
The Management features and tools used for the
Accelerated Stress Benchmarking.
Discussion:
A key component of the Accelerated Stress Benchmarking is the
Management Plane to assess manageability of the router
under stress. The Management Plane defines the Configuration,
Startup Conditions, and Instability Conditions of the
management protocols and features. The Management Plane
includes SNMP, Logging/Debug, Statistics Collection, and
management configuration sessions such as telnet, SSH, and
serial console. SNMP Gets SHOULD be performed continuously.
Management configuration sessions should be open
simultaneously and be repeatedly open and closed. Open
management sessions should have valid and invalid
configuration and show commands entered.
Measurement units:
N/A
Issues:
None
See Also:
Benchmark Planes
Management Plane Configuration Set
Management Plane Startup Conditions
Management Plane Instability Conditions
3.2.4 Security Plane
Definition:
The Security features used during the Accelerated Stress
Benchmarking.
Discussion:
The Security Plane defines the Configuration, Startup
Conditions, and Instability Conditions of the security
features and protocols. The Security Plane includes the
Poretsky and Rao [Page 8]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
ACLs, Firewall, Secure Protocols, and User Login. Tunnels
for those such as IPsec should be established and flapped.
Policies for Firewalls and ACLs should be repeatedly added
and removed from the configuration via telnet, SSH, or
serial management sessions.
Measurement units:
N/A
Issues:
None
See Also:
Benchmark Planes
Security Plane Configuration Set
Security Plane Startup Conditions
Security Plane Instability Conditions
3.3 Startup
3.3.1 Startup Phase
Definition
The portion of the benchmarking test in which the
Startup Conditions are generated with the DUT. This
begins with the attempt to establish the first session
and ends when the last Control Plane session is
established.
Discussion:
The Startup Phase is the first Phase of the benchmarking
test preceding the Instability Phase and Recovery Phase.
It is specified by the Configuration Sets and Startup
Conditions for each Benchmark Plane. The Startup Phase ends
and Instability Phase may begin when the Configuration Sets
are achieved with the DUT.
Measurement Units:
None
Issues:
The 'last control plane session is established' may not
be a sufficient indicator that steady-state is achieved
and Instability Conditions can be applied to begin the
Instability Phase.
See Also:
Benchmark Plane
Configuration Sets
Startup Conditions
Instability Phase
Recovery Phase
Poretsky and Rao [Page 9]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
3.3.2 Benchmarks
3.3.2.1 Stable Aggregate Forwarding Rate
Definition:
Average rate of traffic forwarded by the DUT during the
Startup Phase.
Discussion:
Stable Aggregate Forwarding Rate is the calculated
average of the Aggregate Forwarding Rates measured during
the Startup Phase.
Measurement units:
pps
Issues:
The act of the DUT establishing the Startup Conditions
could influence the forwarding rate in certain
implementations so that this "baseline" for the
remainder of the test is lowered. The alternative is
to change the definition of Stable Aggregate
Forwarding Rate so that it measured during the Startup
Phase, but after Startup Conditions are achieved.
The disadvantage of this definition would be that it
loses measurement of any impact that establishing
Startup Conditions would have on forwarding rate. When
comparing the Startup Aggregate Forwarding Rate benchmark
of two devices it is preferred to know the impact
establishing Startup Conditions has on Forwarding Rate.
The definition was therefore selected so that Stable
Aggregate Forwarding Rate is calculated from measurement
samples throughout the entire Startup Phase.
See Also:
Startup Phase
Aggregate Forwarding Rate
3.3.2.2 Stable Latency
Definition:
Average measured latency of traffic forwarded by the DUT
during the Startup Phase.
Discussion:
Stable Latency is the calculated average Latency during
the Startup Phase.
Measurement units:
seconds
Issues:
None
Poretsky and Rao [Page 10]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
See Also:
Startup Phase
Stable Aggregate Forwarding Rate
3.3.2.3 Stable Session Count
Definition:
Total number of control plane sessions/adjacencies
established and maintained by the DUT during the Startup
Phase and prior to Instability Conditions being initiated.
Discussion:
This measurement should be made after the Control
Plane Startup Conditions are applied to the DUT.
Measurement units:
sessions
Issues:
None
See Also:
Startup Phase
3.3.3 Control Plane
3.3.3.1 Control Plane Configuration Set
Definition:
The routing protocols and scaling values used for the Accelerated
Life Benchmarking.
Discussion:
Control Plane Configuration Set is shown in Figure 2 and specifies
the Routing Protocols, Multicast, and MPLS configuration. Specific
protocols can be enabled or disabled for a benchmark test.
Measurement units:
N/A
Issues:
None
See Also:
Data Plane Configuration Set
Management Configuration Set
Security Configuration Set
Poretsky and Rao [Page 11]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
____________ ____________ ____________
| Routing | | Multicast | | MPLS |
| Protocols |___ | Protocols | __| Protocols |
| | | | | | | |
------------ | ------------ | ------------
| | |
| | |
| \/ |
| ___________ |
| | | |
|------->| DUT |<------|
``| |
-----------
Figure 2. Control Plane Configuration Module
3.3.3.2 Control Plane Startup Conditions
Definition:
Control Plane conditions that occur at the start
of the Accelerated Stress Benchmarking to establish conditions
for the remainder of the test.
Discussion:
Startup Conditions may cause stress on the DUT and produce
failure. Startup Conditions for the Control Plane include
session establishment rate, number of sessions established
and number of routes learned.
Measurement units:
N/A
Issues:
None
See Also:
Startup Conditions
Security Plane Startup Conditions
Control Plane Configuration Set
3.3.4 Data Plane
3.3.4.1 Data Plane Configuration Set
Definition:
The data traffic profile enabled for the Accelerated Stress
Benchmarking.
Discussion:
Data Plane Configuration Set includes the Traffic Profile and
interfaces used for the Accelerated Stress Benchmarking.
Poretsky and Rao [Page 12]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
Measurement Units:
N/A
Issues:
None
See Also:
Traffic Profile
3.3.4.2 Traffic Profile
Definition
The characteristics of the Offered Load to the DUT used for
the Accelerated Stress Benchmarking.
Discussion
The Traffic Profile specifies the number of packet size(s), packet
rate per interface, number of flows, and encapsulation used for the
offered load to the DUT.
Measurement Units:
Traffic Profile is reported as follows:
Parameter Units
--------- ------
Packet Size(s) bytes
Packet Rate(interface) array of packets per second
Number of Flows number
Encapsulation(flow) array of encapsulation type
Issues:
None
See Also:
Data Plane Configuration Set
3.3.5 Management Plane
3.3.5.1 Management Plane Configuration Set
Definition:
The router management features enabled for the
Accelerated Stress Test.
Discussion:
A key component of the Accelerated Stress Test is the Management
Configuration Set to assess manageability of the router under
stress. The Management Configuration Set defines the management
configuration of the DUT. Features that are part of the
Management Configuration Set include Telnet access, SNMP,
Logging/Debug, and Statistics Collection, and services such as
FTP, as shown in Figure 3. These features should be enabled
throughout the Stress test. SNMP Gets should be made continuously
with multiple FTP and Telnet sessions operating simultaneously.
Poretsky and Rao [Page 13]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
FTP sessions should be opened and closed at varying intervals
and get and put files while open. Telnet sessions should be
opened and closed at varying intervals and enter valid and invalid
show and configuration commands while open.
Measurement units:
N/A
Issues:
None
See Also:
Control Plane Configuration Set
Data Plane Configuration Set
Security Plane Configuration Set
____________ ____________
| | | Logging/ |
| SNMP | __| Debug |
| | | | |
------------ | ------------
| |
| |
\/ |
___________ |
| | |
| DUT |<---|
| |
-----------
|
|
\/
___________
| Packet |
| Statistics|
| Collector |
| |
-----------
Figure 3. Management Plane Configuration Set
3.3.6 Security Plane
3.3.6.1 Security Plane Configuration Set
Definition:
Security features and scaling enabled for the Accelerated Stress
Test.
Poretsky and Rao [Page 14]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
Discussion:
The Security Plane Configuration Set includes the configuration
and scaling of ACLs, Firewall, IPsec, and User Access, as shown
in Figure 4. Tunnels should be established and policies
configured. Instability is introduced by flapping tunnels and
configuring and removing policies.
____________ ____________ ____________
| | | Secure | | User |
|ACL/Firewall| | Protocol | __| Access |
| | | | | | |
------------ ------------ | ------------
| | |
| | |
| \/ |
| ___________ |
| | | |
|------->| DUT |<--------|
| |
-----------
Figure 4. Security Configuration Module
Measurement units:
N/A
Issues:
None
See Also:
ACL Configuration Set
Secure Protocol Configuration Set
Password Login Configuration Set
3.3.6.2 Security Plane Startup Conditions
Definition:
Security Plane conditions that occur at the start
of the Accelerated Stress Benchmarking to establish conditions
for the remainder of the test.
Discussion:
Startup Conditions may cause stress on the DUT and produce
failure. Startup Conditions for the Security Plane include
session establishment rate, number of sessions established
and number of policies learned, and number of user access
sessions opened.
Measurement units:
N/A
Issues:
None
Poretsky and Rao [Page 15]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
See Also:
Startup Conditions
Data Plane Startup Conditions
Management Plane Startup Conditions
Security Plane Startup Conditions
3.4 Instability
3.4.1 Instability Phase
Definition:
The portion of the benchmarking test in which the
Instability Conditions are offered to the DUT.
Discussion:
The Instability Phase is the middle Phase of
of the benchmarking test following the Startup
Phase and preceding the Recovery Phase.
Measurement Units:
None
Issues:
None
See Also:
Instability Conditions
Startup Phase
Recovery Phase
3.4.2 Benchmarks
3.4.2.1 Unstable Aggregate Forwarding Rate
Definition:
Rate of traffic forwarded by the DUT during the
Instability Phase.
Discussion:
Unstable Aggregated Forwarding Rate is an instantaneous
measurement of the Aggregate Forwarding Rate during the
Instability Phase.
Measurement units:
pps
Issues:
None
See Also:
Instability Conditions
Aggregate Forwarding Rate
Poretsky and Rao [Page 16]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
3.4.2.2 Degraded Aggregate Forwarding Rate
Definition:
The reduction in Aggregate Forwarding Rate during the
Instability Phase.
Discussion:
The Degraded Aggregate Forwarding Rate is calculated
for each measurement of the Unstable Aggregate
Forwarding Rate. The Degraded Aggregate Forwarding
Rate is calculated by subtracting each measurement
of the Unstable Aggregate Forwarding Rate from the
Stable Aggregate Forwarding Rate, such that
Degraded Forwarding Rate =
Stable Aggregate Forwarding Rate -
Unstable Aggregate Forwarding Rate
Ideally, the Degraded Aggregate Forwarding Rate is zero.
Measurement Units:
pps
Issues:
None
See Also:
Instability Phase
Unstable Aggregate Forwarding Rate
3.4.2.3 Average Degraded Aggregate Forwarding Rate
Definition
DUT Benchmark that is the calculated average of the
obtained Degraded Forwarding Rates.
Discussion:
Measurement Units:
pps
Issues:
None
See Also:
Degraded Aggregate Forwarding Rate
3.4.2.4 Unstable Latency
Definition:
The average increase in measured packet latency during
the Instability Phase compared to the Startup Phase.
Poretsky and Rao [Page 17]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
Discussion:
Latency SHOULD be measured at a fixed interval during the
Instability Phase. Unstable Latency is the difference
between Stable Latency and the average Latency measured
during the Instability Phase. It is expected that there
be an increase in average latency from the Startup Phase
to the Instability phase, but it is possible that the
difference be zero. The Unstable Latency cannot be a
negative number.
Measurement units:
seconds
Issues:
None
See Also:
Instability Phase
Stable Latency
3.4.2.5 Unstable Uncontrolled Sessions Lost
Definition:
Control Plane sessions that are in the down state
but were not intentionally brought down during the
Instability Phase.
Discussion:
The test equipment is able to control protocol
session state with the DUT. The test equipment
is also to monitor for sessions lost with the
DUT which the test equipment itself did not
intentionally bring down.
Measurement units:
sessions
Issues:
None
See Also:
Controlled Session Loss
Uncontrolled Session Loss
3.4.3 Instability Conditions
3.4.3.1 Control Plane Instability Conditions
Definition:
Control Plane conditions that occur during the Accelerated Stress
Benchmark to produce instability and stress the DUT.
Poretsky and Rao [Page 18]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
Discussion:
Control Plane Instability Conditions are experienced by the DUT
after the Startup Conditions have completed. Control Plane
Instability Conditions experienced by the DUT include session
loss, route withdrawal, and route cost changes.
Measurement units:
N/A
Issues:
None
See Also:
Instability Conditions
Data Plane Instability Conditions
Management Plane Instability Conditions
Security Plane Instability Conditions
3.4.3.2 Data Plane Instability Conditions
Definition:
Data Plane conditions that occur during the Accelerated Stress
Benchmark to produce instability and stress the DUT.
Discussion:
Data Plane Instability Conditions are experienced by the DUT
after the Startup Conditions have completed. Data Plane
Instability Conditions experienced by the DUT include interface
shutdown, link loss, and overloaded links.
Measurement units:
N/A
Issues:
None
See Also:
Instability Conditions
Control Plane Instability Conditions
Management Plane Instability Conditions
Security Plane Instability Conditions
3.4.3.3 Management Plane Instability Conditions
Definition:
Management Plane conditions that occur during the Accelerated
Life Benchmark to produce instability and stress the DUT.
Discussion:
Management Plane Instability Conditions are experienced by the DUT
after the Startup Conditions have completed. Management Plane
Instability Conditions experienced by the DUT include repeated
FTP of large files.
Poretsky and Rao [Page 19]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
Measurement units:
N/A
Issues:
None
See Also:
Instability Conditions
Control Plane Instability Conditions
Data Plane Instability Conditions
Security Plane Instability Conditions
3.4.3.4 Security Plane Instability Conditions
Definition:
Security Plane conditions that occur during the Accelerated
Life Benchmark to produce instability and stress the DUT.
Discussion:
Security Plane Instability Conditions are experienced by the DUT
after the Startup Conditions have completed. Security Plane
Instability Conditions experienced by the DUT include session
loss and uninitiated policy changes.
Measurement units:
N/A
Issues:
None
See Also:
Instability Conditions
Control Plane Instability Conditions
Data Plane Instability Conditions
Management Plane Instability Conditions
3.5 Recovery
3.5.1 Recovery Phase
Definition:
The portion of the benchmarking test in which the
Startup Conditions are generated with the DUT, but
the Instability Conditions are no longer offered to
the DUT.
Discussion:
The Recovery Phase is the final Phase of the
benchmarking test following the Startup Phase and
Instability Phase. Startup Conditions must not be
Restarted.
Poretsky and Rao [Page 20]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
Measurement Units:
None
Issues:
None
See Also:
Startup Conditions
Startup Phase
Instability Conditions
Instability Phase
3.5.2 Benchmarks
3.5.2.1 Recovered Aggregate Forwarding Rate
Definition
Rate of traffic forwarded by the DUT during the Recovery
Phase.
Discussion:
Recovered Aggregate Forwarding Rate is an instantaneous
measurement of the Aggregate Forwarding Rate during the
Recovery Phase. Ideally, each measurement of the Recovered
Aggregate Forwarding Rate equals the Stable Aggregate
Forwarding Rate because the Instability Conditions
do not exist in both the Startup and Recovery Phases.
Measurement Units:
pps
Issues:
None
See Also:
Aggregate Forwarding Rate
Recovery Phase
Recovered Aggregate Forwarding Rate
Startup Phase
Stable Aggregate Forwarding Rate
3.5.2.2 Recovered Latency
Definition:
The average increase in measured packet latency during
the Recovery Phase compared to the Startup Phase.
Discussion:
Latency SHOULD be measured at a fixed interval during the
Recovery Phase. Unstable Latency is the difference
between Stable Latency and the average Latency measured
during the Recovery Phase. It is expected that there
Poretsky and Rao [Page 21]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
be no increase in average latency from the Startup Phase
to the Recovery Phase. The Recovered Latency cannot be a
negative number.
Measurement units:
seconds
Issues:
None
See Also:
Recovery Phase
Stable Latency
3.5.2.3 Recovery Time
Definition
The amount of time for the Recovered Aggregate Forwarding
Rate to become equal to the Stable Aggregate Forwarding Rate.
Discussion
Recovery Time is measured beginning at the instant the
Instability Phase ends until the Recovered Aggregate
Forwarding Rate equals the Stable Aggregate Forwarding
Rate for a minimum duration of 180 consecutive seconds.
Measurement Units:
seconds
Issues:
None
See Also:
Recovered Aggregate Forwarding Rate
Stable Aggregate Forwarding Rate
3.5.2.4 Recovered Uncontrolled Control Plane Sessions Lost
Definition:
Control Plane sessions that are in the down state
but were not intentionally brought down during the
Recovery Phase.
Discussion:
The test equipment is able to control protocol
session state with the DUT. The test equipment
is also to monitor for sessions lost with the
DUT which the test equipment itself did not
intentionally bring down.
Measurement units:
sessions
Poretsky and Rao [Page 22]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
Issues:
None
See Also:
Controlled Session Loss
Uncontrolled Session Loss
3.5.2.5 Variability Benchmarks
Definition:
The difference between the measured Benchmarks of the
same DUT over multiple iterations.
Discussion:
Ideally, the benchmarks measured should be the same for
multiple iterations with the same DUT. Configuration
Sets Instability conditions SHOULD be held constant for
this benchmark. Whether the DUT can exhibit such predictable
and repeatable behavior is an important benchmark in itself.
Measurement units:
As applicable to each Benchmark. The results are to be
presented in a table format for successive Iterations.
Ideally, the differences should be zero.
Issues:
None
See Also:
Startup Period
Instability Period
Recovery Period
4. Security Considerations
Documents of this type do not directly effect the security of
the Internet or of corporate networks as long as benchmarking
is not performed on devices or systems connected to operating
networks.
5. References
[1] Bradner, S., Editor, "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, October 1991.
[2] Mandeville, R., "Benchmarking Terminology for LAN Switching
Devices", RFC 2285, June 1998.
[3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999.
[4] "Core Router Evaluation for Higher Availability", Scott
Poretsky, NANOG 25, June 8, 2002, Toronto, CA.
Poretsky and Rao [Page 23]
[5] "Router Stress Testing to Validate Readiness for Network
Deployment", Scott Poretsky, IEEE CQR 2003.
[6] Poretsky, S. and Rao, S., "Methodology for Accelerated
Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-01,
work in progress, October 2004.
6. Author's Address
Scott Poretsky
Quarry Technologies
8 New England Executive Park
Burlington, MA 01803
USA
Phone: + 1 781 395 5090
EMail: sporetsky@quarrytech.com
Shankar Rao
Qwest Communications
Denver, CO
USA
Phone: + 1 303 437 6643
Email: shankar.rao@qwest.com
Appendix 1. White Box Benchmarking Terminology
Minimum Available Memory
Definition:
Minimum DUT Available Memory during the duration of the
Accelerated Stress Test.
Discussion:
It is necessary to monitor DUT memory to measure this
benchmark.
Measurement units:
bytes
Issues:
None
See Also:
Maximum CPU Utilization
Maximum CPU Utilization
Definition:
Maximum DUT CPU utilization during the duration of the
Accelerated Stress Test.
Poretsky and Rao [Page 24]
INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking October 2004
Discussion:
It is necessary to monitor DUT CPU Utilization to measure
this benchmark.
Measurement units: %
Issues:
None
See Also:
Minimum Available Memory
Intellectual Property Statement
The IETF takes no position regarding the validity or scope of any Intel-
lectual Property Rights or other rights that might be claimed to pertain
to the implementation or use of the technology described in this docu-
ment or the extent to which any license under such rights might or might
not be available; nor does it represent that it has made any independent
effort to identify any such rights. Information on the procedures with
respect to rights in RFC documents can be found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an attempt
made to obtain a general license or permission for the use of such
proprietary rights by implementers or users of this specification can be
obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary rights
that may cover technology that may be required to implement this stan-
dard. Please address the information to the IETF at ietf-ipr@ietf.org.
Disclaimer of Warranty
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR
IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMA-
TION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright Statement
Copyright (C) The Internet Society (2004). This document is subject to
the rights, licenses and restrictions contained in BCP 78, and except as
set forth therein, the authors retain all their rights.
Poretsky and Rao [Page 25]
