Network Working Group                         S. Poretsky
 Internet Draft                                NextPoint Networks
 Expires: August 2008
 Intended Status: Informational                Shankar Rao
                                               Qwest Communications

                                               February 25, 2008

             Terminology for Accelerated Stress Benchmarking
                 <draft-ietf-bmwg-acc-bench-term-13.txt>

Intellectual Property Rights (IPR) statement:
   By submitting this Internet-Draft, each author represents that any
   applicable patent or other IPR claims of which he or she is aware
   have been or will be disclosed, and any of which he or she becomes
   aware will be disclosed, in accordance with Section 6 of BCP 79.

Status of this Memo

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as
   Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

Copyright Notice
   Copyright (C) The IETF Trust (2008).

ABSTRACT
   This document provides the Terminology for performing Accelerated
   Stress Benchmarking of networking devices.  The three phases of
   the Stress Test: Startup, Instability and Recovery are defined
   along with the benchmarks and configuration terms associated with
   the each phase.  Also defined are the Benchmark Planes fundamental
   to stress testing configuration, setup and measurement.  The
   terminology is to be used with the companion framework and
   methodology documents.

   Table of Contents
     1. Introduction ............................................... 3
     2. Existing definitions ....................................... 3
     3. Term definitions............................................ 4

Poretsky and Rao                                               [Page 1]


INTERNET-DRAFT        Terminology for Accelerated        February 2008
                         Stress Benchmarking

      3.1 General Terms............................................. 4
        3.1.1 Benchmark Planes...................................... 4
        3.1.2 Configuration Sets.................................... 5
        3.1.3 Startup Conditions.................................... 5
        3.1.4 Instability Conditions................................ 6
        3.1.5 Aggregate Forwarding Rate............................. 6
        3.1.6 Discontinued Sessions................................. 7
        3.1.7 Uncontrolled Session Loss............................. 7
      3.2 Benchmark Planes.......................................... 8
        3.2.1 Control Plane......................................... 8
        3.2.2 Data Plane............................................ 8
        3.2.3 Management Plane...................................... 8
        3.2.4 Security Plane........................................ 9
      3.3 Startup...................................................10
        3.3.1 Startup Phase.........................................10
        3.3.2 Benchmarks............................................10
          3.3.2.1 Stable Aggregate Forwarding Rate..................10
          3.3.2.2 Stable Latency....................................11
          3.3.2.3 Stable Session Count..............................11
        3.3.3 Control Plane.........................................12
          3.3.3.1 Control Plane Configuration Set...................12
          3.3.3.2 Control Plane Startup Conditions..................13
        3.3.4 Data Plane............................................13
          3.3.4.1 Data Plane Configuration Set......................13
          3.3.4.2 Traffic Profile...................................14
        3.3.5 Management Plane......................................14
          3.3.5.1 Management Plane Configuration Set................14
        3.3.6 Security Plane........................................15
          3.3.6.1 Security Plane Configuration Set..................15
          3.3.6.2 Security Plane Startup Conditions.................16
      3.4 Instability...............................................17
        3.4.1 Instability Phase.....................................17
        3.4.2 Benchmarks............................................17
          3.4.2.1 Unstable Aggregate Forwarding Rate................17
          3.4.2.2 Aggregate Forwarding Rate Degradation.............18
          3.4.2.3 Average Aggregate Forwarding Rate Degradation.....18
          3.4.2.4 Unstable Latency..................................19
          3.4.2.5 Unstable Uncontrolled Sessions Lost...............19
        3.4.3 Instability Conditions................................20
          3.4.3.1 Control Plane Instability Conditions..............20
          3.4.3.2 Data Plane Instability Conditions.................20
          3.4.3.3 Management Plane Instability Conditions...........21
          3.4.3.4 Security Plane Instability Conditions.............21
      3.5 Recovery..................................................22
        3.5.1 Recovery Phase........................................22
        3.5.2 Benchmarks............................................22
          3.5.2.1 Recovered Aggregate Forwarding Rate...............22
          3.5.2.2 Recovered Latency.................................23
          3.5.2.3 Recovery Time.....................................23
          3.5.2.4 Recovered Uncontrolled Sessions Lost..............24
          3.5.2.5 Variability Benchmarks............................24
     4. IANA Considerations.........................................25

Poretsky and Rao                                               [Page 2]


INTERNET-DRAFT        Terminology for Accelerated        February 2008
                         Stress Benchmarking

     5. Security Considerations.....................................25
     6. Acknowledgements............................................25
     7. References..................................................25
     8. Author's Address............................................26
     Appendix 1 - White Box Benchmarks..............................26

1. Introduction

   Routers in an operational network are configured with multiple
   protocols and security policies while simultaneously forwarding
   traffic and being managed.  To accurately benchmark a router for
   deployment, it is necessary to test that router under operational
   conditions by simultaneously configuring and scaling network
   protocols and security policies, forwarding traffic, and managing
   the device in a lab environment.  It is useful to accelerate these
   network operational conditions so that the router under test can
   be benchmarked in a lab environment with a shorter test duration.
   Testing a router in accelerated network conditions is known as
   Accelerated Stress Benchmarking.

   This document provides the Terminology for performing Stress
   Benchmarking of networking devices.  The three phases of the Stress
   Test: Startup, Instability and Recovery are defined along with the
   benchmark and configuration terms associated with the each phase.
   Benchmarks for stress testing are defined using the Aggregate
   Forwarding Rate and control plane Session Count during each phase
   of the test.  For each plane, the Configuration Set, Startup
   Conditions, and Instability Conditions are defined.  Also defined are
   the Benchmark Planes fundamental to stress testing configuration,
   setup and measurement.  These are the Control Plane, Data Plane,
   Management Plane and Security Plane. Multiple benchmarks are measured
   for each Benchmark Plane during each Phase.  Benchmarks can be
   compared across multiple planes for the same DUT or at the same
   plane for 2 or more DUTS.  Benchmarks of internal DUT characteristics
   such as memory and CPU utilization (also known as White Box
   benchmarks) are described in Appendix 1, to allow additional
   characterization of DUT behavior.  The terminology is to be used with
   the companion methodology document [4].  The sequence of phases,
   actions, and benchmarks are shown in Table 1.

2.  Existing definitions
   RFC 1242 [1] and RFC 2285 [2] should be consulted before
   attempting to make use of this document. For the sake of clarity
   and continuity this RFC adopts the template for definitions set
   out in Section 2 of RFC 1242. Definitions are indexed and grouped
   together in sections for ease of reference.

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in BCP 14, RFC 2119
   [5].  RFC 2119 defines the use of these key words to help make the
   intent of standards track documents as clear as possible.  While this
   document uses these keywords, it is not a standards track document.

Poretsky and Rao                                               [Page 3]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

   Table 1. Phase Sequence and Benchmarks
   III. Recovery Phase   II. Instability Phase    I. Startup Phase
   <-----------------<---<-------------------<----<--------------<
    Remove Instability   Achieve Configuration    Apply Startup
    Conditions           Set and Apply            Conditions
                         Instability Conditions

    Benchmark:           Benchmark:               Benchmark:
    Recovered Aggregate  Unstable Aggregate       Stable Aggregate
    Forwarding Rate      Forwarding Rate          Forwarding Rate

                         Degraded Aggregate
                         Forwarding Rate

                         Average Degraded
                         Forwarding Rate

    Recovered Latency    Unstable Latency         Startup Latency

    Recovered Uncontrolled Recovered Uncontrolled Stable Session Count
    Sessions Lost          Sessions Lost

    Recovery Time

3. Term definitions
   3.1 General Terms
   3.1.1 Benchmark Planes

      Definition:
      The features, conditions, and behavior for the Accelerated Stress
      Benchmarking.

      Discussion:
      There are four Benchmark Planes: Control Plane, Data Plane,
      Management Plane, and Security Plane as shown in Figure 1.
      Configuration, Startup Conditions, Instability Conditions, and
      Failure Conditions used for each test are defined for each of
      these four Benchmark Planes.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Control Plane
        Data Plane
        Management Plane
        Security Plane

Poretsky and Rao                                               [Page 4]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

         ___________             ___________
        |  Control  |           | Management|
        |   Plane   |___     ___|   Plane   |
        |           |   |   |   |           |
         -----------    |   |    -----------
                        \/  \/                 ___________
                      ___________             | Security  |
                     |           |<-----------|  Plane    |
                     |    DUT    |            |           |
                |--->|           |<---|        -----------
                |     -----------     |
                |                     |
                |     ___________     |
                |    |   Data    |    |
                |--->|   Plane   |<---|
                     |           |
                      -----------

        Figure 1.  Router Accelerated Stress Benchmarking Planes

    3.1.2 Configuration Sets

      Definition:
        The offered load, features, and scaling limits used during the
        Accelerated Stress Benchmarking.

      Discussion:
      There are four Configuration Sets: Control Plane Configuration
      Set, Data Plane Configuration Set, Management Plane Configuration
      Set, and Security Plane Configuration Set.  The minimum
      Configuration Set that MUST be used is discussed in the
      Methodology document [4].

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Control Plane Configuration Set
        Data Plane Configuration Set
        Management Plane Configuration Set
        Security Plane Configuration Set

   3.1.3 Startup Conditions

      Definition:
        Test conditions applied at the start of the Accelerated
        Stress Benchmark to establish conditions for the remainder of
        the test.

Poretsky and Rao                                               [Page 5]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

      Discussion:
        Startup Conditions may cause stress on the DUT and produce
        failure.  Startup Conditions are defined for the Control
        Plane and Security Plane.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Control Plane Startup Conditions
        Data Plane Startup Conditions
        Management Plane Startup Conditions
        Security Plane Startup Conditions

   3.1.4 Instability Conditions

      Definition:
        Test conditions applied during the Accelerated Stress
        Benchmark to produce instability and stress the DUT.

      Discussion:
        Instability Conditions are applied to the DUT after the
        Startup Conditions have completed.  Instability Conditions
        occur for the Control Plane, Data Plane, Management Plane,
        and Security Plane.

      Measurement units:
        N/A

      Issues:  None

     See Also:
        Control Plane Instability Conditions
        Data Plane Instability Conditions
        Management Plane Instability Conditions
        Security Plane Instability Conditions

   3.1.5 Aggregate Forwarding Rate

     Definition:
        Sum of forwarding rates for all interfaces on the
        DUT.

      Discussion:
        Each interface of the DUT forwards traffic at some
        measured rate.  The Aggregate Forwarding Rate is the
        sum of forwarding rates for all interfaces on the DUT.



Poretsky and Rao                                               [Page 6]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

      Measurement units:
        pps

      Issues:
        None

      See Also:
        Startup Phase

   3.1.6 Discontinued Sessions

     Definition:
        Control Plane sessions that are intentionally brought
        down during the Stress test.

      Discussion:
        Discontinued Sessions is performed during the test in
        order to stress the DUT by forcing it to tear down Control
        Plane sessions while handling traffic. It is assumed that
        the test equipment is able to control protocol session
        state with the DUT and is therefore able to introduce
        Discontinued Sessions.

      Measurement units:
        None

      Issues:
        None

      See Also:
        Uncontrolled Session Loss

   3.1.7 Uncontrolled Session Loss
     Definition:
        Control Plane sessions that are in the down state
        but were not intentionally brought down during the
        Stress test.

      Discussion:
        The test equipment is able to control protocol
        session state with the DUT.  The test equipment
        is also to monitor for sessions lost with the
        DUT which the test equipment itself did not
        intentionally bring down.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Discontinued Sessions

Poretsky and Rao                                               [Page 7]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

   3.2 Benchmark Planes

   3.2.1 Control Plane
      Definition:
        The Description of the control protocols enabled for
        the Accelerated Stress Benchmarking.

      Discussion:
        The Control Plane defines the Configuration, Startup
        Conditions, and Instability Conditions of the control
        protocols.  Control Plane protocols MAY include routing
        protocols, multicast protocols, and MPLS protocols.
        These can be enabled or disabled for a benchmark test.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Benchmark Planes
        Control Plane Configuration Set
        Control Plane Startup Conditions
        Control Plane Instability Conditions

  3.2.2 Data Plane
      Definition:
        The data traffic profile used for the Accelerated Stress
        Benchmarking.

      Discussion:
        The Data Plane defines the Configuration, Startup
        Conditions, and Instability Conditions of the data
        traffic.  The Data Plane includes the traffic and
        interface profile.

      Issues:
        None

      Measurement Units:
        N/A

      See Also:
        Benchmark Planes
        Data Plane Configuration Set
        Data Plane Startup Conditions
        Data Plane Instability Conditions

   3.2.3 Management Plane
      Definition:
        The Management features and tools used for the
        Accelerated Stress Benchmarking.

Poretsky and Rao                                               [Page 8]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

      Discussion:
        A key component of the Accelerated Stress Benchmarking is the
        Management Plane to assess manageability of the router
        under stress.  The Management Plane defines the Configuration,
        Startup Conditions, and Instability Conditions of the
        management protocols and features.  The Management Plane
        includes SNMP, Logging/Debug, Statistics Collection, and
        management configuration sessions such as telnet, SSH, and
        serial console.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Benchmark Planes
        Management Plane Configuration Set
        Management Plane Startup Conditions
        Management Plane Instability Conditions

   3.2.4 Security Plane

      Definition:
        The Security features used during the Accelerated Stress
        Benchmarking.

      Discussion:
        The Security Plane defines the Configuration, Startup
        Conditions, and Instability Conditions of the security
        features and protocols.  The Security Plane includes the
        ACLs, Firewall, Secure Protocols, and User Login.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Benchmark Planes
        Security Plane Configuration Set
        Security Plane Startup Conditions
        Security Plane Instability Conditions

Poretsky and Rao                                               [Page 9]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

   3.3 Startup

   3.3.1 Startup Phase

     Definition
        The step of the benchmarking test in which the
        Startup Conditions are generated with the DUT.  This
        begins with the attempt to establish the first session
        and ends when the last Control Plane session is
        established.

     Discussion:
        The Startup Phase is the first Phase of the benchmarking
        test preceding the Instability Phase and Recovery Phase.
        It is specified by the Configuration Sets and Startup
        Conditions for each Benchmark Plane.  The Startup Phase ends
        and Instability Phase MUST begin when the Configuration Sets
        are achieved with the DUT.  The DUT MUST be stable and
        without failure during the Startup Phase to continue to the
        Instability Phase.  If there is failure during the Startup
        Phase then the test MUST be restarted with new Startup
        Conditions.

     Measurement Units:
        None

     Issues:
        None

     See Also:
        Benchmark Plane
        Configuration Sets
        Startup Conditions
        Instability Phase
        Recovery Phase

   3.3.2 Benchmarks
   3.3.2.1 Stable Aggregate Forwarding Rate

     Definition:
        Sum of forwarding rates for all interfaces on the
        DUT during the Startup Phase.

      Discussion:
        The Stable Aggregate Forwarding Rate is calculated from
        measurement samples throughout the entire Startup Phase.
        Stable Aggregate Forwarding Rate is the calculated average
        of the samples measured during the Startup Phase.  It is
        RECOMMENDED that the sample measurements be made on every
        DUT interface every 1 second.

      Measurement units:
        pps

Poretsky and Rao                                              [Page 10]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

      Issues:
        The act of the DUT establishing the Startup Conditions
        could influence the forwarding rate in certain
        implementations so that this "baseline" for the
        remainder of the test is lowered.  The alternative is
        to change the definition of Stable Aggregate
        Forwarding Rate so that it is measured after Startup
        Conditions are achieved.  The disadvantage of that
        definition would be that it loses measurement of any
        impact that establishing Startup  Conditions would have
        on forwarding rate. When comparing the Startup Aggregate
        Forwarding Rate benchmark of two devices it is preferred
        to know the impact establishing Startup Conditions has
        on Forwarding Rate.

      See Also:
        Startup Phase
        Aggregate Forwarding Rate

   3.3.2.2 Stable Latency

      Definition:
        Average measured latency of traffic forwarded by the DUT
        during the Startup Phase.

      Discussion:
        Stable Latency is the calculated average Latency during
        the Startup Phase.

      Measurement units:
        seconds

      Issues:
        None

      See Also:
        Startup Phase
        Stable Aggregate Forwarding Rate

   3.3.2.3 Stable Session Count

     Definition:
        Total number of control plane sessions/adjacencies
        established and maintained by the DUT during the Startup
        Phase and prior to Instability Conditions being initiated.

      Discussion:
        This measurement SHOULD be made after the Control
        Plane Startup Conditions are applied to the DUT.

Poretsky and Rao                                              [Page 11]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

      Measurement units:
        sessions

      Issues:
        None

      See Also:
        Startup Phase

   3.3.3 Control Plane

   3.3.3.1 Control Plane Configuration Set

      Definition:
        The control protocols and scaling values used for the
        Accelerated Stress Benchmarking.

      Discussion:
        Control Plane Configuration Set is represented in Figure 2 and
        specifies protocol configurations for protocols such as, but not
        limitied to, Routing, Multicast, SIP and MPLS.  Specific
        protocols can be enabled or disabled for a benchmark test.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
      Data Plane Configuration Set
      Management Configuration Set
      Security Configuration Set

         ____________            ____________            ____________
        |  Routing   |          |  Multicast |          |    MPLS    |
        |  Protocols |___       |  Protocols |        __|  Protocols |
        |            |   |      |            |        | |            |
         ------------    |       ------------         |  ------------
                         |              |             |
                         |              |             |
                         |             \/             |
                         |         ___________        |
                         |        |           |       |
         ____________    +------->|    DUT    |<------+
        |    SIP     |----------->|           |
        |  Sessions  |             -----------
         ------------

                Figure 2.  Control Plane Configuration Module

Poretsky and Rao                                              [Page 12]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

   3.3.3.2 Control Plane Startup Conditions

      Definition:
        Control Plane conditions that occur at the start
        of the Accelerated Stress Benchmarking to establish conditions
        for the remainder of the test.

      Discussion:
        Startup Conditions may cause stress on the DUT and produce
        failure.  Startup Conditions for the Control Plane include
        session establishment rate, number of sessions established
        and number of routes learned.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Startup Conditions
        Security Plane Startup Conditions
        Control Plane Configuration Set

   3.3.4 Data Plane
   3.3.4.1 Data Plane Configuration Set

      Definition:
        The data traffic profile and interfaces that are enabled for
        the Accelerated Stress Benchmarking.

      Discussion:
        Data Plane Configuration Set includes the Traffic Profile and
        interfaces used for the Accelerated Stress Benchmarking.
        The interface type(s) and number of interfaces for each
        interface type MUST be reported.

      Measurement Units:
        N/A

      Issues: None

      See Also:
        Traffic Profile

Poretsky and Rao                                              [Page 13]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

   3.3.4.2 Traffic Profile
       Definition
        The characteristics of the Offered Load to the DUT on each
        interface for the Accelerated Stress Benchmarking.

      Discussion
        The Traffic Profile specifies the number of packet size(s),
        packet rate, number of flows, and encapsulation on a
        per-interface basis used for the offered load to the DUT.


      Measurement Units:
      Traffic Profile is reported as follows:

        Parameter                    Units
        ---------                    ------
        Packet Size(s)               bytes
        Packet Rate(interface)       array of packets per second
        Aggregate Offered Load       pps
        Number of Flows              number of flows
        Traffic Type                 array of (RTP, UDP, TCP, other)
        Encapsulation(flow)          array of encapsulation type
        Mirroring                    enabled/disabled

      Issues:
        None

      See Also:
        Data Plane Configuration Set

  3.3.5 Management Plane
  3.3.5.1 Management Plane Configuration Set

      Definition:
        The router management features enabled for the
        Accelerated Stress Benchmark.

      Discussion:
        A key component of the Accelerated Stress Benchmark is the
        Management Configuration Set to assess manageability of the
        router under stress.  The Management Configuration Set defines
        the management configuration of the DUT.  Features that are
        part of the Management Configuration Set include access, SNMP,
        Logging/Debug, and Statistics Collection, and services such as
        FTP, as shown in Figure 3. These features SHOULD be enabled
        throughout the Stress test.

Poretsky and Rao                                              [Page 14]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

      Measurement units:
      N/A

      Issues:
         None

      See Also:
      Control Plane Configuration Set
      Data Plane Configuration Set
      Security Plane Configuration Set

                          ____________        ____________
                         |            |      |  Logging/  |
                         |    SNMP    |    __|   Debug    |
                         |            |   |  |            |
                          ------------    |   ------------
                                |         |
                                |         |
                                \/        |
                          ___________     |
                         |           |    |
                         |    DUT    |<---|
                         |           |
                          -----------
                              |
                              |
                              \/
                        ___________
                       |   Packet  |
                       | Statistics|
                       | Collector |
                       |           |
                        -----------

                  Figure 3.  Management Plane Configuration Set

   3.3.6 Security Plane
   3.3.6.1 Security Plane Configuration Set

      Definition:
        Security features and scaling enabled for the Accelerated Stress
        Test.

      Discussion:
        The Security Plane Configuration Set includes the configuration
        and scaling of ACLs, Firewall, IPsec, and User Access, as shown
        in Figure 4.  Tunnels SHOULD be established and policies
        configured.  Instability is introduced by flapping tunnels and
        configuring and removing policies.

Poretsky and Rao                                              [Page 15]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

         ____________      ____________         ____________
        |            |    |   Secure   |       |    User    |
        |ACL/Firewall|    |  Protocol  |     __|   Access   |
        |            |    |            |    |  |            |
         ------------      ------------     |   ------------
             |                  |           |
             |                  |           |
             |                  \/          |
             |         ___________          |
             |        |           |         |
             |------->|    DUT    |<--------|
                      |           |
                       -----------
              Figure 4.  Security Configuration Module

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        ACL Configuration Set
        Secure Protocol Configuration Set
        Password Login Configuration Set

   3.3.6.2 Security Plane Startup Conditions
      Definition:
        Security Plane conditions that occur at the start
        of the Accelerated Stress Benchmarking to establish conditions
        for the remainder of the test.

      Discussion:
        Startup Conditions may cause stress on the DUT and produce
        failure.  Startup Conditions for the Security Plane include
        session establishment rate, number of sessions established
        and number of policies learned, and number of user access
        sessions opened.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Startup Conditions
        Data Plane Startup Conditions
        Management Plane Startup Conditions
        Security Plane Startup Conditions

Poretsky and Rao                                              [Page 16]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

   3.4 Instability

   3.4.1 Instability Phase

     Definition:
        The step of the benchmarking test in which the Instability
        Conditions are offered to the DUT.

     Discussion:
        The Instability Phase is the middle phase of the benchmarking
        test following the Startup Phase and preceding the Recovery
        Phase.  The Startup Phase MUST complete without failure to
        begin the Instability Phase.

     Measurement Units:
        None

    Issues:
        None

     See Also:
        Instability Conditions
        Startup Phase
        Recovery Phase

   3.4.2 Benchmarks
   3.4.2.1 Unstable Aggregate Forwarding Rate

     Definition:
        Rate of traffic forwarded by the DUT during the
        Instability Phase.

      Discussion:
        Unstable Aggregated Forwarding Rate is an instantaneous
        measurement of the Aggregate Forwarding Rate during the
        Instability Phase.

      Measurement units:
        pps

      Issues:
        None

      See Also:
        Instability Conditions
        Aggregate Forwarding Rate

Poretsky and Rao                                              [Page 17]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

   3.4.2.2 Aggregate Forwarding Rate Degradation

      Definition:
        The reduction in Aggregate Forwarding Rate during the
        Instability Phase.

      Discussion:
        The Aggregate Forwarding Rate Degradation is calculated
        for each measurement of the Unstable Aggregate Forwarding
        Rate.  The Aggregate Forwarding Rate Degradation is
        calculated by subtracting each measurement of the Unstable
        Aggregate Forwarding Rate from the Stable Aggregate
        Forwarding Rate, such that

        Aggregate Forwarding Rate Degradation=
        Stable Aggregate Forwarding Rate -
            Unstable Aggregate Forwarding Rate

        Ideally, the Aggregate Forwarding Rate Degradation is zero.

      Measurement Units:
        pps

      Issues:
        None

      See Also:
        Instability Phase
        Unstable Aggregate Forwarding Rate

   3.4.2.3 Average Aggregate Forwarding Rate Degradation

      Definition
        DUT Benchmark that is the calculated average of the
        obtained Degraded Forwarding Rates.

      Discussion:
        Average Aggregate Forwarding Rate Degradation=
        (Sum (Stable Aggregate Forwarding Rate) -
        Sum (Unstable Aggregate Forwarding Rate)) / Number of Samples

      Measurement Units:
        pps

      Issues:
        None

      See Also:
        Aggregate Forwarding Rate Degradation


Poretsky and Rao                                              [Page 18]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

   3.4.2.4 Unstable Latency

      Definition:
        The average increase in measured packet latency during
        the Instability Phase compared to the Startup Phase.

      Discussion:
        Latency SHOULD be measured at a fixed interval during the
        Instability Phase.  Unstable Latency is the difference
        between Stable Latency and the average Latency measured
        during the Instability Phase.  It is expected that there
        be an increase in average latency from the Startup Phase
        to the Instability phase, but it is possible that the
        difference be zero.  The Unstable Latency cannot be a
        negative number.

      Measurement units:
        seconds

      Issues:
        None

      See Also:
        Instability Phase
        Stable Latency

   3.4.2.5 Unstable Uncontrolled Sessions Lost

     Definition:
        Control Plane sessions that are in the down state
        but were not intentionally brought down during the
        Instability Phase.

      Discussion:
        The test equipment is able to control protocol
        session state with the DUT.  The test equipment
        is also to monitor for sessions lost with the
        DUT which the test equipment itself did not
        intentionally bring down.

      Measurement units:
        sessions

      Issues:
        None

      See Also:
        Discontinued Sessions
        Uncontrolled Session Loss

Poretsky and Rao                                              [Page 19]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

   3.4.3 Instability Conditions

   3.4.3.1 Control Plane Instability Conditions

      Definition:
        Control Plane conditions that occur during the Accelerated
        Stress Benchmark to produce instability and stress the DUT.

      Discussion:
        Control Plane Instability Conditions are experienced by the DUT
        after the Startup Conditions have completed.  Control Plane
        Instability Conditions experienced by the DUT include session
        loss, route withdrawal, and route cost changes.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Instability Conditions
        Data Plane Instability Conditions
        Management Plane Instability Conditions
        Security Plane Instability Conditions

   3.4.3.2 Data Plane Instability Conditions
     Definition:
        Data Plane conditions that occur during the Accelerated Stress
        Benchmark to produce instability and stress the DUT.

      Discussion:
        Data Plane Instability Conditions are experienced by the DUT
        after the Startup Conditions have completed.  Data Plane
        Instability Conditions experienced by the DUT include interface
        shutdown, link loss, and overloaded links.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Instability Conditions
        Control Plane Instability Conditions
        Management Plane Instability Conditions
        Security Plane Instability Conditions

Poretsky and Rao                                              [Page 20]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

   3.4.3.3 Management Plane Instability Conditions

      Definition:
        Management Plane conditions that occur during the Accelerated
        Stress Benchmark to produce instability and stress the DUT.

      Discussion:
        Management Plane Instability Conditions are experienced by the
        DUT after the Startup Conditions have completed.  Management
        Plane Instability Conditions experienced by the DUT include
        repeated FTP of large files.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Instability Conditions
        Control Plane Instability Conditions
        Data Plane Instability Conditions
        Security Plane Instability Conditions

   3.4.3.4 Security Plane Instability Conditions

     Definition:
        Security Plane conditions that occur during the Accelerated
        Stress Benchmark to produce instability and stress the DUT.

     Discussion:
        Security Plane Instability Conditions are experienced by the DUT
        after the Startup Conditions have completed.  Security Plane
        Instability Conditions experienced by the DUT include session
        loss and uninitiated policy changes.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Instability Conditions
        Control Plane Instability Conditions
        Data Plane Instability Conditions
        Management Plane Instability Conditions


Poretsky and Rao                                              [Page 21]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

   3.5 Recovery
   3.5.1 Recovery Phase

     Definition:
        The step of the benchmarking test in which the
        Startup Conditions are generated with the DUT, but
        the Instability Conditions are no longer offered to
        the DUT.

     Discussion:
        The Recovery Phase is the final Phase of the
        benchmarking test following the Startup Phase and
        Instability Phase. Startup Conditions MUST NOT be
        Restarted.

     Measurement Units: None

     Issues: None

     See Also:
        Startup Conditions
        Startup Phase
        Instability Conditions
        Instability Phase

   3.5.2 Benchmarks
   3.5.2.1 Recovered Aggregate Forwarding Rate
      Definition
        Rate of traffic forwarded by the DUT during the Recovery
        Phase.

      Discussion:
        Recovered Aggregate Forwarding Rate is an instantaneous
        measurement of the Aggregate Forwarding Rate during the
        Recovery Phase.  Ideally, each measurement of the Recovered
        Aggregate Forwarding Rate equals the Stable Aggregate
        Forwarding Rate because the Instability Conditions
        do not exist in both the Startup and Recovery Phases.

      Measurement Units:
        pps

      Issues: None

      See Also:
        Aggregate Forwarding Rate
        Recovery Phase
        Recovered Aggregate Forwarding Rate
        Startup Phase
        Stable Aggregate Forwarding Rate

Poretsky and Rao                                              [Page 22]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

   3.5.2.2 Recovered Latency

      Definition:
        The average increase in measured packet latency during
        the Recovery Phase compared to the Startup Phase.

      Discussion:
        Latency SHOULD be measured at a fixed interval during the
        Recovery Phase.  Unstable Latency is the difference
        between Stable Latency and the average Latency measured
        during the Recovery Phase.  It is expected that there
        be no increase in average latency from the Startup Phase
        to the Recovery Phase.  The Recovered Latency cannot be a
        negative number.

      Measurement units:
        seconds

      Issues: None

      See Also:
        Recovery Phase
        Stable Latency

   3.5.2.3 Recovery Time

      Definition
        The amount of time for the Recovered Aggregate Forwarding
        Rate to become equal to the Stable Aggregate Forwarding Rate.

      Discussion
        Recovery Time is measured beginning at the instant the
        Instability Phase ends until the Recovered Aggregate
        Forwarding Rate equals the Stable Aggregate Forwarding
        Rate for a minimum duration of 180 consecutive seconds.

      Measurement Units:
        milliseconds

      Issues:
        None

      See Also:
        Recovered Aggregate Forwarding  Rate
        Stable Aggregate Forwarding Rate

Poretsky and Rao                                              [Page 23]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

   3.5.2.4  Recovered Uncontrolled Control Plane Sessions Lost

     Definition:
        Control Plane sessions that are in the down state
        but were not intentionally brought down during the
        Recovery Phase.

      Discussion:
        The test equipment is able to control protocol
        session state with the DUT.  The test equipment
        is also to monitor for sessions lost with the
        DUT which the test equipment itself did not
        intentionally bring down.

      Measurement units:
        sessions

      Issues:
        None

      See Also:
        Discontinued Sessions
        Uncontrolled Session Loss

   3.5.2.5 Variability Benchmarks

      Definition:
        The difference between the measured Benchmarks of the
        same DUT over multiple iterations.

      Discussion:
        Ideally, the measured benchmarks should be the same for multiple
        iterations with the same DUT.  Configuration Sets and
        Instability Conditions MUST be held constant for this
        benchmark.  Whether the DUT can exhibit such predictable and
        repeatable behavior is an important benchmark in itself.

      Measurement units:
        As applicable to each Benchmark. The results are to be
        presented in a table format for successive Iterations.
        Ideally, the differences should be zero.

      Issues:
        None

      See Also:
        Startup Period
        Instability Period
        Recovery Period

Poretsky and Rao                                              [Page 24]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

4. IANA Considerations
   This document requires no IANA considerations.

5. Security Considerations
        Documents of this type do not directly affect the security of
        the Internet or of corporate networks as long as benchmarking
        is not performed on devices or systems connected to operating
        networks.

6. Acknowledgements
   The authors would like to thank the BMWG and particularly
   Al Morton, Jay Karthik, and George Jones for their contributions.

7. References

7.1 Normative References
      [1]   Bradner, S., Editor, "Benchmarking Terminology for Network
            Interconnection Devices", RFC 1242, March 1991.

      [2]   Mandeville, R., "Benchmarking Terminology for LAN Switching
            Devices", RFC 2285, June 1998.

      [3]   Bradner, S. and McQuaid, J., "Benchmarking Methodology for
            Network Interconnect Devices", RFC 2544, March 1999.

      [4]   Poretsky, S. and Rao, S., "Methodology Guidelines for
            Accelerated Stress Benchmarking",
            draft-ietf-bmwg-acc-bench-meth-09, work in progress,
            February 2008.

      [5]   Bradner, S., "Key words for use in RFCs to Indicate
            Requirement Levels", RFC 2119, March 1997.

7.2 Informative References
      [RFC3871] Jones, G., "Operational Security Requirements for Large
           Internet Service Provider (ISP) IP Network Infrastructure.",
           IETF RFC 3871 , September 2004.

      [NANOG25] Poretsky, S., "Core Router Evaluation for Higher
                Availability", NANOG 25, June 8, 2002, Toronto, CA.

      [IEEECQR] Poretsky, S., "Router Stress Testing to Validate
                Readiness for Network Deployment",  IEEE CQR 2003.

Poretsky and Rao                                              [Page 25]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

8. Author's Address

      Scott Poretsky
      NextPoint Networks
      3 Federal Street
      Billerica, MA 01821
      USA
      Phone: + 1 508 439 9008
      EMail: sporetsky@nextpointnetworks.com

      Shankar Rao
      1801 California Street
      8th Floor
      Qwest Communications
      Denver, CO 80202 USA
      Phone: + 1 303 437 6643
      Email: shankar.rao@qwest.com

Appendix 1.  White Box Benchmarking Terminology
   Minimum Available Memory
      Definition:
        Minimum DUT Available Memory during the duration of the
        Accelerated Stress Benchmark.

      Discussion:
        This benchmark enables the assessment of resources in the DUT.
        It is necessary to monitor DUT memory to measure this benchmark.

      Measurement units:
        bytes

      Issues: None

      See Also:
        Maximum CPU Utilization

   Maximum CPU Utilization
      Definition:
        Maximum DUT CPU utilization during the duration of the
        Accelerated Stress Benchmark.

      Discussion:
        This benchmark enables the assessment of resources in the DUT.
        It is necessary to monitor DUT CPU Utilization to measure
        this benchmark.

      Measurement units:  %

      Issues: None

      See Also:
        Minimum Available Memory

Poretsky and Rao                                              [Page 26]


INTERNET-DRAFT        Terminology for Accelerated       February 2008
                         Stress Benchmarking

Full Copyright Statement

   Copyright (C) The IETF Trust (2008).

   This document is subject to the rights, licenses and restrictions
   contained in BCP 78, and except as set forth therein, the authors
   retain all their rights.

   This document and the information contained herein are provided
   on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
   REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE
   IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL
   WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY
   WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE
   ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
   FOR A PARTICULAR PURPOSE.

Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at
   http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at ietf-
   ipr@ietf.org.

Acknowledgement

   Funding for the RFC Editor function is currently provided by the
   Internet Society.

Poretsky and Rao                                             [Page 27]