Network Working Group
   INTERNET-DRAFT
   Expires in: December 2003
                                                   Scott Poretsky
                                                   Avici Systems

                                                   Shankar Rao
                                                   Qwest Communications

                                                   Ray Piatt
                                                   Cable and Wireless

                                                   June 2003

                        Terminology for Benchmarking
                Core Router Software Accelerated Life Testing

                <draft-ietf-bmwg-acc-bench-term-00.txt>


   Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force  (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other
   documents at any time.  It is inappropriate to use Internet-Drafts
   as reference material or to cite them other than as "work in
   progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   ABSTRACT
        This terminology document provides the terms to be used for
   benchmarking router software under accelerated stress conditions.
   A framework is defined to configure routing protocols, security
   policies, traffic forwarding, and management.  Conditions to
   produce instability and accelerate operational conditions are
   also defined.  Benchmarks for evaluating a router subjected to
   the accelerated life test are introduced.  The DUT configuration
   and accelerated stress conditions emulate those of Internet
   Core routers.

   Table of Contents
     1. Introduction ............................................... 2

Poretsky, Rao, Piatt                                            [Page 1]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
                Router Software Accelerated Life Testing
     2. Existing definitions ....................................... 3
     3. Term definitions............................................ 3
      3.1 Configuration Modules..................................... 3
        3.1.1 Control Plane Configuration Module.................... 4
        3.1.2 Data Plane Configuration Module....................... 5
        3.1.3 Management Configuration Module....................... 6
        3.1.4 Security Configuration Module......................... 7
        3.2 Configuration Sets...................................... 7
        3.2.1 Control Plane Configuration Sets...................... 7
        3.2.1.1 Routing Protocol Configuration Set.................. 7
        3.2.1.2 Multicast Protocol Configuration Set................ 8
        3.2.1.3 MPLS Protocol Configuration Set..................... 9
        3.2.2 Data Plane Configuration Set.......................... 9
        3.2.3 Management Configuration Sets.........................10
        3.2.3.1 User Access Configuration Set.......................10
        3.2.3.2 SNMP Configuration Set..............................10
        3.2.3.3 Logging/Debug Configuration Set.....................11
        3.2.3.4 Packet Statistics Collector Configuration...........11
        3.2.4 Security Configuration Set............................11
      3.2.4.1 ACL Configuration Set.................................12
        3.3 Instability Conditions..................................12
        3.3.1 Interface Shutdown Cycling Rate.......................12
        3.3.2 BGP Route Flap Rate...................................13
        3.3.3 IGP Route Flap Rate...................................13
        3.3.4 Route Convergence Due to Better Next-Hop..............14
        3.3.5 LSP Reroute Rate......................................14
        3.3.6 Intended Test Duration................................15
        3.4 Evaluation Benchmarks...................................15
        3.4.1 Run-Time without Error................................15
        3.4.2 Run-Time without Control Plane Error..................16
        3.4.3 Run-Time without Data Plane Error.....................16
        3.4.4 Run-Time without Management Plane Error...............17
        3.4.5 Run-Time without Security Plane Error.................17
     4. Security Considerations.....................................18
     5. References..................................................18
     6. Author's Address............................................18
     7. Full Copyright Statement....................................19
     Appendix 1 - White Box Benchmarks..............................19

   1. Introduction

   Router testing benchmarks have consistently been made in a
   monolithic fashion in which a single protocol or behavior is
   measured in an isolated environment.  It is important to know the
   limits for a router/switch's (hereby referred to as Router) behavior
   for each protocol, however this does not produce a reliable benchmark
   of the router's behavior in a deployed network.  Routers in an
   operational network are simultaneously configured with multiple
   protocols and security policies while forwarding traffic and being
   managed.  To accurately benchmark a router for deployment it is
   necessary to test that router in operational conditions by
   simultaneously configuring the network protocols and security
   policies, sourcing traffic, and managing the router.  It is helpful

Poretsky, Rao, Piatt                                            [Page 2]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
                Router Software Accelerated Life Testing

   to accelerate these network operational conditions so that the
   router under test can be benchmarked with faster test duration.
   Testing a router in accelerated network conditions is known as
   Accelerated Life Testing.  White Box benchmarks are defined in
   Appendix 1.
   Accelerated Life Testing of Routers provides the following benefits:
   1. Evaluation of multiple protocols enabled simultaneously as
      configured in deployed networks
   2. Evaluation of System and Software Stability
   3. Evaluation of Manageability under stressful conditions
   4. Identification of Software Coding bugs such as:
        a. Memory Leaks
        b. Suboptimal CPU Utilization
        c. Coding Logic

   These evaluations are externally observable to the DUT as control
   plane or data plane errors.

   Configuration and operational conditions vary for Core, Peering,
   Aggregate, and Edge Routers.  This document focuses on defining
   the terminology for Accelerated Life Testing of Core Routers.
   Terminology for configuring network conditions, accelerating
   operational parameters, accelerating network instability, and
   evaluating results are provided.

   2.  Existing definitions

   RFC 1242 "Benchmarking Terminology for Network Interconnect
   Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching
   Devices" should be consulted before attempting to make use of this
   document.

   For the sake of clarity and continuity this RFC adopts the template
   for definitions set out in Section 2 of RFC 1242.  Definitions are
   indexed and grouped together in sections for ease of reference.

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED",  "MAY", and "OPTIONAL" in
   this document are to be interpreted as described in RFC 2119.

   3. Term definitions

   3.1 Configuration Modules

      Definition:
        The features and protocols enabled for the Accelerated Life
        Test and conditions created during the test.

      Discussion:
        Features and Protocols are divided into four configuration
        modules as shown in Figure 1.  These are the Control Plane,
        Data Plane, Management Plane, and Security Plane.

Poretsky, Rao, Piatt                                            [Page 3]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

        Definitions for each of these modules are provided in the
        following sections.
         ___________             ___________
        |  Control  |           | Management|
        |   Plane   |___     ___|   Plane   |
        |   Module  |   |   |   |   Module  |
         -----------    |   |    -----------
                        \/  \/                 ___________
                      ___________             | Security  |
                     |           |<-----------|   Plane   |
                     |    DUT    |            |   Module  |
                |--->|           |<---|        -----------
                |     -----------     |
                |                     |
                |     ___________     |
                |    |   Data    |    |
                |--->|   Plane   |<---|
                     |   Module  |
                      -----------

                Figure 1.  Configuration Modules

      Measurement units: N/A

      Issues: None

      See Also:
        Control Plane Configuration Module
        Data Plane Configuration Module
        Management Configuration Module
        Security Configuration Module

   3.1.1 Control Plane Configuration Module

      Definition:
        The control protocols enabled for the Accelerated Life Test.

      Discussion:
        Control Plane Configuration Module is divided into three
        Configuration Sets as shown in Figure 2.  These are Routing
        Protocol, Multicast, and MPLS.  These can be enabled or
        disabled for a benchmark test.  For example, a network operator
        that operates a network with BGP and an IGP without Multicast
        and MPLS protocols can execute a benchmark test with the
        Routing Protocol Configuration Set enabled and the Multicast
        and MPLS Configuration Sets disabled.  Configuration Sets are
        further explained in the next section.

      Measurement units: Enabled/Disabled

      Issues:None


Poretsky, Rao, Piatt                                            [Page 4]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

         ____________            ____________            ____________
        |  Routing   |          |  Multicast |          |    MPLS    |
        |  Protocol  |___       |  Protocol  |        __|  Protocol  |
        | Config Set |  |       | Config Set |       |  | Config Set |
         ------------   |        ------------        |   ------------
                        |               |            |
                        |               |            |
                        |               \/           |
                        |         ___________        |
                        |        |           |       |
                        |------->|    DUT    |<------|
                                 |           |
                                  -----------
        Figure 2.  Control Plane Configuration Module

      See Also:
        Routing Protocol Configuration Set
        Multicast Protocol Configuration Set
        MPLS Protocol Configuration Set


   3.1.2 Data Plane Configuration Module

      Definition:
          The data traffic and interfaces enabled for the Accelerated
          Life Test.

      Discussion:
        Accelerated Life Testing includes protocols, as defined by the
        Control Plane Configuration Module, and data forwarding, as
        defined by the Data Plane Configuration Module.  As shown in
        Figure 3, the Data Plane Configuration Module has a
        corresponding Traffic Profile Configuration Set.

         ___________
        |  Traffic  |
        |  Profile  |___
        | Config Set|   |
         -----------    |
                        \/
                    ____________
                   |            |
                   |    DUT     |
                   |            |
                    ------------

      Measurement Units:
        Enabled/Disabled

      See Also:
        Traffic Profile Configuration Set
        System Configuration Set

Poretsky, Rao, Piatt                                            [Page 5]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

   3.1.3 Management Configuration Module

      Definition:
        The router management features enabled for the
        Accelerated Life Test.

      Discussion:
        A key component of the Accelerated Life Test is the
        Management Configuration Module to assess manageability
        of the router under stress.  The Management Configuration
        Module can be divided into User Access Configuration Set,
        SNMP Configuration Set, Logging/Debug Configuration Set,
        and Packet Statistics Collector Configuration Set as
        shown in Figure 4.

         ____________     ____________        ____________
        |    User    |   |            |      |  Logging/  |
        |   Access   |   |    SNMP    |    __|   Debug    |
        | Config Set |   | Config Set |   |  | Config Set |
         ------------     ------------    |   ------------
              |                 |         |
              |                 |         |
              |                \/         |
              |         ___________       |
              |        |           |      |
              |------->|    DUT    |<-----|
                       |           |
                        -----------
                             |
                             |
                            \/
                        ___________
                       |   Packet  |
                       | Statistics|
                       | Collector |
                       | Config Set|
                        -----------

        Figure 4.  Management Configuration Module

      Measurement units:
        Enabled/Disabled

      Issues:
        None

      See Also:
        User Access Configuration Set
        SNMP Configuration Set
        Logging/Debug Configuration Set
        Packet Statistics Collector Configuration Set


Poretsky, Rao, Piatt                                            [Page 6]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

   3.1.4 Security Configuration Module

      Definition:
        Security features enabled for the Accelerated Life Test.

      Discussion:
        Accelerated Life Testing of Core Router can include
        configuration of  Security features.  The Security
        Configuration Module includes the ACL Configuration
        Set, Secure Protocol Configuration Set, and User Login
        Configuration Set as shown in Figure 5.

         ____________        ____________         ____________
        |            |      |   Secure   |       |    User    |
        |    ACL     |      |  Protocol  |     __|    Login   |
        | Config Set |      | Config Set |    |  | Config Set |
         ------------        ------------     |   ------------
             |                  |             |
             |                  |             |
             |                  \/            |
             |           ___________          |
             |          |           |         |
             |------->  |    DUT    |<--------|
                        |           |
                         -----------
        Figure 5.  Security Configuration Module

      Measurement units:
        Enabled/Disabled

      Issues:
          None

      See Also:
        ACL Configuration Set
        Secure Protocol Configuration Set
        Password Login Configuration Set

   3.2 Configuration Sets

   3.2.1 Control Plane Configuration Sets
   3.2.1.1 Routing Protocol Configuration Set

        Definition:
           Table of Routing Protocols configuration for Accelerated
           Life Test.

        Discussion:
           The Routing Protocol Configuration Set is one of the
           Control Plane Configuration Sets BGP, MBGP, ISIS, OSPF are
           the routing protocols that can be enabled or disabled in
           any combination.  The specific protocol parameters used to

Poretsky, Rao, Piatt                                            [Page 7]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

           establish the test conditions are also provided in the table.

        Measurement units:
           PARAMETER                            UNITS
           BGP                                  Enabled/Disabled
           Number of EBGP Peers                 Peers
           Number of IBGP Peers                 Peers
           Number of BGP Route Instances        Routes
           Number of BGP Installed Routes       Routes

           MBGP                                 Enabled/Disabled
           Number of MBGP Route Instances       Routes
           Number of MBGP Installed Routes      Routes

           ISIS                                 Enabled/Disabled
           ISIS-TE                              Enabled/Disabled
           Number of ISIS Adjacencies           Adjacencies
           Number of ISIS Routes                Routes
           Number of Nodes per Area             Nodes

           OSPF                                 Enabled/Disabled
           OSPF-TE                              Enabled/Disabled
           Number of OSPF Adjacencies           Adjacencies
           Number of OSPF Routes                Routes
           Number of Nodes per Area             Nodes

      Issues:
          None

      See Also:
          Control Plane Configuration Module
          Multicast Protocol Configuration Set
          MPLS Protocol Configuration Set

   3.2.1.2 Multicast Protocol Configuration Set

      Definition:
        Table of Multicast Protocols configuration for Accelerated
        Life Test.

      Discussion:
           The Multicast Protocol Configuration Set is one of the
           Control Plane Configuration Sets PIM-SM and MSDP are
           the routing protocols that can be enabled or disabled in
           any combination.  The specific protocol parameters used to
           establish the test conditions are also provided in the table.

      Measurement units:
           PARAMETER                            UNITS
           PIM-SM                               Enabled/Disabled
           RP                                   Enabled/Disabled
           Number of Multicast Groups           Groups

Poretsky, Rao, Piatt                                            [Page 8]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

           MSDP                                 Enabled/Disabled

      Issues:
          None

      See Also:
          Control Plane Configuration Module
          Routing Protocol Configuration Set
          MPLS Protocol Configuration Set

   3.2.1.3 MPLS Protocol Configuration Set

      Definition:
        Table of MPLS Protocols configuration for Accelerated Life Test.

      Discussion:
        The MPLS Protocol Configuration Set is one of the Control Plane
        Configuration Sets MPLS-TE (RSVP-TE, ISIS-TE, OSPF-TE) and LDP
        are the routing protocols that can be enabled or disabled in any
        combination.  The specific protocol parameters used to establish
        the test conditions are also provided in the table.

      Measurement units:
           PARAMETER                            UNITS
           MPLS-TE
           Number of Ingress Tunnels            Tunnels
           Number of Mid-Point Tunnels          Tunnels
           Number of Egress Tunnels             Tunnels

           LDP
           Number of Sessions                   Sessions
           Number of FECs                       FECs

      Issues:
           None

      See Also:
          Control Plane Configuration Module
          Routing Protocol Configuration Set
          Multicast Protocol Configuration Set

   3.2.2 Data Plane Configuration Sets

   3.2.2.1 Data Plane Configuration Set

      Definition:
        Table of Forwarded Traffic configuration for Accelerated Life
        Test.

      Discussion:

      Measurement units:

Poretsky, Rao, Piatt                                            [Page 9]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

        PARAMETER                               UNITS
        Traffic Forwarding                      Enabled/Disabled
        Packet Size                             Byte Size Distribution
        Number of Flows                         Flows
        Offered Load                            bps (or pps) Aggregate
        Number of Interfaces                    Interfaces

      Issues:
        None

      See Also:
        Data Plane Configuration Module

   3.2.3 Management Configuration Sets
   3.2.3.1 User Access Configuration Set

      Definition:
        Table of User Access methods during Accelerated Life Test.

      Discussion:

      Measurement units:
        PARAMETER                               UNITS
        Telnet Rate                             Sessions/Hour
        FTP Rate                                Sessions/Hour
        Concurrent Telnet Sessions              Sessions
        Concurrent FTP Session                  Sessions
        SSH                                     Enabled/Disabled
        RADIUS                                  Enabled/Disabled
        TACACS                                  Enabled/Disabled

      Issues:
        None

      See Also:
        Management Configuration Module
        SNMP Configuration Set
        Logging/Debug Configuration Set
        Packet Statistics Collector Configuration Set

   3.2.3.2 SNMP Configuration Set

      Definition:
        Table of SNMP Configuration for the Accelerated Life Test.

      Discussion:

      Measurement units:
        SNMP GET Rate                           SNMP Gets/minute

      Issues:
        None

Poretsky, Rao, Piatt                                            [Page 10]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

      See Also:
        Management Configuration Module
        User Access Configuration Set
        Logging/Debug Configuration Set
        Packet Statistics Collector Configuration Set

   3.2.3.3 Logging/Debug Configuration Set

      Definition:
        Table of Logging and Debug configuration for the
        Accelerated Life Test.

      Discussion:

      Measurement units:
        Logging                                 Enabled/Disabled
        Protocol Debug                          Enabled/Disabled


      Issues:
        None

      See Also:
        Management Configuration Module
        User Access Configuration Set
        SNMP Configuration Set
        Packet Statistics Collector Configuration Set

   3.2.3.4 Packet Statistics Collector Configuration Set

     Definition:
        Table of Packet Statistics Collection for the
        Accelerated Life Test.

      Discussion:

      Measurement units:
        Packet Statistics Collector             Enabled/Disabled
        Sampling Rate                           X:1 packets

      Issues:
        None

      See Also:
        Management Configuration Module
        User Access Configuration Set
        SNMP Configuration Set
        Logging/Debug Configuration Set

   3.2.4 Security Configuration Sets


Poretsky, Rao, Piatt                                           [Page 11]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

  3.2.4.1 ACL Configuration Set

      Definition:
        Table of ACL configuration for the Accelerated Life Test

      Discussion:

      Measurement units:
        Access-Control Lists                    Enabled/Disabled
        Number of ACLs                          ACLs
        ACL Depth                               ACEs

      Issues:
        None

      See Also:
        Security Configuration Module

   3.3 Instability Conditions

      Definition:
        Test conditions that occur during the Accelerated Life Test
        that are typical of instability in an operational network.
        These conditions occur at a much greater rate during the
        test.

      Discussion:
        Configuration Modules and Configuration Sets establish the
        configuration of the DUT for the Accelerated Life Test.
        Instability Conditions are events that occur during the
        Accelerated Life Test to stress the router.

      Measurement units:
        N/A

      Issues:
        None

      See Also:
        Interface Shutdown Cycling Rate
        BGP Route Flap Rate
        IGP Route Flap Rate
        Route Convergence Due to Better Next-Hop
        LSP Reroute Rate
        Target Run Time

   3.3.1 Interface Shutdown Rate

      Definition:
        The rate at which physical interfaces are shutdown on the DUT.

      Discussion:

Poretsky, Rao, Piatt                                           [Page 12]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003

           Router Software Accelerated Life Testing
        This instability condition is an administrative shutdown or remote
        loss of carrier for every interface of the DUT.  This produces great
      instability on the DUT due to continuous protocol session loss and
        convergence.

      Measurement units:
        number of interface shutdowns per minute

      Issues:
        None

      See Also:
        Instability Conditions
        Route Flap Rate
        Target Run Time

   3.3.2 BGP Route Flap Rate

      Definition:
        The rate at which BGP routes installed in the route table are
        flapped.

      Discussion:
        This instability condition defines the rate of repeated
        installation, withdrawal,and installation of routes installed in
        the route table. This produces great instability on the DUT due
        to continuous changes to the forwarding table.

      Measurement units:
        routes flapped per second



      Issues:
        None

      See Also:
        Instability Conditions
        Interface Shutdown Cycling Rate
        Target Run Time

  3.3.3 IGP Route Flap Rate

      Definition:
        The rate at which IGP routes installed in the route table are
        flapped.

      Discussion:
        This instability condition defines the rate of repeated
        installation, withdrawal,and installation of routes installed in
        the route table. This produces great instability on the DUT due
        to continuous changes to the forwarding table.

Poretsky, Rao, Piatt                                           [Page 13]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing
      Measurement units:
        routes flapped per second

      Issues:
        None

      See Also:
        Instability Conditions
        Interface Shutdown Cycling Rate
        Target Run Time

  3.3.4 Route Convergence Due to Better Next-Hop

      Definition:
        The rate at which routes in the FIB are replaced by a route
        instance in the RIB with a different next-hop becoming more
        preferred.

      Discussion:
        The route changes are due to attribute and TLV changes. RIB
        contains routes matching those in the FIB.  Convergence due
        to better path can occur for BGP and IGP routes.

      Measurement units:
        more preferred routes per second

      Issues:
        None

      See Also:
        Instability Conditions
        Interface Shutdown Cycling Rate
        Target Run Time

   3.3.5 LSP Reroute Rate

     Definition:
        The rate at which established LSPs experience a failure condition
        and are forced to reroute.

      Discussion:
        This instability condition defines the rate of repeated LSP
        reroutes.  This produces great instability on the DUT due to
        continuous tunnel path calculations and changes to the
        forwarding table.  Rerouting can be performed with Headend
        Reroute, Standby LSP, or Fast Reroute.

      Measurement units:
        routes flapped per second

      Issues:
        None


Poretsky, Rao, Piatt                                           [Page 14]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

      See Also:
        Instability Conditions
        Interface Shutdown Cycling Rate
        Target Run Time

   3.3.6 Intended Test Duration

      Definition:
        The maximum targetted run time for Accelerated Life Test.

      Discussion:
        This is an instability condition because the protocol sessions
        and scaling values must be maintained for this specified time with
        interfaceshutdowns and route flapping.  The test may be halted
        upon reaching the Intended Test Duration.

      Measurement units:
        Hours

      Issues:
        None

      See Also:
        Instability Conditions
        Interface Shutdown Cycling Rate
        Route Flap Rate


   3.4 Evaluation Benchmarks

   3.4.1 Run-Time without Error

      Definition:
        The run-time for the Accelerated Life Test without the DUT
        exhibiting an error in the control plane, data plane, management
        plane, or security plane.

      Discussion:
        For a successful test, the Run-Time Without Error will equal the
        Intended Test Duration.  In the event of any failure, the Run-Time
        Without Error is less than the Intended Test Duration and will equal
        the maximum of the Run-Time without Control Plane Error, Run-Time
        without Data Plane Error, Run-Time without Management Plane Error,
        and Run-Time without Security Plane Error.

      Measurement units:
        Hours and Minutes

      Issues:
        None

      See Also:

Poretsky, Rao, Piatt                                           [Page 15]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

        Intended Test Duration
        Run-Time without Control Plane Error
        Run-Time without Data Plane Error
        Run-Time without Management Plane Error
        Run-Time without Security Plane Error

   3.4.2 Run-Time without Control Plane Error

      Definition:
        The run-time for the Accelerated Life Test without the DUT
        exhibiting an error in the control plane to Routing, Multicast,
        and MPLS Protocol.

      Discussion:
        For a successful test, the Run-Time Without Control Plane Error
        will equal the Run-Time without Error.  In the event of a Control
        Plane failure, the Run-Time Without Control Plane Error is less
        than the Run-Time without Error.

      Measurement units:
        Hours and Minutes

      Issues:
        None

      See Also:
        Intended Test Duration
        Run-Time without Error
        Run-Time without Data Plane Error
        Run-Time without Management Plane Error
        Run-Time without Security Plane Error

   3.4.3 Run-Time without Data Plane Error

       Definition:
        The run-time for the Accelerated Life Test without the DUT
        exhibiting an error in the Data Plane for traffic forwarding
        or DUT interfaces.

      Discussion:
        For a successful test, the Run-Time Without Data Plane Error
        will equal the Run-Time without Error.  In the event of a Data
        Plane failure, the Run-Time Without Data Plane Error is less
        than the Run-Time without Error.

      Measurement units:
        Hours and Minutes

      Issues:
        None

      See Also:

Poretsky, Rao, Piatt                                           [Page 16]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

        Intended Test Duration
        Run-Time without Error
        Run-Time without Control Plane Error
        Run-Time without Management Plane Error
        Run-Time without Security Plane Error

   3.4.4 Run-Time without Management Plane Error

      Definition:
        The run-time for the Accelerated Life Test without the DUT
        exhibiting an error in the Management Plane to User Access,
        SNMP, or Logging/Debug.

      Discussion:
        For a successful test, the Run-Time Without Management Plane
        Error will equal the Run-Time without Error.  In the event of
        a Management Plane failure, the Run-Time Without Management
        Plane Error is less than the Run-Time without Error.

      Measurement units:
        Hours and Minutes

      Issues:
        None

      See Also:
        Intended Test Duration
        Run-Time without Error
        Run-Time without Control Plane Error
        Run-Time without Data Plane Error
        Run-Time without Security Plane Error

   3.4.5 Run-Time without Security Plane Error

      Definition:
        The run-time for the Accelerated Life Test without the DUT
        exhibiting an error in the Security Plane to ACLs.

      Discussion:
        For a successful test, the Run-Time Without Security Plane
        Error will equal the Run-Time without Error.  In the event
        of a Security Plane failure, the  Run-Time Without Control
        Plane Error is less than the Run-Time without Error.

      Measurement units:
        Hours and Minutes

      Issues:
        None

      See Also:
        Intended Test Duration

Poretsky, Rao, Piatt                                           [Page 17]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

        Run-Time without Error
        Run-Time without Control Plane Error
        Run-Time without Data Plane Error
        Run-Time without Management Plane Error

   4. Security Considerations
        Documents of this type do not directly effect the security of
        the Internet or of corporate networks as long as benchmarking
        is not performed on devices or systems connected to operating
        networks.

   5. References

      [1]   Bradner, S., Editor, "Benchmarking Terminology for Network
            Interconnection Devices", RFC 1242, July 1991.

      [2]   Mandeville, R., "Benchmarking Terminology for LAN Switching
            Devices", RFC 2285, June 1998.

      [3]   Bradner, S. and McQuaid, J., "Benchmarking Methodology for
            Network Interconnect Devices", RFC 2544, March 1999.

      [4]   "Core Router Evaluation for Higher Availability", Scott
            Poretsky, NANOG 25, June 8, 2002, Toronto, CA.

      [5]   "Router Stress Testing to Validate Readiness for Network
            Deployment", Scott Poretsky, IEEE CQR 2003.

   6. Author's Address

        Scott Poretsky
        Avici Systems
        101 Billerica Avenue
        N. Billerica, MA 01862
        USA

        Phone: + 1 978 964 2287
        EMail: sporetsky@avici.com

        Shankar Rao
        Qwest Communications
        Denver, CO
        USA

        Phone: + 1 303 437 6643
        Email: srao@qwest.net

        Ray Piatt
        Cable and Wireless
        11700 Plaza America Drive
        Reston, VA 20190
        USA

Poretsky, Rao, Piatt                                           [Page 18]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

        Phone: + 1 703 292 2113
        Email: rpiatt@cw.net

   7.  Full Copyright Statement

        Copyright (C) The Internet Society (1998).  All Rights
        Reserved.

        This document and translations of it may be copied and
        furnished to others, and derivative works that comment on or
        otherwise explain it or assist in its implementation may be
        prepared, copied, published and distributed, in whole or in
        part, without restriction of any kind, provided that the above
        copyright notice and this paragraph are included on all such
        copies and derivative works.  However, this document itself may
        not be modified in any way, such as by removing the copyright
        notice or references to the Internet Society or other Internet
        organizations, except as needed for the purpose of developing
        Internet standards in which case the procedures for copyrights
        defined in the Internet Standards process must be followed, or
        as required to translate it into languages other than English.

        The limited permissions granted above are perpetual and will
        not be revoked by the Internet Society or its successors or
        assigns.  This document and the information contained herein is
        provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE
        INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES,
        EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY
        THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY
        RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
        FOR A PARTICULAR PURPOSE.

   Appendix 1.  White Box Benchmarking Terminology
   Appendix 1.1   Minimum Available Memory

      Definition:
        Minimum DUT Available Memory during the duration of the
        Accelerated Life Test.

      Discussion:
        It is necessary to monitor DUT memory to measure this
        benchmark.

      Measurement units:
        bytes

      Issues:
        None

      See Also:
        Maximum CPU Utilization


Poretsky, Rao, Piatt                                            [Page 19]


INTERNET-DRAFT Terminology for Benchmarking Software Core   June 2003
           Router Software Accelerated Life Testing

   Appendix 1.2 Maximum CPU Utilization

      Definition:
        Maximum DUT CPU utilization during the duration of the
        Accelerated Life Test.

      Discussion:
        It is necessary to monitor DUT CPU Utilization to measure
        this benchmark.

      Measurement units:
        %

      Issues:
        None

      See Also:
        Minimum Available Memory



































Poretsky, Rao, Piatt                                            [Page 20]