Network Working Group                         S. Poretsky
 Internet Draft                                NextPoint Networks
 Expires: August 2008
 Intended Status: Informational                Shankar Rao
                                               Qwest Communications

                                               February 25, 2008

                     Methodology Guidelines for
                   Accelerated Stress Benchmarking
                <draft-ietf-bmwg-acc-bench-meth-09.txt>

Intellectual Property Rights (IPR) statement:
   By submitting this Internet-Draft, each author represents that any
   applicable patent or other IPR claims of which he or she is aware
   have been or will be disclosed, and any of which he or she becomes
   aware will be disclosed, in accordance with Section 6 of BCP 79.

Status of this Memo

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as
   Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

Copyright Notice
   Copyright (C) The IETF Trust (2008).

ABSTRACT
   Routers in an operational network are configured with multiple
   protocols and security policies while simultaneously forwarding
   traffic and being managed.  To accurately benchmark a router for
   deployment it is necessary to test the router in a lab environment
   under accelerated conditions, which is known as Stress Testing.
   This document provides the Methodology Guidelines for performing
   Accelerated Stress Benchmarking of networking devices.
   The methodology is to be used with the companion terminology
   document [4].  These guidelines can be used as the basis for
   additional methodology documents that benchmark stress conditions
   for specific network technologies.

Poretsky and Rao                                               [Page 1]


INTERNET-DRAFT           Methodology Guidelines    February 2008
                      for Accelerated Stress Benchmarking


   Table of Contents
     1. Introduction ............................................... 2
     2. Existing definitions ....................................... 3
     3. Test Setup.................................................. 3
     3.1 Test Topologies............................................ 3
     3.2 Test Considerations........................................ 3
     3.3 Reporting Format........................................... 4
     3.3.1 Configuration Sets....................................... 5
     3.3.2 Startup Conditions....................................... 6
     3.3.3 Instability Conditions................................... 6
     3.3.4 Benchmarks............................................... 7
     4.  Stress Test Procedure...................................... 8
     4.1 General Methodology with Multiple Instability Conditions... 8
     4.2 General Methodology with a Single Instability Condition....10
     5. IANA Considerations.........................................11
     6. Security Considerations.....................................11
     7. Normative References........................................11
     8. Informative References......................................11
     9. Author's Address............................................12

1. Introduction
   Router testing benchmarks have consistently been made in a monolithic
   fashion wherein a single protocol or behavior is measured in an
   isolated environment.  It is important to know the limits for a
   networking device's behavior for each protocol in isolation, however
   this does not produce a reliable benchmark of the device's behavior
   in an operational network.  Routers in an operational network are
   configured with multiple protocols and security policies while
   simultaneously forwarding traffic and being managed.  To accurately
   benchmark a router for deployment it is necessary to test that router
   in operational conditions by simultaneously configuring and scaling
   network protocols and security policies, forwarding traffic, and
   managing the device.  It is helpful to accelerate these network
   operational conditions with Instability Conditions [4] so that the
   networking devices are stress tested.

   This document provides the Methodology for performing Stress
   Benchmarking of networking devices.  Descriptions of Test Topology,
   Benchmarks and Reporting Format are provided in addition to
   procedures for conducting various test cases.  The methodology is
   to be used with the companion terminology document [4].

   Stress Testing of networking devices provides the following benefits:
        1. Evaluation of multiple protocols enabled simultaneously as
        configured in deployed networks
        2. Evaluation of system and software stability
        3. Evaluation of manageability under stressful conditions
        4. Identification of buffer overflow conditions
        5. Identification of software coding bugs such as:
                a. Memory leaks

Poretsky and Rao                                               [Page 2]


INTERNET-DRAFT           Methodology Guidelines    February 2008
                      for Accelerated Stress Benchmarking

                b. Suboptimal CPU utilization
                c. Coding logic

   These benefits produce significant advantages for network operations:
        1.  Increased stability of routers and protocols
        2.  Hardened routers to DoS attacks
        3.  Verified manageability under stress
        4.  Planning router resources for growth and scale

2.  Existing definitions
   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in BCP 14, RFC 2119
   [5].  RFC 2119 defines the use of these key words to help make the
   intent of standards track documents as clear as possible.  While this
   document uses these keywords, this document is not a standards track
   document.

   Terms related to Accelerated Stress Benchmarking are defined in [4].

3. Test Setup
   3.1 Test Topologies
   Figure 1 shows the physical configuration to be used for the
   methodologies provided in this document.  The number of interfaces
   between the tester and DUT will scale depending upon the number of
   control protocol sessions and traffic forwarding interfaces.  A
   separate device may be required to externally manage the device in
   the case that the test equipment does not support such
   functionality.  Figure 2 shows the logical configuration for the
   stress test methodologies.  Each plane MAY be emulated by single or
   multiple test equipment.

   3.2 Test Considerations
   The Accelerated Stress Benchmarking test can be applied in
   service provider test environments to benchmark DUTs under
   stress in an environment that reflects conditions found in
   an operational network.  A particular Configuration Set is
   defined and the DUT is benchmarked using this configuration
   set and the Instability Conditions.  Varying Configuration
   Sets and/or Instability Conditions applied in an iterative
   fashion can provide an accurate characterization of the DUT
   to help determine future network deployments.

   For the management plane SNMP Gets SHOULD be performed
   continuously.  Management sessions SHOULD be open
   simultaneously and be repeatedly open and closed using
   access protocols such as telnet and SSH.  Open management
   sessions SHOULD have valid and invalid configuration and
   show commands entered.  For the security plane, tunnels
   for protocols such as IPsec SHOULD be established and
   flapped.  Policies for Firewalls and ACLs SHOULD be
   repeatedly added and removed via management sessions.

Poretsky and Rao                                               [Page 3]


INTERNET-DRAFT           Methodology Guidelines    February 2008
                      for Accelerated Stress Benchmarking

                                 ___________
                                |   DUT     |
                             ___|Management |
                            |   |           |
                            |    -----------
                           \/
                      ___________
                     |           |
                     |    DUT    |
                |--->|           |<---|
        xN      |     -----------     |    xN
     interfaces |                     | interfaces
                |                     |
                |    |           |    |
                |--->|   Tester  |<---|
                     |           |
                      -----------

                Figure 1. Physical Configuration



         ___________             ___________
        |  Control  |           | Management|
        |   Plane   |___     ___|   Plane   |
        |           |   |   |   |           |
         -----------    |   |    -----------
                       \/  \/                  ___________
                      ___________             | Security  |
                     |           |<-----------|   Plane   |
                     |    DUT    |            |           |
                |--->|           |<---|        -----------
                |     -----------     |
                |                     |
                |     ___________     |
                |    |   Data    |    |
                |--->|   Plane   |<---|
                     |           |
                      -----------

                Figure 2. Logical Configuration


   3.3 Reporting Format
   Each methodology requires reporting of information for test
   repeatability when benchmarking the same or different devices.
   The information that are the Configuration Sets, Instability
   Conditions, and Benchmarks, as defined in [4].  Example
   reporting formats for each are provided below.  Benchmarks
   MUST be reported as provided below.

Poretsky and Rao                                               [Page 4]


INTERNET-DRAFT           Methodology Guidelines    February 2008
                      for Accelerated Stress Benchmarking

   3.3.1 Configuration Sets

   The minimum Configuration Set that MUST be used is as follows:
        PARAMETER                            UNITS
        Number of IGP Adjacencies            Adjacencies
        Number of IGP Routes                 Routes
        Number of Nodes per Area             Nodes
        Number of Areas per Node             Areas
        SNMP GET Rate                        SNMP Gets/minute
        Telnet Establishment Rate            Sessions/Hour
        Concurrent Telnet Sessions           Sessions
        FTP Establishment Rate               Sessions/Hour
        Concurrent FTP Session               Sessions
        SSH Establishment Rate               Sessions/Hour
        Concurrent SSH sessions              Sessions
        DATA TRAFFIC
           Traffic Forwarding                Enabled/Disabled
           Aggregate Offered Load            bps (or pps)
           Number of Ingress Interfaces      interfaces
           Number of Egress Interfaces       interfaces
           Packet Size(s)                    bytes
           Offered Load (interface)          array of bps
           Number of Flows                   flows
           Encapsulation(flow)   array of encapsulation types

   Configuration Sets MAY include and are not limited to the
   following examples.
    Example Routing Protocol Configuration Set-
           PARAMETER                            UNITS
           BGP                                  Enabled/Disabled
           Number of EBGP Peers                 Peers
           Number of IBGP Peers                 Peers
           Number of BGP Route Instances        Routes
           Number of BGP Installed Routes       Routes
           MBGP                                 Enabled/Disabled
           Number of MBGP Route Instances       Routes
           Number of MBGP Installed Routes      Routes
           IGP                                  Enabled/Disabled
           IGP-TE                               Enabled/Disabled
           Number of IGP Adjacencies            Adjacencies
           Number of IGP Routes                 Routes
           Number of Nodes per Area             Nodes
           Number of Areas per Node             Areas

    Example MPLS Protocol Configuration Set-
           PARAMETER                            UNITS
           MPLS-TE                              Enabled/Disabled
           Number of Tunnels as Ingress         Tunnels
           Number of Tunnels as Mid-Point       Tunnels
           Number of Tunnels as Egress          Tunnels
           LDP                                  Enabled/Disabled
           Number of Sessions                   Sessions
           Number of FECs                       FECs

Poretsky and Rao                                               [Page 5]


INTERNET-DRAFT           Methodology Guidelines    February 2008
                      for Accelerated Stress Benchmarking

    Example Multicast Protocol Configuration Set-
           PARAMETER                            UNITS
           PIM-SM                               Enabled/Disabled
           RP                                   Enabled/Disabled
           Number of Multicast Groups           Groups
           MSDP                                 Enabled/Disabled

    Example Data Plane Configuration Set-
        PARAMETER                            UNITS
        Traffic Forwarding                   Enabled/Disabled
        Number of Ingress Interfaces         interfaces
        Number of Egress Interfaces          interfaces

        TRAFFIC PROFILE
        Packet Size(s)               bytes
        Packet Rate(interface)       array of packets per second
        Aggregate Offered Load       pps
        Number of Flows              number of flows
        Traffic Type                 array of (RTP, UDP, TCP, other)
        Encapsulation(flow)          array of encapsulation type
        Mirroring                    enabled/disabled

   Example Management Configuration Set-
        PARAMETER                               UNITS
        SNMP GET Rate                           SNMP Gets/minute
        Logging                                 Enabled/Disabled
        Protocol Debug                          Enabled/Disabled
        Telnet Establishment Rate               Sessions/Hour
        Concurrent Telnet Sessions              Sessions
        FTP Establishment Rate                  Sessions/Hour
        Concurrent FTP Session                  Sessions
        SSH Establishment Rate                  Sessions/Hour
        Concurrent SSH sessions                 Sessions
        Packet Statistics Collector             Enabled/Disabled
        Statistics Sampling Rate                X:1 packets

   Example Security Configuration Set -
        PARAMETER                               UNITS
        Packet Filters                          Enabled/Disabled
        Number of Filters For-Me                filters
        Number of Filter Rules For-Me           rules
        Number of Traffic Filters               filters
        Number of Traffic Filter Rules          rules
        IPsec tunnels                           tunnels
        RADIUS                                  Enabled/Disabled
        TACACS                                  Enabled/Disabled

   Example SIP Configuration Set -
        PARAMETER                               UNITS
        Session Rate                            Sessions per Second
        Media Streams per Session               Streams per session
        Total Sessions                          Sessions

Poretsky and Rao                                               [Page 6]


INTERNET-DRAFT           Methodology Guidelines    February 2008
                      for Accelerated Stress Benchmarking

   3.3.2 Startup Conditions
   Startup Conditions MAY include and are not limited to the following
   examples:
        PARAMETER                               UNITS
        EBGP peering sessions negotiated        Total EBGP Sessions
        IBGP peering sessions negotiated        Total IBGP Sessions
        ISIS adjacencies established            Total ISIS Adjacencies
        ISIS routes learned rate                ISIS Routes per Second
        IPsec tunnels negotiated                Total IPsec Tunnels
        IPsec tunnel establishment rate       IPsec tunnels per second

   3.3.3 Instability Conditions
   Instability Conditions MAY include and are not limited to the
   following examples:
        PARAMETER                               UNITS
        Interface Shutdown Cycling Rate         interfaces per minute
        ISIS Route Flap Rate                    routes per minutes
        LSP Reroute Rate                        LSP per minute
        Overloaded Links                        number
        Amount Links Overloaded                 % of bandwidth
        FTP Rate                                Mb/minute
        IPsec Tunnel Flap Rate                  tunnels per minute
        Filter Policy Changes                   policies per hour
        SSH Session Rate                        SSH sessions per hour
        Telnet Session Rate                     Telnet session per hour
        Command Entry Rate                      Commands per Hour
        Message Flood Rate                      Messages

   3.3.4 Benchmarks

   Benchmarks are as defined in [4] and MUST be reported as follow:
        PARAMETER                               UNITS     PHASE
        Stable Aggregate Forwarding Rate        pps       Startup
        Stable Latency                          seconds   Startup
        Stable Session Count                    sessions  Startup
        Unstable Aggregate Forwarding Rate      pps       Instability
        Degraded Aggregate Forwarding Rate      pps       Instability
        Ave. Degraded Aggregate Forwarding Rate pps       Instability
        Unstable Latency                        seconds   Instability
        Unstable Uncontrolled Sessions Lost     sessions  Instability
        Recovered Aggregate Forwarding Rate     pps       Recovery
        Recovered Latency                       seconds   Recovery
        Recovery Time                           seconds   Recovery
        Recovered Uncontrolled Sessions         sessions  Recovery

Poretsky and Rao                                               [Page 7]


INTERNET-DRAFT           Methodology Guidelines    February 2008
                      for Accelerated Stress Benchmarking

4.  Stress Test Procedure

   4.1 General Methodology with Multiple Instability Conditions

   Objective
   To benchmark the DUT under accelerated stress when there are
   multiple instability conditions.

   Procedure

        1. Report Configuration Set
        2. Begin Startup Conditions with the DUT
        3. Establish Configuration Sets with the DUT
        4. Report Stability Benchmarks
        5. Apply Instability Conditions
        6. Apply Instability Condition specific to test case.
        7. Report Instability Benchmarks
        8. Stop applying all Instability Conditions
        9. Report Recovery Benchmarks
        10. Optional - Change Configuration Set and/or Instability
            Conditions for next iteration

   Expected Results
   Ideally the Forwarding Rates, Latencies, and Session Counts will
   be measured to be the same at each phase.  If no packet or
   session loss occurs then the Instability Conditions MAY be
   increased for a repeated iteration (step 10 of the procedure).

   Example Procedure

       1. Report Configuration Set

           BGP Enabled
           10 EBGP Peers
           30 IBGP Peers
           500K BGP Route Instances
           160K BGP FIB Routes

           ISIS Enabled
           ISIS-TE Disabled
           30 ISIS Adjacencies
           10K ISIS Level-1 Routes
           250 ISIS Nodes per Area

           MPLS Disabled
           IP Multicast Disabled

           IPsec Enabled
           10K IPsec tunnels
           640 Firewall Policies
           100 Firewall Rules per Policy

Poretsky and Rao                                               [Page 8]


INTERNET-DRAFT           Methodology Guidelines    February 2008
                      for Accelerated Stress Benchmarking

           Traffic Forwarding Enabled
           Aggregate Offered Load 10Gbps
           30 Ingress Interfaces
           30 Egress Interfaces
           Packet Size(s) = 64, 128, 256, 512, 1024, 1280, 1518 bytes
           Forwarding Rate[1..30] = 1Gbps
           10000 Flows
           Encapsulation[1..5000] = IPv4
           Encapsulation[5001.10000] = IPsec
           Logging Enabled
           Protocol Debug Disabled
           SNMP Enabled
           SSH Enabled
                 10 Concurrent SSH Sessions
           FTP Enabled
           RADIUS Enabled
           TACACS Disabled
           Packet Statistics Collector Enabled

        2. Begin Startup Conditions with the DUT

           10 EBGP peering sessions negotiated
           30 EBGP peering sessions negotiated
           1K BGP routes learned per second
           30 ISIS Adjacencies
           1K ISIS routes learned per second
           10K IPsec tunnels negotiated

        3. Establish Configuration Sets with the DUT

        4. Report Stability Benchmarks as follow:

           Stable Aggregate Forwarding Rate
           Stable Latency
           Stable Session Count

           It is RECOMMENDED that the benchmarks be measured and
           recorded at one-second intervals.

        5. Apply Instability Conditions

           Interface Shutdown Cycling Rate = 1 interface every 5
                                             minutes
           BGP Session Flap Rate = 1 session every 10 minutes
           BGP Route Flap Rate = 100 routes per minute
           ISIS Route Flap Rate = 100 routes per minute
           IPsec Tunnel Flap Rate = 1 tunnel per minute
           Overloaded Links = 5 of 30
           Amount Links Overloaded = 20%
           SNMP GETs = 1 per sec
           SSH Session Rate = 6 sessions per hour
           SSH Session Duration = 10 minutes
           Command Rate via SSH = 20 commands per minute

Poretsky and Rao                                               [Page 9]


INTERNET-DRAFT           Methodology Guidelines    February 2008
                      for Accelerated Stress Benchmarking

           FTP Restart Rate = 10 continuous transfers (Puts/Gets)
                              per hour
           FTP Transfer Rate = 100 Mbps
           Statistics Sampling Rate = 1:1 packets
           RADIUS Server Loss Rate = 1 per Hour
           RADIUS Server Loss Duration = 3 seconds

        6. Apply Instability Condition specific to test case.

        7. Report Instability Benchmarks as follow:
           Unstable Aggregate Forwarding Rate
           Degraded Aggregate Forwarding Rate
           Ave. Degraded Aggregate Forwarding Rate
           Unstable Latency
           Unstable Uncontrolled Sessions Lost

           It is RECOMMENDED that the benchmarks be measured and
           recorded at one-second intervals.

        8. Stop applying all Instability Conditions

        9. Report Recovery Benchmarks as follow:

           Recovered Aggregate Forwarding Rate
           Recovered Latency
           Recovery Time
           Recovered Uncontrolled Sessions Lost

           It is RECOMMENDED that the benchmarks be measured and
           recorded at one-second intervals.

        10. Optional - Change Configuration Set and/or Instability
            Conditions for next iteration

   4.2 General Methodology with a Single Instability Condition

   Objective
   To benchmark the DUT under accelerated stress when there is a
   single instability conditions.

   Procedure

        1. Report Configuration Set
        2. Begin Startup Conditions with the DUT
        3. Establish Configuration Sets with the DUT
        4. Report Stability Benchmarks
        5. Apply single Instability Condition
        6. Report Instability Benchmarks
        7. Stop applying all Instability Condition
        8. Report Recovery Benchmarks
        9. Optional - Change Configuration Set and/or Instability
            Conditions for next iteration

Poretsky and Rao                                               [Page 10]


INTERNET-DRAFT           Methodology Guidelines    February 2008
                      for Accelerated Stress Benchmarking

   Expected Results
   Ideally the Forwarding Rates, Latencies, and Session Counts will
   be measured to be the same at each phase.  If no packet or session
   loss occurs then the Instability Conditions MAY be increased
   for a repeated iteration (step 10 of the procedure).

5. IANA Considerations
   This document requires no IANA considerations.

6. Security Considerations
        Documents of this type do not directly affect the security of
        the Internet or of corporate networks as long as benchmarking
        is not performed on devices or systems connected to operating
        networks.

7. Normative References

      [1]   Bradner, S., Editor, "Benchmarking Terminology for Network
            Interconnection Devices", RFC 1242, October 1991.

      [2]   Mandeville, R., "Benchmarking Terminology for LAN Switching
            Devices", RFC 2285, October 1998.

      [3]   Bradner, S. and McQuaid, J., "Benchmarking Methodology for
            Network Interconnect Devices", RFC 2544, March 1999.

      [4]   Poretsky, S. and Rao, S., "Terminology for Accelerated
            Stress Benchmarking", draft-ietf-bmwg-acc-bench-term-13,
            work in progress, February 2008.

      [5]   Bradner, S., "Key words for use in RFCs to Indicate
            Requirement Levels", RFC 2119, March 1997.

8. Informative References

      [RFC3871]  RFC 3871 "Operational Security Requirements for Large
            Internet Service Provider (ISP) IP Network Infrastructure.
            G. Jones, Ed.. IETF, September 2004.

      [NANOG25]   "Core Router Evaluation for Higher Availability",
            Scott Poretsky, NANOG 25, October 8, 2002, Toronto, CA.

      [IEEECQR]   "Router Stress Testing to Validate Readiness for
            Network Deployment", Scott Poretsky, IEEE CQR 2003.

      [CONVMETH]   Poretsky, S., "Benchmarking Methodology for IGP Data
            Plane Route Convergence",
            draft-ietf-bmwg-igp-dataplane-conv-meth-15, work in
            progress, February 2008.

Poretsky and Rao                                               [Page 11]


INTERNET-DRAFT           Methodology Guidelines    February 2008
                      for Accelerated Stress Benchmarking

9. Author's Address

      Scott Poretsky
      NextPoint Networks
      3 Federal Street
      Billerica, MA 01821
      USA
      Phone: + 1 508 439 9008
      EMail: sporetsky@nextpointnetworks.com

      Shankar Rao
      1801 California Street
      8th Floor
      Qwest Communications
      Denver, CO 80202
      USA
      Phone: + 1 303 437 6643
      Email: shankar.rao@qwest.com

Poretsky and Rao                                               [Page 12]


INTERNET-DRAFT           Methodology Guidelines    February 2008
                      for Accelerated Stress Benchmarking

Full Copyright Statement

   Copyright (C) The IETF Trust (2008).

   This document is subject to the rights, licenses and restrictions
   contained in BCP 78, and except as set forth therein, the authors
   retain all their rights.

   This document and the information contained herein are provided
   on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE
   REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE
   IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL
   WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY
   WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE
   ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS
   FOR A PARTICULAR PURPOSE.

Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at
   http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at ietf-
   ipr@ietf.org.

Acknowledgement

   Funding for the RFC Editor function is currently provided by the
   Internet Society.

Poretsky and Rao                                              [Page 13]