Benchmarking Methodology WG                                 R.Papneja
     Internet Draft                                                Isocore
     Intended status: Informational
     Expires: April 2011                                           B.Parise
                                                              Cisco Systems
    
                                                                Susan Hares
                                                                     Huawei
    
                                                             October 18, 2010
    
    
    
            Basic BGP Convergence Benchmarking Methodology for Data Plane
                                     Convergence
                    draft-papneja-bgp-basic-dp-convergence-00.txt
    
       Abstract
       BGP is widely deployed and used by several service providers as the
        default Inter AS routing protocol.  It is of utmost importance to
        ensure that when a BGP peer or a downstream link of a BGP peer
        fails, the alternate paths are rapidly used and routes via these
        alternate paths are installed. This document provides the basic BGP
        Benchmarking Methodology using existing BGP Convergence Terminology,
        RFC-4098.
    
       Status of this Memo
    
       This Internet-Draft is submitted to IETF in full conformance with
        the provisions of BCP 78 and BCP 79.
    
       Internet-Drafts are working documents of the Internet Engineering
        Task Force (IETF), its areas, and its working groups.  Note that
        other groups may also distribute working documents as Internet-
        Drafts.
    
       Internet-Drafts are draft documents valid for a maximum of six
        months and may be updated, replaced, or obsoleted by other documents
        at any time.  It is inappropriate to use Internet-Drafts as
        reference material or to cite them other than as "work in progress."
    
       The list of current Internet-Drafts can be accessed at
        http://www.ietf.org/ietf/1id-abstracts.txt
    
       The list of Internet-Draft Shadow Directories can be accessed at
        http://www.ietf.org/shadow.html
    
       This Internet-Draft will expire on April 18, 2009.
    
    
    
     Papneja, et al.    Expires April 18, 2011                 [Page 1]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
    Copyright Notice
    
       Copyright (c) 2010 IETF Trust and the persons identified as the
        document authors.  All rights reserved.
    
        This document is subject to BCP 78 and the IETF Trust's Legal
        Provisions Relating to IETF Documents
        (http://trustee.ietf.org/license-info) in effect on the date of
        publication of this document. Please review these documents
        carefully, as they describe your rights and restrictions with
        respect to this document.  Code Components extracted from this
        document must include Simplified BSD License text as described
        in Section 4.e of the Trust Legal Provisions and are provided
        without warranty as described in the Simplified BSD License.
    
        This document may contain material from IETF Documents or IETF
        Contributions published or made publicly available before November
        10, 2008.  The person(s) controlling the copyright in some of this
        material may not have granted the IETF Trust the right to allow
        modifications of such material outside the IETF Standards Process.
        Without obtaining an adequate license from the person(s)
        controlling the copyright in such materials, this document may not
        be modified outside the IETF Standards Process, and derivative
        works of it may not be created outside the IETF Standards Process,
        except to format it for publication as an RFC or to translate it
        into languages other than English.
    
    Table of Contents
    
        1. Introduction 3
           1.1. Document Scope  5
        2. Existing definitions and requirements 5
        3. Test Topologies    6
           3.1. General Reference Topology 6
        4. Test Considerations 8
           4.1. Number of Peers     8
           4.2. Number of Routes per Peer  8
           4.3. Policy Processing/Reconfiguration 9
           4.4. Configured Parameters (Timers, etc..)          9
           4.5. Interface Types     10
           4.6. Measurement Accuracy 10
           4.7. Measurement Statistics     11
           4.8. Authentication 11
           4.9. Convergence Events  11
           4.10. High Availability  12
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 2]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
       5. Test Cases   12
           5.1. Basic Convergence Tests    12
              5.1.1. RIB-IN Convergence    12
              5.1.2. RIB-OUT Convergence   14
              5.1.3. eBGP Convergence      15
              5.1.4. iBGP Convergence      15
              5.1.5. eBGP Multihop Convergence   16
           5.2. BGP Failure/Convergence Events   17
              5.2.1. Physical Link Failure on DUT End         17
              5.2.2. Physical Link Failure on Remote/Emulator End   18
              5.2.3. ECMP Link Failure on DUT End 19
           5.3. BGP Adjacency Failure (Non-Physical Link Failure) on
           Emulator     19
           5.4. BGP Hard Reset Test cases  20
              5.4.1. BGP Non-Recovering Hard Reset Event on DUT     20
           5.5. BGP Soft Reset  22
           5.6. BGP Route Withdrawal Convergence Time         23
           5.7. BGP Path Attribute Change Convergence Time    25
           5.8. BGP Graceful Restart Convergence Time         26
        6. Reporting Format   28
        7. Security Considerations  30
        8. IANA Considerations 31
        9. References   31
           9.1. Normative References 31
           9.2. Informative References     31
        Authors Addresses    33
    
    
    1. Introduction
    
       This document defines the methodology for benchmarking data plane
        FIB convergence performance of BGP in router and switches for simple
        topologies of 3 or 4 nodes.
    
       The methodology proposed in this document applies to both IPv4 and
        IPv6 and if a particular test is unique to one version, it is marked
        accordingly.  For IPv6 benchmarking the device under test will
        require the support of Multi-Protocol BGP (MP-BGP) [RFC2858,
        RFC2545].
    
       The scope of this companion document is limited to basic BGP
        protocol FIB convergence measurements. BGP extensions outside of
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 3]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
       carrying IPv6 in (MP-BGP) [RFC2858, RFC2545] are outside the scope
        of this document.  Interaction with IGPs (IGP interworking) is
        outside the scope of this document.
    
       1.1 Precise Benchmarking definition
       Since benchmarking is science of precision, let us restate the
        purpose of this document in benchmarking terms. This document
        defines methodology to test
    
       - data plane convergence on a single BGP device that supports the
          BGP [RFC4271] functionality;
    
        - in test topology of 3 or 4 nodes,
    
        - using Basic BGP.
    
       Data plane convergence is defined as the completion of all FIB
        changes so that all forwarded traffic now takes the new proposed
        route.  RFC 4098 defines the terms BGP device, FIB and the forwarded
        traffic. Data plane convergence is different than control plane
        convergence within a node.
    
       Basic BGP is defined as RFC 4271 functional with Multi-Protocol BGP
        (MP-BGP) [RFC2858, RFC2545] for IPv6. The use of other extensions of
        BGP to support layer-2, layer-3 virtual private networks (VPN) are
        out of scope of this document.
    
       The terminology used in this document is defined in [RFC4098]. One
        additional term is defined in this draft: data plane BGP
        convergence.
    
       1.2 Purpose of BGP FIB (data plane) convergence
    
       In the current Internet architecture the Inter-Autonomous System
        (inter-AS) transit is primarily available through BGP. To maintain a
        reliable connectivity within intra-domains or across inter-domains,
        fast recovery from failures remains most critical. To ensure minimal
        traffic losses, many service providers are requiring BGP
        implementations to converge the entire Internet routing table within
        sub-seconds at FIB level.
    
       Furthermore, to compare these numbers amongst various devices,
        service providers are also looking at ways to standardize the
        convergence measurement methods. This document offers test methods
        for simple topologies. These simple tests will provide a quick high-
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 4]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
       level check, of the BGP data plane convergence across multiple
        implementations.
    
       1.2 Control Plane Convergence
    
       The convergence of BGP occurs at two levels: RIB and FIB
        convergence. RFC 4098 defines terms for BGP control plane
        convergence. Methodologies which test control plane convergence are
        out of scope for this draft.
    
       1.3 Benchmarking Testing
    
       In order to ensure that the results obtained in tests are
        repeatable, careful setup of initial conditions and exact steps are
        required.
    
       This document proposes these initial conditions, test steps, and
        result checking.  To ensure uniformity of the results all optional
        parameters SHOULD be disabled and all settings SHOULD be changed to
        default, these may include BGP timers as well.
    
    
    2. Existing definitions and requirements
    
       RFC 1242, "Benchmarking Terminology for Network Interconnect
        Devices" [RFC1242] and RFC 2285, "Benchmarking Terminology for LAN
        Switching Devices" [RFC2285] SHOULD be reviewed in conjunction with
        this document.  WLAN-specific terms and definitions are also
        provided in Clauses 3 and 4 of the IEEE 802.11 standard [802.11].
        Commonly used terms may also be found in RFC 1983 [RFC1983].
    
       For the sake of clarity and continuity, this document adopts the
        general template for benchmarking terminology set out in Section 2
        of RFC 1242.  Definitions are organized in alphabetical order, and
        grouped into sections for ease of reference.
    
       The following terms are assumed to be taken as defined in RFC 1242
        [RFC1242]: Throughput, Latency, Constant Load, Frame Loss Rate, and
        Overhead Behavior.  In addition, the following terms are taken as
        defined in RFC 2285 [RFC2285]: Forwarding Rates, Maximum Forwarding
        Rate, Loads, Device Under Test (DUT), and System Under Test (SUT).
    
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 5]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
       The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
        "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
        document are to be interpreted as described in RFC 2119 [RFC2119].
    
    3. Test Topologies
    
       This section describes simple test setups for use in BGP
        benchmarking tests measuring convergence of the FIB (data plane)
        after the BGP updates has been received.
    
       These simple test nodes have 3 or 4 nodes with the following
        configuration:
    
       1. Basic Test Setup
        2. Three node setup for iBGP or eBGP convergence
        3. Setup for eBGP multihop test scenario
        4. Four node setup for iBGP or eBGP convergence
    
       Individual tests refer to these topologies.
    
       Figures 1-4 use the following conventions
    
       AS-X: Autonomous System X
    
       Loopback Int: Loopback interface on the BGP enabled device
       R2: Helper router
    
    3.1. General Reference Topology
    
       Emulator acts as 1 or more BGP peers for different testcases.
    
        -----------                              -------------
        |          |   traffic interfaces        |            |
        |          |-----------------------1---- | tx         |
        |          |-----------------------2---- | tr1        |
        |          |-----------------------3-----| tr2        |
        |    DUT   |    routing interfaces       | Emulator   |
        |          |                             |            |
        |      Drr1|---------------------------  |Err1        |
        |          |      BGP Peering            |            |
        |      Drr2|---------------------------- |Err2        |
        |          |      BGP Peering            |            |
        -----------                               -------------
    
               Figure 1 Basic Test Setup
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 6]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
    
    
        -------------         -------------           -------------
        |            |        |           |           |           |
        |            |        |           |           |           |
        |   HLP      |        |  DUT      |           | Emulator  |
        |  (AS-X)    |--------| (AS-Y)    |-----------|  (AS-Z)   |
        |            |        |           |           |           |
        |            |        |           |           |           |
        |            |        |           |           |           |
        -------------         -------------           -------------
    
    
         Figure 2 Three Node Setup for eBGP and iBGP Convergence
    
    
    
    
        -------------         -------------           -------------
        |            |        |           |           |           |
        |            |        |           |           |           |
        |    R1      |        |  DUT      |           | Emulator  |
        |  (AS-X)    |--------| (AS-Y)    |-----------|  (AS-Z)   |
        |            |        |           |           |           |
        |            |        |           |           |           |
        |            |        |           |           |           |
        -------------         -------------           -------------
             |Loopback-Int         |Loopback-Int
             |                     |
             +                     +
    
    
    
          Figure 3 BGP Convergence for eBGP Multihop Scenario
    
       ----------      ----------     ----------      ----------
        |         |     |        |     |        |     |         |
        |         |     |        |     |        |     |         |
        |    R1   |     |  DUT2  |     |  DUT1  |     |Emulator |
        | (AS-X)  |-----| (AS-X) |-----| (AS-Y) |-----| (AS-Z)  |
        |         |     |        |     |        |     |         |
        |         |     |        |     |        |     |         |
        |         |     |        |     |        |     |         |
        ----------      ----------     ----------     ----------
    
               Figure 4 Four Node Setup for EBGP and IBGP Convergence
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 7]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
    
    4. Test Considerations
    
       The test cases for measuring convergence for iBGP and eBGP are
        different. Both iBGP and eBGP use different mechanisms to advertise,
        install and learn the routes.  Typically, an iBGP route on the DUT
        is installed and exported only when the next-hop is reachable. For
        eBGP the route is installed on the DUT with the remote interface
        address as the next-hop with the exception of the multihop case.
    
    4.1. Number of Peers
    
       Number of Peers is defined as the number of BGP neighbors or
        sessions the DUT has at the beginning of the test.
       The peers are established before the tests begin.
    
       The relationship could be either, iBGP or eBGP peering depending
        upon the test case requirement.
    
       The DUT establishes one or more BGP sessions with one more emulated
        routers or helper nodes.  Additional peers can be added based on the
        testing requirements. The number of peers enabled during the testing
        should be well documented in the report matrix.
    
    4.2. Number of Routes per Peer
    
       It Number of Routes per Peer is defined as the number of routes
        advertized or learnt by the DUT per session or through neighbor
        relationship with an emulator or helper node. The tester, emulating
        as neighbor MUST advertise at least one route per peer.
    
       Each test must run must identify the route stream in terms of route
        packing, route mixture, and number of routes. This route stream must
        be well documented in the reporting stream. RFC 4098 defines these
        terms.
    
       It is RECOMMENDED that the user may consider advertizing the entire
        current Internet routing table per peering session using an Internet
        route mixture with unique or non-unique routes.
       If multiple peers are used, it is important to precisely document
        the timing sequence between the peer sending routes (as defined in
        RFC 4098).
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 8]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
    4.3. Policy Processing/Reconfiguration
    
       The DUT MUST run one baseline test where policy is Minimum policy as
        defined in RFC 4098. Additional runs may be done with policy set-up
        before the tests begin. Exact policy settings should be documented
        as part of the test.
    
    4.4. Configured Parameters (Timers, etc..)
    
       There are configured parameters and timers that may impact the
        measured BGP convergence times.
    
       The benchmark metrics MAY be measured at any fixed values for these
        configured parameters.
    
       It is RECOMMENDED these configure parameters have two settings: a)
        basic-test, and b)values as expected in the operational network.
       All optional BGP settings MUST be kept consistent across iterations
        of any specific tests
    
       Examples of the configured parameters that may impact measured BGP
        convergence time include, but are not limited to:
    
          1. Interface failure detection timer
          2. BGP Keepalive timer
          3. BGP Holdtime
          4. BGP update delay timer
          5. ConnectRetry timer
          6. TCP Segment Size
          7. Minimum Route Advertisement Interval (MRAI)
          8. MinASOriginationInterval (MAOI)
          9. Route Flap Dampening parameters
          10.   TCP MD5
    
         The basic-test settings for the parameters should be:
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 9]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
            1. Interface failure detection timer (0 ms)
            2. BGP Keepalive timer (1 min)
            3. BGP Holdtime (3 min)
            4. BGP update delay timer (0 s)
            5. ConnectRetry timer (1 s)
            6. TCP Segment Size (4096)
            7. Minimum Route Advertisement Interval (MRAI)(0 s)
            8. MinASOriginationInterval (MAOI) (0 s)
            9. Route Flap Dampening parameters (off)
            10. TCP MD5 (off)
    
    4.5. Interface Types
    
       The type of media dictate which test cases may be executed, each
        interface type has unique mechanism for detecting link failures and
        the speed at which that mechanism operates will influence the
        measurement results. All interfaces MUST be of the same media and
        throughput for each test case.
    
    4.6. Measurement Accuracy
    
       Since observed packet loss is used to measure the route convergence
        time, the time between two successive packets offered to each
        individual route is the highest possible accuracy of any packet-loss
        based measurement.  When packet jitter is much less than the
        convergence time, it is a negligible source of error and hence it
        will be treated as within tolerance.
    
       An exterior measurement on the input media (such Ethernet)is defined
        by this specification.
    
    
    
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 10]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
    4.7. Measurement Statistics
    
       The benchmark measurements may vary for each trial, due to the
        statistical nature of timer expirations, CPU scheduling, etc.
       It is recommended to repeat the test multiple times.  Evaluation of
        the test data must be done with an understanding of generally
        accepted testing practices regarding repeatability, variance and
        statistical significance of a small number of trials.
    
       For any repeated tests that are averaged to remove variance, all
        parameters MUST remain the same.
    
    4.8. Authentication
    
       Authentication in BGP is done using the TCP MD5 Signature Option
        [RFC2385].  The processing of the MD5 hash, particularly in devices
        with a large number of BGP peers and a large amount of update
        traffic, can have an impact on the control plane of the device.  If
        authentication is enabled, it SHOULD be documented correctly in the
        reporting format
    
    4.9. Convergence Events
    
       Convergence events or triggers are defined as abnormal occurrences
        in the network, which initiate route flapping in the network, and
        hence forces the re-convergence of a steady state network. In a real
        network, a series of convergence events may cause convergence
        latency operators desire to test.
    
       These convergence events must be defined in terms of the sequences
        defined in RFC 4098. This basic document begins all tests with a
        router initial set-up.  Additional documents will define BGP data
        plane convergence based on peer initialization.
    
       The convergence events may or may not be tied to the actual failure
       A Soft Reset (RFC 4098) does not clear the RIB or FIB tables.  A
        Hard reset clears the BGP peer sessions, the RIB tables, and FIB
        tables.
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 11]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
    4.10. High Availability
    
       Due to the different Non-Stop-Routing (sometimes referred to High-
        Availability) solutions available from different vendors, it is
        RECOMMENDED that any redundancy available in the routing processors
        should be disabled during the convergence measurements.
    
    5. Test Cases
    
       All tests defined under this section assume the following:
    
         BGP peers should be brought to BGP Peer established state.
    
       a. Furthermore the traffic generation and routing should be verified
          in the topology
    
    5.1. Basic Convergence Tests
       These test cases measure characteristics of a BGP implementation in
        non-failure scenarios like:
    
       a. RIB-IN Convergence
    
       b. RIB-OUT Convergence
    
       c. eBGP Convergence
    
       d. iBGP Convergence
    
    5.1.1. RIB-IN Convergence
    
    Objective:
    
     This test measures the convergence time taken to receive and install a
     route in RIB using BGP
    
     Reference Test Setup:
    
     This test uses the setup as shown in figure 1
    
     Procedure:
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 12]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
         a. All variables affecting Convergence should be set to a basic
             test state (as defined in section 4-4).
          b. Establish BGP adjacency between DUT and peer x of Emulator;
          c. To ensure adjacency establishment, wait for 3 KeepAlives from
             the DUT or a configurable delay before proceeding with the rest
             of the test;
          d. Start the traffic from the Emulator peer-x towards the DUT
             targeted at a routes specified in route mixture (ex. route A)
             Initially no traffic SHOULD be observed on the egress interface
             as the route A is not installed in the forwarding database of
             the DUT.
          e. Advertise route A from the Peer-x to the DUT and record the
             time;
    
             This is Tup(EMx,Rt-A). (nick-name XMT-Rt-time)
    
          f. Record the time when the route-A from Peer-x is received at the
             DUT.
    
             This Tup(DUT,Rt-A).  It is nick named is RCV-Rt-time
    
          g. Record the time when the traffic targeted towards route A is
             received by Emulator on appropriate traffic egress interface.
                rd
             If 3  party route (traffic-egress 2), or BGP peer route
             interfaces.
    
             This is TR(TDx,Rt-A). This is "nick-named" DUT-XMT-Data-Time.
    
          h. The difference between the Tup(TDx,RT-A) and traffic received
             time (TR (TDr, Rt-A) is the FIB Convergence Time for route-A in
             the route mixture.
    
               A full convergence for the route update is the measurement
                between the 1st  route (Route-A) and the last route (Rt-last)
    
                    Route update convergence is
                    TR(TDr, RT-last)- Tup(DUT, Rt-A) or
    
                    (DUT-XMT-Data-Time - RCV-Rt-Time)(rt-A)
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 13]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
    
          Note: It is recommended that a single test with the same route
          mixture be repeated several times. A report should provide the
          Stand deviation of all tests and the average.
    
          Running tests with a varying number of routes and route mixtures
          is important to get a full characterization of a single peer.
    
     5.1.2. RIB-OUT Convergence
    
    Objective:
    
    This test measures the convergence time taken by an implementation to
     receive, install and advertise a route using BGP
    
     Reference Test Setup:
    
     This test uses the setup as shown in figure 2
    
     Procedure:
    
          a. The Helper node (HLP) run same version of BGP as DUT;
          b. All devices MUST be synchronized using NTP or some local
             reference clock;
          c. All configuration variables for HLP, DUT, and Emulator SHOULD
             be set to the same values. These values MAY be basic-test or a
             unique set completely described in the test set-up.
          d. Establish BGP adjacency between DUT and Emulator
          e. Establish BGP adjacency between DUT and Helper Node
          f. To ensure adjacency establishment, wait for 3 KeepAlives from
             the DUT or a configurable delay before proceeding with the rest
             of the test
          g. Start the traffic from the Emulator towards the Helper Node
             targeted at a specific route say route A.
             Initially no traffic SHOULD be observed on the egress interface
             as the route-A is not installed in the forwarding database of
             the DUT
          h. Advertise routeA from the Emulator to the DUT and note the
             time.
             This is Tup(EMx, Route-A). (nick-name EM-XMT-Rt-Time)
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 14]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
         i. Record when Route-A is received by DUT.
    
             This is Tup(DUTr, Route-A). (nick-name DUT-RCV-Rt-Time)
    
          j. Record the time when the ROUTE forward by DUT toward the Helper
             node.
    
             This is Tup(DUTx, Rt-A). (nick-name DUT-XMT-Rt-Time).
    
          k. Record the time when the traffic targeted towards route-A is
             received on the Route Egress Interface toward peer-X.
             This is TR(EMr, Route-A).  (nick-name DUT-XMT-Data Time).
    
    
               FIB convergence = (DUT-RCV-Rt-Time - DUT-XMT-Data-Time).
    
               RIB convergence = (DUT-RCV-Rt-Time - DUT-XMT-Rt-Time).
    
    
             Convergence for a route stream is characterized by
    
               a) Individual route convergence for FIB, RIB
               b) All route convergence of
    
               FIB-convergence =DUT-RCV-Rt-Time(A)-DUT-XMT-Data-Time(last)
    
               RIB-convergence =DUT-RCV-Rt-Time(A)-DUT-XMT-Rt-Time(last)
    
    
    
     5.1.3. eBGP Convergence
    
    Objective:
    
    This test measures the convergence time taken by an implementation to
     receive, install and advertise a route in an eBGP Scenario
    
     Reference Test Setup:
    
     This test uses the setup as shown in figure 2, and the scenarios
     described in RIB-IN and RIB-OUT are applicable to this test case.
    
     5.1.4. iBGP Convergence
    
    Objective:
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 15]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
    
     This test measures the convergence time taken by an implementation to
     receive, install and advertise a route in an iBGP Scenario
    
     Reference Test Setup:
    
     This test uses the setup as shown in figure 2, and the test scenarios
     listed in RIB-IN and RIB-OUT are applicable to this test case.
    
     5.1.5. eBGP Multihop Convergence
    
    Objective
    
     This test measures the convergence time taken by an implementation to
     receive, install and advertise a route in an eBGP Multihop Scenario
    
     Reference Test Setup:
    
     This test uses the setup as shown in figure 3. Two DUTs are used along
     with a helper node.
    
     Procedure:
    
           a. The DUT2 is the same model as DUT and runs the same BGP
               implementation as DUT.
            b. All devices to be synchronized using NTP
            c. All variables affecting Convergence like authentication,
               policies, timers should be set to basic-settings.
            d. All 3 devices, DUT, Emulator and Helper Node are configured
               as different Autonomous Systems
            e. Loopback Interfaces configured on DUT and Helper Node and
               connectivity is established between them using any config
               options available on the DUT
            f. Establish BGP adjacency between DUT1 and Emulator
            g. Establish BGP adjacency between DUT2 and Helper Node
            h. Establish BGP adjacency between DUT 1 and DUT 2
            i. To ensure adjacency establishment, wait for 3 KeepAlives from
               the DUT1 and DUT2 or a configurable delay before proceeding
               with the rest of the test
            j. Start the traffic from the Emulator towards the Helper Node
               targeted at a specific route say routeA.
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 16]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
            k. Initially no traffic SHOULD be observed on the egress
               interface as the routeA is not installed in the forwarding
               database of the DUT
            l. Advertise routeA from the Emulator to the DUT and note the
               time. (Tup(EMx,RouteA)  - this is nicknamed (Route-Rec-time).
    
            m. Record the time when the traffic targeted towards routeA is
               received from Egress Interface of DUT on emulator.
    
               This is TR(EMr,DUT), nicknamed (Data Receive time)
    
            n. The following equation represents the FIB Convergence multi-
               node
                  eBGP Multihop Convergence Time =
                  (Rt-RecTime  - Data-RcvTime).
    
     Note: It is recommended that the test be repeated with varying number
     of routes and route mixtures. With each set route mixture, the test
     should be repeated multiple times. The results should record average,
     mean, Standard Deviation.
    
    5.2. BGP Failure/Convergence Events
    
    5.2.1. Physical Link Failure on DUT End
    
    Objective:
    
     This test measures the route convergence time due to local link failure
     event at DUT's Local Interface
    
     Reference Test Setup:
    
     This test uses the setup as shown in figure 1. Shutdown event is
     defined as an administrative shutdown event on the DUT.
    
     Procedure:
    
           a. All variables affecting Convergence like authentication,
               policies, timers should be set to basic-test policy.
    
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 17]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
           b. Establish 2 BGP adjacencies from DUT to Emulator, one over
               the peer interface and the other using a second peer
               interface.
            c. Advertise the same route, route A over both the adjacencies
               and (Tx1)Interface to be the preferred next hop.
            d. To ensure adjacency establishment, wait for 3 KeepAlives from
               the DUT or a configurable delay before proceeding with the
               rest of the test.
            e. Start the traffic from the Emulator towards the DUT targeted
               at a specific route say route A. Initially traffic would be
               observed on the best egress route (Err1) instead of Trr2
            f. Trigger the shutdown event of Best Egress Interface on DUT
               (Drr1).
            g. Measure the Convergence Time for the event to be detected and
               traffic to be forwarded to Next-Best Egress Interface (rr2).
    
               Time = Data-detect(rr2) - Shutdown time.
    
            h. Stop the offered load and wait for the queues to drain and
               Restart
            i. Bring up the link on DUT Best Egress Interface
            j. Measure the convergence time taken for the traffic to be
               rerouted from (rr2) to Best Interface (rr1)
    
              Time = Data-Detect(rr1)  - Shutdown-time.
    
    
            k. It is recommended that the test be repeated with varying
               number of routes and route mixtures or with number of routes
               & route mixtures closer to what is deployed in operational
               networks
    
    
    
     5.2.2. Physical Link Failure on Remote/Emulator End
    
    Objective:
    
     This test measures the route convergence time due to local link failure
     event at Tester's Local Interface
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 18]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
    Reference Test Setup:
    
     This test uses the setup as shown in figure 1.  Shutdown event is
     defined as shutdown of the local interface of Tester via logical
     shutdown event. The procedure used in 5.2.1 is used for the
     termination.
    
    
    5.2.3. ECMP Link Failure on DUT End
    
    Objective:
    
     This test measures the route convergence time due to local link failure
     event at ECMP Member. The FIB configuration and BGP is set to allow two
     ECMP routes to be installed. However, policy directs the routes to be
     sent only over one of the paths.
    Reference Test Setup:
    
     This test uses the setup as shown in figure 1, and the procedure uses
     5.2.1.
    
    
    5.3. BGP Adjacency Failure (Non-Physical Link Failure) on Emulator
    
    Objective:
    
     This test measures the route convergence time due to BGP Adjacency
     Failure on Emulator
    Reference Test Setup:
    
     This test uses the setup as shown in figure 1
    
    Procedure:
    
           a. All variables affecting Convergence like authentication,
               policies, timers should be basic-policy set.
    
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 19]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
            b. Establish 2 BGP adjacencies from DUT to Emulator, one over
               the Best Egress Interface and the other using the Next-Best
               Egress Interface
            c. Advertise the same route, routeA over both the adjacencies
               and make Best Egress Interface to be the preferred next hop
            d. To ensure adjacency establishment, wait for 3 KeepAlives from
               the DUT or a configurable delay before proceeding with the
               rest of the test
            e. Start the traffic from the Emulator towards the DUT targeted
               at a specific route say routeA. Initially traffic would be
               observed on the Best Egress interface
            f. Remove BGP adjacency via a software adjacency down on the
               Emulator on the Best Egress Interface
    
               Time = BGPadj-down-time - nicknamed BGPpeer-down.
    
    
            g. Measure the Convergence Time for the event to be detected and
               traffic to be forwarded to Next-Best Egress Interface
    
               This time is Tr-rr2 nicknamed - TR2-traffic-on
    
               Convergence = TR2-traffic-on - BGPpeer-down
    
            h. Stop the offered load and wait for the queues to drain and
               Restart
            i. Bring up BGP adjacency on the Emulator over the Best Egress
               Interface
               Time = BGP-adj-up - nicknamed BGPpeer-up
    
    
            j. Measure the convergence time taken for the traffic to be
               rerouted to Best Interface
    
               Time = Tr-rr1 is nicknamed TR1-traffic-on.
    
    
    
     5.4. BGP Hard Reset Test cases
    
    5.4.1. BGP Non-Recovering Hard Reset Event on DUT
    
    Objective:
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 20]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
    
     This test measures the route convergence time due to Hard Reset on the
     DUT
    
    Reference Test Setup:
    
     This test uses the setup as shown in figure 1
    
    Procedure:
    
           a. The requirement for this test case is that the Hard Reset
               Event should be non-recovering and should affect only the
               adjacency between DUT and Emulator on the Best Egress
               Interface
            b. All variables affecting SHOULD be set to basic-test values
            c. Establish 2 BGP adjacencies from DUT to Emulator, one over
               the Best Egress Interface and the other using the Next-Best
               Egress Interface
            d. Advertise the same route, routeA over both the adjacencies
               and make Best Egress Interface to be the preferred next hop
            e. To ensure adjacency establishment, wait for 3 KeepAlives from
               the DUT or a configurable delay before proceeding with the
               rest of the test
            f. Start the traffic from the Emulator towards the DUT targeted
               at a specific route say routeA. Initially traffic would be
               observed on the Best Egress interface
            g. Trigger the Hard Reset event of Best Egress Interface on DUT
            h. Measure the Convergence Time for the event to be detected and
               traffic to be forwarded to Next-Best Egress Interface
    
               Time of convergence = time-traffic flow - time-reset.
    
            i. Stop the offered load and wait for the queues to drain and
               Restart
            j. It is recommended that the test be repeated with varying
               number of routes and route mixtures or with number of routes
               & route mixtures closer to what is deployed in operational
               networks
            k. When varying number of routes are used, convergence Time is
               measured using the Loss Derived method [IGP-Data]
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 21]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
           l. Convergence Time in this scenario is influenced by Failure
               detection time on Tester, BGP Keep Alive Time and routing,
               forwarding table update time
    
    
     5.5. BGP Soft Reset
    
    Objective:
    
     This test measures the route convergence time taken by an
     implementation to service a BGP Route Refresh message and advertise a
     route
    
     Reference Test Setup:
    
     This test uses the setup as shown in figure 2
    
     Procedure:
    
            a. The BGP implementation on DUT & Helper Node needs to support
               BGP Route Refresh Capability [RFC 2918]
            b. All devices to be synchronized using NTP
            c. All variables affecting Convergence like authentication,
               policies, timers should be set to basic-test defaults.
            d. DUT and Helper Node are configured in the same Autonomous
               System whereas Emulator is configured under a different
               Autonomous System
            e. Establish BGP adjacency between DUT and Emulator
            f. Establish BGP adjacency between DUT and Helper Node
            g. To ensure adjacency establishment, wait for 3 KeepAlives from
               the DUT or a configurable delay before proceeding with the
               rest of the test
            h. Configure a policy under BGP on Helper Node to deny routes
               received from DUT
            i. Advertise routeA from the Emulator to the DUT
            j. The DUT will try to advertise the route to Helper Node will
               be denied
            k. Wait for 3 KeepAlives
            l. Start the traffic from the Emulator towards the Helper Node
               targeted at a specific route say routeA. Initially no traffic
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 22]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
              would be observed on the Egress interface, as routeA is not
               present
            m. Remove the policy on Helper Node and issue a Route Refresh
               request towards DUT. Note the timestamp of this event. This
               is the RefreshTime
            n. Record the time when the traffic targeted towards routeA is
               received on the Egress Interface. This is RecTime
            o. The following equation represents the Route Refresh
               Convergence Time per route
                i. Route Refresh Convergence Time = (RecTime -
                    RefreshTime)
    
     5.6. BGP Route Withdrawal Convergence Time
    
    Objective:
    
     This test measures the route convergence time taken by an
     implementation to service a BGP Withdraw message and advertise the
     withdraw
    
     Reference Test Setup:
    
     This test uses the setup as shown in figure 2
    
    
     Procedure:
    
            a. This test consists of 2 steps to determine the Total Withdraw
               Processing Time
            b. Step 1:
                i. All devices to be synchronized using NTP
               ii. All variables should be set to basic-test parameters.
              iii. DUT and Helper Node are configured in the same
                    Autonomous System whereas Emulator is configured under a
                    different Autonomous System
               iv. Establish BGP adjacency between DUT and Emulator
                v. To ensure adjacency establishment, wait for 3 KeepAlives
                    from the DUT or a configurable delay before proceeding
                    with the rest of the test
               vi. Start the traffic from the Emulator towards the DUT
                    targeted at a specific route say routeA. Initially no
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 23]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
                   traffic would be observed on the Egress interface as the
                    routeA is not present on DUT.
              vii. Advertise routeA from the Emulator to the DUT
             viii. The traffic targeted towards routeA is received on the
                    Egress Interface
               ix. Now the Tester sends request to withdraw routeA to DUT.
                    TRx(Awith) nicknamed WdrawTime1
    
                x. Record the time when no traffic is observed on the
                    Egress Interface.
    
                    This is the RouteRemoveTime1(A)
    
                    WdrawConvTime1 = RouteRemoveTime1(A)
    
               xi. The difference between the RouteRemoveTime1 and
                    WdrawTime1 is the WdrawConvTime1
    
    
            c. Step 2:
                i. Continuing from Step 1, re-advertise routeA back to DUT
                    from Tester
               ii. The DUT will try to advertise the routeA to Helper Node
                    (assumption there exists a session between DUT and
                    helper node)
              iii. Start the traffic from the Emulator towards the Helper
                    Node targeted at a specific route say routeA. Traffic
                    would be observed on the Egress interface after routeA
                    is received by the Helper Node
    
                    WATime=time traffic first flows
    
               iv. Now the Tester sends a request to withdraw routeA to
                    DUT. This is the WdrawTime2
    
                    WAWtime-TRx(RouteA) = is nicknamed WdrawTime2
    
                v. DUT processes the withdraw and sends it to Helper Node
               vi. Record the time when no traffic is observed on the
                    Egress Interface of Helper Node. This is the
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 24]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
                   TR-WAW(DUT,RouteA) = RouteRemoveTime2
    
              vii. Total withdraw processing time is
    
                   TotalWdrawTime = ((RouteRemoveTime2 - WdrawTime2) -
                    WdrawConvTime1)
    
    
    
     5.7. BGP Path Attribute Change Convergence Time
    
    Objective:
    
     This test measures the route convergence time taken by an
     implementation to service a BGP Path Attribute Change
    
     Reference Test Setup:
    
     This test uses the setup as shown in figure 1
    
    
     Procedure:
    
            a. This test only applies to Well-Known Mandatory Attributes
               like Origin, AS Path, Next Hop
            b. In each iteration of test only one of these mandatory
               attributes need to be varied whereas the others remain the
               same
            c. All devices to be synchronized using NTP
            d. All variables should be set to basic-test parameters
            e. Advertise the route, routeA over the Best Egress Interface
               only, making it the preferred next hop
            f. To ensure adjacency establishment, wait for 3 KeepAlives from
               the DUT or a configurable delay before proceeding with the
               rest of the test
            g. Start the traffic from the Emulator towards the DUT targeted
               at the specific route say routeA. Initially traffic would be
               observed on the Best Egress interface
            h. Now advertise the same route routeA on the Next-Best Egress
               Interface but by varying one of the well-known mandatory
               attributes to have a preferred value over that interface. The
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 25]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
              other values need to be same as what was advertised on the
               Best-Egress adjacency.
    
               TRx(Path-Change) =  Path Change Event Time
    
            i. Measure the Convergence Time for the event to be detected and
               traffic to be forwarded to Next-Best Egress Interface
    
               DUT(Path-Change, RouteA) = Path-switch time
    
               Convergence = Path-switch time - Path Change Event Time.
    
            j. Stop the offered load and wait for the queues to drain and
               Restart
    
    
    
     5.8. BGP Graceful Restart Convergence Time
    
    Objective:
    
     This test measures the route convergence time taken by an
     implementation during a Graceful Restart Event
    
     Reference Test Setup:
    
     This test uses the setup as shown in figure 4
    
     Procedure:
    
            a. It measures the time taken by an implementation to service a
               BGP Graceful Restart Event and advertise a route
            b. The Helper Nodes are the same model as DUT and run the same
               BGP implementation as DUT
            c. The BGP implementation on DUT & Helper Node needs to support
               BGP Graceful Restart Mechanism [RFC4724]
            d. All devices to be synchronized using NTP
            e. All variables are set to basic-test values.
            f. DUT and Helper Node-1 are configured in the same Autonomous
               System whereas Emulator and Helper Node-2 are configured
               under different Autonomous Systems
            g. Establish BGP adjacency between DUT and Helper Nodes
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 26]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
           h. Establish BGP adjacency between Helper Node-2 and Emulator
            i. To ensure adjacency establishment, wait for 3 KeepAlives from
               the DUT or a configurable delay before proceeding with the
               rest of the test
            j. Configure a policy under BGP on Helper Node-1 to deny routes
               received from DUT
            k. Advertise routeA from the Emulator to Helper Node-2
            l. Helper Node-2 advertises the route to DUT and DUT will try to
               advertise the route to Helper Node-1 which will be denied
            m. Wait for 3 KeepAlives
            n. Start the traffic from the Emulator towards the Helper Node-1
               targeted at the specific route say routeA. Initially no
               traffic would be observed on the Egress interface as the
               routeA is not present
            o. Perform a Graceful Restart Trigger Event on DUT and note the
               time. This is the GREventTime
            p. Remove the policy on Helper Node-1
            q. Record the time when the traffic targeted towards routeA is
               received on the Egress Interface.
    
               TRr(DUT, routeA). This is nicknamed RecTime.
    
            r. The following equation represents the Graceful Restart
               Convergence Time
                i. Graceful Restart Convergence Time = ((GREventTime -
                    RecTime) - RIB-IN)
    
            s. It is assumed in this test case that after a Switchover is
               triggered on the DUT, it will not have any cycles to process
               BGP Refresh messages.
               The reason for this assumption is that there is a narrow
               window of time where after switchover when we remove the
               policy from Helper Node -1, implementations might generate
               Route-Refresh automatically and this request might be
               serviced before the DUT actually switches over and
               reestablishes BGP adjacencies with the peers
    
    
    
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 27]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
    6. Reporting Format
    
       For each test case, it is recommended that the reporting tables
        below are completed and all time values SHOULD be reported with
        resolution as specified in [RFC 4098].
    
    
        Parameter                        Units
        Test case                        Test case number
        Test topology                    1,2,3 or 4
        Parallel links                   Number of parallel links
        Interface type                   GigE, POS, ATM, other
        Convergence Event                Hard reset, Soft reset, link
                                          failure, or other defined
        eBGP sessions                    Number of eBGP sessions
        iBGP sessions                    Number of iBGP sessions
        eBGP neighbor                    Number of eBGP neighbors
        iBGP neighbor                    Number of iBGP neighbors
        Routes per peer                  Number of routes
        Total unique routes              Number of routes
        Total non-unique routes          Number of routes
        IGP configured                   ISIS, OSPF, static, or other
        Route Mixture                    Description of Route mixture
        Route Packing                    Number of routes in an update
        Policy configured                Yes, No
        Packet size offered to the DUT   Bytes
        Offered load                     Packets per second
        Packet sampling interval on      Seconds
        tester
        Forwarding delay threshold       Seconds
        Timer value configured on DUT
           Interface failure indication  Seconds
           delay
           Hold timer                    Seconds
           MinRouteAdvertisementInterval Seconds
           (MRAI)
           MinASOriginationInterval      Seconds
           (MAOI)
           Keepalive                     Seconds
           ConnectRetry                  Seconds
        TCP Parameters for DUT and
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 28]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
       tester
           MSS                           Bytes
           Slow start threshold          Bytes
           Maximum window size           Bytes
    
    
           Test Details:
    
            a. If the Offered Load matches a subset of routes, describe how
               this subset is selected.
            b. Describe how the Convergence Event is applied; does it cause
               instantaneous traffic loss or not.
            c. If there is any policy configured, describe the configured
               policy.
    
           Complete the table below for the initial Convergence Event and
           the reversion Convergence Event.
    
        Parameter                       Unit
        Conversion Event                Initial or reversion
        Traffic Forwarding Metrics
           Total number of packets      Number of packets
           offered to DUT
           Total number of packets      Number of packets
           forwarded by DUT
           Connectivity Packet Loss     Number of packets
           Convergence Packet Loss      Number of packets
           Out-of-order packets         Number of packets
           Duplicate packets            Number of packets
        Convergence Benchmarks
           Rate-derived Method[IGP-
           Data]:
             First route convergence    Seconds
             time
             Full convergence time      Seconds
           Loss-derived Method [IGP-
           Data]:
             Loss-derived convergence   Seconds
             time
           Route-Specific Loss-Derived
           Method:
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 29]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
            Minimum R-S convergence    Seconds
             time
             Maximum R-S convergence    Seconds
             time
             Median R-S convergence     Seconds
             time
             Average R-S convergence    Seconds
             time
    
        Loss of Connectivity Benchmarks
           Loss-derived Method:
             Loss-derived loss of       Seconds
             connectivity period
           Route-Specific loss-derived
           Method:
             Minimum LoC period [n]     Array of seconds
             Minimum Route LoC period   Seconds
             Maximum Route LoC period   Seconds
             Median Route LoC period    Seconds
             Average Route LoC period   Seconds
    
    7. Security Considerations
    
        Benchmarking activities as described in this memo are limited to
        technology characterization using controlled stimuli in a laboratory
        environment, with dedicated address space and the constraints
        specified in the sections above.
    
        The benchmarking network topology will be an independent test setup
        and MUST NOT be connected to devices that may forward the test
        traffic into a production network, or misroute traffic to the test
        management network.
    
        Further, benchmarking is performed on a "black-box" basis, relying
        solely on measurements observable external to the DUT/SUT.
    
        Special capabilities SHOULD NOT exist in the DUT/SUT specifically
        for benchmarking purposes.  Any implications for network security
        arising from the DUT/SUT SHOULD be identical in the lab and in
        production networks.
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 30]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
    8. IANA Considerations
    
       This document requires no IANA considerations.
    
     9. References
    
    9.1. Normative References
    
       [RFC1771] Rekhter, Y. and Li, T., "A Border Gateway Protocol 4(BGP-
                  4)", RFC 4271, March 1995.
       [RFC4098] Berkowitz, H. et al., "Terminology for benchmarking BGP
                  device convergence in control plane", RFC4098, June 2005
       [RFC1242] Bradner, S., "Benchmarking terminology for network
                  interconnection devices", RFC 1242, July 1991.
       [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
                  Requirement Levels", BCP 14, RFC 2119, March 1997.
       [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
                  Network Interconnect Devices", RFC 2544, March 1999
       [IGP-Data] Poretsky, S. et al., "Terminology for benchmarking Link-
                  state IGP data plane convergence," draft-ietf-bmwg-igp-
                  dataplane-conv-term-21, May 2010
    
    9.2. Informative References
    
       [RFC2858] Bates, T. et al., "Multiprotocol extensions for BGP-4,"
                  RFC 2858, June 2000
       [RFC2545] Marques, P. and F. Dupont, "Use of BGP-4 multiprotocol
                  extensions for IPv6 Inter-Domain Routing," RFC2545, March
                  1999
       [RFC3107] Rekhter, Y. and E. Rosen, "Carrying label information in
                  BGP-4," RFC 3107, May 2001
       [RFC2385] Heffernan, A., "Protection of BGP Sessions via the TCP MD5
                  signature option," RFC2385, August 1998
       [RFC2918] Chen, E., "Route Refresh capability for BGP-4," RFC 2918,
                  September 2000
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 31]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
       [RFC4724] Sangli, S. et al., "Graceful restart mechanism for BGP,"
                  RFC 4724, Jan 2007
       [RFC1983] Malkin, G.,"Internet Users' Glossary", RFC 1983,  August
                 1996
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 32]


    Internet-Draft     BGP Convergence Methodology           October 2010
    
    Authors Addresses
    
       Rajiv Papneja
        Isocore
        12359 Sunrise Valley Dr. STE100
        Reston, VA 20191
        Phone: +1.703.860.9273
        Email: rpapneja@isocore.com
    
        Bhavani Parise
        Cisco Systems
        170 West Tasman Drive
        San Jose, CA 95134
        Phone: +408-853-6346
        Email: bhavani@cisco.com
    
       Susan Hares
        Huawei Technologies (USA)
        2330 Central Expressway
        Santa Clara, CA 95050
        Phone: +408-330-4581
        Cell:  +1-734-604-0332
        Email shares@huawei.com
    
       Eric Brendel
        Independent Consultant
        154 3rd St,
        Fair Haven, NJ 07704
        Phone:+1.732.895.1504
        Email: brendel@pektel.com
    
        Mohan Nanduri
        Microsoft
        12012 Sunset Hills Rd.
        Reston, VA 20190
        Phone: +1.703.627.6455
        Email: mnanduri@microsoft.com
    
        Jay Karthik
        Cisco Systems
        170 West Tasman Drive
        San Jose, CA 95134
        Phone: +1.978.319.0527
        Email: jkarthik@cisco.com
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     Papneja, et al.    Expires April 18, 2011               [Page 33]