Network Working Group J. H. Dunn
INTERNET-DRAFT C. E. Martin
Expires: February, 2001 ANC, Inc.
July, 2000
Framework for Router Benchmarking
<draft-ietf-bmwg-rtr-framework-00.txt>
Status of this Memo
This document is an Internet-Draft and is in full conformance with all
provisions of Section 10 of RFC2026. Internet-Drafts are working
documents of the Internet Engineering Task Force (IETF), its areas, and
its working groups. Note that other groups may also distribute working
documents as Internet-Drafts. Internet-Drafts are draft documents valid
for a maximum of six months and may be updated, replaced, or obsoleted
by other documents at any time. It is inappropriate to use Internet-
Drafts as reference material or to cite them other than as "work in
progress." To view the entire list of current Internet-Drafts, please
check the "1id-abstracts.txt" listing contained in the Internet-Drafts
Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net (Northern
Europe), ftp.nis.garr.it (Southern Europe), munnari.oz.au (Pacific Rim),
ftp.ietf.org (US East Coast), or ftp.isi.edu (US West Coast).
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Copyright Notice
Copyright (C) The Internet Society (1999). All Rights Reserved.
Abstract
This memo discusses and proposes a framework for the development of IP
performance benchmarking methodologies in the case of systems under test
(SUT) running IETF routing protocols. The intent of this document is to
facilitate the use of existing metrics developed by the BMWG and other
working groups. This will be accomplished by specifying router
configuration and state parameters and characterizing their effect on IP
packet forwarding in terms of these existing metrics. The terms defined
in this memo will be used in addition to terms defined in RFCs 1242,
2285, and 2544 and 2761. This memo is a product of the Benchmarking
Methodology Working Group (BMWG) of the Internet Engineering Task Force
(IETF).
Dunn & Martin [Page 1]
INTERNET-DRAFT Framework for Router Benchmarking July 2000
I. Background.
1. Introduction.
The purpose of this document is to define a general framework for
particular methodologies to be developed by the Benchmarking Methodology
Working Group (BMWG) of the Operational Requirements Area. This memo
extends existing work, specifically RFC 2330 "Framework for IP
Performance Metrics", May 1998. The goal of this effort is to produce a
set of metrics and methodologies, based on existing metrics, to
characterize the effects of a router's configuration and state on IP
packet forwarding performance as defined in RFC 2544 "Benchmarking
Methodology for Network Interconnect Devices". Metrics will be defined
in accordance with the framework specified in RFC 1242 "Benchmarking
Terminology for Network Interconnect Devices". Methodologies will be
defined in accordance with the framework specified in RFC 2544. This
effort will focus on the effects of the following configuration and
states parameter sets on IP packet forwarding performance:
1. Static configuration parameters, e.g., route cache vs. total
available memory size
2. Dynamic configuration parameters, e.g., BGP4
MinRouteAdvertisementInterval
3. Static states, e.g., response to BGP4 NLRI updates
4. Dynamics states, e.g., response to conflicting BGP4 NLRI updates
(route flapping)
Metrics will be defined which characterize both the impact on IP packet
forwarding performance and router response to route updates based on
these states. Packet forwarding performance assessment will be based on
metrics described in RFCs 1242, 2285 and 2761. The assessment of router
response will be based on the values of MIB objects described in RFC
1850 and the upcoming BGP MIB document. This document will reference
Internet vocabulary, clearly describing Internet components such as
routers, routing protocols, and router MIB element definitions. Any
additional router related vocabulary necessary to develop router metrics
will be defined in this document. Measurement uncertainties and errors
will be described, including how they relate to the analytical framework
shared by many aspects of the Internet engineering discipline.
Dunn & Martin [Page 2]
INTERNET-DRAFT Framework for Router Benchmarking July 2000
2. Existing Definitions.
RFC 1242 "Benchmarking Terminology for Network Interconnect Devices" and
RFC 2330 "Framework for IP Performance Metrics" should be consulted
before attempting to make use of this document. RFC 2544 "Benchmarking
Methodology for Network Interconnect Devices" contains discussions of a
number of terms relevant to the benchmarking of switching devices and
should be consulted. RFC 2285 "Benchmarking Terminology for LAN
Switching Devices" contains a number of terms pertaining to traffic
distributions and datagram interarrival. RFC 2761 "Terminology for ATM
Benchmarking" contains a number terms pertaining to traffic management
[TM4.0, TM4.1]. RFC 1812 "Requirements for IP Version 4 Routers"
contains an excellent glossary with which it is assumed that the reader
is familiar. For the sake of clarity and continuity, this RFC adopts
the template for definitions set out in Section 2 of RFC 1242.
3. Requirements
In this document, the words that are used to define the significance of
each particular requirement are capitalized. These words are: * "MUST"
This word, or the words "REQUIRED" and "SHALL" mean that the item is an
absolute requirement of the specification. * "SHOULD" This word or the
adjective "RECOMMENDED" means that there may exist valid reasons in
particular circumstances to ignore this item, but the full implications
should be understood and the case carefully weighed before choosing a
different course. * "MAY" This word or the adjective "OPTIONAL" means
that this item is truly optional. One vendor may choose to include the
item because a particular marketplace requires it or because it enhances
the product, for example; another vendor may omit the same item.
An implementation is not compliant if it fails to satisfy one or more of
the MUST requirements for the protocols it implements. An
implementation that satisfies all the MUST and all the SHOULD
requirements for its protocols is said to be "unconditionally
compliant"; one that satisfies all the MUST requirements but not all the
SHOULD requirements for its protocols is said to be "conditionally
compliant".
4. Criteria for Router Metrics
RFC 2330 "Framework for IP Performance Metrics" contains a number of
Dunn & Martin [Page 3]
INTERNET-DRAFT Framework for Router Benchmarking July 2000
relevant criteria for performance metrics. In addition, router
performance metrics should be independent of router architecture and
routing protocol. As with IP performance metrics, router metrics will
be dependent on the parameter sets mentioned in section I.1 of this
document. Since all metrics must produce reproducible and self-
consistent results, router metrics are only meaningful in the case where
these parameter sets are stable enough for measurements to be performed.
As a result, auxiliary observations may be required to determine whether
the router is in a stable state.
II. Definitions
The definitions presented in this section have been divided into two
groups. The first group is formal definitions, which are required in
the definitions of the performance metrics but are not themselves
strictly metrics. These definitions are subsumed from other work done
in other working groups both inside and outside the IETF. They are
provided as a courtesy to the reader.
1. Formal Definitions
1.1. Definition Format (from RFC 1242)
Term to be defined.
Definition: The specific definition for the term.
Discussion: A brief discussion of the term, its application and any
restrictions on measurement procedures.
Specification: The working group and document in which the terms are
specified and are listed in the references section.
1.2. Generic Definitions.
1.2.1. Forwarding Decision
Definition: The process by which a router determines, based on information
in the FIB, what the disposition of a packet will be.
Discussion: The forwarding decision is made by the router's forwarder and
is used by the forwarder to correctly forward the packet.
Dunn & Martin [Page 4]
INTERNET-DRAFT Framework for Router Benchmarking July 2000
Specification: N/A
1.2.2. Routing Information Entry
Definition: A single entry in the FIB that provides all information
required for the forwarder to make a forwarding decision.
Discussion: A routing information entry may provide forwarding information
mapping a single network, a class of networks, a specific data link layer
type or other datagram attribute to a unique egress port on the router.
Specification: N/A
1.2.2. Router Port
Definition: A single point of datagram entry and exit, characterized by a
single data link layer connection to other network equipment, access speed
and, perhaps, access controls.
Discussion: A router port is the primary source of datagrams to the
forwarder and destination of datagrams from the forwarder.
Specification: N/A
1.3. Static Configuration Parameters.
Definition: Static Configuration Parameters are those attributes of a
router, which are not altered by information gathered during the operation
of a routing protocol.
1.3.1. Forwarding Information Base Cache
Definition: A dedicated portion of the FIB, which contains the most current
or most often accessed routing information.
Discussion: The cache provides a method for rapidly accessing routing
information without resorting to a full scale search of the FIB.
Specification: N/A
1.3.2. Forwarding Information Base Cache Size
Dunn & Martin [Page 5]
INTERNET-DRAFT Framework for Router Benchmarking July 2000
Definition: The number of routing information entries allocated to the FIB
cache.
Discussion: The size of the cache allocation significantly effects router
performance. If the cache allocation is not large enough, a full scale
search of the FIB is required in order to make forwarding decisions. If
the cache allocation is a significant percentage of the FIB size, the cache
search differs very little from a full scale FIB search.
1.3.3. Forwarding Information Base Size
Definition: The maximum number of routing information entries the FIB can
contain.
Discussion: This parameter impacts the maximum number of reachable
networks supported by the router; however, the relationship between the two
parameter is not always linear.
Specification: N/A
1.4. Dynamic Configuration Parameters.
Definition: Dynamic Configuration Parameters are those attributes of a
router, which are altered by information gathered during the operation of a
routing protocol.
1.3.2. Number of Reachable Networks
Definition: The current number of networks that the routing information in
the FIB allows the router to reach.
Discussion: The number of reachable networks will depend on static
parameters, such as the maximum number of routing entries in the FIB, and
dynamic parmeters, such as whether the FIB accurately reflects the state of
the network and route aggregation.
Specification: N/A
3. Security Considerations.
As this document is solely for providing terminology and describes
Dunn & Martin [Page 6]
INTERNET-DRAFT Framework for Router Benchmarking July 2000
neither a protocol nor an implementation, there are no security
considerations associated with this document.
4. Notices
Internet Engineering Task Force
The IETF takes no position regarding the validity or scope of any
intellectual property or other rights that might be claimed to pertain
to the implementation or use of the technology described in this
document or the extent to which any license under such rights might or
might not be available; neither does it represent that it has made any
effort to identify any such rights. Information on the IETFs procedures
with respect to rights in standards-track and standards-related
documentation can be found in BCP-11. Copies of claims of rights made
available for publication and any assurances of licenses to be made
available, or the result of an attempt made to obtain a general license
or permission for the use of such proprietary rights by implementors or
users of this specification can be obtained from the IETF Secretariat.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary rights,
which may cover technology that may be required to practice this
standard. Please address the information to the IETF Executive
Director.
5. Disclaimer
Copyright (C) The Internet Society (1999). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it or
assist in its implementation may be prepared, copied, published and
distributed, in whole or in part, without restriction of any kind,
provided that the above copyright notice and this paragraph are included
on all such copies and derivative works. However, this document itself
may not be modified in any way, such as by removing the copyright notice
or references to the Internet Society or other Internet organizations,
except as needed for the purpose of developing Internet standards in
which case the procedures for copyrights defined in the Internet
Standards process must be followed, or as required to translate it into
languages other than English. The limited permissions granted above are
Dunn & Martin [Page 7]
INTERNET-DRAFT Framework for Router Benchmarking July 2000
perpetual and will not be revoked by the Internet Society or its
successors or assigns. This document and the information contained
herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE
INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
6. References
[IPv4-Router] Baker, F., "Requirements for IP Version 4 Routers", RFC
1812, June 1995
7. Editors Addresses
Jeffrey Dunn
Advanced Network Consultants, Inc.
4214 Crest Place, Ellicott City, MD 21043 USA
Phone: +1 (410) 750-1700, E-mail: Jeffrey.Dunn@worldnet.att.net
Cynthia Martin
Advanced Network Consultants, Inc.
4214 Crest Place, Ellicott City, MD 21043 USA
Phone: +1 (410) 750-1700, E-mail: Cynthia.E.Martin@worldnet.att.net
Dunn & Martin [Page 8]
