Network Working Group
INTERNET-DRAFT
Expires in: August 2006
Scott Poretsky
Reef Point Systems
Pierre-Yves Geay
Alcatel
February 2006
Methodology for Benchmarking Network-layer
Traffic Control Mechanisms
<draft-ietf-bmwg-dsmmeth-00.txt>
Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Status of this Memo
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as
Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Copyright Notice
Copyright (C) The Internet Society (2006).
Abstract
This document describes the methodology for the benchmarking of
devices that implement traffic control based on IP precedence or
diff-serv code point criteria. The methodology is to be applied
to measurements made on the data plane to evaluate the
performance of the traffic control mechanisms. The methodology
permits the specific traffic control mechanisms and
configuration commands to vary between DUTs. The methodology
uses much of the Terminology defined in [Pp06].
Poretsky and Geay [Page 1]
INTERNET-DRAFT Methodology for Benchmarking February 2006
Network-layer Traffic Control Mechanisms
Table of Contents
1. Introduction ...............................................2
2. Existing definitions .......................................2
3. Test Setup..................................................3
3.1 Test Topologies............................................3
3.2 Test Considerations........................................4
3.3 Reporting Format...........................................5
4. Test Cases..................................................6
4.1 Undifferentiated Response..................................6
4.2 Traffic Control Baseline Performance.......................6
4.3 Traffic Control Performance with Forwarding Congestion.....7
5. IANA Considerations.........................................7
6. Security Considerations.....................................7
7. References..................................................8
8. Author's Address............................................9
9. Full Copyright Statement....................................9
1. Introduction
This document describes the methodology for the benchmarking of
devices that implement traffic control based on IP precedence or
diff-serv code point criteria. The methodology is to be applied
to measurements made on the data plane to evaluate the
performance of the traffic control mechanisms. The methodology
permits the specific traffic control mechanisms and
configuration commands to vary between DUTs. The methodology
uses much of the Terminology defined in [Pp06].
2. Existing definitions
For the sake of clarity and continuity this RFC adopts the
template for definitions set out in Section 2 of RFC 1242.
Definitions are indexed and grouped together in sections for ease
of reference. Reference [Pp06] for benchmarking terminology.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119
[Br97]. RFC 2119 defines the use of these key words to help make the
intent of standards track documents as clear as possible. While this
document uses these keywords, this document is not a standards track
document.
Poretsky and Geay [Page 2]
INTERNET-DRAFT Methodology for Benchmarking February 2006
Network-layer Traffic Control Mechanisms
3. Test Setup
3.1 Test Topologies
Figure 1 shows the Test Topology for benchmarking performance
when Forwarding Congestion does not exist on the egress link.
This topology is to be used when benchmarking the Undifferentiated
Response and the Traffic Control without Forwarding Congestion
Figure 2 shows the Test Topology for benchmarking performance
when Forwarding Congestion does not exist on the egress link.
This topology is to be used when benchmarking the Traffic Control
with Forwarding Congestion. The Forwarding Congestion is
produced by offering load to two ingress interfaces on the DUT
destined for the same single egress interface. The aggregate of
the ingress offered load MUST exceed the Forwarding Capacity of
the egress link to produce Forwarding Congestion.
Expected
Vector
|
|
\/
--------- Offered Vector ---------
| |<--------------------------------| |
| | | |
| | | |
| DUT | | Tester|
| | | |
| |~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>| |
| | Output Vector | |
--------- ---------
Figure 1. Test Topology for Benchmarking
Without Forwarding Congestion
Expected
Vector
|
|
\/
--------- Offered Vector ---------
| |<--------------------------------| |
| | Ingress Links 1,2 | |
| |<--------------------------------| |
| DUT | | Tester|
| | | |
| |~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>| |
| | Output Vector | |
--------- ---------
Figure 2. Test Topology for Benchmarking
With Forwarding Congestion
Poretsky and Geay [Page 3]
INTERNET-DRAFT Methodology for Benchmarking February 2006
Network-layer Traffic Control Mechanisms
3.2 Test Considerations
3.2.1 Routing Configuration
Routing Protocols SHOULD NOT be used. All routing decisions
SHOULD be made based upon pre-configured static routes.
3.2.2 Interface Types
All test cases in this methodology document may be executed with
any interface type. All interfaces MUST be the same media and
Throughput [5,6] for each test case.
3.2.3 Offered Vector
The Offered Vector MUST be configured on the Tester as follows:
a. The Offered Load MUST be the Forwarding Capacity of the
device at a fixed packet size.
b. The Forwarding Capacity MUST be measured at the egress
interface of the DUT
c. Each test case MUST be executed using a single, selectable
packet size. Packet Size is measured in bytes and includes
the IP header and payload. If IPsec packets are used then
the packet size also includes it. Packet Size must be
equal to or less than the interface MTU so that there is no
fragmentation.
d. It is RECOMMENDED that the number of flows used be
1000, 10000, and/or 100000. A flow MUST be identified
by its DSCP, IP Source Address, and IP Destination Address.
e. It is RECOMMENDED that the number of DSCPs used be
1, 2, 3, 4, 6, 8, 16, and/or 64. When the number of DSCPs
is 1 then the Undifferentiated Response is benchmarked.
The actual values of the DSCPs used is selectable.
3.2.4 Test Duration
It is RECOMMENDED that the Test Duration for each test case
includes a minimum of 10 minutes of Offered Load and
Output Vector measurement
3.2.5 Expected Vector
The Expected Vector is configured on the DUT. The Traffic
Control mechanisms and specific configuration commands may
vary between DUTs. Test Cases may be repeated with
variation to the Expected Vector to produce a more
benchmark results.
Poretsky and Geay [Page 4]
INTERNET-DRAFT Methodology for Benchmarking February 2006
Network-layer Traffic Control Mechanisms
3.3 Reporting Format
For each test case, it is recommended that the following
reporting format be completed:
PARAMETERS UNITS
---------- -----
Offered Vector
--------------
Offered Load pps
Number of DSCPs {1..64}
Codepoint Set {0..63, 0..63, ... , x}
Number of Flows {1000, 10000, 100000}
Number of Flows per DSCP Number of Flows/Number of DSCPs
Packet Size bytes
Undifferentiated Response (Number of DSCPs = 1)
-------------------------
Forwarding Capacity pps
Packet Loss packets
Forwarding Delay
Minimum msec
Maximum msec
Average msec
Jitter
Average msec
Peak-to-Peak msec
Out-of-Order Packets packets
Duplicate Packets packets
Expected Vector {for DSCP=n} (as configured on DUT)
----------------------------
Forwarding Capacity pps
Packet Loss packets
Forwarding Delay
Minimum msec
Maximum msec
Average msec
Output Vector {for DSCP=n}
--------------------------
Forwarding Capacity pps
Packet Loss packets
Forwarding Delay
Minimum msec
Maximum msec
Average msec
Jitter
Average msec
Peak-to-Peak msec
Out-of-Order Packets packets
Duplicate Packets packets
Poretsky and Geay [Page 5]
INTERNET-DRAFT Methodology for Benchmarking February 2006
Network-layer Traffic Control Mechanisms
4. Test Cases
4.1 Undifferentiated Response
Purpose:
To establish the baseline performance of the DUT.
Procedure:
1. Configure DUT with Expected Vector.
2. Configure the Tester for the Offered Vector.
Number of DSCPs MUST equal 1 and the
RECOMMENDED DSCP value is 0 (Best Effort).
Use 1000 Flows identified by IP SA/DA. All flows
have the same DSCP value.
3. Using the Test Topology in Figure 1, source the
Offered Load from the Tester to the DUT.
4. Measure and record the Output Vector.
5. Maintain offered load for 10 minutes minimum
to observe possible variations in measurements.
6. Repeat steps 2 through 5 with 10000 and 100000
Flows.
Expected Results:
Forwarding Vector equals the Offered Load. There
is no packet loss and no out-of-order packets.
4.2 Traffic Control Baseline Performance
Purpose:
To benchmark the Output Vectors for a Codepoint Set
without Forwarding Congestion.
Procedure:
1. Configure DUT with Expected Vector for each DSCP in
the Codepoint Set.
2. Configure the Tester for the Offered Vector.
Number of DSCPs MUST 2 or more. Any DSCP values can
be used. Use 1000 Flows identified by IP SA/DA
and DSCP value.
3. Using the Test Topology in Figure 1, source the
Offered Load from the Tester to the DUT.
4. Measure and record the Output Vector for each DSCP
in the Codepoint Set.
5. Maintain offered load for 10 minutes minimum
to observe possible variations in measurements.
6. Repeat steps 2 through 5 with 10000 and 100000
Flows.
7. Increment number of DSCPs used and repeat steps
1 through 6.
Expected Results:
Forwarding Vector equals the Offered Load. There is
no packet loss and no out-of-order packets. Output
vectors match the Expected Vectors for each DSCP in
the Codepoint Set.
Poretsky and Geay [Page 6]
INTERNET-DRAFT Methodology for Benchmarking February 2006
Network-layer Traffic Control Mechanisms
4.3 Traffic Control Performance with Forwarding Congestion
Purpose:
To benchmark the Output Vectors for a Codepoint Set
with Forwarding Congestion.
Procedure:
1. Configure DUT with Expected Vector for each DSCP in
the Codepoint Set.
2. Configure the Tester for the Offered Vector.
Number of DSCPs MUST 2 or more. Any DSCP values can
be used. Use 1000 Flows identified by IP SA/DA
and DSCP value. The Offered Load MUST exceed the
Forwarding Capacity of a single egress link by 25%
using 2 ingress links.
3. Using the Test Topology in Figure 2, source the
Offered Load from the Tester to the DUT. The
aggregate of the ingress offered load MUST exceed
the Forwarding Capacity of the egress link to
produce Forwarding Congestion.
4. Measure and record the Output Vector for each DSCP
in the Codepoint Set.
5. Maintain offered load for 10 minutes minimum
to observe possible variations in measurements.
6. Repeat steps 2 through 5 with 10000 and 100000
Flows.
7. Increment offered load by 25% to 200% maximum.
8. Increment number of DSCPs used and repeat steps
1 through 6.
Expected Results:
Forwarding Vector equals the Offered Load. There is
no packet loss and no out-of-order packets. Output
vectors match the Expected Vectors for each DSCP in
the Codepoint Set.
Poretsky and Geay [Page 7]
INTERNET-DRAFT Methodology for Benchmarking February 2006
Network-layer Traffic Control Mechanisms
5. IANA Considerations
This document requires no IANA considerations.
6. Security Considerations
Documents of this type do not directly affect the security of
the Internet or of corporate networks as long as benchmarking
is not performed on devices or systems connected to
production networks.
Packets with unintended and/or unauthorized DSCP or IP
precedence values may present security issues. Determining
the security consequences of such packets is out of scope for
this document.
7. Acknowledgments
Poretsky and Geay [Page 7]
INTERNET-DRAFT Methodology for Benchmarking February 2006
Network-layer Traffic Control Mechanisms
8. References
8.1 Normative References
[Br91] Bradner, S., "Benchmarking Terminology for Network
Interconnection Devices", RFC 1242, July 1991.
[Br97] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", RFC 2119, March 1997
[Br98] Braden, B., Clark, D., Crowcroft, J., Davie, B.,
Deering, S., Estrin, D., Floyd, S., Jacobson, V.,
Minshall, G., Partridge, C., Peterson, L., Ramakrishnan,
K., Shenker, S., Wroclawski, J. and L. Zhang,
"Recommendations on Queue Management and Congestion
Avoidance in the Internet", RFC 2309, April 1998.
[Ma98] Mandeville, R., "Benchmarking Terminology for LAN
Switching Devices", RFC 2285, July 1998.
[Ni98] Nichols, K., Blake, S., Baker, F., Black, D., "Definition
of the Differentiated Services Field (DS Field) in the
IPv4 and IPv6 Headers", RFC 2474, December 1998.
[Pp06] Perser, J., Poretsky, S., Erramilli, S., and Khurana, S.,
"Terminology for Benchmarking Network-layer Traffic
Control Mechanisms", draft-ieft-bwmg-dsmterm-12, work
in progress, 2006.
8.2 Informative References
[Bl98] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z.,
Weiss, W., "An Architecture for Differentiated Services",
RFC 2475, December 1998.
[Br99] Bradner, S., McQuaid, J. "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999
[Fl93] Floyd, S., and Jacobson, V., "Random Early Detection
gateways for Congestion Avoidance", IEEE/ACM
Transactions on Networking, V.1 N.4, August 1993, p.
397-413. URL "ftp://ftp.ee.lbl.gov/papers/early.pdf".
[Ja99] Jacobson, V., Nichols, K., Poduri, K., "An Expedited
Forwarding PHB", RFC 2598, June 1999
[Ma91] Mankin, A., Ramakrishnan, K., "Gateway Congestion Control
Survey", RFC 1254, August 1991
[Sc96] Schulzrinne, H., Casner, S., Frederick, R., Jacobson, V.,
"RTP: A Transport Protocol for Real-Time Applications",
RFC 1889, January 1996
Poretsky and Geay [Page 8]
INTERNET-DRAFT Methodology for Benchmarking February 2006
Network-layer Traffic Control Mechanisms
9. Authors' Addresses
Scott Poretsky
Reef Point Systems
8 New England Executive Park
Burlington, MA 01803
USA
Phone: + 1 508 439 9008
EMail: sporetsky@reefpoint.com
Pierre-Yves Geay
Alcatel
Orvault, France
Email: Pierre-Yves.Geay@alcatel.fr
Full Copyright Statement
Copyright (C) The Internet Society (2006).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at ietf-
ipr@ietf.org.
Acknowledgement
Funding for the RFC Editor function is currently provided by the
Internet Society.
Poretsky and Geay [Page 9]
