Network Working Group Scott Poretsky
INTERNET-DRAFT Reef Point Systems
Expires in: April 2007
Vijay Gurbani
Lucent Technologies
Carol Davids
Illinois Institute
of Technology
October 2006
Methodology for Benchmarking
SIP Networking Devices
<draft-poretsky-bmwg-sip-bench-meth-00.txt>
Intellectual Property Rights (IPR) statement:
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Status of this Memo
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as
Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Copyright Notice
Copyright (C) The Internet Society (2006).
ABSTRACT
This document describes the methodology for benchmarking Session
Initiation Protocol (SIP) performance as described in Terminology
document [Po06]. The methodology and terminology are to be used
for benchmarking SIP control plane performance with varying control
and media load. Both scale and establishment rate are measured by
control plane performance. The SIP Devices to be benchmarked may
be a single device under test (DUT) or a system under test (SUT).
Benchmarks can be obtained and compared for different types of
devices such as SIP Proxy Server, SBC, P-CSCF, and Server paired
with a Firewall/NAT device.
Poretsky, Gurbani, Davids [Page 1]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
Table of Contents
1. Introduction ...............................................2
2. Existing definitions .......................................2
3. Test Setup..................................................3
3.1 Test Topologies............................................3
3.2 Test Considerations........................................5
3.3 Reporting Format...........................................7
4. Test Cases..................................................7
4.1 Registration Rate.......................................7
4.2 Session Setup Rate......................................8
4.3 Session Setup Rate with Loop Detection Enabled..........9
4.4 Session Setup Rate with Forking.........................9
4.5 Session Setup Rate with Forking and Loop Detection......9
4.6 Session Setup Rate with Media...........................9
4.7 Session Capacity........................................10
4.8 Session Capacity with Media.............................11
4.9 Session Setup Rate with TLS Encrypted SIP...............12
4.10 Session Setup Rate with IPsec Encrypted SIP............12
4.11 Session Setup Rate with SIP Flooding...................12
4.12 IM Rate................................................12
4.13 Presence Rate..........................................12
5. IANA Considerations.........................................13
6. Security Considerations.....................................13
7. Acknowledgements............................................13
8. References..................................................14
9. Author's Address............................................15
10. Full Copyright Statement...................................16
Poretsky, Gurbani, Davids [Page 2]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
1. Introduction
This document describes the methodology for benchmarking Session
Initiation Protocol (SIP) performance as described in Terminology
document [Po06]. The methodology and terminology are to be used
for benchmarking SIP control plane performance with varying control
and media load. Both scale and establishment rate are measured by
control plane performance.
The SIP Devices to be benchmarked may be a single device under test
(DUT) or a system under test (SUT). The DUT is a SIP Server, which
may be any RFC 3261 [Ro02] conforming device. The SUT can be any
device or group of devices containing RFC 3261 conforming
functionality along with Firewall and/or NAT functionality. This
enables benchmarks to be obtained and compared for different types
of devices such as SIP Proxy Server, SBC, P-CSCF, Proxy Server
paired with a Firewall/NAT device, and P-CSCF paired with a
Firewall/NAT device. SIP Associated Media benchmarks can also
be made when testing SUTs.
The test cases covered in this methodology document provide
benchmarks metrics of Registration Rate, SIP Session Setup Rate,
Session Capacity, IM Rate, and Presence Rate. These can be
benchmarked with or without associated Media. Some cases are also
included to cover Forking, Loop detecion, Encrypted SIP, and SIP
Flooding. The test topologies that can be used are described in
the Test Setup section. Topologies are provided for benchmarking
of a DUT or SUT. Benchmarking with Associated Media can be
performed when using a SUT.
SIP permits a wide range of configuration options that are also
explained in the Test Setup section. Benchmark metrics could
possibly be impacted by Associated Media. The selected values for
Session Duration and Media Streams per Session enable benchmark
metrics to be benchmarked without Associated Media. Session Setup
Rate could possibly be impacted by the selected value for Maximum
Sessions Attempted. The benchmark for Session Setup Rate is
measured with a fixed value for Maximum Sessions Attempted.
2. Existing definitions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in BCP 14, RFC 2119.
RFC 2119 defines the use of these key words to help make the
intent of standards track documents as clear as possible. While this
document uses these keywords, this document is not a standards track
document. The term Throughput is defined in RFC 2544.
Terms specific to SIP Performance benchmarking are defined in [Po06].
Poretsky, Gurbani, Davids [Page 3]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
3. Test Setup
3.1 Test Topologies
Figures 1 through 5 below provide various topologies to perform
the SIP Performance Benchmarking. These figures show the Device
Under Test (DUT) to be a single server or a System Under Test (SUT).
Test Topology options to include benchmarking with Associated Media
require use of a SUT and are shown in Figures 4 and 5.
DUT
--------- ---------
| | | |
| | | |
| | SIP | |
|Server |<------------->| Tester|
| | | |
| | | |
| | | |
--------- ---------
Figure 1. Basic SIP Test Topology
SUT
------------------------
--------- --------- ---------
| | | | | |
| | | | | |
| | SIP |Fire- | SIP | |
| Server|<---------------------->| Tester|
| | |Wall | | |
| | | | | |
| | | | | |
--------- --------- ---------
Figure 2. SIP Test Topology with Firewall
SUT
------------------------
--------- --------- ---------
| | | | | |
| | | | | |
| | SIP | NAT | SIP | |
| Server|<---------------------->| Tester|
| | | | | |
| | | | | |
| | | | | |
--------- --------- ---------
Figure 3. SIP Test Topology with NAT Device
Poretsky, Gurbani, Davids [Page 4]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
SUT
------------------------
--------- --------- ---------
| | | | | |
| | | | | |
| | SIP |Fire- | SIP | |
| Server|<---------------------->| Tester|
| | |Wall | | |
| | | | Media | |
| | ---| |---------| |
--------- | --------- ---------
| Media ^
-------------------------|
Figure 4. SIP Test Topology with Media
through Firewall
SUT
------------------------
--------- --------- ---------
| | | | | |
| | | | | |
| | SIP | NAT | SIP | |
| Server|<---------------------->| Tester|
| | | | | |
| | | | Media | |
| | ---| |---------| |
--------- | --------- ---------
| Media ^
-------------------------|
Figure 5. SIP Test Topology with Media
through NAT Device
3.2 Test Considerations
3.2.1 Selection of SIP Transport Protocol
Test cases may be performed with any transport protocol supported
by SIP. This includes, but is not limited to, SIP TCP, SIP UDP,
and TLS. The protocol used for the SIP transport protocol must
be reported with benchmarking results.
3.2.2 Server
The Server is a SIP-speaking device that complies with RFC 3261.
The purpose of this document is to benchmark SIP performance, not
conformance. Conformance to RFC 3261 [Ro02] is assumed for all
tests. The Server may be the DU or a component of a SUT that
includes Firewall and/or NAT functionality. The components of
the SUT may be a single physical device or separate devices.
Poretsky, Gurbani, Davids [Page 5]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
3.2.3 Associated Media
Some tests may require associated media to be present for each SIP
session. The Server is not involved in the forwarding of media.
Associated Media can be benchmarked only with a SUT in which
the media traverses a Firewall, NAT, or Firewall NAT device.
The test topologies to be used when benchmarking SUT performance
for Associated Media are shown in Figures 4 and 5.
3.2.4 Selection of Associated Media Protocol
The test cases specified in this document provide SIP performance
independent of the protocol used for the media stream. Any media
protocol supported by SIP may be used. This includes, but is not
limited to, RTP, RTSP, and SRTP. The protocol used for
Associated Media must be reported with benchmarking results.
3.2.5 Number of Associated Media Streams per SIP Session
Benchmarking results can possibly vary with the number of media
streams per SIP session. When benchmarking a SUT for voice, a
single media stream is used. When benchmarking a SUT for voice
and video, two media streams are used. The number of Associated
Media Streams must be reported with benchmarking results.
3.2.6 Session Duration
SUT performance benchmarks can possibly vary with the duration
of SIP sessions. Session Duration must be reported with
benchmarking results. A Session Duration of zero seconds
indicates transmission of a BYE immediately following successful
SIP establishment indicate by receipt of a 200 OK. An infinite
Session Duration indicates that a BYE is never transmitted.
3.2.7 Attempted Sessions per Second
DUT and SUT performance benchmarks can possibly vary with the
the rate of attempted sessions offered by the Tester. Attempted
Sessions per Second must be reported with benchmarking results.
3.2.8 Stress Testing
The purpose of this document is to benchmark SIP performance, not
system stability under stressful conditions such as a high rate
of Attempted Sessions per Second.
Poretsky, Gurbani, Davids [Page 6]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
3.3 Reporting Format
Test Setup
SIP Transport Protocol = ___________
Session Duration = ___________
Attempted Session Rate = ___________
Maximum Sessions Attempted = ___________
Media Streams per Session = ___________
Media Protocol = ___________
Device Benchmarks
Failed Session Attempts = ___________
Session Capacity = ___________
Zero-Failure Session Setup Rate = ___________
Maximum Retransmits = ___________
Mean Session Setup Delay = ___________
Mean Session TearDown Delay = ___________
4. Test Cases
4.1 Registration Rate
Poretsky, Gurbani, Davids [Page 7]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
4.2 Session Setup Rate
Objective:
To benchmark the maximum session setup rate performance
of the DUT/SUT with zero failures.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Configure Tester for SIP UDP with an Attempted Session Rate =
100 SPS, Session Duration = 0 sec, Maximum Sessions Attempted
= 100,000 and media streams per session=0.
3. Start Tester to initiate SIP Session establishment with the
DUT.
4. Measure Failed Session Attempts and Total Sessions
Established at the Tester.
5. If a Failed Session Attempt is recorded then reduce the
Attempted Session Rate configured on the Tester by 50%.
6. If no Failed Session Attempt is recorded then increase the
Attempted Session Rate configured on the Tester by 50%.
7. Repeat steps 3 through 6 until the Session Setup
Rate is obtained.
Expected Results:
Poretsky, Gurbani, Davids [Page 8]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
4.3 Session Setup Rate with Loop Detection Enabled
4.4 Session Setup Rate with Forking
4.5 Session Setup Rate with Forking and Loop Detection
4.6 Session Setup Rate with Media
Objective:
To benchmark the maximum session setup rate performance
of the SUT with zero failures when Associated Media is
included in the benchmark test.
Procedure:
1. Configure the SUT in the test topology shown in Figure 4 or 5.
2. Configure Tester for SIP UDP with an Attempted Session Rate =
100 SPS, Session Duration = 30 sec, Maximum Sessions Attempted
= 100,000 and media streams per session = 1. The rate of
offered load for each media stream SHOULD be
(eq 1) Offered Load per Media Stream =
Throughput / Maximum Sessions Attempted,
where Throughput is defined in [Ba99].
3. Start Tester to initiate SIP Session establishment with the
SUT and transmit media through the SUT to a destination other
than the server.
4. At the Tester measure Failed Session Attempts, Total Sessions
Established, and Packet Loss [Ba99] of the media.
5. If a Failed Session Attempt or Packet Loss is recorded then
reduce the Attempted Session Rate configured on the Tester by
50%.
6. If no Failed Session Attempt or Packet Loss is recorded then
increase the Attempted Session Rate configured on the Tester
by 50%.
7. Repeat steps 3 through 6 until the Session Setup Rate is
obtained.
8. Repeat steps 1 through 7 for multimedia in which media streams
per session = 2.
Expected Results:
Session Setup Rate results obtained with Associated Media with
any number of media streams per SIP session will be identical
to the Session Setup Rate results obtained without media.
Poretsky, Gurbani, Davids [Page 9]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
4.7 Session Capacity
Objective:
To benchmark the SIP Control Session Capacity of the DUT/SUT.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Configure Tester for SIP UDP with an Attempted Session Rate =
100 SPS, Session Duration = 30 sec, Maximum Sessions Attempted
= 100,000 and media streams per session = 1. The rate of
offered load for each media stream SHOULD be
(eq 1) Offered Load per Media Stream =
Throughput / Maximum Sessions Attempted,
where Throughput is defined in [Ba99].
3. Start Tester to initiate SIP Session establishment with the
SUT and transmit media through the SUT to a destination other
than the server.
4. Measure Failed Session Attempts, Total Sessions Established,
and at the Tester.
5. If a Failed Session Attempt is recorded then reduce the
Maximum Sessions Attempted configured on the Tester by 5,000.
6. If no Failed Session Attempt is recorded then increase the
Maximum Sessions Attempted configured on the Tester by 10,000.
7. Repeat steps 3 through 6 until the Session Capacity is
obtained.
Expected Results:
Poretsky, Gurbani, Davids [Page 10]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
4.8 Session Capacity with Media
Objective:
To benchmark the SIP Control Session Capacity of the SUT
with Associated Media.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Configure Tester for SIP UDP with an Attempted Session Rate =
Zero-Failure Session Setup Rate, Session Duration = 0 sec,
Maximum Sessions Attempted = 10,000 and media streams per
session = 0.
3. Start Tester to initiate SIP Session establishment with the
DUT.
4. Measure Failed Session Attempts, Total Sessions Established,
and Packet Loss [Ba99] at the Tester.
5. If a Failed Session Attempt or Packet Loss is recorded then
reduce the Maximum Sessions Attempted configured on the
Tester by 5,000.
6. If no Failed Session Attempt or Packet Loss is recorded then
increase the Maximum Sessions Attempted configured on the
Tester by 10,000.
7. Repeat steps 3 through 6 until the Session Capacity is
obtained.
8. Repeat steps 1 through 7 for multimedia in which media
streams per session = 2.
Expected Results:
Session Capacity results obtained with Associated Media with
any number of media streams per SIP session will be identical
to the Session Capacity results obtained without media.
Poretsky, Gurbani, Davids [Page 11]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
4.9 Session Setup Rate with TLS Encrypted SIP
4.10 Session Setup Rate with IPsec Encrypted SIP
4.11 Session Setup Rate with SIP Flooding
Objective:
To benchmark the maximum session setup rate performance
of the SUT with zero failures when SIP Flooding is occurring.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
the SUT as shown in Figure 2.
2. Configure Tester for SIP UDP with an Attempted Session Rate =
100 SPS, Session Duration = 0 sec, Maximum Sessions Attempted
= 100,000, Associated Media Streams per session = 0, and
SIP INVITE Message Flood = 500 per second.
3. Start Tester to initiate SIP Session establishment with the
SUT and SIP Flood targetted at the Server.
4. At the Tester measure Failed Session Attempts, Total Sessions
Established, and Packet Loss [Ba99] of the media.
5. If a Failed Session Attempt or Packet Loss is recorded then
reduce the Attempted Session Rate configured on the Tester by
50%.
6. If no Failed Session Attempt or Packet Loss is recorded then
increase the Attempted Session Rate configured on the Tester
by 50%.
7. Repeat steps 3 through 6 until the Session Setup Rate is
obtained.
8. Repeat steps 1 through 7 with SIP INVITE Message Flood =
1000 per second .
Expected Results:
Session Setup Rate results obtained with SIP Flooding may
be degraded.
4.12 IM Rate
4.13 Presence Rate
Poretsky, Gurbani, Davids [Page 12]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
5. IANA Considerations
This document requires no IANA considerations.
6. Security Considerations
Documents of this type do not directly affect the security of
Internet or corporate networks as long as benchmarking is not
performed on devices or systems connected to production
networks. Security threats and how to counter these in SIP
and the media layer is discussed in RFC3261, RFC3550, and
RFC3711 and various other drafts. This document attempts to
formalize a set of common methodology for benchmarking
performance of SIP devices in a lab environment.
7. Acknowledgements
Poretsky, Gurbani, Davids [Page 13]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
8. References
8.1 Normative References
[Ba91] Bradner, S. "Benchmarking Methodology for Network
Interconnection Devices", IETF RFC 1242, July 1991.
[Ba99] Bradner, S. and McQuaid, J., "Benchmarking
Methodology for Network Interconnect Devices",
IETF RFC 2544, March 1999.
[Ma98] Mandeville, R., "Benchmarking Methodology for LAN
Switching Devices", IETF RFC 2285, February 1998.
[Po06] Poretsky, S., Gurbani, V., and Davids, C., "SIP
Performance Benchmarking Terminology",
draft-poretsky-bmwg-sip-term-02, work in progress,
October 2006.
[Ro02] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M. and E. Schooler,
"SIP: Session Initiation Protocol", IETF RFC 3261,
June 2002.
[Ro04] Rosenberg, J., "A Presence Event Package for the
Session Initiation Protocol (SIP)," IETF RFC 3856, August
2004.
[Ga05] Garcia-Martin, M., "Input 3rd-Generation Partnership
Project (3GPP) Release 5 Requirements on the Session
Initiation Protocol (SIP)", IETF RFC 4083, May 2005.
[Sp06] Sparks, R., et al, "Session Initiation Protocol (SIP)
Torture Test Messages", IETF RFC 4475, October 2006.
[Ma06] Malas, D. "SIP Performance Metrics",
draft-malas-performance-metrics-01.txt, work in progress,
October 2006.
[Li06] Lingle, K., Mule, J., Maeng, J., Walker, D.,
"Management Information Base for the Session Initiation
Protocol (SIP)", draft-ietf-sip-mib-10.txt, work in
progress, March 2006.
8.2 Informative References
None
Poretsky, Gurbani, Davids [Page 14]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
9. Author's Address
Scott Poretsky
Reef Point Systems
8 New England Executive Park
Burlington, MA 01803
USA
Phone: + 1 508 439 9008
EMail: sporetsky@reefpoint.com
Vijay Gurbani
2000 Lucent Lane
Lucent Technologies
Room 6G-440
Naperville, IL 60566
USA
Phone: + 1 630 224 0216
Email: vkg@lucent.com
Carol Davids
Illinois Institute of Technology
Rice Campus
201 East Loop Road
Wheaton, IL 60187
USA
Phone: + 1 630 682 6000
Email: davids@iit.edu
Poretsky, Gurbani, Davids [Page 15]
INTERNET-DRAFT Benchmarking Methodology for October 2006
SIP Networking Devices
Full Copyright Statement
Copyright (C) The Internet Society (2006).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at ietf-
ipr@ietf.org.
Acknowledgement
Funding for the RFC Editor function is currently provided by the
Internet Society.
Poretsky, Gurbani, Davids [Page 16]