Benchmarking Methodology Working S. Poretsky
Group Allot Communications
Internet-Draft V. Gurbani
Expires: August 12, 2010 Bell Laboratories, Alcatel-Lucent
C. Davids
Illinois Institute of Technology
February 8, 2010
Methodology for Benchmarking SIP Networking Devices
draft-ietf-bmwg-sip-bench-meth-01
Abstract
This document describes the methodology for benchmarking Session
Initiation Protocol (SIP) performance as described in SIP
benchmarking terminology document. The methodology and terminology
are to be used for benchmarking signaling plane performance with
varying signaling and media load. Both scale and establishment rate
are measured by signaling plane performance. The SIP Devices to be
benchmarked may be a single device under test (DUT) or a system under
test (SUT). Benchmarks can be obtained and compared for different
types of devices such as SIP Proxy Server, SBC, and server paired
with a media relay or Firewall/NAT device.
Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on August 12, 2010.
Poretsky, et al. Expires August 12, 2010 [Page 1]
Internet-Draft SIP Benchmarking Methodology February 2010
Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the BSD License.
Poretsky, et al. Expires August 12, 2010 [Page 2]
Internet-Draft SIP Benchmarking Methodology February 2010
Table of Contents
1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Test Topologies . . . . . . . . . . . . . . . . . . . . . . . 5
4. Test Considerations . . . . . . . . . . . . . . . . . . . . . 7
4.1. Selection of SIP Transport Protocol . . . . . . . . . . . 7
4.2. Server . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.3. Associated Media . . . . . . . . . . . . . . . . . . . . . 8
4.4. Selection of Associated Media Protocol . . . . . . . . . . 8
4.5. Number of Associated Media Streams per SIP Session . . . . 8
4.6. Session Duration . . . . . . . . . . . . . . . . . . . . . 8
4.7. Attempted Sessions per Second . . . . . . . . . . . . . . 9
4.8. Stress Testing . . . . . . . . . . . . . . . . . . . . . . 9
5. Reporting Format . . . . . . . . . . . . . . . . . . . . . . . 9
5.1. Test Setup Report . . . . . . . . . . . . . . . . . . . . 9
5.2. Device Benchmarks for IS . . . . . . . . . . . . . . . . . 10
5.3. Device Benchmarks for NS . . . . . . . . . . . . . . . . . 10
6. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.1. Session Establishment Rate . . . . . . . . . . . . . . . . 10
6.2. Session Establishment Rate with Media . . . . . . . . . . 10
6.3. Session Establishment Rate with Loop Detection Enabled . . 11
6.4. Session Establishment Rate with Forking . . . . . . . . . 12
6.5. Session Establishment Rate with Forking and Loop
Detection . . . . . . . . . . . . . . . . . . . . . . . . 12
6.6. Session Establishment Rate with TLS Encrypted SIP . . . . 13
6.7. Session Establishment Rate with IPsec Encrypted SIP . . . 14
6.8. Session Establishment Rate with SIP Flooding . . . . . . . 14
6.9. Maximum Registration Rate . . . . . . . . . . . . . . . . 15
6.10. Maximum Re-Registration Rate . . . . . . . . . . . . . . . 15
6.11. Maximum IM Rate . . . . . . . . . . . . . . . . . . . . . 16
6.12. Session Capacity without Media . . . . . . . . . . . . . . 16
6.13. Session Capacity with Media . . . . . . . . . . . . . . . 17
6.14. Session Capacity with Media and a Media Relay/NAT
and/or Firewall . . . . . . . . . . . . . . . . . . . . . 18
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18
8. Security Considerations . . . . . . . . . . . . . . . . . . . 18
9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 19
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 19
10.1. Normative References . . . . . . . . . . . . . . . . . . . 19
10.2. Informative References . . . . . . . . . . . . . . . . . . 19
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 19
Poretsky, et al. Expires August 12, 2010 [Page 3]
Internet-Draft SIP Benchmarking Methodology February 2010
1. Terminology
In this document, the key words "MUST", "MUST NOT", "REQUIRED",
"SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT
RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as
described in BCP 14, conforming to [RFC2119] and indicate requirement
levels for compliant implementations.
Terms specific to SIP [RFC3261] performance benchmarking are defined
in [I-D.sip-bench-term].
RFC 2119 defines the use of these key words to help make the intent
of standards track documents as clear as possible. While this
document uses these keywords, this document is not a standards track
document. The term Throughput is defined in [RFC2544].
2. Introduction
This document describes the methodology for benchmarking Session
Initiation Protocol (SIP) performance as described in Terminology
document [I-D.sip-bench-term]. The methodology and terminology are
to be used for benchmarking signaling plane performance with varying
signaling and media load. Both scale and establishment rate are
measured by signaling plane performance.
The SIP Devices to be benchmarked may be a single device under test
(DUT) or a system under test (SUT). The DUT is a SIP Server, which
may be any [RFC3261] conforming device. The SUT can be any device or
group of devices containing RFC 3261 conforming functionality along
with Firewall and/or NAT functionality. This enables benchmarks to
be obtained and compared for different types of devices such as SIP
Proxy Server, SBC, SIP proxy server paired with a media relay or
Firewall/NAT device. SIP Associated Media benchmarks can also be
made when testing SUTs.
The test cases covered in this methodology document provide
benchmarks metrics of Registration Rate, SIP Session Establishment
Rate, Session Capacity, IM Rate, and Presence Rate. These can be
benchmarked with or without associated Media. Some cases are also
included to cover Forking, Loop detecion, Encrypted SIP, and SIP
Flooding. The test topologies that can be used are described in the
Test Setup section. Topologies are provided for benchmarking of a
DUT or SUT. Benchmarking with Associated Media can be performed when
using a SUT.
SIP permits a wide range of configuration options that are also
explained in the Test Setup section. Benchmark metrics could
Poretsky, et al. Expires August 12, 2010 [Page 4]
Internet-Draft SIP Benchmarking Methodology February 2010
possibly be impacted by Associated Media. The selected values for
Session Duration and Media Streams Per Session enable benchmark
metrics to be benchmarked without Associated Media. Session Setup
Rate could possibly be impacted by the selected value for Maximum
Sessions Attempted. The benchmark for Session Establishment Rate is
measured with a fixed value for maximum Session Attempts.
3. Test Topologies
Figures 1 through 6 below provide various topologies to perform the
SIP Performance Benchmarking. These figures show the Device Under
Test (DUT) to be a single server or a System Under Test (SUT). Test
Topology options to include benchmarking with Associated Media
require use of a SUT and are shown in Figures 4 and 5.
DUT
--------- ---------
| | | |
| | | |
| | SIP | |
|Server |<------------->| Tester|
| | | |
| | | |
| | | |
--------- ---------
Figure 1: Basic SIP Test Topology
SUT
------------------------
--------- --------- ---------
| | | | | |
| | | | | |
| | SIP |Fire- | SIP | |
| Server|<---------------------->| Tester|
| | |Wall | | |
| | | | | |
| | | | | |
--------- --------- ---------
Figure 2: SIP Test Topology with Firewall
Poretsky, et al. Expires August 12, 2010 [Page 5]
Internet-Draft SIP Benchmarking Methodology February 2010
SUT
------------------------
--------- --------- ---------
| | | | | |
| | | | | |
| | SIP | NAT | SIP | |
| Server|<---------------------->| Tester|
| | | | | |
| | | | | |
| | | | | |
--------- --------- ---------
Figure 3: SIP Test Topology with NAT Device
SUT
------------------------
--------- --------- ---------
| | | | | |
| | | | | |
| | SIP |Fire- | SIP | |
| Server|<---------------------->| Tester|
| | |Wall | | |
| | | | Media | |
| | ---| |---------| |
--------- | --------- ---------
| Media ^
-------------------------|
Figure 4: SIP Test Topology with Media through Firewall
Poretsky, et al. Expires August 12, 2010 [Page 6]
Internet-Draft SIP Benchmarking Methodology February 2010
SUT
------------------------
--------- --------- ---------
| | | | | |
| | | | | |
| | SIP | NAT | SIP | |
| Server|<---------------------->| Tester|
| | | | | |
| | | | Media | |
| | ---| |---------| |
--------- | --------- ---------
| Media ^
-------------------------|
Figure 5: SIP Test Topology with Media through NAT Device
SUT
------------------------
--------- --------- ---------
| | | | | |
| | | | | |
| | SIP |Media | SIP | |
| Server|<---------------------->| Tester|
| | |Relay | | |
| | | | Media | |
| | ---| |---------| |
--------- | --------- ---------
| Media ^
-------------------------|
Figure 6: SIP Test Topology with Media Through a Media Relay
4. Test Considerations
4.1. Selection of SIP Transport Protocol
Discussion:
Test cases may be performed with any transport protocol supported
by SIP. This includes, but is not limited to, SIP TCP, SIP UDP,
and TLS. The protocol used for the SIP transport protocol must be
reported with benchmarking results.
Poretsky, et al. Expires August 12, 2010 [Page 7]
Internet-Draft SIP Benchmarking Methodology February 2010
4.2. Server
Discussion:
The Server is a SIP-speaking device that complies with RFC 3261.
The purpose of this document is to benchmark SIP performance, not
conformance. Conformance to [RFC3261] is assumed for all tests.
The Server may be the DUT or a component of a SUT that includes
Firewall and/or NAT functionality. The components of the SUT may
be a single physical device or separate devices.
4.3. Associated Media
Discussion:
Some tests may require associated media to be present for each SIP
session. The Server is not involved in the forwarding of media.
Associated Media can be benchmarked only with a SUT in which the
media traverses a Media Relay, Firewall, NAT, or Firewall/NAT
device. The test topologies to be used when benchmarking SUT
performance for Associated Media are shown in Figures 4 and 5, in
which the SIP signaling is bidirectional and the Associated Media
is unidirectional.
4.4. Selection of Associated Media Protocol
Discussion:
The test cases specified in this document provide SIP performance
independent of the protocol used for the media stream. Any media
protocol supported by SIP may be used. This includes, but is not
limited to, RTP, RTSP, and SRTP. The protocol used for Associated
Media must be reported with benchmarking results.
4.5. Number of Associated Media Streams per SIP Session
Discussion:
Benchmarking results may vary with the number of media streams per
SIP session. When benchmarking a SUT for voice, a single media
stream is used. When benchmarking a SUT for voice and video, two
media streams are used. The number of Associated Media Streams
must be reported with benchmarking results.
4.6. Session Duration
Poretsky, et al. Expires August 12, 2010 [Page 8]
Internet-Draft SIP Benchmarking Methodology February 2010
Discussion:
SUT performance benchmarks may vary with the duration of SIP
sessions. Session Duration must be reported with benchmarking
results. A Session Duration of zero seconds indicates
transmission of a BYE immediately following successful SIP
establishment indicate by receipt of a 200 OK. An infinite
Session Duration indicates that a BYE is never transmitted.
4.7. Attempted Sessions per Second
Discussion:
DUT and SUT performance benchmarks may vary with the the rate of
attempted sessions offered by the Tester. Attempted Sessions per
Second must be reported with benchmarking results.
4.8. Stress Testing
Discussion:
The purpose of this document is to benchmark SIP performance, not
system stability under stressful conditions such as a high rate of
Attempted Sessions per Second.
5. Reporting Format
5.1. Test Setup Report
SIP Transport Protocol = ___________________________
Session Attempt Rate = _____________________________
IS Media Attempt Rate = ____________________________
Total Sessions Attempted = _________________________
Media Streams Per Session = _______________________
Associated Media Protocol = _______________________
Media Packet Size = _______________________________
Media Offered Load = ______________________________
Media Session Hold Time = _________________________
Establishment Threshold Time = ____________________
Loop Detecting Option = ___________________________
Forking Option
Number of endpoints request sent to = ___________
Type of forking = _______________________________
Note: Total Sessions Attempted is used in the calculation of the
Session Establishment Performance ([I-D.sip-bench-term], Section
3.4.5). It is the number of session attempts ([I-D.sip-bench-term],
Section 3.1.6) that will be made over the duration of the test.
Poretsky, et al. Expires August 12, 2010 [Page 9]
Internet-Draft SIP Benchmarking Methodology February 2010
5.2. Device Benchmarks for IS
Registration Rate = _______________________________
Re-registration Rate = ____________________________
Session Capacity = _________________________________
Session Overload Capacity = ________________________
Session Establishment Rate = ______________________
Session Establishment Performance = _______________
Session Attempt Delay = ___________________________
Session Disconnect Delay = ________________________
5.3. Device Benchmarks for NS
IM Rate = _______________________________
6. Test Cases
6.1. Session Establishment Rate
Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Configure Tester for SIP UDP with an Session Attempt Rate =
100 SPS, maximum Session Attempts = 100,000 and Media Streams
Per Session=0.
3. Start Tester to initiate SIP Session establishment with the
DUT.
4. Measure Session Attempt Failures and total Established
Sessions at the Tester.
5. If a Session Attempt Failure is recorded then reduce the
Session Attempt Rate configured on the Tester by 50%.
6. If no Session Attempt Failure is recorded then increase the
Session Attempt Rate configured on the Tester by 50%.
7. Repeat steps 3 through 6 until the Session Establishment Rate
is obtained and recorded.
Expected Results: This is the scenario to obtain the maximum Session
Establishment Rate of the DUT/SUT.
6.2. Session Establishment Rate with Media
Poretsky, et al. Expires August 12, 2010 [Page 10]
Internet-Draft SIP Benchmarking Methodology February 2010
Objective:
To benchmark the Session Establishment Rate of the SUT with zero
failures when Associated Media is included in the benchmark test.
Procedure:
1. Configure the SUT in the test topology shown in Figure 4, 5 or
6.
2. Configure Tester for SIP UDP with an Session Attempt Rate =
100 SPS, maximum Session Attempts = 100,000 and Media Streams
Per Session = 1. The rate of offered load for each media
stream SHOULD be (eq 1) Offered Load per Media Stream =
Throughput / maximum sessions attempted, where Throughput is
defined in [RFC2544].
3. Start Tester to initiate SIP Session establishment with the
SUT and transmit media through the SUT to a destination other
than the server.
4. At the Tester measure Session Attempt Failures, total
Established Sessions, and Packet Loss [RFC2544] of the media.
5. If a Session Attempt Failure or Packet Loss is recorded then
reduce the Session Attempt Rate configured on the Tester by
50%.
6. If no Session Attempt Failure or Packet Loss is recorded then
increase the Session Attempt Rate configured on the Tester by
50%.
7. Repeat steps 3 through 6 until the Session Establishment Rate
is obtained and recorded.
8. Repeat steps 1 through 7 for multimedia in which Media Streams
Per Session = 2.
Expected Results: Session Establishment Rate results obtained with
Associated Media with any number of media streams per SIP session
are expected to be identical to the Session Establishment Rate
results obtained without media in the case where the server is
running on a platform separate from the platform on which the
Media Relay, NAT or Firewall is running. Session Establishment
Rate results obtained with Associated Media may be lower than
those obtained without media in the case where the server and the
NAT, Firewall or Media Relay are running on the same platform.
6.3. Session Establishment Rate with Loop Detection Enabled
Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when the Loop Detection option is enabled.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Configure Tester for SIP UDP with an Session Attempt Rate =
100 SPS, maximum Session Attempts = 100,000 and Media Streams
Per Session=0.
Poretsky, et al. Expires August 12, 2010 [Page 11]
Internet-Draft SIP Benchmarking Methodology February 2010
3. Turn on the Loop Detection option in the DUT or SUT.
4. Start Tester to initiate SIP Session establishment with the
DUT.
5. Measure Session Attempt Failures and total Established
Sessions at the Tester.
6. If a Session Attempt Failure is recorded then reduce the
Session Attempt Rate configured on the Tester by 50%.
7. If no Session Attempt Failure is recorded then increase the
Session Attempt Rate configured on the Tester by 50%.
8. Repeat steps 4 through 7 until the Session Establishment Rate
is obtained and recorded.
Expected Results: Session Establishment Rate results obtained with
Loop Detection may be lower than those obtained without Loop
Detection enabled.
6.4. Session Establishment Rate with Forking
Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when the Forking Option is enabled.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Configure Tester for SIP UDP with an Session Attempt Rate =
100 SPS, maximum Session Attempts = 100,000 and Media Streams
Per Session=0.
3. Set the number of endpoints that will receive the forked
invitation to a value of 2 or more (subsequent tests may
increase this value at the discretion of the tester.)
4. Start Tester to initiate SIP Session establishment with the
DUT.
5. Measure Session Attempt Failures and total Established
Sessions at the Tester.
6. If a Session Attempt Failure is recorded then reduce the
Session Attempt Rate configured on the Tester by 50%.
7. If no Session Attempt Failure is recorded then increase the
Session Attempt Rate configured on the Tester by 50%.
8. Repeat steps 4 through 7 until the Session Establishment Rate
is obtained and recorded.
Expected Results: Session Establishment Rate results obtained with
Forking may be lower than those obtained without Forking enabled.
6.5. Session Establishment Rate with Forking and Loop Detection
Poretsky, et al. Expires August 12, 2010 [Page 12]
Internet-Draft SIP Benchmarking Methodology February 2010
Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when both the Forking and Loop Detection Options are
enabled.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Configure Tester for SIP UDP with an Session Attempt Rate =
100 SPS, maximum Session Attempts = 100,000 and Media Streams
Per Session=0.
3. Start Tester to initiate SIP Session establishment with the
DUT.
4. Enable the Loop Detection Options on the DUT.
5. Set the number of endpoints that will receive the forked
invitation to a value of 2 or more (subsequent tests may
increase this value at the discretion of the tester.)
6. Measure Session Attempt Failures and total Established
Sessions at the Tester.
7. If a Session Attempt Failure is recorded then reduce the
Session Attempt Rate configured on the Tester by 50%.
8. If no Session Attempt Failure is recorded then increase the
Session Attempt Rate configured on the Tester by 50%.
9. Repeat steps 4 through 7 until the Session Establishment Rate
is obtained and recorded.
Expected Results: Session Establishment Rate results obtained with
Forking and Loop Detection may be lower than those obtained with
only Forking or Loop Detection enabled.
6.6. Session Establishment Rate with TLS Encrypted SIP
Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when using TLS encrypted SIP.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Configure Tester for SIP TCP, enable TLS, Session Attempt Rate
= 100 SPS, maximum Session Attempts = 100,000 and Media
Streams Per Session = 0.
3. Start Tester to initiate SIP Session establishment with the
DUT.
4. Measure Session Attempt Failures and total Established
Sessions at the Tester.
5. If a Session Attempt Failure is recorded then reduce the
Session Attempt Rate configured on the Tester by 50%.
6. If no Session Attempt Failure is recorded then increase the
Session Attempt Rate configured on the Tester by 50%.
Poretsky, et al. Expires August 12, 2010 [Page 13]
Internet-Draft SIP Benchmarking Methodology February 2010
7. Repeat steps 3 through 6 until the Session Establishment Rate
is obtained and recorded.
Expected Results: Session Establishment Rate results obtained with
TLS Encrypted SIP may be lower than those obtained with plaintext
SIP.
6.7. Session Establishment Rate with IPsec Encrypted SIP
Objective:
To benchmark the Session Establishment Rate of the DUT/SUT with
zero failures when using IPsec Encryoted SIP.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Configure Tester for SIP TCP, enable IPSec, Session Attempt
Rate = 100 SPS, maximum Session Attempts = 100,000 and Media
Streams Per Session = 0.
3. Start Tester to initiate SIP Session establishment with the
DUT.
4. Measure Session Attempt Failures and total Established
Sessions at the Tester.
5. If a Session Attempt Failure is recorded then reduce the
Session Attempt Rate configured on the Tester by 50%.
6. If no Session Attempt Failure is recorded then increase the
Session Attempt Rate configured on the Tester by 50%.
7. Repeat steps 3 through 6 until the Session Establishment Rate
is obtained and recorded.
Expected Results: Session Establishment Rate results obtained with
IPSec Encrypted SIP may be lower than those obtained with
plaintext SIP.
6.8. Session Establishment Rate with SIP Flooding
Objective:
To benchmark the Session Establishment Rate of the SUT with zero
failures when SIP Flooding is occurring.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
the SUT as shown in Figure 2.
2. Configure Tester for SIP UDP with an Session Attempt Rate =
100 SPS, maximum Session Attempts = 100,000, Associated Media
Streams Per Session = 0, and SIP INVITE Message Flood = 500
per second.
3. Start Tester to initiate SIP Session establishment with the
SUT and SIP Flood targetted at the Server.
4. At the Tester measure Session Attempt Failures, total
Established Sessions, and Packet Loss [RFC2544] of the media.
Poretsky, et al. Expires August 12, 2010 [Page 14]
Internet-Draft SIP Benchmarking Methodology February 2010
5. If a Session Attempt Failure or Packet Loss is recorded then
reduce the Session Attempt Rate configured on the Tester by
50%.
6. If no Session Attempt Failure or Packet Loss is recorded then
increase the Session Attempt Rate configured on the Tester by
50%.
7. Repeat steps 3 through 6 until the Session Establishment Rate
is obtained and recorded.
8. Repeat steps 1 through 7 with SIP INVITE Message Flood = 1000
per second.
Expected Results: Session Establishment Rate results obtained with
SIP Flooding may be degraded.
6.9. Maximum Registration Rate
Objective:
To benchmark the maximum registration rate of the DUT/SUT with
zero failures.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Configure Tester for SIP UDP with an attempted Registration
Rate = 100 SPS and maximum registrations attempted = 100,000.
3. Set the registration timeout value to at least 3600 seconds.
4. At the Tester measure failed registration attempts, total
registrations and packet loss.
5. If a Failed Registration Attempt or Packet Loss is recorded
then reduce the Attempted Registration Rate configured on the
Tester by 50%.
6. If no Failed Registration or Packet Loss is recorded then
increase the Attempted Registration Rate configured on the
Tester by 50%.
7. Repeat steps 5 and 6 until the all registrations have
succeeded. This number is obtained and recorded.
Expected Results:
6.10. Maximum Re-Registration Rate
Objective:
To benchmark the maximum re-registration rate of the DUT/SUT with
zero failures.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Execute test detailed in Section 6.9 to register the endpoints
with the registrar. The rest of the steps below MUST be
performed at least 5 minutes after, but no more than 15
minutes after the test performed in Section 6.9.
Poretsky, et al. Expires August 12, 2010 [Page 15]
Internet-Draft SIP Benchmarking Methodology February 2010
3. Configure Tester for SIP UDP with an attempted Registration
Rate = 100 SPS and maximum registrations attempted = 100,000.
4. Configure Tester to re-register the same address-of-records
that were registered in Section 6.9.
5. At the Tester measure failed registration attempts, total
registrations and packet loss.
6. If a Failed Registration Attempt or Packet Loss is recorded
then reduce the Attempted Registration Rate configured on the
Tester by 50%.
7. If no Failed Registration or Packet Loss is recorded then
increase the Attempted Registration Rate configured on the
Tester by 50%.
8. Repeat steps 6 and 7 until the all re-registrations have
succeeded. This number is obtained and recorded.
Expected Results: The rate should be at least equal to but not more
than the result of Section 6.9.
6.11. Maximum IM Rate
Objective:
To benchmark the maximum IM rate of the SUT with zero failures.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Configure Tester for SIP UDP with an Attempted IM Rate = 100
SPS, Maximum IM Attempted = 100,000.
3. At the Tester measure Failed IM Attempts, Total IM and Packet
Loss.
4. If a Failed IM Attempt or Packet Loss is recorded then reduce
the Attempted IM Rate configured on the Tester by 50%.
5. If no Failed IM or Packet Loss is recorded then increase the
Attempted IM Rate configured on the Tester by 50%.
6. Repeat steps 3 through 6 until the Maximum IM Rate is obtained
and recorded.
Expected Results:
6.12. Session Capacity without Media
Objective:
To benchmark the Session Capacity of the SUT without Associated
Media.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Configure Tester for SIP UDP with an Session Attempt Rate =
Session Establishment Rate, maximum Session Attempts = 10,000
and Media Streams Per Session = 0.
Poretsky, et al. Expires August 12, 2010 [Page 16]
Internet-Draft SIP Benchmarking Methodology February 2010
3. Start Tester to initiate SIP Session establishment with the
DUT.
4. Measure Session Attempt Failures, total Established Sessions,
and Packet Loss [RFC2544] at the Tester.
5. If a Session Attempt Failure or Packet Loss is recorded then
reduce the maximum Session Attempts configured on the Tester
by 5,000.
6. If no Session Attempt Failure or Packet Loss is recorded then
increase the maximum Session Attempts configured on the Tester
by 10,000.
7. Repeat steps 3 through 6 until the Session Capacity is
obtained and recorded.
8. Repeat steps 1 through 7 for multimedia in which media streams
per session = 2.
Expected Results: This is the scenario to obtain the maximum Session
Capacity of the DUT/SUT.
6.13. Session Capacity with Media
Objective:
To benchmark the session capacity of the DUT/SUT with Associated
Media.
Procedure:
1. Configure the DUT in the test topology shown in Figure 1 or
SUT as shown in Figures 2 or 3.
2. Configure Tester for SIP UDP with a Session Attempt Rate = 100
SPS, Session Duration = 30 sec, maximum Session Attempts =
100,000 and Media Streams Per Session = 1.
NOTE: The total offered load to the DUT/SUT SHOULD be equal
to the Throughput of the DUT/SUT as defined in [RFC2544].
The offered load to the DUT/SUT for each media stream
SHOULD be equal to
Throughput/Maximum Session Attemps.
3. Start Tester to initiate SIP Session establishment with the
SUT and transmit media through the SUT to a destination other
than the server.
4. Measure Session Attempt Failures and total Established
Sessions at the Tester.
5. If a Session Attempt Failure is recorded then reduce the
maximum Session Attempts configured on the Tester by 5,000.
6. If no Session Attempt Failure is recorded then increase the
maximum Session Attempts configured on the Tester by 10,000.
7. Repeat steps 3 through 6 until the Session Capacity is
obtained and recorded.
Poretsky, et al. Expires August 12, 2010 [Page 17]
Internet-Draft SIP Benchmarking Methodology February 2010
Expected Results: Session Capacity results obtained with Associated
Media with any number of media streams per SIP session will be
identical to the Session Capacity results obtained without media.
6.14. Session Capacity with Media and a Media Relay/NAT and/or Firewall
Objective:
To benchmark the Session Establishment Rate of the SUT with
Associated Media.
Procedure:
1. Configure the SUT in the test topology shown in Figure 4, 5 or
6.
2. Configure Tester for SIP UDP with a Session Attempt Rate = 100
SPS, Session Duration = 30 sec, maximum Session Attempts =
100,000 and Media Streams Per Session = 1.
NOTE: The offered load for each media stream SHOULD be as
in Equation 1.
3. Start Tester to initiate SIP Session establishment with the
SUT and transmit media through the SUT to a destination other
than the server.
4. Measure Session Attempt Failures and total Established
Sessions at the Tester.
5. If a Session Attempt Failure is recorded then reduce the
maximum Session Attempts configured on the Tester by 5,000.
6. If no Session Attempt Failure is recorded then increase the
maximum Session Attempts configured on the Tester by 10,000.
7. Repeat steps 3 through 6 until the Session Capacity is
obtained and recorded.
Expected Results: Session Capacity results obtained with Associated
Media with any number of media streams per SIP session may be
lower than the Session Capacity without Media result if the Media
Relay, NAT or Firewall is sharing a platform with the server.
7. IANA Considerations
This document does not requires any IANA considerations.
8. Security Considerations
Documents of this type do not directly affect the security of
Internet or corporate networks as long as benchmarking is not
performed on devices or systems connected to production networks.
Security threats and how to counter these in SIP and the media layer
is discussed in RFC3261, RFC3550, and RFC3711 and various other
drafts. This document attempts to formalize a set of common
methodology for benchmarking performance of SIP devices in a lab
Poretsky, et al. Expires August 12, 2010 [Page 18]
Internet-Draft SIP Benchmarking Methodology February 2010
environment.
9. Acknowledgments
The authors would like to thank Keith Drage and Daryl Malas for their
contributions to this document.
10. References
10.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999.
[I-D.sip-bench-term]
Poretsky, S., Gurbani, V., and C. Davids, "SIP Performance
Benchmarking Terminology",
draft-ietf-bmwg-sip-bench-term-01 (work in progress),
January 2010.
10.2. Informative References
[RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston,
A., Peterson, J., Sparks, R., Handley, M., and E.
Schooler, "SIP: Session Initiation Protocol", RFC 3261,
June 2002.
Authors' Addresses
Scott Poretsky
Allot Communications
300 TradeCenter, Suite 4680
Woburn, MA 08101
USA
Phone: +1 508 309 2179
Email: sporetsky@allot.com
Poretsky, et al. Expires August 12, 2010 [Page 19]
Internet-Draft SIP Benchmarking Methodology February 2010
Vijay K. Gurbani
Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane
Rm 9C-533
Naperville, IL 60566
USA
Phone: +1 630 224 0216
Email: vkg@bell-labs.com
Carol Davids
Illinois Institute of Technology
201 East Loop Road
Wheaton, IL 60187
USA
Email: davids@iit.edu
Poretsky, et al. Expires August 12, 2010 [Page 20]
