Terminology for Benchmarking Session Initiation Protocol (SIP) Devices: Basic Session Setup and Registration
draft-ietf-bmwg-sip-bench-term-12

[Ballot comment]
Thanks for adding text to address my comment:

The security considerations look fine, but in reading the draft, it takes up until that section to learn that the tests are not performed on a production network and the follow on draft says they are intended for a lab environment.  I think it would be useful to add this assumption into section 2.1 of this draft - that the benchmarks are intended for lab environments or not intended to be used on production networks as a DoS attack is possible using these methods.

[Ballot comment]
I agree with Alissa's comment on media.  Suggest:
OLD: "Any media protocol MAY be used."
NEW: "The format of the media is determined by the SDP attributes for the 'm' line in question."

The phrase "media control protocol" is used in Section 3.1.3.  I assume this is meant to mean RTCP?  If so, it would be useful to note this.

[Ballot comment]
= Section 2.1 =
"A DUT MAY also include a B2BUA, SBC functionality."

This seems like a strange use of normative language (especially because the "must" earlier in the same bullet is not normative). I think the intended meaning here was actually "may."

= Section 3.1.4 =
"Any media protocol MAY be used."

Same comment as above -- what other choice is there if any media protocol is acceptable? I think this should be "may."

= Section 3.3.2 =
"Following the default value of T1 (500ms) specified in
      the table and a constant multiplier of 64 gives a value of 32
      seconds for this timer (i.e., 500ms * 64 = 32s)."
   
Is this just meant to be an example? Not sure what its purpose is. And it seems really long.

[Ballot comment]
The security considerations look fine, but in reading the draft, it takes up until that section to learn that the tests are not performed on a production network and the follow on draft says they are intended for a lab environment.  I think it would be useful to add this assumption into section 2.1 of this draft - that the benchmarks are intended for lab environments or not intended to be used on production networks as a DoS attack is possible using these methods.


Thanks.

IESG/Authors/WG Chairs:

IANA has reviewed draft-ietf-bmwg-sip-bench-term-11, which is currently in Last Call, and has the following comments:

We understand that, upon approval of this document, there are no
IANA Actions that need completion.

While it is helpful for the IANA Considerations section of the document to remain in place upon publication, if the authors prefer to remove it, IANA doesn't object.

If this assessment is not accurate, please respond as soon as possible.

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (Terminology for Benchmarking Session Initiation Protocol (SIP) Devices: Basic session setup and registration) to Informational RFC


The IESG has received a request from the Benchmarking Methodology WG
(bmwg) to consider the following document:
- 'Terminology for Benchmarking Session Initiation Protocol (SIP)
  Devices: Basic session setup and registration'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2014-10-08. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This document provides a terminology for benchmarking the Session
  Initiation Protocol (SIP) performance of devices.  Methodology
  related to benchmarking SIP devices is described in the companion
  methodology document.  Using these two documents, benchmarks can be
  obtained and compared for different types of devices such as SIP
  Proxy Servers, Registrars and Session Border Controllers.  The term
  "performance" in this context means the capacity of the device-under-
  test (DUT) to process SIP messages.  Media streams are used only to
  study how they impact the signaling behavior.  The intent of the two
  documents is to provide a normalized set of tests that will enable an
  objective comparison of the capacity of SIP devices.  Test setup
  parameters and a methodology is necessary because SIP allows a wide
  range of configuration and operational conditions that can influence
  performance benchmark measurements.  A standard terminology and
  methodology will ensure that benchmarks have consistent definition
  and were obtained following the same procedures.




The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-bmwg-sip-bench-term/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-bmwg-sip-bench-term/ballot/


No IPR declarations have been submitted directly on this I-D.

This is a publication request for:

draft-ietf-bmwg-sip-bench-term -11 2014-07-02 Active


Al Morton is the Document Shepherd, and prepared this form.

(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)? Why is
this the proper type of RFC? Is this type of RFC indicated in the
title page header?

Informational, as indicated on the title page. All BMWG RFCs are
traditionally Informational, in part because they do not define
protocols and the traditional conditions for Stds track advancement
did not apply. However, they are specifications and the RFC 2119 terms
are applicable to identify the level of requirements.

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary:

All networking devices have a limited capacity to serve their purpose.
In some cases these limits can be ascertained by counting physical
features (e.g., interface card slots), but in other cases standardized
tests are required to be sure that all vendors count their
protocol-handling capacity in the same way, to avoid specmanship. This
draft addresses one such case, where the SIP session-serving capacity
of a device can only be discovered and rigorously compared with other
devices through isolated laboratory testing.

This document describes the methodology for benchmarking Session -or-
This document describes the terminology for benchmarking Session
Initiation Protocol (SIP) performance as described in SIP benchmarking
terminology document. The methodology and terminology are to be used
for benchmarking signaling plane performance with varying signaling
and media load. Both scale and establishment rate are measured by
signaling plane performance. The SIP Devices to be benchmarked may be
a single device under test or a system under test. Benchmarks can be
obtained and compared for different types of devices such as SIP Proxy
Server, Session Border Controller, and server paired with a media
relay or Firewall/NAT device.

Working Group Summary:

There were periods of intense and constructive feedback on this draft,
but also several pauses in progress during development. The most
lively discussions were prompted by presentation of actual test
results using the draft methods, which require significant time
investment but are well- worth the result. These drafts serve a useful
purpose for the industry.

Document Quality:

There are existing implementations of the method, as noted above.

Dale Worley conducted an early review, following BMWG's request of the
RAI area. Dales's comments were addressed in version 05. Henning
Schulzrinne commented on the original work proposal.

Personnel:

Who is the Document Shepherd? Who is the Responsible Area Director? Al
Morton is Shepherd, Joel Jaeggli is Responsible AD.

(3) Briefly describe the review of this document that was performed by
the Document Shepherd. If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

The shepherd has reviewed the drafts many times, and his comments are
in the BMWG archive.

(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed? No.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

No. Cross-area review has been obtained, however it impossible to get
the attention of everyone who considers themselves a SIP expert.

(6) Describe any specific concerns or issues that the Document
Shepherd has with this document that the Responsible Area Director
and/or the IESG should be aware of? For example, perhaps he or she is
uncomfortable with certain parts of the document, or has concerns
whether there really is a need for it. In any event, if the WG has
discussed those issues and has indicated that it still wishes to
advance the document, detail those concerns here.

No concerns, this is still a valuable memo, as mentioned above.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP
78 and BCP 79 have already been filed. If not, explain why?

There are not outstanding IPR disclosures, according to the authors.

(8) Has an IPR disclosure been filed that references this document? If
so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

No.

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it?

Although the comments and review intensity was highly variable, it now
appears that the WG is satisfied. The first WGLC was completed on 5
April 2010 with comments. The second WGLC was completed on 18 May 2012
with comments. The third WGLC was completed on 10 Dec 2012 with
comments, and the 1st Pub Request. A IETF Last Call followed, and
completed on 30 Jan 2013 with comments. A fourth WGLC was completed 11
June 2014 with comments from expert review. The current versions (11)
address Dale Worley's RAI area early review and Robert Spark's
reviews. The fifth WGLC completed quietly on July 14th, 2014.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See http://www.ietf.org/tools/idnits/ and the
Internet-Drafts Checklist). Boilerplate checks are not enough; this
check needs to be thorough.

Nits are warnings requiring no action for these drafts.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

N/A

(13) Have all references within this document been identified as
either normative or informative?

Yes.

(14) Are there normative references to documents that are not ready
for advancement or are otherwise in an unclear state? If such
normative references exist, what is the plan for their completion?

The -term and -meth drafts are proceeding toward publication as a
pair.

(15) Are there downward normative references references (see RFC
3967)? If so, list these downward references to support the Area
Director in the Last Call procedure.

No.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are
not listed in the Abstract and Introduction, explain why, and point to
the part of the document where the relationship of this document to
the other RFCs is discussed. If this information is not in the
document, explain why the WG considers it unnecessary.

No.

(17) Describe the Document Shepherd's review of the IANA
considerations section, especially with regard to its consistency with
the body of the document. Confirm that all protocol extensions that
the document makes are associated with the appropriate reservations in
IANA registries. Confirm that any referenced IANA registries have been
clearly identified. Confirm that newly created IANA registries include
a detailed specification of the initial contents for the registry,
that allocations procedures for future registrations are defined, and
a reasonable name for the new registry has been suggested (see RFC
5226).

No requests of IANA.

(18) List any new IANA registries that require Expert Review for
future allocations. Provide any public guidance that the IESG would
find useful in selecting the IANA Experts for these new registries.

N/A

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

N/A

This is a publication request for:
draft-ietf-bmwg-sip-bench-meth -11 2014-07-02  Active
draft-ietf-bmwg-sip-bench-term -11 2014-07-02  Active


Al Morton is the Document Shepherd, and prepared this form.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational, as indicated on the title page.
All BMWG RFCs are traditionally Informational,
in part because they do not define protocols and
the traditional conditions for Stds track advancement
did not apply.  However, they are specifications and
the RFC 2119 terms are applicable to identify the
level of requirements.

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

All networking devices have a limited capacity to serve their
purpose. In some cases these limits can be ascertained by counting
physical features (e.g., interface card slots), but in other cases
standardized tests are required to be sure that all vendors count
their protocol-handling capacity in the same way, to avoid specmanship.
This draft addresses one such case, where the SIP session-serving
capacity of a device can only be discovered and rigorously compared
with other devices through isolated laboratory testing.

This document describes the methodology for benchmarking Session
-or-
This document describes the terminology for benchmarking Session
Initiation Protocol (SIP) performance as described in SIP
benchmarking terminology document.  The methodology and terminology
are to be used for benchmarking signaling plane performance with
varying signaling and media load.  Both scale and establishment rate
are measured by signaling plane performance.  The SIP Devices to be
benchmarked may be a single device under test or a system under
test.  Benchmarks can be obtained and compared for different
types of devices such as SIP Proxy Server, Session Border Controller,
and server paired with a media relay or Firewall/NAT device.

Working Group Summary:

There were periods of intense and constructive feedback on this draft,
but also several pauses in progress during development. The most lively
discussions were prompted by presentation of actual test results using
the draft methods, which require significant time investment but are well-
worth the result. These drafts serve a useful purpose for the industry.

Document Quality:

There are existing implementations of the method, as noted above.

Dale Worley conducted an early review, following BMWG's request
of the RAI area.  Dales's comments were addressed in version 05.
Henning Schulzrinne commented on the original work proposal.

Personnel:

Who is the Document Shepherd? Who is the Responsible Area Director?
Al Morton is Shepherd, Joel Jaeggli is Responsible AD.

(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

The shepherd has reviewed the drafts many times, and his comments are
in the BMWG archive.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?
No.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

No. Cross-area review has been obtained, however it impossible to get
the attention of everyone who considers themselves a SIP expert.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

No concerns, this is still a valuable memo, as mentioned above.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

There are not outstanding IPR disclosures, according to the authors.

(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No.

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Although the comments and review intensity was highly variable,
it now appears that the WG is satisfied. 
The first WGLC was completed on 5 April 2010 with comments.
The second WGLC was completed on 18 May 2012 with comments.
The third WGLC was completed on 10 Dec 2012 with comments, and the 1st Pub Request.
A IETF Last Call followed, and completed on 30 Jan 2013 with comments.
A fourth WGLC was completed 11 June 2014 with comments from expert review.
The current versions (11) address Dale Worley's RAI area early review
and Robert Spark's reviews.
The fifth WGLC completed quietly on July 14th, 2014.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

Nits are warnings requiring no action for these drafts.

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, media type, and URI type reviews.

N/A

(13) Have all references within this document been identified as either normative or informative?

Yes.

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

The -term and -meth drafts are proceeding toward publication as a pair.

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

No.

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No.

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 5226).

No requests of IANA.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

N/A

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, etc.

N/A

This is a publication request for:
draft-ietf-bmwg-sip-bench-meth -11 2014-07-02  Active
draft-ietf-bmwg-sip-bench-term -11 2014-07-02  Active


Al Morton is the Document Shepherd, and prepared this form.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational, as indicated on the title page.
All BMWG RFCs are traditionally Informational,
in part because they do not define protocols and
the traditional conditions for Stds track advancement
did not apply.  However, they are specifications and
the RFC 2119 terms are applicable to identify the
level of requirements.

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

All networking devices have a limited capacity to serve their
purpose. In some cases these limits can be ascertained by counting
physical features (e.g., interface card slots), but in other cases
standardized tests are required to be sure that all vendors count
their protocol-handling capacity in the same way, to avoid specmanship.
This draft addresses one such case, where the SIP session-serving
capacity of a device can only be discovered and rigorously compared
with other devices through isolated laboratory testing.

This document describes the methodology for benchmarking Session
-or-
This document describes the terminology for benchmarking Session
Initiation Protocol (SIP) performance as described in SIP
benchmarking terminology document.  The methodology and terminology
are to be used for benchmarking signaling plane performance with
varying signaling and media load.  Both scale and establishment rate
are measured by signaling plane performance.  The SIP Devices to be
benchmarked may be a single device under test or a system under
test.  Benchmarks can be obtained and compared for different
types of devices such as SIP Proxy Server, Session Border Controller,
and server paired with a media relay or Firewall/NAT device.

Working Group Summary:

There were periods of intense and constructive feedback on this draft,
but also several pauses in progress during development. The most lively
discussions were prompted by presentation of actual test results using
the draft methods, which require significant time investment but are well-
worth the result. These drafts serve a useful purpose for the industry.

Document Quality:

There are existing implementations of the method, as noted above.

Dale Worley conducted an early review, following BMWG's request
of the RAI area.  Dales's comments were addressed in version 05.
Henning Schulzrinne commented on the original work proposal.

Personnel:

Who is the Document Shepherd? Who is the Responsible Area Director?
Al Morton is Shepherd, Joel Jaeggli is Responsible AD.

(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

The shepherd has reviewed the drafts many times, and his comments are
in the BMWG archive.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?
No.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

No. Cross-area review has been obtained, however it impossible to get
the attention of everyone who considers themselves a SIP expert.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

No concerns, this is still a valuable memo, as mentioned above.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

There are not outstanding IPR disclosures, according to the authors.

(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No.

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Although the comments and review intensity was highly variable,
it now appears that the WG is satisfied. 
The first WGLC was completed on 5 April 2010 with comments.
The second WGLC was completed on 18 May 2012 with comments.
The third WGLC was completed on 10 Dec 2012 with comments, and the 1st Pub Request.
A IETF Last Call followed, and completed on 30 Jan 2013 with comments.
A fourth WGLC was completed 11 June 2014 with comments from expert review.
The current versions (11) address Dale Worley's RAI area early review
and Robert Spark's reviews.
The fifth WGLC completed quietly on July 14th, 2014.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

Nits are warnings requiring no action for these drafts.

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, media type, and URI type reviews.

N/A

(13) Have all references within this document been identified as either normative or informative?

Yes.

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

The -term and -meth drafts are proceeding toward publication as a pair.

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

No.

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No.

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 5226).

No requests of IANA.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

N/A

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, etc.

N/A

After consulting with the chairs I am sending this document back to the working group. we will revist the document when a revision occurs and the document completes an new WGLC

IANA has reviewed draft-ietf-bmwg-sip-bench-term-08, which is currently
in Last Call, and has the following comments:

IANA understands that, upon approval of this document, there are no
IANA Actions that need completion.

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Subject: Last Call:  (Terminology for Benchmarking Session Initiation Protocol (SIP) Networking Devices) to Informational RFC


The IESG has received a request from the Benchmarking Methodology WG
(bmwg) to consider the following document:
- 'Terminology for Benchmarking Session Initiation Protocol (SIP)
  Networking Devices'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2013-01-30. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This document provides a terminology for benchmarking the SIP
  performance of networking devices.  The term performance in this
  context means the capacity of the device- or system-under-test to
  process SIP messages.  Terms are included for test components, test
  setup parameters, and performance benchmark metrics for black-box
  benchmarking of SIP networking devices.  The performance benchmark
  metrics are obtained for the SIP signaling plane only.  The terms are
  intended for use in a companion methodology document for
  characterizing the performance of a SIP networking device under a
  variety of conditions.  The intent of the two documents is to enable
  a comparison of the capacity of SIP networking devices.  Test setup
  parameters and a methodology document are necessary because SIP
  allows a wide range of configuration and operational conditions that
  can influence performance benchmark measurements.  A standard
  terminology and methodology will ensure that benchmarks have
  consistent definition and were obtained following the same
  procedures.




The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-bmwg-sip-bench-term/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-bmwg-sip-bench-term/ballot/


No IPR declarations have been submitted directly on this I-D.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet
Standard, Informational, Experimental, or Historic)? Why is this the proper
type of RFC? Is this type of RFC indicated in the title page header?

Informational, as indicated on the title page.
All BMWG RFCs are traditionally Informational,
in part because they do not define protocols and
the traditional conditions for Stds track advancement
did not apply.  However, they are specifications and
the RFC 2119 terms are applicable to identify the
level of requirements.

(2) The IESG approval announcement includes a Document Announcement Write-Up.
Please provide such a Document Announcement Write-Up. Recent examples can be
found in the "Action" announcements for approved documents. The approval
announcement contains the following sections:

Technical Summary:

All networking devices have a limited capacity to serve their
purpose. In some cases these limits can be ascertained by counting
physical features (e.g., interface card slots), but in other cases
standard ized tests are required to be sure that all vendors count
their protocol-handling capacity in the same way, to avoid specmanship.
This draft addresses one such case, where the SIP session-serving
capacity of a device can only be discovered and rigorously compared
with other devices through isolated laboratory testing.

This document describes the methodology for benchmarking Session
-or-
This document describes the terminology for benchmarking Session
Initiation Protocol (SIP) performance as described in SIP
benchmarking terminology document.  The methodology and terminology
are to be used for benchmarking signaling plane performance with
varying signaling and media load.  Both scale and establishment rate
are measured by signaling plane performance.  The SIP Devices to be
benchmarked may be a single device under test or a system under
test.  Benchmarks can be obtained and compared for different
types of devices such as SIP Proxy Server, Session Border Controller,
and server paired with a media relay or Firewall/NAT device.

Working Group Summary:

There were periods of intense and constructive feedback on this draft,
but also several pauses in progress during development. The most lively
discussions were prompted by presentation of actual test results using
the draft methods, which require significant time investment but are well-
worth the result. These drafts serve a useful purpose for the industry.

Document Quality:

There are existing implementations of the method, as noted above.

Dale Worley conducted an early review, following BMWG's request
of the RAI area.  Dales's comments were addressed in version 05.
Henning Schulzrinne commented on the original work proposal.

Personnel:

Who is the Document Shepherd? Who is the Responsible Area Director?
Al Morton is Shepherd, Ron Bonica is Responsible AD.

(3) Briefly describe the review of this document that was performed by the
Document Shepherd. If this version of the document is not ready for
publication, please explain why the document is being forwarded to the IESG.

The shepherd has reviewed the drafts many times, and his comments are
in the BMWG archive.  The last review resulted in post-WGLC revisions.

(4) Does the document Shepherd have any concerns about the depth or breadth of
the reviews that have been performed?
No.

(5) Do portions of the document need review from a particular or from broader
perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or
internationalization? If so, describe the review that took place.

No. Cross-area review has been obtained, however it impossible to get
the attention of everyone who considers themselves a SIP expert.

(6) Describe any specific concerns or issues that the Document Shepherd has
with this document that the Responsible Area Director and/or the IESG should be
aware of? For example, perhaps he or she is uncomfortable with certain parts of
the document, or has concerns whether there really is a need for it. In any
event, if the WG has discussed those issues and has indicated that it still
wishes to advance the document, detail those concerns here.

No concerns, this is still a valuable memo, as mentioned above.

(7) Has each author confirmed that any and all appropriate IPR disclosures
required for full conformance with the provisions of BCP 78 and BCP 79 have
already been filed. If not, explain why?

There are no outstanding IPR disclosures, according to the authors.

(8) Has an IPR disclosure been filed that references this document? If so,
summarize any WG discussion and conclusion regarding the IPR disclosures.

No.

(9) How solid is the WG consensus behind this document? Does it represent the
strong concurrence of a few individuals, with others being silent, or does the
WG as a whole understand and agree with it?

Although the comments and review intensity was highly variable,
it now appears that the WG is satisfied.  The 3rd WGLC went quietly.
The first WGLC was completed on 5 April 2010 with comments.
The second WGLC was completed on 18 May 2012 with comments.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent?
If so, please summarise the areas of conflict in separate email messages to the
Responsible Area Director. (It should be in a separate email because this
questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this document.
(See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist).
Boilerplate checks are not enough; this check needs to be thorough.

Nits (version 2.12.13) are warnings requiring no action for this draft.

(12) Describe how the document meets any required formal review criteria, such
as the MIB Doctor, media type, and URI type reviews.

N/A

(13) Have all references within this document been identified as either
normative or informative?

Yes.

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative references
exist, what is the plan for their completion?

The -term and -meth drafts are proceeding to publication as a pair.

(15) Are there downward normative references references (see RFC 3967)? If so,
list these downward references to support the Area Director in the Last Call
procedure.

No.

(16) Will publication of this document change the status of any existing RFCs?
Are those RFCs listed on the title page header, listed in the abstract, and
discussed in the introduction? If the RFCs are not listed in the Abstract and
Introduction, explain why, and point to the part of the document where the
relationship of this document to the other RFCs is discussed. If this
information is not in the document, explain why the WG considers it
unnecessary.

No.

(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes are
associated with the appropriate reservations in IANA registries. Confirm that
any referenced IANA registries have been clearly identified. Confirm that newly
created IANA registries include a detailed specification of the initial
contents for the registry, that allocations procedures for future registrations
are defined, and a reasonable name for the new registry has been suggested (see
RFC 5226).

No requests of IANA.

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find useful in
selecting the IANA Experts for these new registries.

N/A

(19) Describe reviews and automated checks performed by the Document Shepherd
to validate sections of the document written in a formal language, such as XML
code, BNF rules, MIB definitions, etc.

N/A

This is a publication request for:
  draft-ietf-bmwg-sip-bench-meth -07 2012-01-06  Active
  draft-ietf-bmwg-sip-bench-term -07 2012-01-06  Active

using the shepherding form dated 24 February 2012, now available from
http://www.ietf.org/iesg/template/doc-writeup.html

Al Morton is the Document Shepherd, and prepared this form.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational, as indicated on the title page.
All BMWG RFCs are traditionally Informational,
in part because they do not define protocols and
the traditional conditions for Stds track advancement
did not apply.  However, they are specifications and
the RFC 2119 terms are applicable to identify the
level of requirements.

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

All networking devices have a limited capacity to serve their
purpose. In some cases these limits can be ascertained by counting
physical features (e.g., interface card slots), but in other cases
standard ized tests are required to be sure that all vendors count
their protocol-handling capacity in the same way, to avoid specmanship.
This draft addresses one such case, where the SIP session-serving
capacity of a device can only be discovered and rigorously compared
with other devices through isolated laboratory testing.

This document describes the methodology for benchmarking Session
-or-
This document describes the terminology for benchmarking Session
Initiation Protocol (SIP) performance as described in SIP
benchmarking terminology document.  The methodology and terminology
are to be used for benchmarking signaling plane performance with
varying signaling and media load.  Both scale and establishment rate
are measured by signaling plane performance.  The SIP Devices to be
benchmarked may be a single device under test or a system under
test.  Benchmarks can be obtained and compared for different
types of devices such as SIP Proxy Server, Session Border Controller,
and server paired with a media relay or Firewall/NAT device.

Working Group Summary:

There were periods of intense and constructive feedback on this draft,
but also several pauses in progress during development. The most lively
discussions were prompted by presentation of actual test results using
the draft methods, which require significant time investment but are well-
worth the result. These drafts serve a useful purpose for the industry.

Document Quality:

There are existing implementations of the method, as noted above.

Dale Worley conducted an early review, following BMWG's request
of the RAI area.  Dales's comments were addressed in version 05.
Henning Schulzrinne commented on the original work proposal.

Personnel:

Who is the Document Shepherd? Who is the Responsible Area Director?
Al Morton is Shepherd, Ron Bonica is Responsible AD.

(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

The shepherd has reviewed the drafts many times, and his comments are
in the BMWG archive.  The last review resulted in post-WGLC revisions.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?
No.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

No. Cross-area review has been obtained, however it impossible to get
the attention of everyone who considers themselves a SIP expert.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

No concerns, this is still a valuable memo, as mentioned above.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

There are no outstanding IPR disclosures, according to the authors.

(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No.

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Although the comments and review intensity was highly variable,
it now appears that the WG is satisfied.  The 3rd WGLC went quietly.
The first WGLC was completed on 5 April 2010 with comments.
The second WGLC was completed on 18 May 2012 with comments.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

Nits (version 2.12.13) are warnings requiring no action for this draft.

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, media type, and URI type reviews.

N/A

(13) Have all references within this document been identified as either normative or informative?

Yes.

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

The -term and -meth drafts are proceeding to publication as a pair.

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

No.

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No.

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 5226).

No requests of IANA.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

N/A

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, etc.

N/A