Session Initiation Protocol Event Package for Voice Quality Reporting
draft-ietf-sipping-rtcp-summary-13

[Ballot comment]
1. Section 4.4 talks about subscription duration but not about when to cancel a subscription (by either the reporter or the collector); for example, it might be appropriate to recommend cancelling the subscription when the voice session had ended?

2. The RemoteID "bill@elpmaxe.org" does not conform to RFC 2606.

3. The references to draft-ietf-sipping-config-framework and RFC 5389 appear to be informative, not normative.

[Ballot comment]
1. Section 4.4 talks about subscription duration but not about when to cancel a subscription (by either the reporter or the collector); for example, it might be appropriate to recommend cancelling the subscription when the voice session had ended?

2. The RemoteID "bill@elpmaxe.org" does not conform to RFC 2606.

3. The references to draft-ietf-sipping-config-framework and RFC 5389 appear to be informative, not normative.

[Ballot comment]
1. Section 4.4 talks about subscription duration but not about when to cancel a subscription (by either the reporter or the collector); for example, it might be appropriate to recommend cancelling the subscription when the voice session had ended?

2. The RemoteID "bill@elpmaxe.org" does not conform to RFC 2606.

3. The references to draft-ietf-sipping-config-framework and RFC 5389 appear to be informative, not normative.

[Ballot comment]
1) in section 3.2, where is "unmanaged" defined?

2) in 3.2, s/recommended/RECOMMENDED/

3) in 3.3., where is report bodies defined?

4) in 4.5, "always shares" - with whom?

5) in 4.6.2.3.8, where is "fmtp" defined?

[Ballot discuss]
1) in 3.2 the reporter should send an options message to ensure support of the publish message; what happens if should the reporter do if it is not supported?

2) in 4.6.2.4, s/can/MAY/

[Ballot comment]
Some of the comments in the Gen-ART Review by Scott Brim were
  not addressed.  Please consider them.

  In Secrion 3.4 (Overload Avoidance) has small editorial issues

  - Typo: "MUST send a 503 Service Unavailable or other 5xx esponse"

  - "on these responses and respect the retry after time interval."
    Perhaps it should say "Retry-After."

[Ballot comment]
[Updated as per draft-ietf-sipping-rtcp-summary-10.txt]

The document shepherd/sponsoring AD should rerun
on the ABNF from this document.


      ; SignalLevel will normally be a negative value
      ; the absence of the negative sign indicates a positive value

I think "." is missing at the end of the previous line.

      ; where the signal level is negative, the sign MUST be
      ; included.This metric applies to the speech signal decoded
      ; from the received packet stream.

      SignalLevel = "SL" EQUAL (["-"] 1*2DIGIT)



      RLQEstAlg = "RLQEstAlg" EQUAL word ; "P.564", or other

Some pointer where P.564 is defined would be good hear.


4.6.2.2.  Timestamps

  Following SIP and other IETF convention, timestamps are provided in
  Coordinated Universal Time (UTC) using the ABNF format provided in
  RFC 3339 [7].

ABNF for timestamps allows for timezones, so it would be good to mention
in the ABNF section that timezones other than "Z" are not allowed.



4.6.2.11.2.  RLQEstAlg

  This field provides a text name for the algorithm used to estimate
  ListeningQualityR.

Is there an IANA registry for such algorithms, or does this field
contain free form text?

4.6.2.11.4.  RCQEstAlg

  This field provides a text name for the algorithm used to estimate
  ConversationalQualityR.

As above.

4.6.2.11.6.  ExtRInEstAlg

  This field provides a text name for the algorithm used to estimate
  ExternalR-In.

As above.

4.6.2.11.8.  ExtROutEstAlg

  This field provides a text name for the algorithm used to estimate
  ExternalR-Out.  Conversion of RFC3611 reported MOS scores for use in
  reporting MOS-LQ and MOS-CQ MUST be performed by dividing the RFC3611
  reported value by 10 if this value is less than or equal to 50 or
  omitting the MOS-xQ parameter if the RFC3611 reported value is 127
  (which indicates unavailable).

All but the first sentence don't seem to belong to this section.

4.6.2.11.11.  MOSCQEstAlg

  This field provides a text name for the algorithm used to estimate
  MOS-CQ.

Is there an IANA registry for such algorithms, or does this field
contain free form text?

[Ballot comment]
; SignalLevel will normally be a negative value
      ; the absence of the negative sign indicates a positive value

I think "." is missing at the end of the previous line.

      ; where the signal level is negative, the sign MUST be
      ; included.This metric applies to the speech signal decoded
      ; from the received packet stream.

      SignalLevel = "SL" EQUAL (["-"] 1*2DIGIT)



      RLQEstAlg = "RLQEstAlg" EQUAL word ; "P.564", or other

Some pointer where P.564 is defined would be good hear.


4.6.2.2.  Timestamps

  Following SIP and other IETF convention, timestamps are provided in
  Coordinated Universal Time (UTC) using the ABNF format provided in
  RFC 3339 [7].

ABNF for timestamps allows for timezones, so it would be good to mention
in the ABNF section that timezones other than "Z" are not allowed.



4.6.2.11.2.  RLQEstAlg

  This field provides a text name for the algorithm used to estimate
  ListeningQualityR.

Is there an IANA registry for such algorithms, or does this field
contain free form text?

4.6.2.11.4.  RCQEstAlg

  This field provides a text name for the algorithm used to estimate
  ConversationalQualityR.

As above.

4.6.2.11.6.  ExtRInEstAlg

  This field provides a text name for the algorithm used to estimate
  ExternalR-In.

As above.

4.6.2.11.8.  ExtROutEstAlg

  This field provides a text name for the algorithm used to estimate
  ExternalR-Out.  Conversion of RFC3611 reported MOS scores for use in
  reporting MOS-LQ and MOS-CQ MUST be performed by dividing the RFC3611
  reported value by 10 if this value is less than or equal to 50 or
  omitting the MOS-xQ parameter if the RFC3611 reported value is 127
  (which indicates unavailable).

All but the first sentence don't seem to belong to this section.

4.6.2.11.11.  MOSCQEstAlg

  This field provides a text name for the algorithm used to estimate
  MOS-CQ.

Is there an IANA registry for such algorithms, or does this field
contain free form text?

[Ballot discuss]
[Updated as per draft-ietf-sipping-rtcp-summary-09.txt - only 1 comment was addressed in this version, so my DISCUSS still stands.]

1). Multiple errors in the ABNF reported by BAP :

      Metrics = TimeStamps CRLF

        [SessionDescription CRLF]
          CallID CRLF
          FromID CRLF

Invalid (not allowed by ABNF syntax) empty line after "Metrics = ..."

      Ssrc        = "SSRC" EQUAL ( ''0x'' 1*8HEXDIG)

''0x'' is not a valid ABNF production. You either need to use "0x"
(but note that all strings in ABNF are case insensitive, so this would
match "0X"), or use a sequence of dot separated decimal or hex numbers
(as per RFC 5234, section 2.3): %x30.78

      JitterBuffer = "JitterBuffer" HCOLON

          [JitterBufferAdaptive WSP]
          [JitterBufferRate WSP]
          [JitterBufferNominal WSP]

Invalid empty line.

      NetworkPacketLossRate =

        "NLR" EQUAL (1*3DIGIT ["." 1*2DIGIT]) ;percentage

Invalid empty line.

      JitterBufferDiscardRate =

        "JDR" EQUAL (1*3DIGIT ["." 1*2DIGIT]) ;percentage

Invalid empty line.

      BurstGapLoss = "BurstGapLoss" HCOLON

          [BurstLossDensity WSP]
          [BurstDuration WSP]
          [GapLossDensity WSP]
          [GapDuration WSP]
          [MinimumGapThreshold]
          *(WSP Extension)

Invalid empty line.

      partial-time = time-hour ":" time-minute ":" time-second
      [time-secfrac]

I think this line wrapped, so BAP complains about indentation.


CallTerm ABNF production is not defined and I can't find it in RFC 3261.

2).
4.6.2.9.4.  One Way Delay

  This value SHOULD be measured using the methods defined in IETF
  RFC2679.  The parameter is expressed in milliseconds.

This should be a Normative reference.


3).
5.2.  application/vq-rtcp-xr MIME Registration

  Encoding considerations: text

According to Section 4.8 of RFC 4288, this value can only contain one of the following values:

7bit/8bit/binary/framed

So please the one which is appropriate for this media type

4).

1.2.  Usage Scenarios

  Configuration of usage of the event package is not covered in this
  document.  It is the recommendation of this document that the SIP
  configuration framework [8] be used.  The authors have defined a
  configuration dataset that would facilitate this support in section
  5.8.

There is no section 5.8. There is section 4.8, but it says:

4.8.  Configuration Dataset for vq-rtcpxr Events

  It is the suggestion of the authors that the SIP configuration
  framework [8] be used to establish the necessary parameters for usage
  of vq-rtcpxr events.  A dataset for this purpose should be designed
  and documented in a separate draft upon completion of the framework.

So the two sections are inconsistent.

IANA comments:

Action #1:

Upon approval of this document, IANA will make the following assignment
in the "Session Initiation Protocol (SIP) Event Types Namespace"
registry at
http://www.iana.org/assignments/sip-events

Package Name Type Contact Reference
-------------------------- ---------------- ---------------- --------
-vq-rtcpx package Amy Pendleton  [RFC-sipping-rtcp-
summary-08]


Action #2:

Upon approval of this document, IANA will register the following
application media type at
http://www.iana.org/assignments/media-types/application/

vq-rtcpxr

[Ballot comment]
Unclear on the use use of references within the ABNF comments.
Sometimes they are used, and sometimes not.
Suspect that (as per MIB Modules) they should not be used.

Wonder if the Proto Writeup should be updated. This would certainly have caught Alexey's issues with the ABNF.

[Ballot comment]
; PayloadDesc provides a text description of the codec
      ; The abbreviated Standard name for the codec SHOULD be used
      ; (for example "G.729A")

      PayloadDesc  = "PD" EQUAL (word / DQUOTE word-plus DQUOTE)

While the character repertoire allowed by the ABNF is quite limited,
it would be good to say which language is used for PayloadDesc.

If you want to allow more than one language here, this would require
a language tag identifier as per BCP 18 (RFC 2277).

(Same comment regarding text in Section 4.6.2.3.2.)


      ; SignalLevel will normally be a negative value
      ; the absence of the negative sign indicates a positive value

I think "." is missing at the end of the previous line.

      ; where the signal level is negative, the sign MUST be
      ; included.This metric applies to the speech signal decoded
      ; from the received packet stream.

      SignalLevel = "SL" EQUAL (["-"] 1*2DIGIT)



      RLQEstAlg = "RLQEstAlg" EQUAL word ; "P.564", or other

Some pointer where P.564 is defined would be good hear.


4.6.2.2.  Timestamps

  Following SIP and other IETF convention, timestamps are provided in
  Coordinated Universal Time (UTC) using the ABNF format provided in
  RFC 3339 [7].

ABNF for timestamps allows for timezones, so it would be good to mention
in the ABNF section that timezones other than "Z" are not allowed.



4.6.2.11.2.  RLQEstAlg

  This field provides a text name for the algorithm used to estimate
  ListeningQualityR.

Is there an IANA registry for such algorithms, or does this field
contain free form text?

4.6.2.11.4.  RCQEstAlg

  This field provides a text name for the algorithm used to estimate
  ConversationalQualityR.

As above.

4.6.2.11.6.  ExtRInEstAlg

  This field provides a text name for the algorithm used to estimate
  ExternalR-In.

As above.

4.6.2.11.8.  ExtROutEstAlg

  This field provides a text name for the algorithm used to estimate
  ExternalR-Out.  Conversion of RFC3611 reported MOS scores for use in
  reporting MOS-LQ and MOS-CQ MUST be performed by dividing the RFC3611
  reported value by 10 if this value is less than or equal to 50 or
  omitting the MOS-xQ parameter if the RFC3611 reported value is 127
  (which indicates unavailable).

All but the first sentence don't seem to belong to this section.

4.6.2.11.11.  MOSCQEstAlg

  This field provides a text name for the algorithm used to estimate
  MOS-CQ.

Is there an IANA registry for such algorithms, or does this field
contain free form text?

[Ballot discuss]
1). Multiple errors in the ABNF reported by BAP :

      Metrics = TimeStamps CRLF

        [SessionDescription CRLF]
          CallID CRLF
          FromID CRLF

Invalid (not allowed by ABNF syntax) empty line after "Metrics = ..."

      Ssrc        = "SSRC" EQUAL ( ''0x'' 1*8HEXDIG)

''0x'' is not a valid ABNF production. You either need to use "0x"
(but note that all strings in ABNF are case insensitive, so this would
match "0X"), or use a sequence of dot separated decimal or hex numbers
(as per RFC 5234, section 2.3): %x30.78

      JitterBuffer = "JitterBuffer" HCOLON

          [JitterBufferAdaptive WSP]
          [JitterBufferRate WSP]
          [JitterBufferNominal WSP]

Invalid empty line.

      NetworkPacketLossRate =

        "NLR" EQUAL (1*3DIGIT ["." 1*2DIGIT]) ;percentage

Invalid empty line.

      JitterBufferDiscardRate =

        "JDR" EQUAL (1*3DIGIT ["." 1*2DIGIT]) ;percentage

Invalid empty line.

      BurstGapLoss = "BurstGapLoss" HCOLON

          [BurstLossDensity WSP]
          [BurstDuration WSP]
          [GapLossDensity WSP]
          [GapDuration WSP]
          [MinimumGapThreshold]
          *(WSP Extension)

Invalid empty line.

      partial-time = time-hour ":" time-minute ":" time-second
      [time-secfrac]

I think this line wrapped, so BAP complains about indentation.


CallTerm ABNF production is not defined and I can't find it in RFC 3261.

2).
4.6.2.9.4.  One Way Delay

  This value SHOULD be measured using the methods defined in IETF
  RFC2679.  The parameter is expressed in milliseconds.

This should be a Normative reference.


3).
5.2.  application/vq-rtcp-xr MIME Registration

  Encoding considerations: text

According to Section 4.8 of RFC 4288, this value can only contain one of the following values:

7bit/8bit/binary/framed

So please the one which is appropriate for this media type

4).

1.2.  Usage Scenarios

  Configuration of usage of the event package is not covered in this
  document.  It is the recommendation of this document that the SIP
  configuration framework [8] be used.  The authors have defined a
  configuration dataset that would facilitate this support in section
  5.8.

There is no section 5.8. There is section 4.8, but it says:

4.8.  Configuration Dataset for vq-rtcpxr Events

  It is the suggestion of the authors that the SIP configuration
  framework [8] be used to establish the necessary parameters for usage
  of vq-rtcpxr events.  A dataset for this purpose should be designed
  and documented in a separate draft upon completion of the framework.

So the two sections are inconsistent.

[Ballot discuss]
1). Multiple errors in the ABNF reported by BAP :

      Metrics = TimeStamps CRLF

        [SessionDescription CRLF]
          CallID CRLF
          FromID CRLF

Invalid (not allowed by ABNF syntax) empty line after "Metrics = ..."

      Ssrc        = "SSRC" EQUAL ( ''0x'' 1*8HEXDIG)

''0x'' is not a valid ABNF production. You either need to use "0x"
(but note that all strings in ABNF are case insensitive, so this would
match "0X"), or use a sequence of dot separated decimal or hex numbers
(as per RFC 5234, section 2.3): %x30.78

      JitterBuffer = "JitterBuffer" HCOLON

          [JitterBufferAdaptive WSP]
          [JitterBufferRate WSP]
          [JitterBufferNominal WSP]

Invalid empty line.

      NetworkPacketLossRate =

        "NLR" EQUAL (1*3DIGIT ["." 1*2DIGIT]) ;percentage

Invalid empty line.

      JitterBufferDiscardRate =

        "JDR" EQUAL (1*3DIGIT ["." 1*2DIGIT]) ;percentage

Invalid empty line.

      BurstGapLoss = "BurstGapLoss" HCOLON

          [BurstLossDensity WSP]
          [BurstDuration WSP]
          [GapLossDensity WSP]
          [GapDuration WSP]
          [MinimumGapThreshold]
          *(WSP Extension)

Invalid empty line.

      partial-time = time-hour ":" time-minute ":" time-second
      [time-secfrac]

I think this line wrapped, so BAP complains about indentation.


CallTerm ABNF production is not defined and I can't find it in RFC 3261.

2).
5.2.  application/vq-rtcp-xr MIME Registration

  Encoding considerations: text

According to Section 4.8 of RFC 4288, this value can only contain one of the following values:

7bit/8bit/binary/framed

So please the one which is appropriate for this media type

[Ballot discuss]
5.2.  application/vq-rtcp-xr MIME Registration

  Encoding considerations: text

According to Section 4.8 of RFC 4288, this value can only contain one of the following values:

7bit/8bit/binary/framed

So please the one which is appropriate for this media type

[Ballot comment]
Section 4.6.2.3 and 4.6.2.13 - I do not understand what is the usage of the following:

Payload Desc

This parameter is not mapped from any specific SDP
or RTP field; provides a text description of the
Payload Type/codec.

  RLQEstAlg

  This field provides a text description of the algorithm used
  to estimate ListeningQualityR. 

  RCQEstAlg

  This field provides a text description of the algorithm used
  to estimate ConversationalQualityR

How is interoperability ensured without a proper register that defines what text applies to each algorithm?

[Ballot comment]
As the authors don't want to change the format at all to address the issues I don't see how good resolution of the issues can be done. Rather than spending more time on this document to fix things that aren't fixed in a good way I are abstaining.

The main issues that aren't really addressed, only worked around are:
- SSRC multiplexing model
- Do not handles dynamic parameters in a good way
- Parameters for whole call are not statistically processed in a really usable way. Instead of an average over all, better statics should be provied, like max, min, avg, variation.

[Ballot discuss]
Support Magnus discuss

Adding one more ... Section 4.8 would likely need to wait for the dataset work to be further along and require a normative reference to it. As this is unlikely to happen soon, I would suggest moving section 4.8 into a separate draft.

[Ballot discuss]
From my IETF last call comments that hasn't been addressed:
(Removed the ones that has been addressed)


3. Section 4.6.1

"  SessionDescription  = "SessionDesc" COLON
      [PayloadType WSP]
      [PayloadDesc WSP]
      [SampleRate WSP]
      [FrameDuration WSP]
      [FrameOctets WSP]
      [FramesPerPacket WSP]
      [PacketsPerSecond WSP]
      [FmtpOptions WSP]
      [PacketLossConcealment WSP]
      [SilenceSuppressionState]
      *(WSP Extension)

  ; PayloadType provides the PT parameter used in the RTP packets
  ; i.e. the codec used for decoding received RTP packets
  ; It is recommended that IANA registered values are used
  ; where possible.
  PayloadType  = "PT" EQUAL (1*3DIGIT)

  ; PayloadDesc provides a text description of the codec
  ; It is recommended that IANA registered names are used
  ; where possible.
  PayloadDesc  = "PD" EQUAL word"

This description doesn't seem to work. It doesn't provide a way of binding a payload type to an actual description of the codec configured. It may depend on a failure to consider the use of multiple payload types within a RTP session.

4. Section 4.6.1.

"  ; SampleRate provides the rate at which voice was sampled
  ; in the case of narrowband codecs, the value will typically be 8000
  SampleRate = "SR" EQUAL (1*5DIGIT)"

This should also be bound on a per payload type basis.

5. Section 4.6.1.

"  ; FrameDuration can be combined with the FramesPerPacket to determine
  ; the packetization rate; the units for this are milliseconds.
  FrameDuration = "FD" EQUAL (1*3DIGIT)

This is a static value for many codecs. However there exist some codecs that can manipulate this runtime. Thus if one like to have this working then one should have this as a reported value.

6. Section 4.6.1.

  ; FrameOctets provides the number of octets in each frame
  ; Used where FrameDuration is not available
  FrameOctets  = "FO" EQUAL (1*4DIGIT)

  ; FramesPerPacket provides the number of frames in each RTP packet
  FramesPerPacket = "FPP" EQUAL (1*2DIGIT)

  ; Packets per second provides the number of packets, including one or
  ; more frames within each, that are transmitted per second
  PacketsPerSecond = "PPS" EQUAL (1*5DIGIT)"

These parameters are all dynamic ones and can vary with time to adapt to transport sessions. Some are even so dynamic that they can change on a packet by packet basis. Thus a complete different structure and definitions are needed to handle this.

7. Section 4.6.1.

"  ; FMTP options from SDP.  Note that the parameter is deliniated
  ; by " " to avoid parsing issues in transitioning between SDP and
  ; SIP parsing
  FmtpOptions = "FMTP" EQUAL DQUOTE word-plus DQUOTE"

This also needs to be on per payload type basis. With the possibility to add multiple ones.

8. Section 4.6.1.

I somewhat question the SSRC multiplexing model used in this report format. Is it the intention that "SessionReport" applies on the complete RTP session as the name hints at or as the fact is on a single source. Because with current structure you need to send individual reports for each SSRC that takes part of the same session. And in that duplicate a lot of information related to the configuration. Please explain.


9. Section 4.6.1:

  ; LocalAddr provides the IP address, port and ssrc for the
  ; session from the perspective of the endpoint/UA which is
  ; sending the report
  LocalAddr  = "LocalAddr" COLON IPAddress WSP Port WSP Ssrc

  ; RemoteAddr provides the IP address, port and ssrc for the
  ; session from the perspective of the peer of the endpoint/UA
  ; that is sending the report
  RemoteAddr  = "RemoteAddr" COLON IPAddress WSP Port WSP Ssrc

These definitions are not correct by RTP definitions. SSRC represents sources in an RTP session. They do have source and destination address and ports. However depending on how you are who does the reporting the destination may or may not match the reporting entities address. Multicast is one such example. But if one actually like translators or mixers to use this package then one needs to separate the roles correctly. The right way of characterize a RTP flow from a particular source would in my view be to:

1. Be explicit on the identify of the reporting entity, and please consider that they may not have an SSRC even if most will. (They will if they are sending RTCP). But there are reporting entities that only consumes RTCP and do not send it and thus do not need an SSRC.

2. Use source address/port tuple and destination address/port tuple plus SSRC for identifying each flow.

That way one can avoid the confusion about who is reporting and on what in most cases. It might still not be crystal clear in systems that employ translators as they rewrite source and destination tuples. But the SSRC will be maintained over such entities.

The extension of this is that one needs to be able to provide the report data on per reporter basis for each source of an RTP flow. Thus some multi-level hierarchy is needed to handle this correctly. The current ones assumes a single remote peer point that can provides RTCP data that one may like to summarize in this report. Which totally fails in multi-party environments using multicast or translators.


10. Section 4.6.1:

One more thing about the identification of who's data and for what. Please consider how a REPORTER can provide data for the stream it sent itself. Currently it seems that this stream will be impossible to include in the report as you must have local data which you don't have. You can only have remote data for that stream.


11. section 4.6.1:

  ; RoundTripDelay is recommended to be measured as defined in
  ; RTCP, RFC 3550.
  RoundTripDelay = "RTD" EQUAL (1*5DIGIT) ;0-65535

  ; OneWayDelay is recommended to be measured according to
  ; recommendations provided by the IPPM working group but may be
  ; based on alternative measurement recommendations
  OneWayDelay = "OWD" EQUAL (1*5DIGIT) ;0-65535

These measurements are lacking a clear definition or allows for alternative definitions to be used. I can't see how this provide interoperable usage of the measurement. The method for the measurement should be clear and precise so that is useful.


13. Section 4.6.2:

Aggregation of the metrics over the reporting interval. In RTCP XR the reported value it the value of the measurement at the time of sending the RTCP packet. However in this summary report I think you need to clarify what values actually should be used. Especially if it is the intention that a peer should include summary of received RTCP packets from other peers. Because to me it seems that using this metric will require one to run additional instances of the measurement process then for the RTCP reports or for SessionReport or the IntervalReport due to the need to reset the values at the start time of the metrics.

So please clarify how the different metrics should be derived before being written into the different type of report types.

To make it clear about this example. Lets use Round Trip Delay as an example. What value should be included in a SessionReport? RTT values may vary substantially during a session. It also is instantaneous value for each received RTCP RR from ones peer. So it is the latest for the session duration, the largest, the average, the median, some other statistical processing. And should I calculate this different in IntervalReport or in a AlertReport?

[Ballot discuss]
Adding one more ... Section 4.8 would likely need to wait for the dataset work to be further along and require a normative reference to it. As this is unlikely to happen soon, I would suggest moving section 4.8 into a separate draft.

[Ballot discuss]
1. Introduction

'the event package is designed to be extensible for use with any RTP application.  The negotiation of such extensions is not defined in this recommendation.'

Saying that the event package is extensible and leaving 'the negotiation of such extensions' completely out of the scope seems under-defined. I assume that a COLLECTOR needs to understand everything that a REPORTER needs to send. Without any capability negotiation or version number how can a future extensible REPORTER make sure that the COLLECTOR it deals with will make any sense of what is being sent?


2. Section 3.2

I believe that the overload avoidance should be prevented by adding to the configuration profile the method or methods of overload avoidence that the REPORTER must implement and the time interval between sending thresholds, so that a reporting system can dynamically adapt to COLLECTOR and network status.

Also limiting the MTU of the reports can be considered as a supplementary method for Overload avoidance.

3. Section 4.4 - The definition of the Subscription Duration is missing a default "Expires" value as per RFC 3265, Section 4.4

4. Section 4.6.1, page 7

  ; PayloadDesc provides a text description of the codec
  ; It is recommended that IANA registered names are used
  ; where possible.

What IANA register is refered here?

5. The following definitions in 4.6.11, page 10 are inconsistent with the mapping defined in 4.2.6.10:

; RoundTripDelay is recommended to be measured as defined in
  ; RTCP, RFC 3550.
  RoundTripDelay = "RTD" EQUAL (1*5DIGIT) ;0-65535
     
  ; OneWayDelay is recommended to be measured according to
  ; recommendations provided by the IPPM working group but may be
  ; based on alternative measurement recommendations
  OneWayDelay = "OWD" EQUAL (1*5DIGIT) ;0-65535
 
  ; Interarrival Jitter is recommended to be measured as defined
  ; in RTCP, RFC 3550, but may be based on alternatives
  InterarrivalJitter = "IAJ" EQUAL (1*5DIGIT) ;0-65535

I also do not understand what 'may be based on alternative measurement recommendations' means. How can interoperability be ensured if alternative (and not specified!) methods are allowed?

6. Section 4.6.2.3 and 4.6.2.13 - I do not understand what is the usage of the following:

Payload Desc

This parameter is not mapped from any specific SDP
or RTP field; provides a text description of the
Payload Type/codec.

  RLQEstAlg

  This field provides a text description of the algorithm used
  to estimate ListeningQualityR. 

  RCQEstAlg

  This field provides a text description of the algorithm used
  to estimate ConversationalQualityR

How is interoperability ensured without a proper register that defines what text applies to each algorithm?

7. Section 4.2.6.10 - How can interoperability be ensured if more than one method of measurement is acceptable without any means of knowing which one is used for the following:

This value may be measured based on IETF IPPM recommendations
or may be calculated as described in RFC3611:

8. Section 4.2.6.13 -

  This field does not have a direct mapping from RFC3611 but
  is expected to be provided in the RTCP High Resolution
  (RTCP HR) draft [x].

[x] needs to be defined and added to the reference list. However it points to a I-D which is only at draft-00 in AVT. Does really this document need to hang on that draft, or should it rather directly refer ITU-T G.107?

9. The references section is missing lots of normative references, also [5] appears twice.

[Ballot comment]
1. In the introduction section RTCP standard is reference [5] and not [3]

2. A renumbering of the sections probably happened between versions, but in a few instances the configuration dataset is refered to be in section 5.8 instead of section 4.8 where it can be found now.

3. Section 3:

The REPORTER shall not send any
  vq-rtcpxr events where a COLLECTOR has not been configured. 

I cannot compile this phrase. What does it mean? If a COLLECTOR was not configured in the REPORTER there is no way to send information to that collector anyway. If this means that the REPORTER should wait for the COLLECTOR to be fully configured, how does the REPORTER know when and if this happened? And in any case shall not needs to be capitalized, right?

3. It would be nice if the order of the mapping definitons in 4.6.2 would be identical with the order metrics show up in the ABNF syntax definition in 4.6.1

4. 4.6.2.3 - missing reference for payload type, probably RFC 3551

5. missing reference for FMTP and expansion

6. One Way Delay - if the reference to IPPM recommendation stays then refer to RFC 2679

7. Provide references for ITU-T G.1020 and G.107

8. 4.2.6.10 - missing 2119 keyword capitalization (It is RECOMMENDED ...)

[Ballot comment]
In 4.2.6.10, the specifications for both Round Trip Delay and End System Delay end with

"This parameter does no require any conversion."

Perhaps "no" should be "not" in both cases?

[Ballot discuss]
Section 4.5 identifies three notify bodies: "a session report, an interval session
report, and a alert report."

The subsequent paragraphs also describes a "threshold report".  From the context,
I presume that "threshold report" is a synonym for "alert report", but I wouldn't
risk my own money on it.  If they are the same, please use "alert report" throughout.
If not, please explain the difference.

[Ballot comment]
1. In the introduction section RTCP standard is reference [5] and not [3]

2. A renumbering of the sections probably happened between versions, but in a few instances the configuration dataset is refered to be in section 5.8 instead of section 4.8 where it can be found now.

3. Section 3:

The REPORTER shall not send any
  vq-rtcpxr events where a COLLECTOR has not been configured. 

I cannot compile this phrase. What does it mean? If a COLLECTOR was not configured in the REPORTER there is no way to send information to that collector anyway. If this means that the REPORTER should wait for the COLLECTOR to be fully configured, how does the REPORTER know when and if this happened? And in any case shall not needs to be capitalized, right?

3. It would be nice if the order of the mapping definitons in 4.6.2 would be identical with the order metrics show up in the ABNF syntax definition in 4.6.1

4. 4.6.2.3 - missing reference for payload type, probably RFC 3551

5. missing reference for FMTP and expansion

6. One Way Delay - if the reference to IPPM recommendation stays then refer to RFC 2679

7. Provide references for ITU-T G.1020 and G.107

[Ballot discuss]
1. Introduction

'the event package is designed to be extensible for use with any RTP application.  The negotiation of such extensions is not defined in this recommendation.'

Saying that the event package is extensible and leaving 'the negotiation of such extensions' completely out of the scope seems under-defined. I assume that a COLLECTOR needs to understand everything that a REPORTER needs to send. Without any capability negotiation or version number how can a future extensible REPORTER make sure that the COLLECTOR it deals with will make any sense of what is being sent?


2. Section 3.2

I believe that the overload avoidance should be prevented by adding to the configuration profile the method or methods of overload avoidence that the REPORTER must implement and the time interval between sending thresholds, so that a reporting system can dynamically adapt to COLLECTOR and network status.

Also limiting the MTU of the reports can be considered as a supplementary method for Overload avoidance.

3. Section 4.4 - The definition of the Subscription Duration is missing a default "Expires" value as per RFC 3265, Section 4.4

4. Section 4.6.1, page 7

  ; PayloadDesc provides a text description of the codec
  ; It is recommended that IANA registered names are used
  ; where possible.

What IANA register is refered here?

5. The following definitions in 4.6.11, page 10 are inconsistent with the mapping defined in 4.2.6.10:

; RoundTripDelay is recommended to be measured as defined in
  ; RTCP, RFC 3550.
  RoundTripDelay = "RTD" EQUAL (1*5DIGIT) ;0-65535
     
  ; OneWayDelay is recommended to be measured according to
  ; recommendations provided by the IPPM working group but may be
  ; based on alternative measurement recommendations
  OneWayDelay = "OWD" EQUAL (1*5DIGIT) ;0-65535
 
  ; Interarrival Jitter is recommended to be measured as defined
  ; in RTCP, RFC 3550, but may be based on alternatives
  InterarrivalJitter = "IAJ" EQUAL (1*5DIGIT) ;0-65535

I also do not understand what 'may be based on alternative measurement recommendations' means. How can interoperability be ensured if alternative (and not specified!) methods are allowed?

6. Section 4.6.2.3 and 4.6.2.13 - I do not understand what is the usage of the following:

Payload Desc

This parameter is not mapped from any specific SDP
or RTP field; provides a text description of the
Payload Type/codec.

  RLQEstAlg

  This field provides a text description of the algorithm used
  to estimate ListeningQualityR. 

  RCQEstAlg

  This field provides a text description of the algorithm used
  to estimate ConversationalQualityR

How is interoperability ensured without a proper register that defines what text applies to each algorithm?

7. Section 4.2.6.10 - How can interoperability be ensured if more than one method of measurement is acceptable without any means of knowing which one is used for the following:

This value may be measured based on IETF IPPM recommendations
or may be calculated as described in RFC3611:

8. Section 4.2.6.13 -

  This field does not have a direct mapping from RFC3611 but
  is expected to be provided in the RTCP High Resolution
  (RTCP HR) draft [x].

[x] needs to be defined and added to the reference list. However it points to a I-D which is only at draft-00 in AVT. Does really this document need to hang on that draft, or should it rather directly refer ITU-T G.107?

8. The references section is missing lots of normative references, also [5] appears twice.

[Ballot comment]
Document has a long list of idnits.

Section 4.6., paragraph 1:
>    The formal syntax definitions described in this
>    section are expressed in the Augmented BNF [6] format used in SIP [2],

  RFC2234 [6] was obsoleted by RFC 4234.

Section 4.6.1, paragraph 38:
>    ; Mean Absolute Jitter is recommended to be measured as defined
>    ; by ITU-T G.1020 where it is known as MAPDV

  Need to add a reference to ITU-T G.1020.

Section 4.6.2, paragraph 0:
>  4.6.2  Parameter Definitions and Mappings

  Using RFC2119 terminology would make the definitions in this section
  less ambiguous.

Section 4.6.2.7, paragraph 2:
>  For conversion, see "General mapping percentages from 8
>  bit, fixed point numbers".

  See where? (Also below.)

Section 4.2.6.10, paragraph 6:
>    This value may be measured based on IETF IPPM recommendations
>    or may be calculated as described in RFC3611:

  Cite the relevant IDs or RFCs from IPPM.

Section 4.2.6.13, paragraph 2:
>    This field does not have a direct mapping from RFC3611 but
>    is expected to be provided in the RTCP High Resolution
>    (RTCP HR) draft [x].

  What draft is [x]?

Section 4.2.6.13, paragraph 3:
>    The scale used will typically be ITU-T G.107 compliant (0-100) but
>    can be greater where wideband audio codecs are used.

  Need to cite G.107.

[Ballot discuss]
Section 4.8, paragraph 1:
> It is the suggestion of the authors that the SIP configuration
> framework [8] be used to establish the necessary parameters for usage
> of vq-rtcpxr events.  A dataset for this purpose is provided below:

  DISCUSS: XML doesn't validate in xmllint.

[Ballot comment]
2. Section 3.3:

  "The REPORTER shall populate the Request-URI of the PUBLISH method
  with the address of the resource (AOR) of the COLLECTOR."

Okay, maybe my lack of SIP knowledge shows here. But should this sentence really talk about PUBLISH? Or is it a copy paste error?

[Ballot discuss]
From my IETF last call comments that hasn't been addressed:

1. Section 3.2:

I think there needs to be some details about how on actually configures and ensure that this type of overload situation doesn't happens. Lets ensure that we don't have to retrofit overload / congestion handling also to this protocol. Instead all mechanisms should be in place to ensure that interoperable function exist so that one can prevent overload.


3. Section 4.6.1

"  SessionDescription  = "SessionDesc" COLON
      [PayloadType WSP]
      [PayloadDesc WSP]
      [SampleRate WSP]
      [FrameDuration WSP]
      [FrameOctets WSP]
      [FramesPerPacket WSP]
      [PacketsPerSecond WSP]
      [FmtpOptions WSP]
      [PacketLossConcealment WSP]
      [SilenceSuppressionState]
      *(WSP Extension)

  ; PayloadType provides the PT parameter used in the RTP packets
  ; i.e. the codec used for decoding received RTP packets
  ; It is recommended that IANA registered values are used
  ; where possible.
  PayloadType  = "PT" EQUAL (1*3DIGIT)

  ; PayloadDesc provides a text description of the codec
  ; It is recommended that IANA registered names are used
  ; where possible.
  PayloadDesc  = "PD" EQUAL word"

This description doesn't seem to work. It doesn't provide a way of binding a payload type to an actual description of the codec configured. It may depend on a failure to consider the use of multiple payload types within a RTP session.

4. Section 4.6.1.

"  ; SampleRate provides the rate at which voice was sampled
  ; in the case of narrowband codecs, the value will typically be 8000
  SampleRate = "SR" EQUAL (1*5DIGIT)"

This should also be bound on a per payload type basis. I also think that only 1*5 digits are to little. That doesn't allow for usage with 192kHz sampling rate that may occur in a pro-audio application.

5. Section 4.6.1.

"  ; FrameDuration can be combined with the FramesPerPacket to determine
  ; the packetization rate; the units for this are milliseconds.
  FrameDuration = "FD" EQUAL (1*3DIGIT)

This is a static value for many codecs. However there exist some codecs that can manipulate this runtime. Thus if one like to have this working then one should have this as a reported value.

6. Section 4.6.1.

  ; FrameOctets provides the number of octets in each frame
  ; Used where FrameDuration is not available
  FrameOctets  = "FO" EQUAL (1*4DIGIT)

  ; FramesPerPacket provides the number of frames in each RTP packet
  FramesPerPacket = "FPP" EQUAL (1*2DIGIT)

  ; Packets per second provides the number of packets, including one or
  ; more frames within each, that are transmitted per second
  PacketsPerSecond = "PPS" EQUAL (1*5DIGIT)"

These parameters are all dynamic ones and can vary with time to adapt to transport sessions. Some are even so dynamic that they can change on a packet by packet basis. Thus a complete different structure and definitions are needed to handle this.

7. Section 4.6.1.

"  ; FMTP options from SDP.  Note that the parameter is deliniated
  ; by " " to avoid parsing issues in transitioning between SDP and
  ; SIP parsing
  FmtpOptions = "FMTP" EQUAL DQUOTE word-plus DQUOTE"

This also needs to be on per payload type basis. With the possibility to add multiple ones.

8. Section 4.6.1.

I somewhat question the SSRC multiplexing model used in this report format. Is it the intention that "SessionReport" applies on the complete RTP session as the name hints at or as the fact is on a single source. Because with current structure you need to send individual reports for each SSRC that takes part of the same session. And in that duplicate a lot of information related to the configuration. Please explain.


9. Section 4.6.1:

  ; LocalAddr provides the IP address, port and ssrc for the
  ; session from the perspective of the endpoint/UA which is
  ; sending the report
  LocalAddr  = "LocalAddr" COLON IPAddress WSP Port WSP Ssrc

  ; RemoteAddr provides the IP address, port and ssrc for the
  ; session from the perspective of the peer of the endpoint/UA
  ; that is sending the report
  RemoteAddr  = "RemoteAddr" COLON IPAddress WSP Port WSP Ssrc

These definitions are not correct by RTP definitions. SSRC represents sources in an RTP session. They do have source and destination address and ports. However depending on how you are who does the reporting the destination may or may not match the reporting entities address. Multicast is one such example. But if one actually like translators or mixers to use this package then one needs to separate the roles correctly. The right way of characterize a RTP flow from a particular source would in my view be to:

1. Be explicit on the identify of the reporting entity, and please consider that they may not have an SSRC even if most will. (They will if they are sending RTCP). But there are reporting entities that only consumes RTCP and do not send it and thus do not need an SSRC.

2. Use source address/port tuple and destination address/port tuple plus SSRC for identifying each flow.

That way one can avoid the confusion about who is reporting and on what in most cases. It might still not be crystal clear in systems that employ translators as they rewrite source and destination tuples. But the SSRC will be maintained over such entities.

The extension of this is that one needs to be able to provide the report data on per reporter basis for each source of an RTP flow. Thus some multi-level hierarchy is needed to handle this correctly. The current ones assumes a single remote peer point that can provides RTCP data that one may like to summarize in this report. Which totally fails in multi-party environments using multicast or translators.


10. Section 4.6.1:

One more thing about the identification of who's data and for what. Please consider how a REPORTER can provide data for the stream it sent itself. Currently it seems that this stream will be impossible to include in the report as you must have local data which you don't have. You can only have remote data for that stream.


11. section 4.6.1:

  ; RoundTripDelay is recommended to be measured as defined in
  ; RTCP, RFC 3550.
  RoundTripDelay = "RTD" EQUAL (1*5DIGIT) ;0-65535

  ; OneWayDelay is recommended to be measured according to
  ; recommendations provided by the IPPM working group but may be
  ; based on alternative measurement recommendations
  OneWayDelay = "OWD" EQUAL (1*5DIGIT) ;0-65535

These measurements are lacking a clear definition or allows for alternative definitions to be used. I can't see how this provide interoperable usage of the measurement. The method for the measurement should be clear and precise so that is useful.

12. Section 4.6.2.2:

Following SIP and other IETF convention, timestamps are provided
in Coordinated Universal Time (UTC) using the ABNF format provided
in IETF RFC3339 [x].

Reference number is missing.

13. Section 4.6.2:

Aggregation of the metrics over the reporting interval. In RTCP XR the reported value it the value of the measurement at the time of sending the RTCP packet. However in this summary report I think you need to clarify what values actually should be used. Especially if it is the intention that a peer should include summary of received RTCP packets from other peers. Because to me it seems that using this metric will require one to run additional instances of the measurement process then for the RTCP reports or for SessionReport or the IntervalReport due to the need to reset the values at the start time of the metrics.

So please clarify how the different metrics should be derived before being written into the different type of report types.

To make it clear about this example. Lets use Round Trip Delay as an example. What value should be included in a SessionReport? RTT values may vary substantially during a session. It also is instantaneous value for each received RTCP RR from ones peer. So it is the latest for the session duration, the largest, the average, the median, some other statistical processing. And should I calculate this different in IntervalReport or in a AlertReport?

14. Section 8.

Reference [6] needs to be updated to RFC 4234.

IANA Last Call Comments:

Upon approval of this document, the IANA will take the
following Actions:

Action 1 (section 5.1):

Upon approval of this document, the IANA will make the
following assignments in the "SIP Event Types Namespace"
registry located at

http://www.iana.org/assignments/sip-events

Package Name Type Contact Reference
---------------- -------- ------- ---------
vq-rtcpxr package Amy Pendleton [RFC-sipping-rtcp-summary-02]


Action 2 (section 5.2):

Upon approval of this document, the IANA will make the
following assignments in the "Application Media Types"
registry located at

http://www.iana.org/assignments/media-types/application/

Name Reference
-------- ---------
vq-rtcpxr [RFC-sipping-rtcp-summary-02]


We understand the above to be the only IANA Actions for
this document.

PROTO Write-up

1.a) Have the chairs personally reviewed this version of the Internet
Draft (ID), and in particular, do they believe this ID is ready
to forward to the IESG for publication?

The three SIPPING WG chairs have reviewed the document and believe it is
ready for publication. Gonzalo Camarillo is its PROTO shepherd.

1.b) Has the document had adequate review from both key WG members
and key non-WG members? Do you have any concerns about the
depth or breadth of the reviews that have been performed?

The draft has been thoroughly reviewed by a number of SIPPING members.

1.c) Do you have concerns that the document needs more review from a
particular (broader) perspective (e.g., security, operational
complexity, someone familiar with AAA, etc.)?

We do not have any particular concern in that respect.

1.d) Do you have any specific concerns/issues with this document that
you believe the ADs and/or IESG should be aware of? For
example, perhaps you are uncomfortable with certain parts of the
document, or have concerns whether there really is a need for
it. In any event, if your issues have been discussed in the WG
and the WG has indicated it that it still wishes to advance the
document, detail those concerns in the write-up.

We feel comfortable with the document. However, we would propose to
change the title of the draft to:

Session Initiation Protocol (SIP) Event Package for Voice Quality Reporting


1.e) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with
others being silent, or does the WG as a whole understand and
agree with it?

Our original intention was to advance this document as an AD-sponsored
individual contribution because, when it was first submitted, there was
not enough interest in the WG. However, as time went by, a large part of
the WG became fairly interested in this document. Consequently, we
adopted it as a WG item.

1.f) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in
separate email to the Responsible Area Director.

Nobody has done anything to stop this document.

1.g) Have the chairs verified that the document adheres to all of the
ID nits? (see http://www.ietf.org/ID-Checklist.html).

Yes, it does.

1.h) Is the document split into normative and informative references?
Are there normative references to IDs, where the IDs are not
also ready for advancement or are otherwise in an unclear state?
(note here that the RFC editor will not publish an RFC with
normative references to IDs, it will delay publication until all
such IDs are also ready for publication as RFCs.)

All the normative references are RFCs.

1.i) For Standards Track and BCP documents, the IESG approval
announcement includes a write-up section with the following
sections:

* Technical Summary

This document defines a SIP event package that enables the collection
and reporting of metrics that measure the quality for RTP-based Voice
over IP (VoIP) sessions.

* Working Group Summary

The WG analyzed several approaches to implement quality reporting in SIP
(e.g., forking RTCP, new header fields, and new body types). Finally,
the WG decided to use a SIP event package for that purpose.

* Protocol Quality

Allison Mankin is the Responsible Area Director. The WG chair shepherd
for the document is Gonzalo Camarillo.