Guidelines for Use of the RTP Monitoring Framework
draft-ietf-avtcore-monarch-22

[Ballot comment]
Thanks for addressing all my points.

For the record, I want to stress I didn't request, part of my review, the addition of the following sentence (added in version 22) part of my review:

  New RTCP XR report block definitions should not define new performance
  metrics, but should rather refer to metrics defined elsewhere

[Ballot comment]
Thanks for addressing all my points. I'll clear the DISCUSS

For the record, I want to stress I didn't request, part of my review, the addition of the following sentence (added in version 22) part of my review:

  New RTCP XR report block definitions should not define new performance
  metrics, but should rather refer to metrics defined elsewhere

[Ballot discuss]
1. I'm confused by the scope of this document
The title says: Guidelines for Use of the RTP Monitoring Framework
The abstract says:
  This memo proposes an extensible RTP monitoring framework for
  extending RTP Control Protocol (RTCP) with a new RTCP Extended
  Reports (XR) block type to report new metrics regarding media
  transmission or reception quality.
The introduction says:
  The objective of this document is to describe an extensible RTP
  monitoring framework to provide a small number of re-usable Quality
  of Service (QoS) / QoE metrics which facilitate reduced
  implementation costs and help maximize inter-operability.

So is it both about new metric definitions AND guidelines how to encode them?
It seems the focus is on the second part only.
In such a guidelines document, I was expecting a few words about the new metrics: definitions, existing registry, reusing definitions, etc...
Then, I was then expecting this document to refer to RFC 6390.
Something similar to the http://tools.ietf.org/html/draft-ietf-xrblock-rtcp-xr-qoe-02 section 1.3, which mentions:

1.3. Performance Metrics Framework
  The Performance Metrics Framework [RFC6390] provides guidance on the
  definition and specification of performance metrics.  Metrics
  described in this draft either reference external definitions or
  define metrics generally in accordance with the guidelines in
  [RFC6390].

But, even with a stronger statement: "must follow RFC6390"

Also, I believe that some aspects of the document should really be based on RFC6390.
For example,
"Composed metrics" is really the PMOL "Composed Performance Metrics"
"interval metrics" is the PMOL "Measurement Timing".
If the "interval metrics" and "composed metrics" are not defined in one of the RTCP document, I'd suggest using the PMOL term.
Idem for "sampled metrics", which should be based on the PMOL "Measurement Timing"

2. I believe the relationship with RFC 5968 is not clear.
Does this document contain a superset of the guidelines in RFC 5968, specifically regarding monitor?
I believe a clear statement is required. Something such as "all the guidelines from RFC 5968 must apply on top of the guidelines in this document"

3. Actually, my entire point is more a DISCUSS-DISCUSS, for both this draft and draft-ietf-xrblock-rtcp-xr-pdv-05.txt.
Sorry to pick on these two drafts, but we need to have an IESG performance metrics discussion.
Where does the list of performance metric definitions come from at the IETF?
We have multiple sources:
- IPPM for IP performance metrics
- RTCP for RTP performance metrics:
  Definitions in the document themselves or potentially referencing some other SDOs
  Example: http://tools.ietf.org/html/draft-ietf-xrblock-rtcp-xr-pdv-05

  bits 014-011
            0: MAPDV2, Clause 6.2.3.2 of [G.1020],
            1: 2-point PDV, Clause 6.2.4 of [Y.1540].

- PMOL: Performance Metrics at Other Layers, with
  RFC 6076 on Basic Telephony SIP End-to-End Performance Metrics
- IPFIX will one day or the other exports performance metrics.
  I see for example http://tools.ietf.org/html/draft-akhter-opsawg-perfmon-ipfix-03
  It's again a redefinition, and it should not be!

My concerns are that we start to define performance metrics in different parts of the IETF, without consistency.

We have defined RFC 6390 on "Guidelines for Considering New Performance Metric Development", which ask for specific definition
See http://tools.ietf.org/html/rfc6390#section-5.4.4

I believe that the IETF should at least:
- define the performance metrics in a consistent way according to RFC6390.
- document those performance metrics in a single location

So my questions are:
- are we defining the performance metrics the right way within the IETF?
- where is this shared repository of performance metrics (at least for the ones created in the IETF)?
- is the PMOL directorate (RFC 6390) used effectively?

[Ballot discuss]
1. I'm confused by the scope of this document
The title says: Guidelines for Use of the RTP Monitoring Framework
The abstract says:
  This memo proposes an extensible RTP monitoring framework for
  extending RTP Control Protocol (RTCP) with a new RTCP Extended
  Reports (XR) block type to report new metrics regarding media
  transmission or reception quality.
The introduction says:
  The objective of this document is to describe an extensible RTP
  monitoring framework to provide a small number of re-usable Quality
  of Service (QoS) / QoE metrics which facilitate reduced
  implementation costs and help maximize inter-operability.

So is it both about new metric definitions AND guidelines how to encode them?
It seems the focus is on the second part only.
In such a guidelines document, I was expecting a few words about the new metrics: definitions, existing registry, reusing definitions, etc...
Then, I was then expecting this document to refer to RFC 6390.
Something similar to the http://tools.ietf.org/html/draft-ietf-xrblock-rtcp-xr-qoe-02 section 1.3, which mentions:

1.3. Performance Metrics Framework
  The Performance Metrics Framework [RFC6390] provides guidance on the
  definition and specification of performance metrics.  Metrics
  described in this draft either reference external definitions or
  define metrics generally in accordance with the guidelines in
  [RFC6390].

But, even with a stronger statement: "must follow RFC6390"

Also, I believe that some aspects of the document should really be based on RFC6390.
For example,
"Composed metrics" is really the PMOL "Composed Performance Metrics"
"interval metrics" is the PMOL "Measurement Timing".
If the "interval metrics" and "composed metrics" are not defined in one of the RTCP document, I'd suggest using the PMOL term.
Idem for "sampled metrics", which should be based on the PMOL "Measurement Timing"

2. I believe the relationship with RFC 5968 is not clear.
Does this document contain a superset of the guidelines in RFC 5968, specifically regarding monitor?
I believe a clear statement is required. Something such as "all the guidelines from RFC 5968 must apply on top of the guidelines in this document"

3. Actually, my entire point is more a DISCUSS-DISCUSS, for both this draft and draft-ietf-xrblock-rtcp-xr-pdv-05.txt.
Sorry to pick on these two drafts, but we need to have an IESG performance metrics discussion.
Where does the list of performance metric definitions come from at the IETF?
We have multiple sources:
- IPPM for IP performance metrics
- RTCP for RTP performance metrics:
  Definitions in the document themselves or potentially referencing some other SDOs
  Example: http://tools.ietf.org/html/draft-ietf-xrblock-rtcp-xr-pdv-05

  bits 014-011
            0: MAPDV2, Clause 6.2.3.2 of [G.1020],
            1: 2-point PDV, Clause 6.2.4 of [Y.1540].

- PMOL: Performance Metrics at Other Layers, with
  RFC 6076 on Basic Telephony SIP End-to-End Performance Metrics
- IPFIX will one day or the other exports performance metrics.
  I see for example http://tools.ietf.org/html/draft-akhter-opsawg-perfmon-ipfix-03
  It's again a redefinition, and it should not be!

My concerns are that we start to define performance metrics in different parts of the IETF, without consistency.

We have defined RFC 6390 on "Guidelines for Considering New Performance Metric Development", which ask for specific definition
See http://tools.ietf.org/html/rfc6390#section-5.4.5

I believe that the IETF should at least:
- define the performance metrics in a consistent way according to RFC6390.
- document those performance metrics in a single location

So my questions are:
- are we defining the performance metrics the right way within the IETF?
- where is this shared repository of performance metrics (at least for the ones created in the IETF)?
- is the PMOL directorate (RFC 6390) used effectively?

[Ballot comment]
-
OLD:
  There are many ways in which the performance of an RTP session can be
  monitored.  These include RTP-based mechanisms such as the RTP SNMP
  MIB [RFC2959], or the Session Initiation Protocol (SIP) event package
  for RTCP summary reports [RFC6035], or non-RTP mechanisms such as
  generic MIBs, NetFlow, IPFix, and so on.
NEW:
  There are many ways in which the performance of an RTP session can be
  monitored.  These include RTP-based mechanisms such as the RTP
  MIB module [RFC2959], or the Session Initiation Protocol (SIP) event package
  for RTCP summary reports [RFC6035], or non-RTP mechanisms such as
  generic MIBs, NetFlow [RFC3954], IPFIX [RFC5101] [RFC5102], and so on.

- Not sure why you rename the Monitor [RFC3550] to RTP Monitor.
I search for RTP Monitor in RFC 3550 and could not find it, obviously!

- Add an extra comma after "on reception quality" in the following text
  Both the RTCP or RTCP XR can be extended to transport
  these metrics, e.g., the basic RTCP Reception Report (RR) [RFC3550]
  that conveys reception statistics (i.e., transport level statistics)
  for multiple RTP media streams, the RTCP XRs [RFC3611] that
  supplement the existing RTCP packets and provide more detailed
  feedback on reception quality and RTCP NACK [RFC4585] that provides
  feedback on the RTP sequence numbers for a subset of the lost packets
  or all the currently lost packets.


- Somewhere in the section 6, I would refer to RFC 5481 for PDV/IPDV.
Al Morton and I created that RFC as they were much confusion about PDV/IPDV and because some different terms were used. For example, see RFC 5481 section 1.
  There are many ways to formulate packet delay variation metrics for
  the Internet and other packet-based networks.  The IETF itself has
  several specifications for delay variation [RFC3393], sometimes
  called jitter [RFC3550] or even inter-arrival jitter [RFC3550], and
  these have achieved wide adoption.
  ...

That would be ideal (up to you) if the draft would use: delay variation instead of jitter.
See some justifications in Section 1 of RFC 5481.

- Minor comment: take it or leave it.
Slightly confusing sentences containing QoE, as QoE is at the same time a acronym and a reference
  "that can be used
  to correlate the metrics, provide end to end service visibility and
  measure and monitor Quality of Experience (QoE) [RFC6390]."

  "One example of such metrics is the QoE Metric
  specified in QoE metric reporting Block [QOE]."

Proposal
OLD:

  "One example of such metrics is the QoE Metric
  specified in QoE metric reporting Block [QOE]."


NEW:

  "One example of such metrics is the QoE Metric
  specified in QoE metric reporting Block [QOE_BLOCK]."

[Ballot comment]

Saying "encryption of the monitoring report is recommended"
seems a bit trite. I'm not asking that you define precisely how
to secure all possible RTP deployment choices, but perhaps the
right thing to do here is to say that these metrics SHOULD be
secured to the same extent as the RTP flows that they measure.
(Or some such.)

How could you encrypt traffic for a 3rd party monitor without
knowing who that monitor is? That seems somewhat impossible
in general. So, as pointed out by the secdir review [1] the
document should at least recognise the problem and maybe
describe some environments where it can in fact be solved.

  [1] http://www.ietf.org/mail-archive/web/secdir/current/msg03465.html

[Ballot comment]

  The authors report than changes are pending to handle the editorial
  comments raised in the Gen-ART Review by Meral Shirazipour on
  31-Jul-2012.  I hope the updated I-D will be posted prior to IESG
  approval of this document.

IANA has reviewed draft-ietf-avtcore-monarch-17, which is currently in Last Call, and has the following comments:

IANA understands that, upon approval of this document, there are no
IANA actions which must be completed.

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Subject: Last Call:  (Guidelines for Use of the RTP Monitoring Framework) to Informational RFC


The IESG has received a request from the Audio/Video Transport Core
Maintenance WG (avtcore) to consider the following document:
- 'Guidelines for Use of the RTP Monitoring Framework'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2012-08-02. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This memo proposes an extensible RTP monitoring framework for
  extending RTP Control Protocol (RTCP) with a new RTCP Extended
  Reports (XR) block type to report new metrics regarding media
  transmission or reception quality.  In this framework, a new XR block
  should contain a single metric or a small number of metrics relevant
  to a single parameter of interest or concern, rather than containing
  a number of metrics which attempt to provide full coverage of all
  those parameters of concern to a specific application.  Applications
  may then "mix and match" to create a set of blocks which covers their
  set of concerns.  Where possible, a specific block should be designed
  to be re-usable across more than one application, for example, for
  all of voice, streaming audio and video.




The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-avtcore-monarch/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-avtcore-monarch/ballot/


No IPR declarations have been submitted directly on this I-D.

Writeup for draft-ietf-avtcore-monarch-17


(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)? Why
is this the proper type of RFC? Is this type of RFC indicated in the
title page header?

Informational

This document is informational as it is describing the general workings
of monitoring in RTP. It does discuss how some identified issues for how
is prefered to be extended by the WG. BCP status was discussed but had
no significant support in the WG.


(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

This memo describes the extensible RTP monitoring framework for
using RTP Control Protocol (RTCP) with new RTCP Extended
Reports (XR) block type to report new metrics regarding media
transmission or reception quality. In this framework, any new XR block
should contain a single metric or a small number of metrics relevant
to a single parameter of interest or concern, rather than containing
a number of metrics which attempt to provide full coverage of all
those parameters of concern to a specific application. Applications
may then "mix and match" to create a set of blocks which covers their
set of concerns. Where possible, a specific block should be designed
to be re-usable across more than one application, for example, for
all of voice, streaming audio and video.

Working Group Summary

This document was jointly last called with XRBLOCK WG, one of
the primary consumer of these architectural considerations. It was
significantly rewritten as the result of the AD review as it didn't
meet the goals of an architecture specificaiton, and was more focused
on the issues which the WG had come to consensus on. The changes
has been WG last called and consensus exist for moving forward
with these changes.

Document Quality

There are ongoing usage of the principles for extensions described in
this document within the XRBLOCK WG. The document has been reviewed
by a number of people in both AVTCORE and XRBLOCK WG. The AD found
a significant scope problem which has been resolved.

Personnel

Magnus Westerlund is the Document Shepherd and the Responsible Area
Director is Robert Sparks


(3) Briefly describe the review of this document that was performed by
the Document Shepherd. If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

The shepherd reviewed the document very carefully in version -09 prior to
the WG last call. He has since reviewed the changes with each new version.
The WG last call and the changes after the WG last call has been verified
with the WG before requesting publication. The AD provided feedback
resulting in significant changes that has been reviewed and discussed on
the mailing list. Those changes has then be called consensus on. Before
requesting publication a second time.


(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?

No, this document has gotten reasonably good levels of review.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

No special review has taken place beyond the one from XRBLOCK WG as the
main consumers of the document.


(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

No concerns.


(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

Yes, the shepherd has confirmed with all authors.


(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

No IPR disclosures.


(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it?

It is fairly wide consensus with clearly review from more individuals
than just the usual suspects.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

ID-nits tool lists two issues with outdated I-D references. But that is
due to passage of time since submission 20 days ago.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

No formal review required for this document.


(13) Have all references within this document been identified as
either normative or informative?

Only informative references used.


(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

No normative references.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in the
Last Call procedure.

No.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

No change to other documents already published.

(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

As this document doesn't define any protocol, nor registry etc there is
no need for any IANA actions.


(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

None.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

No formal language.

Writeup for draft-ietf-avtcore-monarch-13


(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)?  Why
is this the proper type of RFC?  Is this type of RFC indicated in the
title page header?

Informational


(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

  This memo proposes an architecture for extending RTP Control Protocol
  (RTCP) with a new RTCP Extended Reports (XR) (RFC3611) block type to
  report new metrics regarding media transmission or reception quality,
  following RTCP guideline established in RFC5968.  This memo suggests
  that a new block should contain a single metric or a small number of
  metrics relevant to a single parameter of interest or concern, rather
  than containing a number of metrics which attempt to provide full
  coverage of all those parameters of concern to a specific
  application.  Applications may then "mix and match" to create a set
  of blocks which covers their set of concerns.  Where possible, a
  specific block should be designed to be re-usable across more than
  one application, for example, for all of voice, streaming audio and
  video.

Working Group Summary

  This document has been jointly last called with XRBLOCK WG one of
  the primary consumer of these architectural considerations.

Document Quality

  There are ongoing usage of the architectural principles described in
  this document within the XRBLOCK WG. The document has been reviewed
  by a number of people in both AVTCORE and XRBLOCK WG.
 
Personnel

  Magnus Westerlund is the Document Shepherd and the Responsible Area
  Director is Robert Sparks
 
 
(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

The shepherd reviewed the document very carefully in version -09 prior to
the WG last call. He has since reviewed the changes with each new version.
The WG last call and the changes after the WG last call has been verified
with the WG before requesting publication. Thus the shepherd believes the
document to be ready.


(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?

No, this document has gotten reasonably good levels of review.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

No special review has taken place beyond the one from XRBLOCK WG as the
main consumers of the document.


(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

No concerns.


(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

Yes, the shepherd has confirmed with all authors.


(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

No IPR disclosures.


(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

It is fairly wide consensus with clearly review from more individuals
than just the usual suspects.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

ID-nits tool lists no issues. There exist mentioning in the abstract of
both RFC3611 and RFC5968 which could be considered citations.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

No formal review required for this document.


(13) Have all references within this document been identified as
either normative or informative?

Only informative references used.


(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

No normative references.

(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in the
Last Call procedure.

No.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

No change to other documents already published.

(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

As this document doesn't define any protocol, nor registry etc there is
no need for any IANA actions.


(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

None.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

No formal language.