Considerations for Selecting RTP Control Protocol (RTCP) Extended Report (XR) Metrics for the WebRTC Statistics API
draft-ietf-xrblock-rtcweb-rtcp-xr-metrics-10

Thanks to the authors and contributors on this document. I have two substantive
comments, and a handful of minor nits.

---------------------------------------------------------------------------

§5:

>  application can use these metrics to calculate the Mean Opinion Score
>  (MoS) values or Media Delivery Index (MDI) for their services.

"...calculate Estimated Mean Opinion Score (eMOS) values..."

(MOS scores necessarily require human subjective evaluation of audio under
carefully controlled circumstances -- any computer-generated metrics are
inherently estimations, and should not be called MOS scores.)

---------------------------------------------------------------------------

§5.3.1:

>  Error-resilience mechanisms, like RTP retransmission or FEC, are
>  optional in RTCWEB because the overhead of the repair bits adding to
>  the original streams.

This is a little misleading. draft-ietf-rtcweb-rtp-usage says:

   WebRTC endpoints MUST follow the recommendations for FEC use given in
   [I-D.ietf-rtcweb-fec].

And, in turn, draft-ietf-rtcweb-fec says:

   To support the functionality recommended above, implementations MUST
   be able to receive and make use of the relevant FEC formats for their
   supported audio codecs, and MUST indicate this support, as described
   in Section 4.  Use of these formats when sending, as mentioned above,
   is RECOMMENDED.

Rather than trying to reiterate this somewhat nuanced situation in *this*
document, I recommend removing the entire sentence and fixing the rest of the
paragraph. If you choose to keep it, please make sure it more clearly explains
the level of FEC support required of RTCWEB implementations.


===========================================================================

The remainder of my comments are grammatical or editorial nits that should be
fixed before progressing the document.

---------------------------------------------------------------------------

§3:

>  The RTCP Sender Reports (SRs) and Receiver Reports (RRs) [RFC3550]
>  exposes the basic metrics...

"expose"

>  However, these metrics provides...

"provide"


>  For example, it may be useful to distinguish between
>  packets lost and packets discarded due to late arrival, even though
>  they have the same impact on the multimedia quality, it helps in
>  identifying and diagnosing issues.

This is a run-on sentence. Suggest: "...due to late arrival. Even though..."

>  The WebRTC application extracts the statistic from the browser by
>  querying the getStats() API [W3C.WD-webrtc-20161124], but the browser
>  currently only reports the local variables i.e., the statistics
>  related to the outgoing RTP media streams and the incoming RTP media
>  streams.

I don't think the "currently" in this sentence is actually true any longer; and
I expect it to get increasingly less true as time goes on. Consider rephrasing.

>  [I-D.ietf-rtcweb-rtp-usage] does not mandate the use of
>  any RTCP XRs and since their usage is optional.

This is not a sentence. Maybe remove "since"?

---------------------------------------------------------------------------

§4:

>  For example, an application may choose
>  to poll the stack for statistics every 1 second, in this case the
>  underlying stack local will return the current snapshot of the local
>  statistics (for incoming and outgoing media streams).

This is a comma splice. Consider: "...every second. In this case..."

>  However it may
>  return the same remote statistics as before for the remote
>  statistics

Add a comma after "However".

---------------------------------------------------------------------------

§5:

>  Since following metrics are all defined in RTCP XR which is not
>  mandated in WebRTC, all of them are local.

"Since the following..."
       ^^^

"...RTCP XR, which is..."
           ^

>  However, if RTCP XR is
>  supported by negotiation between two browsers, following metrics can
>  also be generated remotely and be sent to local by RTCP XR packets.

"...two browsers, the following..."
                  ^^^

>  Following metrics are classified into 3 categories: network impact

"The following metrics..."
 ^^^


>  viewpoint of application, e.g., bit rate, frames rate or jitter

"...frame rate..."

>  WebRTC
>  application can use these metrics to calculate the Mean Opinion Score

"applications"


---------------------------------------------------------------------------

§5.1.1:

>  lost and duplicated packets.  Lost packets counts are useful for

"Lost packet counts..."

---------------------------------------------------------------------------

§5.1.3:

>  the two run length vectors to identify congestion-related losses,
>  i.e., a router queue became overloaded causing delays and then

Replace "i.e." ("that is") with "e.g." ("for example").

---------------------------------------------------------------------------

§5.2.1:

>  The metric reports the cumulative size of the packets discarded in
>  the interval, it is complementary to number of discarded packets.

"...the interval. It is complementary..."

---------------------------------------------------------------------------

§5.2.2:

>  For
>  audio streams, a single RTP packet may contain one or multiple audio
>  frames, each of which has a fixed length.

I don't think this is generally true. Even in the case of WebRTC, I believe the
default mode of operation for Opus is VBR, which results in frame sizes that
vary from frame to frame.

>  The metrics in this category includes: number of discarded key

"...category include:..."

---------------------------------------------------------------------------


§5.3.1:

>  The un-repaired packets count and repaired loss count defined in

"...unrepaired packet count..."

>  packets to lost packets.  Including this kind of metrics helps the

Choose either "...these kinds of metrics..." or "...this kind of metric..."

---------------------------------------------------------------------------

§5.3.2:

>  related losses, i.e., a router queue became overloaded causing delays

Replace "i.e." with "e.g."

---------------------------------------------------------------------------

§6:

>  identifiers relevant to RTP media in WebRTC.  These group of
>  identifiers are defined on a ReportGroup corresponding to an
>  Synchronization source (SSRC).  In practice the application need to

"This group of identifiers..."
 ^^^^

"...corresponding to a synchronization source..."
                     ^

"...the application needs to..."
                    ^^^^^

>  For XR metrics, it depends on
>  two factors: 1) if it measured at the endpoint, 2) if it reported by
>  the endpoint in an XR report.

"...if it is measured..."
          ^^

"...if it is reported..."
          ^^

---------------------------------------------------------------------------

§11.2:

>  [W3C.WD-webrtc-20161124]
>             Sporny, M. and D. Longley, "WebRTC 1.0: Real-time
>             Communication Between Browsers", World Wide Web Consortium
>             WD WD-webrtc-20161124, November 2016,
>             <https://www.w3.org/TR/2016/WD-webrtc-20161124>.

The authorship of this document is incorrect (neither Sporny nor Longley are
authors on WebRTC).

The URL points to an outdated version of the specification. Please update to
https://www.w3.org/TR/2017/CR-webrtc-20171102/


>  [W3C.WD-webrtc-stats-20161214]
>             Alvestrand, H. and V. Singh, "Identifiers for
>             WebRTC&#039;s Statistics API", World Wide Web Consortium
>             WD WD-webrtc-stats-20161214, December 2016,
>             <https://www.w3.org/TR/2016/WD-webrtc-stats-20161214>.

Please replace "&#039;" with an apostrophe.

Please update the URL to https://www.w3.org/TR/2018/WD-webrtc-stats-20180519/

Thanks for the well-written document. A few comments mostly on references:

- Maybe also provide an (informative) reference to draft-ietf-rtcweb-overview-19 on the first occurrence of WebRTC...?

- Is ReportGroup defined in another RFC? If so, please provide reference in ther terminology section.

- Maybe provide (informative) references for Mean Opinion Score (MoS) and Media Delivery Index (MDI), as well as "Media Loss Rate (MLR) of MDI" maybe.

- It seems to me that at least [W3C.WD-webrtc-stats-20161214] should be a normative reference.

- I wonder if it would make sense to refer to some of the IPPM RFCs that define these or similar metrics?

Thanks for producing this document. 

I did have one comment (and I apologize in advance that this is a particular hot button for me).

I understand that you can do things with these metrics that you can't do with previous metrics, but I wish there was a clearer description of "if you want to understand X, you should implement this metric" earlier in the document. The impression the reader is left with, is "if you're doing RTCWeb, you should implement this metric", but I'm guessing that at least some of these metrics would be useful for people who weren't doing RTCWeb, but need to measure something that people who are doing RTCWeb also need to measure, and if that's true, a short section that helped them figure that out would be helpful.