Evaluating Congestion Control for Interactive Real-Time Media
draft-ietf-rmcat-eval-criteria-14

[Ballot comment]
Section 1

  media flow's throughput.  Furthermore, the proposed algorithms are
  expected to operate within the envelope of the circuit breakers
  defined in RFC8083 [RFC8083].

The "proposed algorithms" are the congestion-control schemes, not the
evaluation procedures, right?

Section 3.1

  If the congestion control implements, retransmissions or FEC, the

nit: no comma (first one)

Section 4.4

It's too bad that we don't have more specific guidance to give on loss
generation modeling.

Section 4.5.1

  path.  Due to this, if a packet becomes overly delayed, the packets
  after it on that flow are also delayed.  This is especially true for

Doesn't this imply that the real PDV data poitns are not independent and
that we should expect simple PDF modeling to produce inaccurate or
unphysical results?

[Ballot comment]
I agree with the comments about Section 3.1 by Alexey and Adam.

With respect to Éric’s comment about Section 2, the verb is the last word, “apply”.  Only, the subject in “terminology”, which takes a singular verb, so it should be “applies”.

[Ballot comment]
Thanks for the work performed on this document. I agree with Alexey that Section 3.1 needs to either be fleshed out or removed, as the current specification is insufficiently specified to create interoperable analysis tools. You might look to RFC 6873 for an example of the degree of precision that is called for here.

[Ballot comment]
"Sample video test sequences are available at: [xiph-seq] and
  [HEVC-seq].  The following two video streams are the recommended
  minimum for testing: Foreman and FourPeople."

Are there minimum recommended ones at [xiph-seq]? Both of the ones listed appear to be at [HEVC-seq].

[Ballot comment]
Thank you for this document.

I understand that section 3.1 is not serving the main purpose of this document, but I don't think it is implementable as written. In particular:

3.1.  RTP Log Format

  Having a common log format simplifies running analyses across and
  comparing different measurements.  The log file should be tab or
  comma separated containing the following details:

          Send or receive timestamp (unix)
          RTP payload type
          SSRC
          RTP sequence no
          RTP timestamp
          marker bit
          payload size

Is this sufficient inambiguous to be useful for interoperability? I.e. does each field has a single textual format? How is "marker bit" encoded?
Is each line CRLF or LF terminated?

[Ballot discuss]
Section 5.2.  Per “Sample video test sequences are available at: [xiph-seq] and [HEVC-seq].  The following two video streams are the recommended minimum for testing: Foreman and FourPeople.”, these test sequences seems underspecified.

** Is the “recommended” here intended to be normative?  There is no RFC2119 boiler plate in this document to guide the parsing of the text.

** From the text, there wasn’t much precision in where to find these recommended videos (Foreman and FourPeople).  At the url pointed to by [HEVC-seq], I found the filenames “FourPeople_1280x720_60.yuv” and “foreman15_4000.yuv”, is that them?

** Is it expected for http://www.netlab.tkk.fi/~varun/test_sequences/foreman15_4000.yuv to be 0 bytes?  I tried on 03/03/2020 at ~0950 EST

** Give that that one of the recommended urls doesn’t work even before this draft is published, I have great reservation with keeping a normative “recommended” to such external repositories.  However, providing pointers to repositories of “sample video test sequences” makes sense to me and is helpful.

[Ballot comment]
** Section 3.1.  The purpose of this log isn’t clear.  What is the relationship of it to the metrics described in the previous section?  Where does it fit into the measurement workflow?  Is this constructed on a per packet capture file basis?

** References:
-- Section 3.  Consider adding a citation for tcpdump and wireshark

-- Section 4.4.  Consider adding a citation for the “Bilbert-Elliot” model

** Editorial:
-- Section 3.  Recommend explicitly spelling out PCAP as packet capture.

-- Section 4.5. s/is is/is/

[Ballot comment]
Thank you for the work put into this document. It is indeed important to be able to compare apples with apples.

I have a generic comment, is the case of multi-path / multi-home considered in this document ?

Nevertheless, please find below some non-blocking COMMENTs (and I would appreciate a response from the authors) and NITS.

I hope that this helps to improve the document,

Regards,

-éric

== COMMENTS ==

-- Section 3 --
Please add references to PCAP format and expand the acronym.

-- Section 3.1 --
AFAIK, Unix timestamps have a 1-second accuracy but this document requires an accurancy of 200 msec. Is it expected that Unix timestamp are enough ?


== NITS ==

-- Section 2 --
The sentence "The terminology defined ..." is not a sentence as there is no verb :-)

(Via drafts-lastcall-comment@iana.org): IESG/Authors/WG Chairs:

The IANA Functions Operator has reviewed draft-ietf-rmcat-eval-criteria-11.txt, which is currently in Last Call, and has the following comments:

We understand that this document doesn't require any registry actions.

While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, we do not object.

If this assessment is not accurate, please respond as soon as possible.

Thank you,

Michelle Cotton
IANA Services

The following Last Call announcement was sent out (ends 2020-02-26):

From: The IESG
To: IETF-Announce
CC: rmcat-chairs@ietf.org, varun.singh@iki.fi, draft-ietf-rmcat-eval-criteria@ietf.org, Colin Perkins , rmcat@ietf.org, ietf@kuehlewind.net, Martin Stiemerling , csp@csperkins.org
Reply-To: last-call@ietf.org
Sender:
Subject: Last Call:  (Evaluating Congestion Control for Interactive Real-time Media) to Informational RFC


The IESG has received a request from the RTP Media Congestion Avoidance
Techniques WG (rmcat) to consider the following document: - 'Evaluating
Congestion Control for Interactive Real-time Media'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2020-02-26. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


  The Real-time Transport Protocol (RTP) is used to transmit media in
  telephony and video conferencing applications.  This document
  describes the guidelines to evaluate new congestion control
  algorithms for interactive point-to-point real-time media.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-rmcat-eval-criteria/

IESG discussion can be tracked via
https://datatracker.ietf.org/doc/draft-ietf-rmcat-eval-criteria/ballot/


No IPR declarations have been submitted directly on this I-D.

Notification list changed to Martin Stiemerling <mls.ietf@gmail.com>, Colin Perkins <csp@csperkins.org>, varun.singh@iki.fi from Martin Stiemerling <mls.ietf@gmail.com>, Colin Perkins <csp@csperkins.org>

> As required by RFC 4858, this is the current template for the Document
> Shepherd Write-Up.
>
> Changes are expected over time. This version is dated 24 February 2012.
>
> (1) What type of RFC is being requested (BCP, Proposed Standard,
> Internet Standard, Informational, Experimental, or Historic)?  Why
> is this the proper type of RFC?  Is this type of RFC indicated in the
> title page header?

  An Informational RFC is being requested. This is indicated on the title
  page of the draft. An informational RFC is appropriate, since the draft
  provides guidelines for evaluating implementations of congestion control
  algorithms. While intended to be useful for implementers and designers of
  new algorithms, such evaluation is not needed for interoperability.

> (2) The IESG approval announcement includes a Document Announcement
> Write-Up. Please provide such a Document Announcement Write-Up. Recent
> examples can be found in the "Action" announcements for approved
> documents. The approval announcement contains the following sections:
>
> Technical Summary

  The Real-time Transport Protocol (RTP) is used to transmit media in
  telephony and video conferencing applications. This draft describes
  guidelines for how to evaluate new congestion control algorithms for
  interactive point-to-point real-time media.

> Working Group Summary
>
>  Was there anything in WG process that is worth noting? For
>  example, was there controversy about particular points or
>  were there decisions where the consensus was particularly
>  rough?

  This draft started early in the lifetime of the working group. There was
  some extensive discussion leading up to its adoption as a working group
  draft at IETF 89. Since then, the scope has gradually narrowed and some
  work has been split out into draft-ietf-rmcat-eval-test, but there has
  been little controversy - slow progress has been rather a sign that the
  group has focussed on congestion control algorithm design, rather than
  on revising the evaluation criteria.

> Document Quality
>
>  Are there existing implementations of the protocol? Have a
>  significant number of vendors indicated their plan to
>  implement the specification? Are there any reviewers that
>  merit special mention as having done a thorough review,
>  e.g., one that resulted in important changes or a
>  conclusion that the document had no substantive issues? If
>  there was a MIB Doctor, Media Type or other expert review,
>  what was its course (briefly)? In the case of a Media Type
>  review, on what date was the request posted?

  No media type, MIB doctor, or similar expert review needed.  The working
  draft has seen several evaluations of congestion control algorithms such
  as NADA and SCReAM, and these have been based on the evaluation criteria
  described in this draft, and in the other evaluation drafts. The criteria
  seem mature and have been implemented.


> Personnel
>
>  Who is the Document Shepherd? Who is the Responsible Area
>  Director?

  The shepherd is Colin Perkins. The responsible area directory is Mirja
  Kühlewind.

> (3) Briefly describe the review of this document that was performed by
> the Document Shepherd.  If this version of the document is not ready
> for publication, please explain why the document is being forwarded to
> the IESG.

  Colin Perkins reviewed the draft for WG last call in July 2018, finding
  a reasonable number of nits. The only major concern was the lack of a
  security considerations section; this was discussed in following RMCAT
  meetings and the draft revised.

> (4) Does the document Shepherd have any concerns about the depth or
> breadth of the reviews that have been performed?

  No concerns. The draft has received extension review and implementation
  over many years.

> (5) Do portions of the document need review from a particular or from
> broader perspective, e.g., security, operational complexity, AAA, DNS,
> DHCP, XML, or internationalization? If so, describe the review that
> took place.

  The draft focusses on evaluation of congestion control algorithms. The
  RMCAT WG has appropriate experience to evaluate this. There is nothing
  needing specialist review from other areas.


> (6) Describe any specific concerns or issues that the Document Shepherd
> has with this document that the Responsible Area Director and/or the
> IESG should be aware of? For example, perhaps he or she is uncomfortable
> with certain parts of the document, or has concerns whether there really
> is a need for it. In any event, if the WG has discussed those issues and
> has indicated that it still wishes to advance the document, detail those
> concerns here.

  No concerns.

> (7) Has each author confirmed that any and all appropriate IPR
> disclosures required for full conformance with the provisions of BCP 78
> and BCP 79 have already been filed. If not, explain why.

  All authors have confirmed that no IPR declarations are needed.

> (8) Has an IPR disclosure been filed that references this document?
> If so, summarize any WG discussion and conclusion regarding the IPR
> disclosures.

  No IPR disclosures filed.


> (9) How solid is the WG consensus behind this document? Does it
> represent the strong concurrence of a few individuals, with others
> being silent, or does the WG as a whole understand and agree with it? 

  There is strong consensus.


> (10) Has anyone threatened an appeal or otherwise indicated extreme
> discontent? If so, please summarise the areas of conflict in separate
> email messages to the Responsible Area Director. (It should be in a
> separate email because this questionnaire is publicly available.)

  No such discontent.

> (11) Identify any ID nits the Document Shepherd has found in this
> document. (See https://www.ietf.org/tools/idnits/ and the Internet-Drafts
> Checklist). Boilerplate checks are not enough; this check needs to be
> thorough.

  No nits.

> (12) Describe how the document meets any required formal review
> criteria, such as the MIB Doctor, media type, and URI type reviews.

  None needed.

> (13) Have all references within this document been identified as
> either normative or informative?

  Yes.

> (14) Are there normative references to documents that are not ready for
> advancement or are otherwise in an unclear state? If such normative
> references exist, what is the plan for their completion?

  All normative references are to RFCs or drafts already submitted for
  publication.

> (15) Are there downward normative references references (see RFC 3967)?
> If so, list these downward references to support the Area Director in
> the Last Call procedure.

  No such references.


> (16) Will publication of this document change the status of any
> existing RFCs? Are those RFCs listed on the title page header, listed
> in the abstract, and discussed in the introduction? If the RFCs are not
> listed in the Abstract and Introduction, explain why, and point to the
> part of the document where the relationship of this document to the
> other RFCs is discussed. If this information is not in the document,
> explain why the WG considers it unnecessary.

  No such updates.


> (17) Describe the Document Shepherd's review of the IANA considerations
> section, especially with regard to its consistency with the body of the
> document. Confirm that all protocol extensions that the document makes
> are associated with the appropriate reservations in IANA registries.
> Confirm that any referenced IANA registries have been clearly
> identified. Confirm that newly created IANA registries include a
> detailed specification of the initial contents for the registry, that
> allocations procedures for future registrations are defined, and a
> reasonable name for the new registry has been suggested (see RFC 8126).

  No IANA actions needed.

> (18) List any new IANA registries that require Expert Review for future
> allocations. Provide any public guidance that the IESG would find
> useful in selecting the IANA Experts for these new registries.

  No IANA actions needed.

> (19) Describe reviews and automated checks performed by the Document
> Shepherd to validate sections of the document written in a formal
> language, such as XML code, BNF rules, MIB definitions, etc.

  None needed.

> As required by RFC 4858, this is the current template for the Document
> Shepherd Write-Up.
>
> Changes are expected over time. This version is dated 24 February 2012.
>
> (1) What type of RFC is being requested (BCP, Proposed Standard,
> Internet Standard, Informational, Experimental, or Historic)?  Why
> is this the proper type of RFC?  Is this type of RFC indicated in the
> title page header?

  An Informational RFC is being requested. This is indicated on the title
  page of the draft. An informational RFC is appropriate, since the draft
  provides guidelines for evaluating implementations of congestion control
  algorithms. While intended to be useful for implementers and designers of
  new algorithms, such evaluation is not needed for interoperability.

> (2) The IESG approval announcement includes a Document Announcement
> Write-Up. Please provide such a Document Announcement Write-Up. Recent
> examples can be found in the "Action" announcements for approved
> documents. The approval announcement contains the following sections:
>
> Technical Summary

  The Real-time Transport Protocol (RTP) is used to transmit media in
  telephony and video conferencing applications. This draft describes
  guidelines for how to evaluate new congestion control algorithms for
  interactive point-to-point real-time media.

> Working Group Summary
>
>  Was there anything in WG process that is worth noting? For
>  example, was there controversy about particular points or
>  were there decisions where the consensus was particularly
>  rough?

  This draft started early in the lifetime of the working group. There was
  some extensive discussion leading up to its adoption as a working group
  draft at IETF 89. Since then, the scope has gradually narrowed and some
  work has been split out into draft-ietf-rmcat-eval-test, but there has
  been little controversy - slow progress has been rather a sign that the
  group has focussed on congestion control algorithm design, rather than
  on revising the evaluation criteria.

> Document Quality
>
>  Are there existing implementations of the protocol? Have a
>  significant number of vendors indicated their plan to
>  implement the specification? Are there any reviewers that
>  merit special mention as having done a thorough review,
>  e.g., one that resulted in important changes or a
>  conclusion that the document had no substantive issues? If
>  there was a MIB Doctor, Media Type or other expert review,
>  what was its course (briefly)? In the case of a Media Type
>  review, on what date was the request posted?

  No media type, MIB doctor, or similar expert review needed.  The working
  draft has seen several evaluations of congestion control algorithms such
  as NADA and SCReAM, and these have been based on the evaluation criteria
  described in this draft, and in the other evaluation drafts. The criteria
  seem mature and have been implemented.


> Personnel
>
>  Who is the Document Shepherd? Who is the Responsible Area
>  Director?

  The shepherd is Colin Perkins. The responsible area directory is Mirja
  Kühlewind.

> (3) Briefly describe the review of this document that was performed by
> the Document Shepherd.  If this version of the document is not ready
> for publication, please explain why the document is being forwarded to
> the IESG.

  Colin Perkins reviewed the draft for WG last call in July 2018, finding
  a reasonable number of nits. The only major concern was the lack of a
  security considerations section; this was discussed in following RMCAT
  meetings and the draft revised.

> (4) Does the document Shepherd have any concerns about the depth or
> breadth of the reviews that have been performed?

  No concerns. The draft has received extension review and implementation
  over many years.

> (5) Do portions of the document need review from a particular or from
> broader perspective, e.g., security, operational complexity, AAA, DNS,
> DHCP, XML, or internationalization? If so, describe the review that
> took place.

  The draft focusses on evaluation of congestion control algorithms. The
  RMCAT WG has appropriate experience to evaluate this. There is nothing
  needing specialist review from other areas.


> (6) Describe any specific concerns or issues that the Document Shepherd
> has with this document that the Responsible Area Director and/or the
> IESG should be aware of? For example, perhaps he or she is uncomfortable
> with certain parts of the document, or has concerns whether there really
> is a need for it. In any event, if the WG has discussed those issues and
> has indicated that it still wishes to advance the document, detail those
> concerns here.

  No concerns.

> (7) Has each author confirmed that any and all appropriate IPR
> disclosures required for full conformance with the provisions of BCP 78
> and BCP 79 have already been filed. If not, explain why.

  All authors have confirmed that no IPR declarations are needed.

> (8) Has an IPR disclosure been filed that references this document?
> If so, summarize any WG discussion and conclusion regarding the IPR
> disclosures.

  No IPR disclosures filed.


> (9) How solid is the WG consensus behind this document? Does it
> represent the strong concurrence of a few individuals, with others
> being silent, or does the WG as a whole understand and agree with it? 

  There is strong consensus.


> (10) Has anyone threatened an appeal or otherwise indicated extreme
> discontent? If so, please summarise the areas of conflict in separate
> email messages to the Responsible Area Director. (It should be in a
> separate email because this questionnaire is publicly available.)

  No such discontent.

> (11) Identify any ID nits the Document Shepherd has found in this
> document. (See https://www.ietf.org/tools/idnits/ and the Internet-Drafts
> Checklist). Boilerplate checks are not enough; this check needs to be
> thorough.

  No nits.

> (12) Describe how the document meets any required formal review
> criteria, such as the MIB Doctor, media type, and URI type reviews.

  None needed.

> (13) Have all references within this document been identified as
> either normative or informative?

  Yes.

> (14) Are there normative references to documents that are not ready for
> advancement or are otherwise in an unclear state? If such normative
> references exist, what is the plan for their completion?

  All normative references are to RFCs or drafts already submitted for
  publication.

> (15) Are there downward normative references references (see RFC 3967)?
> If so, list these downward references to support the Area Director in
> the Last Call procedure.

  No such references.


> (16) Will publication of this document change the status of any
> existing RFCs? Are those RFCs listed on the title page header, listed
> in the abstract, and discussed in the introduction? If the RFCs are not
> listed in the Abstract and Introduction, explain why, and point to the
> part of the document where the relationship of this document to the
> other RFCs is discussed. If this information is not in the document,
> explain why the WG considers it unnecessary.

  No such updates.


> (17) Describe the Document Shepherd's review of the IANA considerations
> section, especially with regard to its consistency with the body of the
> document. Confirm that all protocol extensions that the document makes
> are associated with the appropriate reservations in IANA registries.
> Confirm that any referenced IANA registries have been clearly
> identified. Confirm that newly created IANA registries include a
> detailed specification of the initial contents for the registry, that
> allocations procedures for future registrations are defined, and a
> reasonable name for the new registry has been suggested (see RFC 8126).

  No IANA actions needed.

> (18) List any new IANA registries that require Expert Review for future
> allocations. Provide any public guidance that the IESG would find
> useful in selecting the IANA Experts for these new registries.

  No IANA actions needed.

> (19) Describe reviews and automated checks performed by the Document
> Shepherd to validate sections of the document written in a formal
> language, such as XML code, BNF rules, MIB definitions, etc.

  None needed.

WG last call completed some time ago, and the updated draft was submitted prior to IETF 105 (adding security considerations as discussed). Will now send to IESG.