Video Codec Requirements and Evaluation Methodology

Note: This ballot was opened for revision 09 and is now closed.

Alvaro Retana No Objection

Comment (2019-06-12 for -09)
§4.2 mentions "Reference software provided to the NETVC WG for candidate codecs", but, if I understand correctly, the WG will not evaluate the candidates.

Benjamin Kaduk (was Discuss) No Objection

Comment (2019-06-13 for -09)
No email
send info
Clearing my Discuss as we had a brief discussion on the IESG telechat, and
the responsible AD will follow the discussion with the tsvart reviewer.

Original Discuss portion:

I support Roman's Discuss.

I am sympathetic to the tsv-art reviewer's concerns that this document
is focused on video technology of 5 years ago and may lack relevant in
the current world.  I don't intend to hold a Discuss point for any
specific resolution, but I do think the IESG should discuss whether this
concern affects the value of publishing this document as an RFC.


Original Comment portion (unchanged)

Section 2.1

What do "PAM" and "RA" mean?  Moving Appendix A earlier (before Section
2) or referring to it from the Introduction would be helpful.  Note that
RFC style is to expand on first use...

       . High Dynamic Range (HDR), Wide Color Gamut (WCG), high
          resolution (currently, up to 4K), high frame-rate content are
          important use cases, the codec should be able to encode such
          content efficiently.

nits: missing "and" in serial list, and the last comma is a comma splice.

Section 2.5

[Google didn't help me find reference [9].]

Section 2.6

The (long) list in Section 2.5 includes "cloud gaming"; how much overlap
does that have with this service?

Section 3.1, 3.2

What is the difference between "General Requirements" and "Basic

ection 3.2.1

Is "Exemplary input source formats" supposed to just be an example, or
an indication of the pinnacle of possible values?

Section 4.1

   Initially, for the codec selected as a reference one (e.g., HEVC or
   VP9), a set of 10 QP (quantization parameter) values should be
   specified (in a separate document on Internet video codec testing)
   and corresponding quality values should be calculated. [...]

This seems to suggest ("Initially", "for the codec selected") that the
evaulation requirements are not yet complete.  Are they intended to be a
single set of requirements for the codec's development, or customized to
some per-application requirements?  (Is there a reference needed to
ongoing work to solidify these requirements?)

   QP'k = argmin { abs(Q'i(QP'i) - Qk(QPk)) },
          i in R

I would suggest defining the argmin function.

It's surprising to see no reference to draft-ietf-netvc-testing from
this document.

Section 6

I don't see much need for a "Conclusions" section of this nature, in
this document.

Appendix B

Defining (e.g.) "high dynamic range" and "wide color gamut" with respect
to "normal" or "conventional" mechanisms does not really provide a
stable and archival reference for comparison.

Roman Danyliw (was Discuss) No Objection

Comment (2019-11-21)
No email
send info
Thank you for addressing my DISCUSS and COMMENT items.

Éric Vyncke No Objection

Comment (2019-06-12 for -09)
Thank you all for the work put into this document. Most of the content was way above my technical knowledge though; I would appreciate a reply to the two comments below.


-- Section 2.1 --

Please expand/explain 'PAM', 'RA', 'EDTV', ... before first use in table 1.

-- Section 4 --

This section is only on 'compression performance' and as I read it about compression ratio and nothing is said about packet-loss robustness which is identified in previous section as important. Am I missing something ?


(Adam Roach; former steering group member) Yes

Yes ( for -09)
No email
send info

(Alissa Cooper; former steering group member) No Objection

No Objection (2019-06-12 for -09)
Please respond to the Gen-ART review.

I think sections 4.2 and 6 should be removed.

(Barry Leiba; former steering group member) No Objection

No Objection ( for -09)
No email
send info

(Deborah Brungard; former steering group member) No Objection

No Objection ( for -09)
No email
send info

(Mirja Kühlewind; former steering group member) Abstain

Abstain (2019-06-05 for -09)
I would expect that this document also mentions the interaction with congestion control, e.g. see RMCAT. All I can find is this, which seems very high level:
"3.1.7. Specifications providing integration with system and delivery
   layers should be developed."

Further the security consideration section is rather brief while it actually does mention security requirements. In a requirement document I would have expected that these requirements are clearly spelled out and explained, as well as risks/attacks are discussed.

And why is section 4 in this document and not in draft-ietf-netvc-testing?

Another minor comment:
Sec 2: "for instance, wired channels are considerably more error-
   free than wireless channels and therefore require different QoS
I don't think this is generally true as most mobile networks repair losses on the lower layers.