Minutes IETF101: netvc
minutes-101-netvc-00

NETVC WG Session at IETF 101 in London
Monday, March 19, 2018  15:50-17:20

Area Director: Adam Roach
Chairs: Mo Zanaty, Natasha Rooney

Note Taker:  Matthew A. Miller, Nathan Egge
Jabber Scribe: Jonathan Lennox

----------------------------------------------------------------------------------------
Notes from Matt Miller
----------------------------------------------------------------------------------------

Administrivia, Status of Requirements and Testing Documents (Charis, 10 minutes)

Testing and Evaluation Criteria (Thomas Daede, 10 minutes)

* (Mo Zanaty) MSSSIM vs SSIM?
  - updated version of VMAF, draft has not been updated to address that
  - if the results have a large discrepency, perform subjective testing

* (Mo Zanaty) close (publish) the doc now?
  - once the open issue is cleared out
  - (Tim Terriberry) there is some issues on rate control, but it doesn't have
  to hold publication

* does this get a WGLC?
  - never did a WGLC because it wasn't supposed to stop changing

ACTIONS:
    * Thomas Daede to publish an update
    * WGLC once that update is published

Thor/AV1 Update and Comparisons (Steinar Midtskogen, 20 minutes)

* (Stephen?) what's the relationship between Thor and AV1?
  - when AV1 started: VP9 + parts of Daala and Thor

* (Mo Zanaty) Is the plan to keep Thor going here?
  - It depends on what happens with AV1

* (Mo Zanaty) What's the pratical (real-time encoding) gap from H264 (720p @
30fps)?
  - possible with Thor with a few cores
  - none of AV1 implementations can do this yet
  - checked the references Jonnathan was using, and there's a 2% gain

ACTIONS: N/A

The xvc Video Codec (Jonatan Samuelsson, 30 minutes)

* (Mo Zanaty) did you design those based on suspecting IPR encumberence, or by
the technology - want make every piece disableable - whatever is practical to
separate is separated; not just IPR encumberence or technology

* (Tim Terriberry) there are other ways to mitigate risk, such as paying a
bunch of money to laywers, or agreements with companies to have royalty-free
terms.  Thse strategy here is to mitigate the future by trying to turn things
off.  And trying to make money on the high-end.  This is ok if it weren't for
past images (e.g., Firefox).  To be widely deployable is you need to do that
risk mitigation work up front. - We send out requests for patent declarations -
IOW, we have to do all the IPR work * (Tim Terriberry) have you actually filed
an IPR declaration to license royalty-free?  The charter works on the
royalty-free baseline, and how good that performance is.
   - "I'm not sure ..."
   - This group doesn't require royalty-free?
   + if the compression is really good and the terms very reasonable, maybe: if
   the compression is not good ad the term heavy, NO) - We can't "legally" say
   it must be royalty-free - Different participants have different thresholds
   for licensing, but favoring the largest deployment
* (Thomas Daede) Any comments test conditions or feedback on xvc testing tools
and frameworks? - our results were based on the uncompressed framework - we
have tested with others, but we have seen much more divergence - w/r/t HEVC,
it's easier to talk about feature disparity * (Thomas Daede) Is this on Github?
   - work on version 1.1 is on Github, but version 2.0 is not released yet
* (Ben Schwartz) How you think the management of a dynamic codec will work with
a standards organization?
   - Think the IETF is more agile that others, so can update as needed
   - we guarantee disabling tools within 60 days (and licensing is with the
   Corporation)
* (Cullen Jennings) I like the enabling/disabling of tools idea; allow
companies to enable/disable things as needed * IPR codec litigtation is for
many reason; if for money then it starts near expiration to maximize gains *
(Harald) Have you actually had experience with dealing with IPR holders
   - we have not had any reports
   - (Stephen ??) My experience has been somewhat different (more specifics vs
   "you [defendent] figure it out") - (Haralld) I like the idea, but am curious
   how it works in practice (we don't know yet)
* (Stephen) Most work on a additive from baseline, but this is substractive
from baseline.  Maybe this group will take this, which I don't know.  But then
the question is if we can get the granularity to balance performance/trolls *
(Ben Schwartz) Cullen has been speaking about other architectures on media, and
using runtime plugins of codecs.  If xvc is not a good fit for netvc, it might
be good to participate in whatever the "new media" work ends up doing

* (Mo Zanaty) You are in continuous dialog -- do you think enough vendors will
agree to baseline terms - no, but there's not much use then there's not much
interest in defining the licensing terms - we would not make a decision on
something close to the charter objectives.  We cannot adopt something without a
rough idea of what the performant baseline is * Do you think you can figure out
the rf baseline before the next meeting? - we have some idea what it looks
like, but we haven't gotten commitements - it's very difficult for us to say it
is royalty-free * Are the current toolsets closer to the HEVC + extras?
   - (missed)
* You may need to check on the latest to re-evaluate your baseline (correction:
only 2% off, not 80% off)

ACTIONS:
    * WG to review the document and have more discussion

----------------------------------------------------------------------------------------
Notes from Nathan Egge:
 ----------------------------------------------------------------------------------------

--- Milestone's and Status (Mo Zanaty, Natasha Rooney) ---

Mo: It seems unlikely that we will have a submitted codec even if we extend for
4 months, let us discuss later.

Natasha Rooney: We will have a short meeting today.

Mo: No Daala update today (removed from the agenda).

--- Testing (Thomas Daede) ---

Thomas: Draft has not had updates since the last meeting.

Mo: I don't see VMAF and there was also a discussion about dropping MS SSIM.

Thomas: Netflix has created an updated version of VMAF (called harmonic).  This
update resolves issues related to the score saturating.

Mo: Does the testing infrastructure have new VMAF integrated?

Thomas: It does not.

Mo: So the draft and infrastructure needs updating before we can close this out.

Mo: What about the Fast SSIM v. MS SSIM issue?

Thomas: These produce different results so we don't want to drop one of them. 
The draft says if you see drastically different results, you should use some
other kind of testing like subjective.

Mo: So after the VMAF issue, does anyone have any other issues we need to
address before we close (publish) the draft.

Thomas: If we are going to add VMAF leave it open until next meeting.

Tim: If I recall, we were going to add tests specifically for rate control.  If
we are not going to address that, we could potentially just publish.  We could
even publish without having another working group meeting.

Mo: If anyone feels like we cannot publish now, please speak now (or on the
mailing list).  I think the preference of the editor will be to close it out
once we resolve VMAF issue.

Jonathan Lennox: Has this been through WG last call.

Mo: No, it has not.  The intent was to keep it open while it was changing.  We
will certainly have a long WG last call period.

Mo: For the notes: Leave the document open waiting for the VMAF revision.  Then
have a WG last call barring any other comments on the list asking to keep it
open.

Mo: As a tool to help evaluate results, when all the metrics agree it is a no
brainer.  But when you see there is divergence or significant differences in
the metrics, is it useful to try to categorize that on specific content.  So we
can tell when metrics should be ignored, or taken with a grain of salt.  Can we
look at test clips and figure out how to classify things?  Is this worth trying
to do?

Thomas: We could, we would need to compare this to subjective results to get a
good correlation there.  There have only been a few subjective tests on loop
filters, so we could make some conclusions there.  But no one has done
subjective tests on other tools.

Mo: This could be good for developers, as certain classes of content have
different metrics.

Thomas: I have also considered defining a subset of metrics, or a priority to
some metrics, and eliminate some that are not useful.  In particular, the PSNR
Cb and PSNR Cr are redundant with CIEDE 2000.  All of the APSNR ones are mostly
redundant.

--- Thor Update (Steinar Midtskogen) ---

Steinar: I will also be pretty short, not much has happened since last time we
met.  No new tools, only optimizations and bug fixes.

Stephan Wenger: What is the relationship between Thor and AV1?

Steinar: When the Aliance for Open Media was formed, there were three
candidates: VP9, Thor and Daala.  So AV1 is VP9 and parts of Daala and Thor. 
Some of the development on AV1 made it back into Thor.

Stephan Wenger: So what is the plan for Thor?  Is this group just going to
rubber stamp AV1?

Mo: Let's hold off on that discussion until after the XVC presentation.  As
Steinar shows here, AV1 could potentially fit the bill, but practically the
complexity is not there.  What is the gap to make it practical for real time?

Steinar: (interrupted)

Mo: To get a 30fps at reasonable resolutions.

Steinar: For AV1 there are not any current implementations doing that, which I
am aware of.  For Thor, I think it is possible with existing CPUs.

Mo: You are showing frames per minute for AV1, so we are two orders of
magnitude off real-time.

Steinar: It is certainly possible to get real-time encoding for Thor.  With AV1
it is not proven yet.  It will probably take time for implementations to be
able to do that.

Steinar: I'd also like to note that I picked a bad reference on our last update
and wanted to make sure that Jonatan was using as a bad reference.  Since
February there has been a 2% gain so it seems like he has not picked a bad
reference.

--- XVC Video Codec (Jonatan Samuelsson) ---

Jonatan: Developed by Divideon, open source, exploring possibility of having a
RF baseline.

Mo: Question about the selection of the tools controlled by the flags, did you
design those by speculation on the areas that may be effected by patents, or
did you look at coding tools when making the choices.

Jonatan: Good question, we tried to have as many flags as possible with as many
options as possible, so you can turn off as many things as possible.

Mo: So it is not based on IPR or technology, it is based on turning off as many
things as are possible.

Jonatan: Yes we have flags that turn off things which are not at risk, like
bi-prediction.

Tim: (On slide 6) Your argument is that high performance tools are newer and
have high risk and that low performance tools are older and have low risk and
have based XVC on this.  However, there are other ways to mitigate risk, e.g.
paying lawyers to do patent research, or paying to license other technologies. 
I just want to be clear that you have done neither of these things.  Your
strategy is to just turn off the components.

Stephan: It is actually worse than that, these people [Divideon] plan to make
some money on this at the high operating points.

Tim: We'll get to that in a second.  I think that strategy would be fine if we
didn't live in a world with past damages.  For example, if I have been shipping
this codec for months, I'm liable for past damages.

Tim: In order to make this viable you will need to do some of the other risk
mitigation strategies up front.

Jonatan: One of the reasons we brought this codec to the IETF is to get more
information, and maybe ask for patent declarations on the technology.

Tim: I can understand that, but it sounds like what you are asking is for us to
do all that work to determine if it is viable for RF use cases.

Tim: The other question is, you have talked about the possibility of there
being an RF baseline.  Have you actually filed an IP declaration and agreed to
license your technology on a RF basis.

Stephan: I don't know to what extent you are willing to disclose anything.  At
this point you have not published a draft and are under no obligation to do so.
 Tim don't push you luck.

[Note, there is a draft here:
https://datatracker.ietf.org/doc/draft-samuelsson-netvc-xvc/]

Tim (let me clarify): If there are things that you don't want to contribute on
a RF basis, we would have to take those out of the current codec to understand
what the performance is.  This group is interested in only the RF baseline. 
And having metrics for how much the RF baseline would be impacted by making
those changes, would be necessary to understand.

Jonatan: My understanding is that we are not only considering royalty free
codecs in this working group.

Mo: The charter does not exclude royalty bearing codecs, however you would need
to demonstrate gains to make it worth including.  We don't want anything worse
than HEVC or other codecs risk-wise.

Tim: If you go read the charter we talk about how we will follow BCP 79 (...)
in order to prefer selections that are RF.  We cannot make RF a codec
requirement as the IETF is unable to make that determination, it would require
a court to do so.

Stephan: <question about does Tim mean a no-money license>

Tim: As someone who ships open source code that others can redistribute, the
only acceptable license to use would be no-money.  Other people might have
other business models.

Thomas Daede: I was very happy with your result slides, I thought you did a
good job following the testing draft procedure.  I was curious if you had any
comments about the testing draft, and if you had any problems following it with
XVC.

Thomas Daede: I am also interested in how you got the results.  Were there any
coding tools that specifically give you the gains, or is it just the sum of the
parts?

Jonatan:  When we did this testing we used AWCY and it was very good to work
with.  When we have tested with other methodologies you do get some outliers,
there are individual sequences where AV1 does better than XVC (some screen
content sequences).  We also see some difference depending on which metric we
are looking at.  I don't have any specific comment on the test conditions (your
first question).

Jonatan: When it comes to what is actually bringing the performance gains, I
don't have enough insight into AV1 to comment on that.  Compared to HM or HEVC,
it is much easier to compare.  I have a list (slide ?), these are features that
do not exist in HEVC, and this is where we see the performance benefit.

Thomas: I have one other question, did you do your testing with version 2.0 or
1.0 of XVC

Jonatan: The testing is with all these tools enabled.

Thomas: Are all these tools up on github?

Jonatan: We have not released 2.0 yet because we have not determined which
tools will go in it.

Ben Schwartz: I wanted to ask how you imagine mapping a highly dynamic codec
(something that changes frequently over time) into an IETF RFC which does not
change much over time.

Jonatan: That is a very good question.  I guess I might be more agile than
other standards organizations and simply update more frequently.  We don't
envision that though, we think it will be somewhat rare that you would need to
remove tools.

Mo: From the chairs, I don't think there would be technical problem (we have
many WGs that have lived too long).  I don't see a problem with having drafts
coming in over time.

Ben: Your company's website has listed 60 days as the time-frame for disclosure
of tools.  And that you will be in charge of change control.

Jonatan: Yes, we will disclose within 60 days.

Cullen: One of the things I like about this is the ability to turn off tools
individually.  One of the things I am involved in is looking at if we think
patents are obvious or not, so it could be useful on an individual company
basis to be able to turn off features.

Cullen: Another comment, for companies, when it is just a money case, typically
the optimal strategy is to wait until a few years before the patent is
expiring.  So you may have to wait many many years to see if there is a problem.

Harald Alvestrand: I am curious to see how your process has worked.  When I
have run into this in the past, I have had comments to the effect of "I have
patents in your technology" full stop.  When I ask "Well tell me which
technology?", they have said "That is your problem"

Jonatan: We would still do the disclosure (interrupted)

Stephan: I have had experience in this area.  These days patent disclosures
have to be more specific.  I think that it would be very useful to be able to
turn off this one tool with a nasty patent troll.  However, video codec
engineers will tell you it is not easy to do.

Stephan: Standards groups however would still like to have this ability because
it will discourage patent trolls from coming after a standard.

Stephan: Those who worked on H263 you may recall we ran out of letters when
making all the different variations of the format.  Within a vendor cross
compatiblity worked, but across vendors there was fall back to the baseline. 
Assuming that this group would want something like this, the question is can we
get the granularity right.  On the one hand, broad enough to not hurt the
performance, but narrow enough to rule out the troll.  This is what 263 did not
get right.

Ben Schwartz: I thought Cullen would ask this, but he didn't so I will.  Cullen
has been speaking in other WG's about new architectures for media and formats,
and has reinvigorated the idea of runtime toggleable codecs, e.g., being able
to download the codec on the fly and use it for that session.  If XVC is not
something that this WG is interested in, you may want to consider other working
groups.

Mo: From the floor, you mentioned something in the draft about continuous
dialog with patent holders.  Do you have a rough feel for if you will have
enough of those committing to RF baseline that performance would be at least
Thor like?

Jonatan: What I can say is it is kind of a chicken and egg problem.  The
feedback I get is that as long as there is no interest in using XVC, there is
no commitment to make the claims to define the licensing terms.  The response
that we get so far, many organizations want to lay low to see where this is
going first.  See who is going to use it.

Mo: From a chair point of view, we would not make a decision to hum on
something that will not meet the charter requirements.  Unless we know what the
performance is for an RF baseline, we cannot make a decision to use this over
something like Thor.  Is it possible that you will go off and make that
determination before the next meeting?

Jonatan: We have some ideas of what it would look like, but the question is
would it really be Royalty Free.  It is very difficult for our company to be
able to say that this is Royalty Free.

Mo: Fair answer.  Would you say that the list of RF tools, does it look like
JVT or HM + something different.

Jonatan:  A lot of tools resemble HEVC and some of the additional tools look
like JVT.  The basic technology is very much on that track.

Mo: Double check with Steinar that the blip in performance that happened when
someone tried to make AV1 faster is not what you were comparing against?  It
looked like your dates may overlap.

Tim: If you heard Steinar in his presentation, he went back and had a look and
that blip was 2% worse than the tip. but not 20%

Mo: Thanks Tim.  I think there are a lot of good things in XVC that could
contribute to NETVC or the IETF.  In particular having a regular cadence of
releases might line up nicely with browser releases and that would let us
regularly update the codec in WebRTC.  The idea of having a dynamically updated
codec makes sense in the RT case, but the big question is what performance can
you get that is still RF.