Summary ------------ First, the chairs gave an update on status. Liaisons have been sent to ITU, 3gpp, MPEG on where we are. Some unhappiness was expressed that the MPEG meeting just ended and we should have sent it earlier. The chairs mentioned that they have given four weeks for the other SDOs to come back with comments on the Opus spec - till August 26. Concerns were raised that they would not have time to react since many do not meet before then. The chairs presented updated milestones which were accepted. Status of the requirements document was discussed - an update has been sent which addresses IESG comments. Greg Maxwell presented testing to date. This includes a new test since the last meeting, based on a paper which has not yet been published. Also a document is now available which includes all of the test results (draft-valin-codec-results). Christian Hoene presented his views on additional testing needed. We then had open discussion on what more testing is needed. The chairs said there needs to be a volunteer and a date. The conclusion was that CHristian Hoene and Jan Skoglund volunteered to do more testing. Suggestions had been made to test tandeming, tonal languages, multiple speakers, and validation on objective metrics. We agreed to go offline and come back to the list with specifics on who will test what and when, within the next week. The guidelines document was discussed. The main issue is around characterization and testing. The conclusion of the discussion is that the group agreed to add a milestone which includes testing results for all tests done since inception of this project until one year after RFC of Opus is produced, and call that our testing document. The guidelines document will be updated to reflect this process. THe working group agreed to adopt draft-valin-codec-results as a working group item. The WGLC comments on Opus were discussed. We agreed to keep the framing within the Opus spec itself. We agreed to keep the padding feature, keep the 120ms limit on packet size, and add an optional last frame length. Detailed notes ------------------ CODEC WG (Thursday, July 28, 2011) Chair status update (chairs - 10mins): Note well Agenda bash Liaison review * OPUS held until 4 week LS period, which ends 26 August. * Comment: MPEG-LA and other groups aren't meeting til November and usually these LS are discussed in meetings. * Keith: Did you send LS to 3GPP SA4? Yes. Updated milestones Requirements doc: has completed IESG review 2 discusses. Testing ------- Discussion of test results we have to date (Greg Maxwell - 10min) draft-valin-codec-results-00 Nokia Intespeech 2011: - Paper available upon request (limited distribution will be public in August) * Opus did well at high bit rates * Opus did worse at lower bit rates Comments: * There is a strong debate about mixed b/w testing, which is still a topic of discussion * ACR09(?) testing methodology is not standardized : just exists in papers. * The testing didn't use any mmru's (type of anchors) spanning a range of quality, thus hard to reproduce. * Main conclusions derived from average only score, thus questionable. ANSI would be happy to provide raw data. * Jean-Marc : WG is not chartered for testing across b/w * Greg: results are very consistent between Nokia and Google. * K: Really need to consider how big is the likelihood that OPUS isn't meeting requirements * JR: highlights that the objective is testing to allow WG to make decision whether to progress or modify OPUS document. * EKR: these tests show OPUS did pretty good? Any chance OPUS "sucks ass". High confidence that OPUS is pretty good. Testing Considerations (Christian Hoene - 10min) draft-hoene-codec-quality-01 What is still missing * Most important: 1. two voices at the same time : stereo/binaural 2. Smooth bit rate changes/no artifacts of change mode changes 3. Time stretching and shortening (if this feature is going to be supported) - to support same level of quality as GIPS * Others:.... Codec characterization :should be done later Open Discussion: what more testing are participants willing to do? (Chairs : 30min) * Want a list of tests (with a owner and timeline) if folks think past testing has not been sufficient Comments: * Tim Derryberry : have lots of informal test results. Greg has data on bit error test. * Christian : hasn't been done by third party or validated. * Greg: Question in room about which tests were done to produce results? Bit error, high packet lost, fuzz testing, etc. by developers that haven't been publicized in a document. Have posted to ML. 3rd party tests are great, but who will do them? * Alistair(?): question for characterization. Objective testing? * Greg: have done SNR and listening tests, for example. * Jean-Marc: Formal subjective testing is nice. Lot of code has been in Skype for years. * Roni: if you want to state it's better than some other thing, then it should go through same testing as others. Fine to say it's good enough. * JDR: look at document and let us know if words are inaccurate. Tell us what to say. * Roni: should say that tests were as good as IETF could do * James: as a practical matter, it would be helpful if there are independent groups that can replicate the test results presented here. More likley to get uptake. * Paul: Has any testing been done tandeming with other codecs? * Christian: This use to be an important question. Current opinion is that it's not so important here. * Paul: if there are Gws (to PSTN : e.g., Int'l), then might have problems. End to end is fine. This has been done with Finnish speakers : has it been done with other nationalities. * Kevin Fleming: One of our implementors put this in Asterisk in 4 days part time. * Jean-Marc: Tandeming is not the main use case. * Paul: Any objective testing for ESQ, PAQ? * Greg: yes - look at charts from last meeting * EKR: If this doc is almost done, how long will it take to run more tests? * Cullen: folks have said not that long. However, no one has volunteered to do more tests. * Christian: we will run some more tests. * What tests? * Christian: stereo voice and two voices * Jean-Marc: haven't done stereo with multiple voices. Has been extensively tested with music. * Christian: was this CELT or SILK. * Jean-Marc: CELT. * Christian: Have you tested everything? What has not been tested? What makes sense to test so that I can do it? * Cullen: Here's someone offering to test. What do people want tested? * Paul: Tandem, against performance testing equipment, characterizing one way delay (latency path-to-path, end-to-end). Operators stay within G.114 ITU-T transmission plan. * Jean-Marc: delay didn't require testing. 5ms+frame size+ computational delay * Paul: what is the default packetization period. * JM: AVT issue. 20 seems reasonable. Other can be configured and signaled inband. * Paul: with 20msec+5msec, this won't meet G.711 * Cullen: Doesn't G.114 only apply to narrowband? * Paul: yes, but you've said this could be used in narrowband. * (Google): we're willing to do more testing. We can decide what to do on ML. * JR: need input from WG now as to what testing should be done. Need to know whether folks want to pass this doc as proposed standard. * Anisse: Issue of selectable delay. Why would I ever use 40msec if 20 gives the same quality? Results showing quality that show it depends on frame size. * JM: there has been 20 vs 40 testing, etc. Quality increases with increasing frame size. * Anisse: Testplan was not well received. . Need to test codec for various languages. * Paul: Japanese in particular can cause problems. * JM: it's harder to agree on test plans than to do the tests. Whoever tests decides what they want to test. * Cullen: certainly people will test what they want to test. But, Google and Christian volunteered to do some testing. Can you guys do this in Japanese? * Paul Coverdale: it's explicitly stated that two languages will be tested and that one be something like Chinese. * Christian: have a Chinese student. * Michael: know there is an ITU-T database with rich content. But, license requires that you are using it to develop an ITU-T codec. Need an LS to SG-12. Think they would be willing. * Stephan: can ask informally. SG-16 turf may be able to be done with a quick phone call. * Christian: ? About proposal to setup a document by someone independent to do the tests. * JR: right thing to do is for you to get a 3rd party to do this, write a draft and then the WG can consider. Can't make decision without resources or document. Cullen: don't see anyone else stepping forward for testing. Haven't talked about timing - a rough estimate would be good. * Google: Chinese testing will take more time. Note: more discussion on using ITU-T database for codec testing. - Christian: need an LS to MPEG for AALC to make a comparison to OPUS. - Anisse: Do not think this is possible. ITU-T has tried in the past. - JM: this has been discussed on ML before. How to get sample and not test code? Can we provide them input samples? - Stephan: It is unlikely that MPEG will cooperate. Conclusion: chairs will follow-up with Christian and Google. Will post dates and tests to be done to ML. Document Progress ----------------- Discussion of any WGLC issues on Guidelines (Jean-Marc Valin - 15min) draft-ietf-codec-guidelines-02 Comments: * Erik: what about codec-results document? * Cullen: have heard from several people that we should have a document to capture test results. Proposal: 1) We take doc as a WG item as a starting point. Keep alive even as OPUS progresses. Once we get good data, then publish. 2) Need someone that isn't a codec author to serve as editor. Propose Christian as editor * JR: need to amend wording from slide for section 2 to point to the codec-results document. * Paul C: has seen testing along the way. Normally, when you think you've finalized your codec * JR: proposing taking document now (for SILk, CELT, OPUS)... * Paul C: don't mind background on early versions. Need to characterization on final tests on final codec. * Cullen: not final til it's an RFC. Will have a proposed standard, then draft standard and then full standard. * Chair proposal: Do we want to adopt codec-results as a WG document? * Roni: what is the purpose? Can leave this as an individual document. Do you want this doc to go to publication? * Cullen: yes, at some point. It must reflect WG consensus. * Michael: seems lunacy to specify performance results unless * you specify the version/date. There are always * trade-offs. Codec might change over time : parameters can * change - e.g., CELT with noise code. * Stephen: Is this document making any assessment as to validity of test procedures? * Cullen: No. It's collecting the tests that were done. * Stephen: Does this include agreement that methodology was correct? * JR: No. Just state methodology. Hum vote: Do we want to adopt draft-valin-codec-results to track tests before publication and after? Conclusion: Majority support. One hum against. Question: Can we use this doc as a reference in the guidelines document? Conclusion: No objections. Summary: Guidelines doc complete. Comments: * Roni: you may need to keep WG open if you keep that document alive. * Cullen: will publish at some point. * Roni: there is a dependency/issue. Discussion of any WGLC issues on Opus (Koen Vos and Jean-Marc Valin - 45min) draft-ietf-codec-opus-07 Padding - Comments: * Matthew K: Supports recommendation to keep (padding) for use in SRTP, non-RTP transport. * Justin--- * EKR: in favor for a security reason he can't recollect. * Michael R: You can still add padding at RTP layer. * Christian: What is content of padding? Shouldn't you keep this open? * Matthew K: There are side channel attacks that are enabled if you allow other than 0. * Jonathan L: what is the overhead to support padding? * JM: zero. If we didn't pad, there would be an unused bit * Christian: you can't split padding from framing. Framing requires this feature. Framing - Comments: * Christian: clarified his concern - doesn't see need for this in codec spec and rtp specification. Why same mechanism twice? * JM: if done at RTP level, then have overhead to signal packing. In code, it's signaled at the same place as mode switching, so zero overhead when done in OPUS layer. * Matthew K: frame packing is a good idea. * Greg: this was implemented with very low overhead. Wouldn't be possible in RTP packetization scheme. * Roni: there are other codecs with framing inside the codec itself running in RTP. * Tim Terriberry: if don't specify framing in codec, then different protocols will re-invent their own. * Jonathan L: brought up an issue on ML wrt middleboxes (combining consecutive packets). Is changing mode something you would do rarely? * Stephen: we still do have RTP packetization. JR: seem to have consensus to have framing in codec. Frame packing (1/2) - Comments: Add optional "last frame length" * Cullen: anyone against this? Conclusion: No one had concerns. Then discussion (not captured in detail) about what this really means. Implication on padding? * JM: there would be no signaling bits for more than 120ms, but think that is too much anyways. * Christian: disagrees. Thinks this limit is not too high. There are requirements in delay tolerant networks. * Skype: haven't needed over that (60 msec) Cullen: Anyone have issues with Framing (2) proposal? Conclusion: everyone supports proposal