RMCAT SESSION I: Evaluation and Interfaces Monday, Nov 2, Afternoon session II 15:20-16:50 Note takers: Coline Perkins Maxim Proshin Alan Ford ## Chairs presentation # draft-ietf-rmcat-wireless-tests updates: draft-fu-rmcat-wifi-test-case updates to be merged into draft-ietf-rmcat-wireless-tests. Zahed plans to submit the merged draft this week # draft-zhu-rmcat-video-traffic-source-02 Updates: Varun: what is holding up the traffic source draft? Mirja: Did not see an update, and authors didn't request WG adoption Karen: presentation on Friday, and expect an adoption call then # draft-singh-rmcat-adaptive-fec-02 Updates: Varun: unclear how to proceed with draft-singh-rmcat-adaptive-fec Mirja: Varun should chase people up # draft-singh-rmcat-cc-app-interactions Updates: Varun: expects to have draft-singh-rmcat-cc-app-interactions updated this week. Will have Stefan Holmer as co-author # draft-ietf-rmcat-eval-criteria-04: Varun: we put some text this week earlier together with Stefan and hopefully it will be ready this week. The document needs some updates since Dallas. Short TCP model added (as agreed in Prague). I'm working closely with Stefan, will add Stefan as co-author, to add Google’s jitter model. More metrics to be added as people request them (no feedback received yet). draft-zhu-rmcat-video-traffic-source-02 and draft-ietf-rmcat-eval-criteria-04: Mirja: you are still going to keep those two documents separated, right? Varun: I think that reasonable way forward, but on the other hand it is applicable here so maybe there is no need to separate Mirja: I think the video sequences can be useful in general Varun: yes, it is ok to try to separate Mirja: What about the selection of video sequences for the trace-based video modelling. Such information should go into the evaluation draft ? Zahed: that needs a separate discussion I guess. We need to select the sequences that we should really use. Varun: moving it from test to criteria is a sensible and should not be controversial Zahed: just moving that [what is that?] (text is not a problem) Varun : there is another wg (netvc) that works on the similar problem, maybe we should figure it out and join the work or provide comments Karen: do you ask for some support or you want to do it personally. Varun: I don't do it personally, but I would appreciate if someone follows it or maybe I'll do it Mirja: any additional contribution from Stefan ? Stefan: yep, we will also add # Milestones: Mirja: one new milestone (adopt SBD and Coupled CC) has been met since last IETF. The milestone on coupled CC and SBD has been corrected to refer to experimental publication. A bunch of milestones are coming up very soon Karen: just to emphasize that if we want to meet deadlines, we need to review and put some pressure to improve them before the next IETF. There is the possibility to have WGLC go out on the list before the next IETF. # Reviews of -rmcat-cc-codec-interactions and -rmcat-eval-test: draft-ietf-rmcat-cc-codec-interactions-01 Karen: the next goal is to provide the document to rtcweb for comments We are looking for names to review it and also it would be good to receive commitments right now. 22 or 29 January? Agreed with deadline for review comments January 29 2016: - Stefan Holmer - Bo Burman - Michael Ramalho - Randell Jesup draft-ietf-rmcat-eval-test-02 Agreed with deadline for review comments January 29 2016: - Safiqul Islam (with a special eye on test cases for Coupled Congestion control) - Stefan Holmer - Randell Jesup # AOB: RMCAT Fridays meeting has conflict with perc, netvc wgs. Should be avoided next time. ## Updates on RMCAT test cases draft-ietf-rmcat-eval-test-02 and draft-ietf-rmcat-wireless-tests-00, Zahed Sarker Varun: does the draft need to give more detail on how the UDP packets are generated? Packet spacing, size, etc., to meet a particular bit rate, since there are multiple ways of meeting a bitrate, that have different overheads and trade-offs. Zahed: not discussed yet. ACTION Varun: eval criteria draft needs to clarify background UDP traffic behaviour (packet spacing, rate, etc.) Wireless tests: Has anyone implemented the Wi-Fi test cases? [a couple of "yes" including Stefan] Mirja: for the basic test cases, we picked a subset that are mandatory for candidates to test against, left some optional. Do we also need to do this for WiFi? Zahed: yes, think so Mirja: there were issues with the LTE simulations, since not everyone has the Ericsson simulator. Zahed: have clarified how to use NS3 to do this, and what parameters to test Varun: have they been compared? Zahed: not explicitly, but seems clear that they will be different, since the two simulators vary in propagation models, behaviour at edge of cells, etc. Varun: should we require everyone test LTE using NS3, to allow comparison? Does this make sense, given we know that the NS3 model is limited. Unclear. Delay discussion until other people try to implement the wireless tests, to see if the differences become important. Michael: I remember that for the basic test cases we decided to split the wireless out because the doc was long from the beginning. Should we do the same for wireless test cases? Zahed: yes, it could be two parts: wifi test cases and LTE test case, but we decided not to do this. Michael: but for LTE test cases you don't have simulation results, right? Zahed: Has tried. Very hard to implement in a physical testbed; possible in a simulator. Have updated the draft with information about LENA (LTE) simulator Varun: I see differences between simulation results since last time and in behavio (some speculation about additional simulations). Zahed: Ericsson LTE simulator it is much more complex. Michael: We should use test cases and say if it is safe or not Zahed: if you want me to use my test cases to improve LENA, then my answer is that LENA is good enough Ken Carlberg: no discussion on potential oscillation when, e.g., dropping from 4G to 3G; should this sort of handover be in scope? Zahed: no, we don't consider this Ken: doesn't discuss overload between GBR and non-GBR traffic, and how towers deal with non-GBR traffic in this case? Zahed: can add this, but thinks it's an extreme corner case, and not something we need to model ## RTP Extensions for Transport-wide Congestion Control draft-holmer-rmcat-transport-wide-cc-extensions-01, Stefan Holmer RTP Header extension: Perkins: if we are to do this, suggest making the sequence number longer, since we already have codecs for which the 16 bit seq num is too small Stefan: good point. Varun: how does this interact with existing send-timestamp? Stefan: this arrival time can replace send-timestamp. Varun: Also interesting to look at impact on MPRTP. RTCP Message: David Hayes: several things want to use PT 205? can use different FMT values Varun: what SSRC is used here? It’s transport wide, but needs a sender SSRC Colin: better to feedback on a per stream basis; can aggregate across the flows to get transport-wide statistics Varun: let's work out what information is needed, and work it out in avtext. Stefan: goal = feed back to sender arrival time of each packet received Colin: have XR blocks that do this already! Michael Ramalho: sympathetic to the desire to aggregate; unsure it works. What is transport-wide in the world of middleboxes? e.g simulcasting to a switch that sends to different targets? Stefan: we have a transport between two peers; one peer may be a node in the cloud that terminates the session. It's between these peers, not necessarily end to end. Randell: how does transport-wide numbering works when different streams have different DSCP values, and hence different congestion contexts? Also could we do this is just RTP header extensions? Stefan: thinks you want to include all the streams in the estimate - need to run several instances of the transport-wide cc in the flow. Separate "transports" for separate DSCP values Mirja: how does this relate to coupled congestion control and shared bottleneck detection? Zahed: doesn't get the dscp issue - isn't this the same as coupled cc? Harald: sender should get the feedback, determine what packets it relates to, and adapt the stream in response Colin: sure, but that's argument for per-ssrc feedback Colin: we should do this on a per-stream basis, rather than having people need to combine/uncombine. Should work on a per-SSRC basis. Plus DSCP demux is not required since DSCP will be static per stream. The entire adaptation seems to work per ssrc, but the feedback is all mixed up for the transport Karen: notes that not all candidate algorithms operate at the sender, unclear this works for all Stefen: but nada can be changed so these calculations are done on the sender. Jörg Ott: is the run-length encoding different to all the other RLEs we have in RTP? It should be, to reduce complexity. Stefan: Not sure. We should make it the same. Varun: what state is required at sender? Does the sender need to track the send times for the transport sequence numbers across all the flows Stefan: need to know what time every packet was sent. Marat Zhanikeev: rather than deltas, consider network calculus, and maybe different representations? Chairs: Does the wg feel that we should give the usage of a generic feedback mechanism for sender side only RMCAT CC a try? Zahed: scream looking into whether they can use generic feedback; needs to give feedback on whether this works; requirements first Colin: wants to see work on requirements first. What do all the candidates need? What is available in existing mechanisms? Only after that, think about solutions Varun: again, wants to see requirements ## Shared Bottleneck Detection draft-ietf-rmcat-sbd-03, David Hayes Colin: can this work with standard rtcp feedback (sr/rr)? No, needs extensions Mirja: need to decide if we ship the algorithm as an experimental draft, or expand the draft to include the packet formats. Decision discussed was not to specify the feedback packet format in this draft, but possibly do this later for standards track. **** RMCAT SESSION II: Congestion Control Candidates Friday, Nov 6, Friday Morning session I 9:00-11:30 Note taker: Brian Trammell ## Intro & Wrap-Up Monday meeting Chairs: At the RMCAT session on Monday and supported by emails since we have the impression that the wg is ready to * Give usage of a common feedback message for sender side only RMCAT CC's _a try_. We should have a discussion to confirm this and we should agree on a way forward. A proposed way forward can be : *1* Requirements to be considered in each CC algs draft *2* Analyse required feedback rates and timing as well as content and point to existing remedies and/or what new needed Varun: Rate and timing does not automatically fall out of feedback (?). Up to the receivers to decide what to do. Mechanisms already in place for minimum and maximum rate. Karen: Mechanisms exist so they can negotiate feedback rate ? Varun: Before we say generic feedback, question is what is content. We already have XRBLOCKs. Karen: First, do we want something common and generic. Second, do we already have that, or something new? Varun: Intuition, use things that are there. These might be verbose. When we have more experience, e.g. granularity, then we can compress a bit. Can speak from experience developing after the fact, many XRBLOCKs already exist that have sufficient information. Karen: Starting on analysis for generic feedback. Yes if you want to do that we have a WG draft to make the analysis in. Do we want to do this now or have more experience? Zahed: We can negotiate some parameters, we have a fraction of bandwidth for feedback. Intervals, you can guess. Doable. Start with XRBLOCK, ask what's missing. Which is the WG document? Karen: ietf-rmcat-rtp-cc-feedback. Only question, do we want to start this work now? Mirja: If you have your own feedback, receiver has to support it... Isn't it valuable to make ?  Zahed: I don't know if the receiver supports XRBLOCK. That can't be the fallback. For all algs in RMCAT they fall back to RTCP report. Randall (virtual): Try to get info back from proposed implementations on what feedback their algorithms need, useful for shared feedback story. Might require new messages, might not. Even if we can reuse XRBLOCK, unlikely that a client not designed for XRBLOCK will use. Don't want to lock in too early on data expected from client... maybe too late... Need to think about feedback stream interaction with forward stream in the same direction. Xiaoqing: Many benefits in a common receiver. Want to start with a common receiver... Mirja: Question to speaker: Willing to revisited the decision and change the algo? Xiaoqing: Assuming per packet feedback, Karen: making a description of the requirements of this, making this in your document or somewhere else ? Xiaoqing: One list of req for common receiver to early. Premature to make these requirements. Provide specific spec for the feedback mechanism for the current decision and point to different directions Mirja: Where to put the requirements? Xiaoqing: Three candidates, each different flavor of feedback mechanism. Easier, to put up detailed mechanisms of feedback, find out after what/how to merge Karen: Where to put it? Speaker: (didn't get answer) Mirja: Results possible based on theory or real testing feedback needed Xiaoqing: Done some studies on impact of feedback, can share theoretical and practical results. Nothing concrete to recommend.  Brian T.: Generic observation about use of xr block. Go the common feedback route, make sure that feedback is compatible with xr block. xr block not sufficient to use here, having the metrics compatible with feedback. Measurement agents can understand xr block and feedback. Zahed: We talk about information we consider for feedback in the draft already.  Mirja: Do you also have information about how often you need the feedback? Zahed: In the document. Scream started with per-packet feedback. Noticed problem with frequency of per-packet feedback with rtcp. Generic treatment is a short-term solution, what's the standardization benefit if we have one alg? Karen: There was not assumption that there will be one algorithm. Zahed: Should we really clearly say there will be more than one? Then this work is really important. Karen: We don't know. Impression I get from this discussion, need to have each CC alg explain the feedback and rate they need. Scream already does this. Mirja: Sufficient to put this in the appendix. Zahed: Format of generic feedback message should wait. Michael Ramallo: Also started per-packet, did something different. Question is compressibility. If per-packet can be effectively compressed, then everyone gets what they want.  Stefan Holmer: Not all about interaction between these algorithms. Even when we have standardized the algs, they won't be final. Very active field, we need to not have a constrained feedback format. Karen: That's a requirement for how we do it.  Varun: We should do something in this area. Summary? Karen: We now ask for all algorithm proposals to summarize feedback requirements in their drafts. We won't necessarily analyze this now. Mirja: We'll submit all algorithms as experimental first without generic feedback. Generic feedback is a requirement for proposed standard revisions. ## Update on Video Traffic Source Model for RMCAT Evaluation draft-zhu-rmcat-video-traffic-source-02, Xiaoqing Zhu Mirja: Integrated into ns-3? Xiaoqing: Yes, codec is just C++ though Mirja: Others integrating into own simulations? Just curious Zahed: Yes. All results are based on trace-based models. Not doing statistical models yet, plan to analyze. No idea when we'll implement. Mirja: Test cases don't argue for one or the other. Stefan: We have some something similar, simpler. Seems reasonable that we could integrate this code into simulations. Mirja: Hum to support adoption HUMMMMMMM Mirja: Hum to not support adoption SOUND OF PEOPLE TYPING BUT NO HUM Mirja: Okay, clear support, will confirm adoption on the list. ## Coupled congestion control with GCC draft-ietf-rmcat-coupled-cc-00, Safiqul Islam Zahed: eval test cases don't talk about CCC. Will there be a variant of eval test cases for multiple flows for CCC? Safiqul: We can make this a special case of the test case, add a paragraph to say what changes for CCC (i.e., two rmcat flows instead of one vs tcp) Zahed: Would be interesting to know for each test case what the requirements are to exercise CCC. Mirja: On Monday we discussed wireless test cases. Do you plan to look at those? Safiqul: Haven't decided yet. Karen: Will you also test coupling between rmcat and other flows? Safiqul: Started work recently on coupling data and media channels. Do we think it would be a good idea to add this to the draft? Karen: This is a requirement for the rmcat group. Safiqul: Once we start doing tests, we can come up with test cases to test this kind of coupling and share with test case editors Zahed: That'd be great ## Update on GCC draft-ietf-rmcat-gcc-01, Stefan Holmer Randall: Thought about removing the packet size term: big frames are all a bunch of big packets followed by a smaller one. Might revisit this decision with low bandwidth networks and streams where each frame fits into a packet. Zahed: Agree with Randall's point Mirja: Work in progress or are you looking for reviews...? Stefan: Makes sense to wait until we fix the loss-based controller. Mirja: After you make changes, rerun the test cases. ## Updated evaluation results on NADA draft-ietf-rmcat-nada-01, Xiaoqing Zhu Zahed (q. on 5.1 variable cap): what video traffic model? Xiaoqing: Perfect Zahed: On ramp-up, need to think that the codec might not be perfect. Randall (on AQM): no aqm entry in eval-criteria, we will need to work well with it. thanks for doing these tests. (on 5.1 variable cap): reqt's mention potential faster ramp-up, since quality of call at the beginning is important to people. Want to make QBOUND variable. (...?) Xiaoqing: Very conservative on ramping up if it sees queueing delay, threshold is 10ms. QBOUND not bound on queue, it's just an aggressiveness parameter. Stefan: Maybe explain this is the case in the draft. Stefan (on impact of parameter values): ETA of 2 means you react mainly to changes in queueing delay? Xiaoqing: Yes. Stefan: This makes it a bit more similar to gcc. Stefan: Have you evaluated how using queueing delay? Xiaoqing: An earlier version of NADA used OWD to drive all interactions -- but then feedback depends on path propagation delay, which is hard for rtt fairness.  Mirja: Thanks to Stefan for review. Zahed, have you reviewed? Zahed: Yes. Just need to send the message. Mirja: How close are you to last call? Xiaoqing: Wait for reviews. Many open issues are just things to be aware of. Mirja: Option after next revision to go for WGLC. We don't have to wait for the next meeting. Karen: Plans to test on the Internet? Xiaoqing: Safe to try out. Haven't considered doing so though. ## Updates on SCReAM - An implementation experience draft-ietf-rmcat-scream-cc-02, Zahed Sarker Stefan (on test case 5.1): this is a local queue? Zahed: yes, red line. Stefan: There's a bottleneck in the test case then. Xiaoqing: Two clarifying questions. RTP queue above 500ms, net queue about 500ms, these are additive? Zahed: Not necessarily Xiaoqing: Would like to see end-to-end delay. Zahed: Blue line is end-to-end delay. Xiaoqing: Then how does the red line go over the blue one? What happens to a packet that experiences 500ms of RTP queueing? What happens at overshoot? If it arrives at the receiver Zahed + Brian + Xiaoqing + Michael: A packet that enters the RTP queue at peak delay also experiences the RTP peak queue, then hits the network queue (time-shifted). Instantaneous end to end delay would be good to have here too. Randall: on delay in codec: is this a delay in the codec's response to its controls, or in the various feedback decision pathways? Zahed: This is the first one -- calculate bit rate at input and output. Maybe we configured the codec wrong. Randall: Many ways to configure. Documenting configuration useful. This chart does not look right to me. Stefan: We have done this simulation too, and we did not experience the same problems, so possibly configuration. Many of the encoders running on mobiles don't have many knobs. May not give you what you need. Zahed: That's the point we're trying to make. Mirja: Update is ready for last call. Review volunteers: Xiaoqing, Stefan, Mo, and Varun Varun: Are you seeing deployments of scream in the wild? Is it merged to master in openwebrtc? Zahed: There's a flag. Mirja: Are you aware of users? Zahed: Hard to tell. Varun, you have data? Varun: I assume it's GCC. Mirja: You can look at feedback format Varun: We're not in path. Is scream on master? I'd like to try it out. Zahed: On master as of last week. Mirja: How close to WGLC? Zahed: We're ready. Should there be a deadline? Karen: We want reviews this year. ACTION Reviewers: Xiaoqing, Stefan, Mo, and Varun Document for review is present draft-ietf-rmcat-scream-cc-02. Zahed: We'll submit a separate draft on feedback measures. Mirja: You should re-add a bit of text about what information should be in the feedback. There's no format fot the extension, just a list of fields. Varun: Didn't we finish this discussion Mirja: Conclusion was that draft should say what requirements are for feedback, description of content and timing. Looked at the draft, there needs to be more text on this in the scream draft. ## Considerations for draft-singh-rmcat-cc-app-interactions No draft yet, Varun & Stefan Karen: Do we have any idea of what statistics the controller provides? Varun: Standardized from W3C. Target bitrate, estimated bitrate, etc. Karen: It's a point for the algorithms, they need to instrument. Varun: From a statistics PoV it's mainly performance metrics. In terms of the configuration parameters. Karen: might have some overlap with the codec-interactions draft. Needs to be addresssed as soon as there is a first version of this draft. Mirja: slides are uploaded now, please provide feedback on the list! Harald: There is some work on interfaces in webrtc and other WGs, so it'd good to coordinate