Benchmarking Methodology WG (BMWG) Tuesday, November 6, 2012 1520-1650 Afternoon Session II Salon B OPS bmwg BMWG met with 34 people in attendance, in a 1.5 hr session on Tuesday afternoon. The RFC 2544 AS draft is finally approved, this took far more time and resources than it should have, and has impacted chair-time. Most/all comments have been addressed in the SIP Benchmarking drafts, and revised drafts have been made available for this meeting (at the deadline). WGLC is next, after an editorial update. The Content-Aware benchmarking method has been (on-hold, while the authors secure their non-standards job changes. There has been progress in the recent past, so no concern now, but we still need the terminology draft to catch-up. There is concern that, even though most testers use IMIX, there is no ability to extrapolate results to other traffic mixes or even to slightly different circumstances (so why do it?, because the fixed size tests produce interpretable results, but are clearly artificial). There was agreement to add a disclaimer about extensibility, and the psuedo=random will be updated as planned, and then the Next Step is WG Last Call for IMIX Genome draft. The BGP convergence benchmarking will be adopted on the charter, but there is a need to identify a RTG area advisor. (ACTION - Al) There is a consensus to consult the operator community on new work, such as the Data Center Bridge draft and other new proposals. (ACTION - Ron, with help from Al) The WG will pursue recognition of the new and currently chartered work, and revise milestones for approval. In particular, the power benchmarking received comments as planned, and there appears to be "new blood" willing to join in the effort. There is a draft in development on Software Upgrade Benchmarking with multiple vendor authors, working on the "how" of upgrade operation. There was new work described on Traffic Management Benchmarking, and there is a limited relationship to the diffserv benchmarking work of the past, and the Content Aware Benchmarking now. The BMWG also reviewed a Liaison from the MEF on Service Activation Testing, and there was consensus on the short Liaison Reply text previously sent to the list, so it can now be communicated. (ACTION - Al) Also, need to update protection benchmarking milestone as DONE (Oct '12) (ACTION - Al) DETAILED NOTES Benchmarking Methodology WG (BMWG) Tuesday, November 6, 2012 1520-1650 Afternoon Session II Salon B OPS bmwg Remote Participation: http://www.ietf.org/meeting/85/remote-participation.html Bill Cerveny taking notes; Al Morton (chair) monitoring jabber 0. Agenda Bashing Al asked if anyone had IPR to disclose, no one indicated they had any to disclose. 1. WG Status and Milestones 1A. Approved: 'RFC 2544 Applicability Statement: Use on Production Networks Considered Harmful' (draft-ietf-bmwg-2544-as-08.txt) as Informational RFC Drafts not presented at this meeting: Supplementary BMWG Page: http://home.comcast.net/~acmacm/BMWG Additional contributors will be working on "draft-manral-bmwg-power-usage-02.txt, according to Al Morton. Ron Bonica [AD] noted that there appears that an IPv6 neighbor discovery benchmarking document would be of interest, and that he would be willing to mentor someone who worked on this document. 1B. Revised draft: Status: IESG Evaluation, on telechat agenda for Nov 11 1C. Call for review and adoption: Basic BGP Convergence Benchmarking Methodology status draft-papneja-bgp-basic-dp-convergence draft-varlashkin-router-conv-bench Status: Four people indicated support, should we add a RTG advisor? Al Morton said there has been a call for adoption. Al has asked if anyone additional would be interested in adopting draft-papneja-bgp-basic-dp-convergence. Al said BMWG would pull this document onto the charter and will find a RTG advisor. Ram Krishnan said he would review the draft. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 2. SIP performance benchmarking terms and methods Presenter: Vijay Gurbani http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-meth-05 http://tools.ietf.org/html/draft-ietf-bmwg-sip-bench-term-05 WGLC comments, including Expert review now addressed in revised drafts. Vijay Gurbani, Alcatel-Lucent, presented slides ... Al said it was worthwhile to have one more WG last call and try to get it done this year (2012). "If you review this, I will review your drafts" -- Vijay Gurbani Al - Need to have the SIPCORE group review this drafts. Barry Constantine will review the drafts. (item 3 removed during agenda bashing) 4. IMIX Genome Presenter: Al Comments on methods to determine the mix (Aamer) were addressed Next Step: WGLC http://tools.ietf.org/wg/bmwg/draft-ietf-bmwg-imix-genome Al Morton presented slides ... Scott Bradner -- We spent a lot of time talking about real world mixes; to boil down that discussion ... the real world would not be consistent; the real world mix of this type is counterproductive. This is a really great way of doing something that I'm not sure should be done. I would strongly recommend putting in disclaimers. Al -- Will look at adding a disclaimer. Ram -- Are headers and payload being adjusted or just payload? Al -- Just payload Ram -- You may want to vary headers as well. Al -- That comment is relevant to content-aware testing. Tim Copley -- I think it is valuable. Al -- We will have a modification to include this disclaimer, after which there will be a WG last call. 5. Liaison Reply to MEF on ETH Service Activation Original statement in: https://datatracker.ietf.org/liaison/1177/ Proposed response on the list: http://www.ietf.org/mail-archive/web/bmwg/current/msg02673.html Al reviewed proposed response. He took notge that there were no replies on the list, decalsre consensus and said he will send the response. 6. Milestone Review and New Work Proposal Review/Matrix Al said he will send mail to Ron about marking "Terminology for Protection Benchmarking" as done in October 2012. Al said he would like for people to review the "Data Center Bridge" draft and get this moving. Ram suggested that this be circulated to the data center community. Al acknowledged this may be something to circulate in the data center area Scott Bradner -- noted that in principle, the curren RFC 2544 method holds, despite the lack of loss. In a specific time window, there is an intended amount of traffic that should be sent. If a lesser amount has been recieved, then resources have been maxed out, and this is an equivalent to loss. Ron Bonica -- It would be a very easy thing to send an e-mail to NANOG, this is what we're doing and if there is anything we should be doing. Ron -- Operators not well represented at IETF; ask them what are interested in ... Scott -- Operators aren't here but they don't know they should be here. If they find work useful, they may participate. Telling operators there is something that may help them, may help them to get involved. Al -- Ron and Al to work on something for NANOG community, since Ron regularly attends NANOG, at least. draft-player-dcb-benchmarking is at revision 04, updated 4/18/2011 http://tools.ietf.org/html/draft-player-dcb-benchmarking-04 7. Software Upgrade Benchmarking document Presenter: Sarah Banks Following previous WG discussions, a draft is in preparation which clarifies the details of the proposal. Document is coming. If anyone has ideas or feedback, talk to Sarah Banks (e-mail, etc). 8. Traffic Management Benchmarking Presenter: Barry Constantine New work proposal, draft in preparation Barry presented slides ... Scott B -- After a lot of testing, in my experience, back-to-back testing is mostly useless. I would not suggest back-to-back testing as particularly useful. Scott -- Need to have a defined pattern of traffic to be sent through, then measure the traffic pattern that comes out. This is an area that should be extremely useful; we aren't there yet. Comment from Steve Wright -- Understanding the dynamics of reconfiguring the network to change traffic management would be useful. Scott -- That type of test has been talked about in the past. Scott -- Policing may not be the right term to use. Al -- Look at some previous related RFCs before preparing the draft. http://tools.ietf.org/html/draft-ietf-bmwg-dsmmeth-02 and RFC 4689 (draft-ietf-bmwg-dsmterm)