Minutes of Benchmarking Methodology WG (BMWG) TUESDAY, March 20, 2018 0930-1200 Morning Session I Palace C OPS bmwg Remote Participation: http://www.ietf.org/meeting/101/index/index.html http://www.ietf.org/meeting/101/remote-participation.html -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 0. Agenda Bashing adopted as-is 1. WG Status (Chairs) (Brief status below) Two Drafts remain on our current Charter Interim Meeting Report (March 1) - discussed NG-firewall benchmarking methodology in preparation of this meeting There are many proposals for new work: Network service layer abstract model, back2back frame testing, network virtualization platforms, ... 2. Charter and Milestones (Chairs) No new RFCs this time, but there were 6 last time! - milestones see presentation slide 6 Supplementary BMWG page http://bmwg.encrypted.net/ 3. Benchmarking Methodology for SDN Controller Performance Presenter: Sarah Banks (or Bhuvaneswaran Vengainathan) On IESG Aganda for April 19, 2018 https://tools.ietf.org/html/draft-ietf-bmwg-sdn-controller-benchmark-term-09 https://tools.ietf.org/html/draft-ietf-bmwg-sdn-controller-benchmark-meth-08 The chair reported that the Drafts have emerged from IETF Last Call, comments have been addressed, and they are on the April 6 IESG agenda. 4. Benchmarking Methodology for EVPN and PBB-EVPN Presenter: Sudhin Jacob Related Draft: https://tools.ietf.org/html/draft-kishjac-bmwg-evpntest-09 Sudhin presented the current status. No questions/comments were raised. Al confirmed that the draft is 3/4 adopted: is well on its way to adoption. Further suggestions for development are encouraged from the WG. Al mentioned he aims to recruit additional contributors, and Sudin welcomed contributions. Continuing Proposals: 5. Proposal on Network Service Layer Abstract Model Presenter: Sean Wu Related Draft: https://tools.ietf.org/html/draft-xwu-bmwg-nslam-00 Plan is to update to include NFV Considerations (no slides available). Sean gave a presentation remotely during the Singapore meeting. Draft attempts to model service layer so that it could be tested. Challenge: make it more specific to benchmarking. Being worked on. Sean has indicated that work is in progrèss, expecting to complete the benchmarking example this month. 6. Updates for the Back-to-back Frame Benchmark Presenter: Al Morton Related Draft: https://tools.ietf.org/html/draft-morton-bmwg-b2b-frame-01 Background Slides: https://wiki.opnfv.org/download/attachments/10293193/VSPERF-Dataplane-Perf-Cap-Bench.pptx?api=v2 https://wiki.opnfv.org/display/vsperf/Traffic+Generator+Testing Al presented the current draft and provided some background for the methodology. He raised two main questions for discussion during the meeting: 1. Should a particular search algorithm be included? 2. Should the search include trial repetition whenever frame loss is observed, to avoid the effects of background loss (unrelated to buffer overflow)? Carsten: should the search produce the optimum result? or minimum result? Al: If the phenomenon is not responsible for or part of the buffering, then it’s worth repeating, or doing long duration test to see how often the background process causes loss. Al will propose a search with repetition after this meeting. 7. Updates to Firewall Benchmarking for Modern Firewalls Presenter: Carsten Rossenhövel Related RFC: https://tools.ietf.org/html/rfc3511 Related Draft: https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-02 Related Web page: https://www.netsecopen.org/ https://www.netsecopen.org/about Want to strongly improve the benchmarks for these types of devices, and the repeatability of results. NetSecOpen will conduct a certification program intended afterwards. Al: any thoughts about putting more detail in the definition of binary search? (Carsten: yes) Al: what about repetition due to errors? Carsten: binary search is a time consuming method. Can become days of testing waiting for reseats. Anything better than binary? Linear? even more iterations. Al: might be a way to improve test time by tightening search range based on previous results Al: Comment on test results acceptance criteria. We have to be careful with the wording. Can’t declare pass/fail, seldom set objectives either. Allow others to set thresholds. Some thresholds are possible: RFC2544 with 0% loss as an example. 0 is a number set by IETF. Carsten: removing these from draft would lessen value. Understood that it’s difficult to agree on. Maybe an annex. Al: maybe getting rid of the words “acceptance criteria” will help as a start, but we need to discuss this further. Carsten: bringing in people from industry for help with draft construction. Al: this is great proposed schedule: draft numbers are off by +1 Will refer to a NIST database for attacks. Al: that’s great Carsten: What’s next? Al: need to adopt our next charter. Once that happens, things will be able to move ahead. Trying to get interim meetings to advance this work in order to get close to July stable draft. Carsten: want to use results in NetSecOpen soon Al: IESG approval steps would not likely be until August if WG consensus is achieved in July because of process timelines IETF #102 in Montreal might be a good time for last WG updates and finishing touches Carsten: requesting people review and contribute 8. Return of VNF Benchmarking Methodology Presenter: Raphael Vicente Rosa Related Draft: https://tools.ietf.org/html/draft-rosa-bmwg-vnfbench-01 Related Publication: (IEEE Comm Mag) https://www.dropbox.com/s/n9niziuhi648scj/2017-rosa-taking.pdf?dl=0 Raphael presented the draft. Al suggested to coordinate parameters used with each entity in the test setup. Al further suggested to delineate the vSwitch performance from the VNF application-layer performance. Open source code will be released in the second half of 2018 for both vSwitch and vIMS benchmarking; it is programmed in Python and self-contained, Raphael said. 9. Considerations for Benchmarking Network Virtualization Platforms Presenter: Samuel Kommu Related Draft: https://tools.ietf.org/html/draft-skommu-bmwg-nvp-01.txt NEW-ish Draft with minimal exposure, lots of comments at IETF-98 WG Discussion: - not covered - but Samuel has asked for review during teh Interim meeting. 10. ETSI NFV Liaison on NFVI Benchmarking Normative Specification Presenter: Al Morton Issues worked-out with omission from statements@ietf.org: https://datatracker.ietf.org/liaison/1565/ Related message to the WG list: https://mailarchive.ietf.org/arch/msg/bmwg/1_jPsNUf9q1ObXRwF-_MlZKiE5w Comments can be collected from BMWG list and replied as a WG, or sent directly to the chairs of ETSI NFV TST WG (see the liaison text). Al presented the status of the ETSI NFV TST009 work item. There were no comments during the session. ACTION: Al will send a request for comment to the list, with a 1 month deadline. 11. Re-chartering BMWG Chairs and AD Advisor Review the Revised Text (and Milestones): https://mailarchive.ietf.org/arch/msg/bmwg/3g13ODStfbq3H1TrRIwrRTIb0Ao https://mailarchive.ietf.org/arch/msg/bmwg/xE7YzPe7j61x-bgzUX9ETdYU53E Al went over the new charter briefly. Warren Kumari asked whether the last sentence ("Ideally, BMWG should communicate with the operations community through organizations such as NANOG, RIPE, and APRICOT") is applicable or might be better removed. Al explained we would like to get this communication established again, that this is a key connection for this WG and the OPS Area. and that this was the main emphasis of the OPS-DIR at our meeting Monday! Pierre asked regarding a milestone for ETSI NFV TST009-related benchmarking. Al commented that there is no active draft for that topic in BMWG and we should avoid overlapping work for now. Raphael requested editing the milestone to "General VNF Benchmarking". Carsten comented that such a generic document does not sound useful because its scope is too wide. Al proposed the subject of the milestone to become: "General VNF Benchmarking Automation". There were no objections. The charter was agreed unanimously by hum, with no objections, no abstains. ACTION: There will be a one-week comment period on the list, then this goes to Warren and the IESG for approval. LAST. AOB No topics. Session closed at 1156. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-