Minutes IETF101: bmwg
minutes-101-bmwg-00

Minutes of Benchmarking Methodology WG (BMWG)
TUESDAY, March 20, 2018
0930-1200  Morning Session I
Palace C               OPS     bmwg

Remote Participation:
http://www.ietf.org/meeting/101/index/index.html
http://www.ietf.org/meeting/101/remote-participation.html

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

0. Agenda Bashing
adopted as-is

1. WG Status  (Chairs)
   (Brief status below)
Two Drafts remain on our current Charter

Interim Meeting Report (March 1) - discussed NG-firewall benchmarking
methodology in preparation of this meeting

There are many proposals for new work: Network service layer abstract model,
back2back frame testing, network virtualization platforms, ...

2. Charter and Milestones (Chairs)
No new RFCs this time, but there were 6 last time!
 - milestones see presentation slide 6
Supplementary BMWG page http://bmwg.encrypted.net/

3. Benchmarking Methodology for SDN Controller Performance
   Presenter: Sarah Banks (or Bhuvaneswaran Vengainathan)
   On IESG Aganda for April 19, 2018
https://tools.ietf.org/html/draft-ietf-bmwg-sdn-controller-benchmark-term-09
https://tools.ietf.org/html/draft-ietf-bmwg-sdn-controller-benchmark-meth-08
The chair reported that the Drafts have emerged from IETF Last Call,
comments have been addressed, and they are on the April 6 IESG agenda.

4. Benchmarking Methodology for EVPN and PBB-EVPN
   Presenter:  Sudhin Jacob
   Related Draft:
   https://tools.ietf.org/html/draft-kishjac-bmwg-evpntest-09
Sudhin presented the current status.  No questions/comments were raised.
Al confirmed that the draft is 3/4 adopted: is well on its way to adoption.
Further suggestions for development are encouraged from the WG.
Al mentioned he aims to recruit additional contributors, and Sudin
welcomed contributions.

Continuing Proposals:

5. Proposal on Network Service Layer Abstract Model
   Presenter: Sean Wu
   Related Draft:
   https://tools.ietf.org/html/draft-xwu-bmwg-nslam-00
   Plan is to update to include NFV Considerations (no slides available).
   Sean gave a presentation remotely during the Singapore meeting.
   Draft attempts to model service layer so that it could be tested.
   Challenge: make it more specific to benchmarking. Being worked on.
   Sean has indicated that work is in progrèss, expecting to complete the
   benchmarking example this month.

6. Updates for the Back-to-back Frame Benchmark
   Presenter: Al Morton
   Related Draft:
   https://tools.ietf.org/html/draft-morton-bmwg-b2b-frame-01
   Background Slides:
   https://wiki.opnfv.org/download/attachments/10293193/VSPERF-Dataplane-Perf-Cap-Bench.pptx?api=v2
   https://wiki.opnfv.org/display/vsperf/Traffic+Generator+Testing Al presented
   the current draft and provided some background for the methodology.  He
   raised two main questions for discussion during the meeting:
       1. Should a particular search algorithm be included?
       2. Should the search include trial repetition whenever frame loss is
       observed, to avoid the effects of background loss (unrelated to buffer
       overflow)?

    Carsten: should the search produce the optimum result? or minimum result?

    Al: If the phenomenon is not responsible for or part of the buffering,
        then it’s worth repeating, or doing long duration test to see how often
        the background process causes loss.

    Al will propose a search with repetition after this meeting.

7. Updates to Firewall Benchmarking for Modern Firewalls
   Presenter: Carsten Rossenhövel
   Related RFC:  https://tools.ietf.org/html/rfc3511
   Related Draft:
   https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-02
   Related Web page:
   https://www.netsecopen.org/
   https://www.netsecopen.org/about
   Want to strongly improve the benchmarks for these types of devices, and the
   repeatability of results.
   NetSecOpen will conduct a certification program intended afterwards.
   Al: any thoughts about putting more detail in the definition of binary
   search? (Carsten: yes) Al: what about repetition due to errors? Carsten:
   binary search is a time consuming method. Can become days of testing waiting
   for reseats. Anything better than binary? Linear? even more iterations. Al:
   might be a way to improve test time by tightening search range based on
   previous results Al: Comment on test results acceptance criteria. We have to
   be careful with the wording. Can’t declare pass/fail, seldom set objectives
   either. Allow others to set thresholds. Some thresholds are possible:
   RFC2544 with 0% loss as an example. 0 is a number set by IETF. Carsten:
   removing these from draft would lessen value. Understood that it’s difficult
   to agree on. Maybe an annex. Al: maybe getting rid of the words “acceptance
   criteria” will help as a start, but we need to discuss this further.
   Carsten: bringing in people from industry for help with draft construction.
   Al: this is great proposed schedule: draft numbers are off by +1 Will refer
   to a NIST database for attacks. Al: that’s great Carsten: What’s next?

   Al: need to adopt our next charter. Once that happens, things will be able
   to move ahead. Trying to get interim meetings to advance this work in order
   to get close to July stable draft.

   Carsten: want to use results in NetSecOpen soon

   Al: IESG approval steps would not likely be until August if WG consensus is
   achieved in July because of process timelines

   IETF #102 in Montreal might be a good time for last WG updates and finishing
   touches

   Carsten: requesting people review and contribute

8. Return of VNF Benchmarking Methodology
   Presenter:  Raphael Vicente Rosa
   Related Draft:
   https://tools.ietf.org/html/draft-rosa-bmwg-vnfbench-01
   Related Publication:  (IEEE Comm Mag)
   https://www.dropbox.com/s/n9niziuhi648scj/2017-rosa-taking.pdf?dl=0
   Raphael presented the draft.  Al suggested to coordinate parameters used
   with each entity in the test setup.  Al further suggested to delineate the
   vSwitch performance from the VNF application-layer performance. Open source
   code will be released in the second half of 2018 for both vSwitch and vIMS
   benchmarking; it is programmed in Python and self-contained, Raphael said.

9. Considerations for Benchmarking Network Virtualization Platforms
   Presenter: Samuel Kommu
   Related Draft:
   https://tools.ietf.org/html/draft-skommu-bmwg-nvp-01.txt
   NEW-ish Draft with minimal exposure, lots of comments at IETF-98

WG Discussion:
- not covered - but Samuel has asked for review during teh Interim meeting.

10. ETSI NFV Liaison on NFVI Benchmarking Normative Specification
   Presenter: Al Morton
   Issues worked-out with omission from statements@ietf.org:
   https://datatracker.ietf.org/liaison/1565/
   Related message to the WG list:
   https://mailarchive.ietf.org/arch/msg/bmwg/1_jPsNUf9q1ObXRwF-_MlZKiE5w
   Comments can be collected from BMWG list and replied as a WG, or sent
   directly to the chairs of ETSI NFV TST WG (see the liaison text). Al
   presented the status of the ETSI NFV TST009 work item. There were no
   comments during the session.

ACTION: Al will send a request for comment to the list,
with a 1 month deadline.

11. Re-chartering BMWG
   Chairs and AD Advisor
   Review the Revised Text (and Milestones):
   https://mailarchive.ietf.org/arch/msg/bmwg/3g13ODStfbq3H1TrRIwrRTIb0Ao
   https://mailarchive.ietf.org/arch/msg/bmwg/xE7YzPe7j61x-bgzUX9ETdYU53E
   Al went over the new charter briefly.  Warren Kumari asked whether the last
   sentence ("Ideally, BMWG should communicate with the operations community
   through organizations such as NANOG, RIPE, and APRICOT") is applicable or
   might be better removed.
   Al explained we would like to get this communication established again,
   that this is a key connection for this WG and the OPS Area.
   and that this was the main emphasis of the OPS-DIR at our meeting Monday!
   Pierre asked regarding a milestone for ETSI NFV TST009-related benchmarking.
   Al commented that there is no active draft for that topic in BMWG
   and we should avoid overlapping work for now.
   Raphael requested editing the milestone to "General VNF Benchmarking".
   Carsten comented that such a generic document does not sound useful
   because its scope is too wide.
   Al proposed the subject of the milestone to become:
   "General VNF Benchmarking Automation". There were no objections.

   The charter was agreed unanimously by hum, with no objections, no abstains.

ACTION: There will be a one-week comment period on the list, then this goes to
   Warren and the IESG for approval.

LAST. AOB
No topics.  Session closed at 1156.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-