Minutes IETF101: bmwg
||Minutes IETF101: bmwg
Minutes of Benchmarking Methodology WG (BMWG)
TUESDAY, March 20, 2018
0930-1200 Morning Session I
Palace C OPS bmwg
0. Agenda Bashing
1. WG Status (Chairs)
(Brief status below)
Two Drafts remain on our current Charter
Interim Meeting Report (March 1) - discussed NG-firewall benchmarking
methodology in preparation of this meeting
There are many proposals for new work: Network service layer abstract model,
back2back frame testing, network virtualization platforms, ...
2. Charter and Milestones (Chairs)
No new RFCs this time, but there were 6 last time!
- milestones see presentation slide 6
Supplementary BMWG page http://bmwg.encrypted.net/
3. Benchmarking Methodology for SDN Controller Performance
Presenter: Sarah Banks (or Bhuvaneswaran Vengainathan)
On IESG Aganda for April 19, 2018
The chair reported that the Drafts have emerged from IETF Last Call,
comments have been addressed, and they are on the April 6 IESG agenda.
4. Benchmarking Methodology for EVPN and PBB-EVPN
Presenter: Sudhin Jacob
Sudhin presented the current status. No questions/comments were raised.
Al confirmed that the draft is 3/4 adopted: is well on its way to adoption.
Further suggestions for development are encouraged from the WG.
Al mentioned he aims to recruit additional contributors, and Sudin
5. Proposal on Network Service Layer Abstract Model
Presenter: Sean Wu
Plan is to update to include NFV Considerations (no slides available).
Sean gave a presentation remotely during the Singapore meeting.
Draft attempts to model service layer so that it could be tested.
Challenge: make it more specific to benchmarking. Being worked on.
Sean has indicated that work is in progrèss, expecting to complete the
benchmarking example this month.
6. Updates for the Back-to-back Frame Benchmark
Presenter: Al Morton
https://wiki.opnfv.org/display/vsperf/Traffic+Generator+Testing Al presented
the current draft and provided some background for the methodology. He
raised two main questions for discussion during the meeting:
1. Should a particular search algorithm be included?
2. Should the search include trial repetition whenever frame loss is
observed, to avoid the effects of background loss (unrelated to buffer
Carsten: should the search produce the optimum result? or minimum result?
Al: If the phenomenon is not responsible for or part of the buffering,
then it’s worth repeating, or doing long duration test to see how often
the background process causes loss.
Al will propose a search with repetition after this meeting.
7. Updates to Firewall Benchmarking for Modern Firewalls
Presenter: Carsten Rossenhövel
Related RFC: https://tools.ietf.org/html/rfc3511
Related Web page:
Want to strongly improve the benchmarks for these types of devices, and the
repeatability of results.
NetSecOpen will conduct a certification program intended afterwards.
Al: any thoughts about putting more detail in the definition of binary
search? (Carsten: yes) Al: what about repetition due to errors? Carsten:
binary search is a time consuming method. Can become days of testing waiting
for reseats. Anything better than binary? Linear? even more iterations. Al:
might be a way to improve test time by tightening search range based on
previous results Al: Comment on test results acceptance criteria. We have to
be careful with the wording. Can’t declare pass/fail, seldom set objectives
either. Allow others to set thresholds. Some thresholds are possible:
RFC2544 with 0% loss as an example. 0 is a number set by IETF. Carsten:
removing these from draft would lessen value. Understood that it’s difficult
to agree on. Maybe an annex. Al: maybe getting rid of the words “acceptance
criteria” will help as a start, but we need to discuss this further.
Carsten: bringing in people from industry for help with draft construction.
Al: this is great proposed schedule: draft numbers are off by +1 Will refer
to a NIST database for attacks. Al: that’s great Carsten: What’s next?
Al: need to adopt our next charter. Once that happens, things will be able
to move ahead. Trying to get interim meetings to advance this work in order
to get close to July stable draft.
Carsten: want to use results in NetSecOpen soon
Al: IESG approval steps would not likely be until August if WG consensus is
achieved in July because of process timelines
IETF #102 in Montreal might be a good time for last WG updates and finishing
Carsten: requesting people review and contribute
8. Return of VNF Benchmarking Methodology
Presenter: Raphael Vicente Rosa
Related Publication: (IEEE Comm Mag)
Raphael presented the draft. Al suggested to coordinate parameters used
with each entity in the test setup. Al further suggested to delineate the
vSwitch performance from the VNF application-layer performance. Open source
code will be released in the second half of 2018 for both vSwitch and vIMS
benchmarking; it is programmed in Python and self-contained, Raphael said.
9. Considerations for Benchmarking Network Virtualization Platforms
Presenter: Samuel Kommu
NEW-ish Draft with minimal exposure, lots of comments at IETF-98
- not covered - but Samuel has asked for review during teh Interim meeting.
10. ETSI NFV Liaison on NFVI Benchmarking Normative Specification
Presenter: Al Morton
Issues worked-out with omission from firstname.lastname@example.org:
Related message to the WG list:
Comments can be collected from BMWG list and replied as a WG, or sent
directly to the chairs of ETSI NFV TST WG (see the liaison text). Al
presented the status of the ETSI NFV TST009 work item. There were no
comments during the session.
ACTION: Al will send a request for comment to the list,
with a 1 month deadline.
11. Re-chartering BMWG
Chairs and AD Advisor
Review the Revised Text (and Milestones):
Al went over the new charter briefly. Warren Kumari asked whether the last
sentence ("Ideally, BMWG should communicate with the operations community
through organizations such as NANOG, RIPE, and APRICOT") is applicable or
might be better removed.
Al explained we would like to get this communication established again,
that this is a key connection for this WG and the OPS Area.
and that this was the main emphasis of the OPS-DIR at our meeting Monday!
Pierre asked regarding a milestone for ETSI NFV TST009-related benchmarking.
Al commented that there is no active draft for that topic in BMWG
and we should avoid overlapping work for now.
Raphael requested editing the milestone to "General VNF Benchmarking".
Carsten comented that such a generic document does not sound useful
because its scope is too wide.
Al proposed the subject of the milestone to become:
"General VNF Benchmarking Automation". There were no objections.
The charter was agreed unanimously by hum, with no objections, no abstains.
ACTION: There will be a one-week comment period on the list, then this goes to
Warren and the IESG for approval.
No topics. Session closed at 1156.