Minutes IETF103: bmwg

Meeting Minutes Benchmarking Methodology (bmwg) WG
Title Minutes IETF103: bmwg
State Active
Other versions plain text
Last updated 2018-11-18

Meeting Minutes

   Benchmarking Methodology WG (BMWG)

THURSDAY, November 8, 2018
0900-1100 (UTC+7)  Morning Session I
Room    OPS     bmwg

Remote Participation:


BMWG met with 19 people present and 2 remote participants.
Al Morton chaired the meeting, and Sudhin Jacob took the minutes.
The Call for adoption on Next-Gen FIrewall Draft has just completed, three
people offered support (and review) at the meeting, in addition to 5 or more on
the list before the meeting).  The WG chairs will consider the WG input and
determine the outcome of the call.

The WG EVPN and EVPN-PBB draft has made much progress addressing comments.
The WG chairs will consider the WG input and determine if a WG Last Call is
currently a good next step (not withstanding the many comments delivered at the
meeting today, which would be considered WGLC comments during the comment
period, as appropriate). There are also two new EVPN-related benchmarking
proposals available as new drafts.

The WG used the session for considerable discussion of buffer-size measurement
topics, including the present draft to update RFC 2544, which has benefited
from previous comments and additional experimentation.
The WG chairs will consider the WG input and determine if a call for
WG Adoption is a reasonable next step.

There were several other new proposals in draft form, and a discussion of
the cross-over between benchmarking test methods and Internet access methods,
prompted by a Liaison from ITU-T SG 12 that describes a plan to evaluate these
methods in a scientific way.  Initial Lab results were shared with the WG, and
there was good interest and discussion.  The meeting ended precisely on-time!


0. Agenda Bashing

1. WG Status  (Chairs)
   (Brief status below)
There are many proposals for new work...
No comments on the working group status.

2. Charter and Milestones (Chairs)

ETSI NFV GS on NFVI Benchmarking Normative Specification Published

We had a draft of this spec available for comment in BMWG during the first half
of the year. Now BWMG can appreciate what was accomplished here,
in our field of interest.

3. Benchmarking Methodology for Network Security Device Performance
   Presenter: Samaresh Nair
   Related RFC:  https://tools.ietf.org/html/rfc3511
   Related Draft:
   Related Web page:

Sudhin Jacob support adoption.
Also, Mike Ackermann and Barak Gafni added support as tghe meeting closed.

Al: After the POC testing, what was something you learned from testing program,
something specific in the draft?
This is a request to share the results.
Samaresh said he will share the results/answers in the future.
He will provide info on the changes that are made.

ACTION  Chairs - call consensus on adoption.

4. Benchmarking Methodology for EVPN and PBB-EVPN
   Presenter:  Sudhin Jacob
   Related Draft:

Barak do you intend to test Cntrl and dataplane?
SJ - yes, both, BGP type 2 routes MAC learning
Barak - do you include both layer 2 and layer 3 prefixes?
       suggest to add type 5 scaling - believes this is needed.
SJ: Had this comment, Decided not to add.  Also not multicast.
    Maybe add another draft to cover these points.
Barak - what underlays are included? MPLS? others?
SJ: overlay is like a container, we measure what is in the container.
Barak - underlay configuration may affect the overlay performance -
         when you go to the lab, it matters!
SJ: Type 6 and 7 multicast considered for new draft?

ACTION - Chairs consider WGLC for this draft.

Related, NEW EVPN proposals:

5. Benchmarking Methodology for EVPN VPWS
   Presenter:  Sudhin Jacob
   Related Draft:
 Al asked ... ansewer: Scale and SOAK tests do test the dataplane
 Barak very important to test with number of peers and number of underlay
 destinations SJ VNI and others: there are in-scale tests, Overlay VRFs are
 considered. SJ will look at Type 5 for this

6. Benchmarking Methodology for EVPN Multihoming Restor. & Mass Withdrawal
   Presenter:  Al Morton
   Related Drafts:
  SJ has a comment on configuration parameters for mass withdrawl
  Al make some suggestions, we'll be glad to incorporate.
  Barak - how is trafic split between PE1 and PE2?  we'll find oyut

Continuing Proposals (topics may be added):

7. Updates for Back-to-back Frame Benchmark & OPNFV Plugfest/VSPERF Testing
   Presenter: Al Morton
   Related Draft:
   OPNFV Plugfest (Fraser Release) VSPERF Testing Slides

   Background Slides:

 Al made the point that Yoshiaki's input had been incorporated in this version.
Further, that all open issues with respect to search algorithms are now closed,
through reference to ETSI GS NFVTST009 on NFVI Benchmarking.
No other comments.
Al said will consider the details for adoption of this draft.

ACTION  Chairs - consider call for WG adoption.

8. New Buffer assessment method for RFC 8239 Data Center Benchmarking
   Author: Yoshiaki Itou
   Mail List References:

Barak there are issues in 100g ethernet port testing,
Barak suggested to decouple the pipeline versus buffer.
Al there a lot of good benchmarks. Al suggested that Barak write down the ideas.
Al said Barak is bringing a lot of expertise to the WG - much appreciated!
Al if it 12 and 12 ok, it seems to be the linear case.
Barak once you operate with pauses on some vendors SWE,
it will change the architecture of buffers, and
you will be getting wrong results.
Barak question I like previous method pushing the two ports to one.
Al 1 percent over-subscription show different results, 24 and 58 frames,
so there must be some buffer which is not filled by the 1 percent flow.
Barak the test/results have to be based internal architecture of switch
(?? whitebox ??).

NEW Proposals:

9. New Adaptive Search Algorithm from FD.io CSIT
   Presenter: Vratko Polak, Maciek Konstantynowicz
   Related Draft:

Al requested to add bmwg in the draft name. Agreed!
There are also improved versions of binary search, comparison would be good.
The WG may continue to update RFC 2544 in a piece-wise way.

10.New Long-term tests for Loss characterization from FD.io CSIT
   Presenter: Vratko Polak, Maciek Konstantynowicz
   Related Draft:

Same comment about filename.
Al says there are improvements coming to RFC 2544
like long term loss measurement. We should compare the
efficiency of the different techniques, as they emerge.

BMWG Communications:

11.Liaison from ITU-T SG 12:
   Presenter: Al Morton

Michel UDP cap limits. Wolfgang from DT told that many CPE has TCP
prioritization. Doug Grinkemeyer has indicated interest to join the work.


Barak:  said consider may be to extend the work on the RFC to measure buffer
size. Fairness between two ports is important, and he commented this has
interest in the industry. Al: There is also the QoS aspect, within multi-port

Al: You said there is interest the industry.
are you willing to bring-n tests to characterize the fairness?
are you interested in bringing a new draft doccument?

Barack said he will try.

AD Ignas with the last word:
We need to face the reality many topics discussed in the ietf is not known the
industry. Need to achieve some visibility at Operator's forums, RIPE and others.

Last point:
Mike i may have missed the next generation firewall discussion,
I am interested in that. Do I talk to you?
Al I suggest you read the Internet draft.
Al 3 people shown interest in this at the meeting, and about 5 on the list.