# Benchmarking Methodology Working Group (bmwg) - IETF 118 Prague   {#benchmarking-methodology-working-group-bmwg---ietf-118-prague}

Date and Time: 06/11/2023 Monday 9:30-11:30 CET  
Location: Hilton Prague, Amsterdam  
Chairs: Sarah Banks, Giuseppe Fioccola  
AD: Warren Kumari  
Zulip Stream: https://zulip.ietf.org/login/#narrow/stream/bmwg

## Note taker(s), Jabber, IPR   {#note-takers-jabber-ipr}

Paolo Volpato volunteered.

## Administrative and WG Status (Chairs)   {#administrative-and-wg-status-chairs}

*   WG Adoptions and Proposals
*   Welcome and self-introduction from Giuseppe

Agenda discussed:

*   WGLC is coming for Stateful NATxy Gateways BM draft
*   Request to revive the EVPN draft has been declined by the WG (lack
    of support)
*   Segment Routing drafts will remain as separate drafts (mailing list
    discussion and feedback from authors)

BMWG activity presented and milestone revised.

## WG Drafts   {#wg-drafts}

### Multiple Loss Ratio Search and YANG Model, Maciek Konstantynowicz and Vratko Polak   {#multiple-loss-ratio-search-and-yang-model-maciek-konstantynowicz-and-vratko-polak}

[draft-ietf-bmwg-mlrsearch][1]

*   Maciek and Vratko presented
    (https://datatracker.ietf.org/meeting/118/materials/slides-118-bmwg-2-multiple-loss-ratio-search-and-yang-model).

It was presented the update to the draft. They are not asking yet for
WGLC but are looking for reviewers/co-authors and feedback from testers
and developers.  
It is a big problem to have a reliable test in a virtualized
environment.  
If time permits, slot to present the results of open source testing.  
Acknowledgment to Al Morton.

Carsten Rossenhoevel: good work to continue. Software and virtual
environment need longer test duration, too short test duration may give
wrong results. This does not influence the draft itself, but it is
suggested to add a note on that.

Maciek Konstantynowicz: good comment, even the perfect SW in a shared
environment is not isolated. It is about SUT and DUT that should help in
isolation. MLRserach comes with parameters from where you can get result
compatible with RFC 2544. This represents test duration, not the actual
duration.

Vratko Polak: it is all about splitting the noise from useful signals in
the spectrum, if interested, please give input.

Carsten Rossenhoevel: if the draft becomes RFC it aims to amend RFC
2544. So, industry argues we use RFC 2544. If you write something, then
that result becomes normative. The particular test time may be good for
one environment but not for another.

Maciek Konstantynowicz: We want to standardize it, but we need feedback.
We give value to what you say, but the time value of MLRsearch is
configurable. Experts will customize it for the situation.  
We did not ask for WGLC, but we aim to get support from the WG.

### Benchmarking Methodology for Stateful NATxy Gateways using RFC 4814 Pseudorandom Port numbers, Gabor Lencse   {#benchmarking-methodology-for-stateful-natxy-gateways-using-rfc-4814-pseudorandom-port-numbers-gabor-lencse}

[draft-ietf-bmwg-benchmarking-stateful][2]

*   Gabor presented
    (https://datatracker.ietf.org/meeting/118/materials/slides-118-bmwg-3-benchmarking-methodology-for-stateful-natxy-gateways-using-rfc-4814-pseudorandom-port-numbers).

Presented the summary of the proposal.  
Asking for WGLC.  
No questions.

Giuseppe Fioccola: WGLC will be done on the mailing list.

## Proposals   {#proposals}

### Recommendations for using Multiple IP addresses in Benchmarking Tests, Gabor Lencse   {#recommendations-for-using-multiple-ip-addresses-in-benchmarking-tests-gabor-lencse}

[draft-lencse-bmwg-multiple-ip-addresses][3]

*   Gabor presented
    (https://datatracker.ietf.org/meeting/118/materials/slides-118-bmwg-4-recommendations-for-using-multiple-ip-addresses-in-benchmarking-tests)

Problem described and solution on IP address pseudoradomization
introduced. It is important to have the hash function with random
fields, all forwarding engines now are multi-core.  
Question to the audience to acknowledge if the problem exists. IPv4
pools are typically smaller than IPv6.

Maciek Konstantynowicz: regarding the problem statement with the 1st RSS
and 2nd RSS implementations, port numbers and protocols use that for
entropy.  
In synthetic tests, they may not represent real systems. The problem of
using address ranges is interesting. This is also for stateful cases and
we should understand how prescriptive it should be to define the
benchmark.  
In general, it is good advice to use bigger IP pools. It looks like a
useful augmentation of RFC 2544.

Vratko Polak: it is important to have repeatable results, but different
environments may not permit it. Tests are related to encapsulated flows,
with different technologies. Sometimes the tests use multiple
encapsulations.

Gabor Lencse: We need to improve the RSS part.

Sarah Banks: it's a decision based on implementation. You design the
test based on the SUT or tester.

Gabor Lencse: software systems need to load all cores.

Maciek Konstantynowicz: if we go ahead with this work, including RSS,
repeatability might be impacted.  
Cores are open systems and are configurable. In SW you specify the
resources, so I agree partially with Sarah.

Sarah Banks: thanks for clarification. As a tester you figure out what
you have. BMWG stayed in RFC 2544 era for long. Need to have a
conversation.

### Considerations for Benchmarking Network Performance in Containerized Infrastructures, Tran Minh Ngoc   {#considerations-for-benchmarking-network-performance-in-containerized-infrastructures-tran-minh-ngoc}

[draft-dcn-bmwg-containerized-infra][4]

*   Tran presented
    (https://datatracker.ietf.org/meeting/118/materials/slides-118-bmwg-5-considerations-for-benchmarking-network-performance-in-containerized-infrastructures).

Update from -11 to -13 presented.

Giuseppe Fioccola: thanks for addressing the comments

Maciek Konstantynowicz: thanks for presenting. Vratko provided the
specific comments. We try to do containerized network functions test. We
faced the repeatability of tests. Kubernetes open source setup
containerization setup in a repeatable manner is an issue. Do you have a
section on this?

Gabor Lencse: support the work.

Maciek Konstantynowicz: also support.

Giuseppe Fioccola: let's do a poll to understand how many people have
read the draft.

Sarah Banks: thanks to 7 people who read the draft. Adoption should
happen in the mailing list.

### Benchmarking methodology for MPLS Segment Routing, Paolo Volpato   {#benchmarking-methodology-for-mpls-segment-routing-paolo-volpato}

[draft-vfv-bmwg-srmpls-bench-meth][5]

*   Paolo presented
    (https://datatracker.ietf.org/meeting/118/materials/slides-118-bmwg-6-benchmarking-methodology-for-mpls-segment-routing).

The last version incorporates almost all text from referenced RFCs.

Gabor Lencse: concern on 2 testers, suggest to add logic to manage both
testers. Suggested to revise the figure about the DUT test

Paolo Volpato: Sure, it will be added to the next version.

Maciek Konstantynowicz: asking for other SR functions to test and
highlighted that the two drafts on SR may also be merged.

Paolo Volpato: we considered the services for other documents and
authors still think that 2 separate drafts are better.

Qi Zhang: Do we have new metrics for the SR. Any test results?

Paolo Volpato: yes, new parameters described in a dedicated section. 3rd
party test results are expected.

Carsten Rossenhoevel: insist on merging the draft because 95% is common.
SR-MPLS tests have a small incremental value from MPLS BM tests.

Maciek Konstantynowicz: support drafts merge. Asking to expand to
functions. We are also testing SR-MPLS and we can share some results.

Giuseppe Fioccola: discussion to continue on the list.

### Benchmarking methodology for IPv6 Segment Routing, Eduard Vasilenko   {#benchmarking-methodology-for-ipv6-segment-routing-eduard-vasilenko}

[draft-vfv-bmwg-srv6-bench-meth][6]

*   Eduard presented
    (https://datatracker.ietf.org/meeting/118/materials/slides-118-bmwg-7-benchmarking-methodology-for-ipv6-segment-routing).

Overview and scope presented. Why to keep services out of the initial
scope? Because in particular for SRv6 there are too many. It would
become an unmanageable document. If you want more, please be specific on
the services.  
As an opinion, the two drafts should be kept separated so that a tester
can choose what they are interested into.

Carsten Rossenhoevel: one example of test areas for SRv6 is compressed
SIDs. No EVPN.

Eduard Vasilenko: apart from it, what else?

Sarah Banks: We asked the list to discuss whether to merge or not to
merge SR drafts.

Giuseppe Fioccola: let's continue on the list.

### Problems and requirements of evaluation methodology for integrated space and terrestrial networks, Zeqi Lai   {#problems-and-requirements-of-evaluation-methodology-for-integrated-space-and-terrestrial-networks-zeqi-lai}

[draft-lai-bmwg-istn-methodology][7]

*   Zeqi presented
    (https://datatracker.ietf.org/meeting/118/materials/slides-118-bmwg-8-benchmarking-methodology-for-reliable-transport-protocols-in-integrated-space-and-terrestrial-networks).

Proposal presented.

Sarah Banks: thanks for showing up. Please send questions to the list.

## Presentations   {#presentations}

### FD.io CSIT Performance Dashboard, Maciek Konstantynowicz and Vratko Polak   {#fdio-csit-performance-dashboard-maciek-konstantynowicz-and-vratko-polak}

[draft-ietf-bmwg-network-tester-cfg][8]

*   Vratko/Maciek presented
    (https://datatracker.ietf.org/meeting/118/materials/slides-118-bmwg-9-fdio-csit-performance-dashboard).

Giuseppe Fioccola showed the first slide of Maciek's presentation on
FD.io CSIT performance dashboard.  
Maciek proposes a side meeting on this topic and possibly an interim
meeting.

Sarah Banks: let's further discuss about interim meetings and let's try
to reach consensus on whether to merge the two SR-related drafts.  
Thanks for the interesting session.



[1]: https://datatracker.ietf.org/doc/draft-ietf-bmwg-mlrsearch/
[2]: https://datatracker.ietf.org/doc/draft-ietf-bmwg-benchmarking-stateful/
[3]: https://datatracker.ietf.org/doc/draft-lencse-bmwg-multiple-ip-addresses/
[4]: https://datatracker.ietf.org/doc/draft-dcn-bmwg-containerized-infra/
[5]: https://datatracker.ietf.org/doc/draft-vfv-bmwg-srmpls-bench-meth/
[6]: https://datatracker.ietf.org/doc/draft-vfv-bmwg-srv6-bench-meth/
[7]: https://datatracker.ietf.org/doc/draft-lai-bmwg-istn-methodology/
[8]: https://datatracker.ietf.org/doc/draft-ietf-bmwg-network-tester-cfg/