Minutes IETF109: bmwg
minutes-109-bmwg-00

BMWG Session (IETF-109)
Thursday, November 19, 2020 (UTC+07)
12:00-14:00 Thursday Session I (Room 4)
https://datatracker.ietf.org/meeting/109/agenda/

Bill Cerveny taking notes

Summary https://trac.ietf.org/trac/ops/wiki/IETF109summary  (detailed minutes
below) Benchmarking Methodology WG met with about 24 people joining. Bill
Cerveny kindly took the minutes

WG Drafts

    Brian Monkman volunteered to Review the EVPN WG draft for Editorial
    improvement, un-blocking the progress back to AD-review. The Next
    Generation Security Device Benchmarking WG Draft has made much progress
    this year and will see a WG Last Call on version 06 the Back2back Frame
    Benchmark partial update of RFC 2544 completed AD-review and is on it's way
    to LC and IESG (Dec 17)

Proposals

    With all WG drafts completing work (or nearly so), it's time to consider
    new work carefully There will be a WG adoption call on the list for the
    time-saving Multiple Loss Ratio Search algorithm draft Good synergies
    identified between YANG Traffic gen control and VNF test automation drafts,
    with comment exchanges on the list Progress on Cloud-Containerized
    Networking Benchmarking Considerations earned through "getting hands dirty"
    with real test experience. 5G transport benchmarking needs to have some
    further investigation to avoid overlap with the planned control-plane and
    data-plane work: IETF TEAS WG, ETSI TC INT, and O-RAN WG9 (data-plane) were
    mentioned. Also 3GPP, obviously. It would be good to have test equipment
    vendors more involved. New test results with another BMWG-method-compliant
    open source test tool were presented, and the tool was offered for others
    to try.

Scheduling and Calendar

Apparently there were agenda/calendar problems: it was reported post-meeting
that the IETF Agenda failed to do any conversion to UTC mode when selected
(continued to show Bangkok time). The chair made every effort to mention the
actual times in other zones as part of communications prior to the meeting,
including a "meeting starts now" message to the list. However, there were still
cases where issues/confusion caused presenters to miss the meeting completely.
(This year, I have heard many people say that calendars make them cry, but I
currently have no tears left to cry over calendar/meeting/time-zone issues, and
no expertise to do anything about the problems.)

WG Status
No questions on status
WG Drafts:

    EVPN

    https://tools.ietf.org/html/draft-ietf-bmwg-evpntest-06
    status:

        Returned to WG for proof reading

        Completed Sec 1+2 proof reading plus suggestions for automated tools

        Draft was updated on August 7 (most recent diffs are spaces added
        after sentences in sec 3 and beyond)

        Brian Monkman will review sections of this draft and report to list.

    Next Generation Firewall Benchmarking
    https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-05
    status:
        Security Effectiveness: Configuration info and Appendix on Method
        Draft was updated on Oct 30

    7 people have read draft
    Brian might change title for WGLC.
    Bill Cerveny will send his comments for grammar, etc. to Brian Monkman

    Back-to-Back Frame (Update to RFC2544)
    https://tools.ietf.org/html/draft-ietf-bmwg-b2b-frame-03
    status:
        Reminder of Nov 2019 comments and additional comments on-list:
        https://mailarchive.ietf.org/arch/msg/bmwg/FNOWgx5FjFgubYNBI3bOwOciDh0/
        All November and May 2020 comments/clarifications addressed on-list.
        Comments resulted from use of the Benchmark over last year
        Quiet WGLC, with a supporting comment from Scott Bradner (no changes).
        AD-Review Nov 9, addressed in new version after black-out lifted.

Proposals:

    A YANG Data Model for Network Interconnect Tester Management
    https://tools.ietf.org/html/draft-vassilev-bmwg-network-interconnect-tester-04
    status:
        Draft updated Sept 9, added mechanism for real time synchronization of
        traffic generation open-source/hardware implementation of
        generator/analyzer at the hackathon.

No one raised hand that they have read above draft. Al said readers were needed.

    Methodology for VNF Benchmarking Automation
    https://tools.ietf.org/html/draft-rosa-bmwg-vnfbench-06
    status:
        Draft updated Oct 20, major re-factor in content, giving more maturity
        and clarifying its core concepts and proposal, added YANG model(s)
        Comment from Vladimir Vassilev on the list: review in-progress, needs
        more info

    5G transport network benchmarking
    https://tools.ietf.org/html/draft-contreras-bmwg-5g-02
    status:
        Draft updated Nov 2: control plane, methodology, network slices/teas WG

Questions:
Carsten Rossenhoevel: Luis, two questions: 1) Is IETF the right place for this
work item, or should it rather be standardized in 3GPP or ETSI (or O-RAN)? I am
active in O-RAN WG9 which works on something potentially similar.

Luis said yes. Could be complementary with ORAN, not duplicating work

Carsten: 2) Since this WG is about benchmarking, what do you plan to define in
the draft to support RAN benchmarking? This would require advanced emulators.
Testing with real User Equipment (UEs) would not work well for benchmarking I
think?

Luis: We will look at the problem from an IETF technology perspective, not from
the radio part, for instance.

Carsten:Thank you for your response, Luis. With regards to test tools, I
suggest to compare with the next-gen-firewall document. We also added new test
methodology which requires test tools to support. I would suggest you work with
test tool vendors to ensure that the methodology you plan to define is
implementable (in good IETF sense to first conduct a full proof of concept
before an RFC is approved). I am quite worried that testing with single UEs
does not do justice to the problem and that the document might become a dead
horse iif there is no way to implement e2e benchmarking across RAN and 5GCore
in reality.

Al:We need follow-up on these activities with other groups. Investigate overlap
with ORAN. Getting someone from test equipment vendors on BMWG list would be
good.

Luis: We don’t have the answers yet, we need to review and propose work then.

    Considerations for Benchmarking Network Performance in Containerized
    Infrastructures
    https://tools.ietf.org/html/draft-dcn-bmwg-containerized-infra-05
        Draft updated Nov 2: New section describing a benchmarking experiment
        Results from the recent hackathon on SR-IOV/DPDK and hugepages settings.

Question from Benson Muite: NUMA allocation has several modes. Can one pin to
specific cores? Do NUMA and other hardware specific effects need to be included
in the specification or is there some way to make the specification applicable
to a wide range of hardware?

Sun Kj: (didn’t catch everything) There may be some hardware tests for that.

Al: It would be good if the draft had some generations and recommendations for
further development for metrics and methodologies.

Sun Kj said he would attempt to do this.

Al: I encourage you to keep working on this. We always get interesting feedback
from results in meetings.

    Discussion on these topics needed (where does BMWG’s interest exist)

    Multiple Loss Ratio Search
    https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch-03
    status:
        Comments (many questions) on the list:

Gabor Lencse: Will review above draft.

Al: We will adopt above draft (Al: Confirm!), not hearing any dissention.
https://mailarchive.ietf.org/arch/msg/bmwg/DdEqW8kT54-PNtiXFNv3FYHh8go/

    Probabilistic Loss Ratio Search
    https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch-03
    status:

    Network Function Service Density
    draft-mkonstan-nf-service-density (expired),
    revisit the overall problem space, explore tighter collaboration options
    status:
    sent e-mail with questions, set-up a chance to explore with many Orgs.

    Al: Will hold discussion of above on list.

    An Upgrade to Benchmarking Methodology for Network Interconnect Devices
    https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-rfc2544-bis-00
    status:
        no updated text, but several exchanges on the list:
        https://mailarchive.ietf.org/arch/msg/bmwg/t3TKnwaPCqSAbox2Zdq61zrmZNA/
        https://mailarchive.ietf.org/arch/msg/bmwg/sg28EfrZrs8J1mcWnrlBx-mpDHU/
        https://mailarchive.ietf.org/arch/msg/bmwg/OZe3Gm2ik12C3KE_I-hcTOoSh_s/

    Al: Will discuss above on list.
        Recent experiments with siitperf, the RFC 8219-compliant SIIT tester

-=-=-=-=-=-=- EVPN work on hold -=-=-=-=-=-

    Benchmarking Methodology for EVPN VPWS
    https://tools.ietf.org/html/draft-kishjac-bmwg-evpnvpwstest-05
    status:
        topic on-hold (new draft)

    Benchmarking Methodology for EVPN Multi-casting
    https://tools.ietf.org/html/draft-vikjac-bmwg-evpnmultest-05
    status:
        topic on-hold (new draft)

    Benchmarks and Methods for Multihomed EVPN
    https://tools.ietf.org/html/draft-morton-bmwg-multihome-evpn-04
    status:
        topic on hold (new draft)

AOB:

-=-=-=-=-=-=-=-=-