BMWG Session (IETF-109) Thursday, November 19, 2020 (UTC+07) 12:00-14:00 Thursday Session I (Room 4) https://datatracker.ietf.org/meeting/109/agenda/ Bill Cerveny taking notes Summary https://trac.ietf.org/trac/ops/wiki/IETF109summary (detailed minutes below) Benchmarking Methodology WG met with about 24 people joining. Bill Cerveny kindly took the minutes WG Drafts Brian Monkman volunteered to Review the EVPN WG draft for Editorial improvement, un-blocking the progress back to AD-review. The Next Generation Security Device Benchmarking WG Draft has made much progress this year and will see a WG Last Call on version 06 the Back2back Frame Benchmark partial update of RFC 2544 completed AD-review and is on it's way to LC and IESG (Dec 17) Proposals With all WG drafts completing work (or nearly so), it's time to consider new work carefully There will be a WG adoption call on the list for the time-saving Multiple Loss Ratio Search algorithm draft Good synergies identified between YANG Traffic gen control and VNF test automation drafts, with comment exchanges on the list Progress on Cloud-Containerized Networking Benchmarking Considerations earned through "getting hands dirty" with real test experience. 5G transport benchmarking needs to have some further investigation to avoid overlap with the planned control-plane and data-plane work: IETF TEAS WG, ETSI TC INT, and O-RAN WG9 (data-plane) were mentioned. Also 3GPP, obviously. It would be good to have test equipment vendors more involved. New test results with another BMWG-method-compliant open source test tool were presented, and the tool was offered for others to try. Scheduling and Calendar Apparently there were agenda/calendar problems: it was reported post-meeting that the IETF Agenda failed to do any conversion to UTC mode when selected (continued to show Bangkok time). The chair made every effort to mention the actual times in other zones as part of communications prior to the meeting, including a "meeting starts now" message to the list. However, there were still cases where issues/confusion caused presenters to miss the meeting completely. (This year, I have heard many people say that calendars make them cry, but I currently have no tears left to cry over calendar/meeting/time-zone issues, and no expertise to do anything about the problems.) WG Status No questions on status WG Drafts: EVPN https://tools.ietf.org/html/draft-ietf-bmwg-evpntest-06 status: Returned to WG for proof reading Completed Sec 1+2 proof reading plus suggestions for automated tools Draft was updated on August 7 (most recent diffs are spaces added after sentences in sec 3 and beyond) Brian Monkman will review sections of this draft and report to list. Next Generation Firewall Benchmarking https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-05 status: Security Effectiveness: Configuration info and Appendix on Method Draft was updated on Oct 30 7 people have read draft … Brian might change title for WGLC. Bill Cerveny will send his comments for grammar, etc. to Brian Monkman Back-to-Back Frame (Update to RFC2544) https://tools.ietf.org/html/draft-ietf-bmwg-b2b-frame-03 status: Reminder of Nov 2019 comments and additional comments on-list: https://mailarchive.ietf.org/arch/msg/bmwg/FNOWgx5FjFgubYNBI3bOwOciDh0/ All November and May 2020 comments/clarifications addressed on-list. Comments resulted from use of the Benchmark over last year Quiet WGLC, with a supporting comment from Scott Bradner (no changes). AD-Review Nov 9, addressed in new version after black-out lifted. Proposals: A YANG Data Model for Network Interconnect Tester Management https://tools.ietf.org/html/draft-vassilev-bmwg-network-interconnect-tester-04 status: Draft updated Sept 9, added mechanism for real time synchronization of traffic generation open-source/hardware implementation of generator/analyzer at the hackathon. No one raised hand that they have read above draft. Al said readers were needed. Methodology for VNF Benchmarking Automation https://tools.ietf.org/html/draft-rosa-bmwg-vnfbench-06 status: Draft updated Oct 20, major re-factor in content, giving more maturity and clarifying its core concepts and proposal, added YANG model(s) Comment from Vladimir Vassilev on the list: review in-progress, needs more info… 5G transport network benchmarking https://tools.ietf.org/html/draft-contreras-bmwg-5g-02 status: Draft updated Nov 2: control plane, methodology, network slices/teas WG Questions: Carsten Rossenhoevel: Luis, two questions: 1) Is IETF the right place for this work item, or should it rather be standardized in 3GPP or ETSI (or O-RAN)? I am active in O-RAN WG9 which works on something potentially similar. Luis said yes. Could be complementary with ORAN, not duplicating work Carsten: 2) Since this WG is about benchmarking, what do you plan to define in the draft to support RAN benchmarking? This would require advanced emulators. Testing with real User Equipment (UEs) would not work well for benchmarking I think? Luis: We will look at the problem from an IETF technology perspective, not from the radio part, for instance. Carsten:Thank you for your response, Luis. With regards to test tools, I suggest to compare with the next-gen-firewall document. We also added new test methodology which requires test tools to support. I would suggest you work with test tool vendors to ensure that the methodology you plan to define is implementable (in good IETF sense to first conduct a full proof of concept before an RFC is approved). I am quite worried that testing with single UEs does not do justice to the problem and that the document might become a dead horse iif there is no way to implement e2e benchmarking across RAN and 5GCore in reality. Al:We need follow-up on these activities with other groups. Investigate overlap with ORAN. Getting someone from test equipment vendors on BMWG list would be good. Luis: We don’t have the answers yet, we need to review and propose work then. Considerations for Benchmarking Network Performance in Containerized Infrastructures https://tools.ietf.org/html/draft-dcn-bmwg-containerized-infra-05 Draft updated Nov 2: New section describing a benchmarking experiment Results from the recent hackathon on SR-IOV/DPDK and hugepages settings. Question from Benson Muite: NUMA allocation has several modes. Can one pin to specific cores? Do NUMA and other hardware specific effects need to be included in the specification or is there some way to make the specification applicable to a wide range of hardware? Sun Kj: (didn’t catch everything) There may be some hardware tests for that. Al: It would be good if the draft had some generations and recommendations for further development for metrics and methodologies. Sun Kj said he would attempt to do this. Al: I encourage you to keep working on this. We always get interesting feedback from results in meetings. Discussion on these topics needed (where does BMWG’s interest exist) Multiple Loss Ratio Search https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch-03 status: Comments (many questions) on the list: Gabor Lencse: Will review above draft. Al: We will adopt above draft (Al: Confirm!), not hearing any dissention. https://mailarchive.ietf.org/arch/msg/bmwg/DdEqW8kT54-PNtiXFNv3FYHh8go/ Probabilistic Loss Ratio Search https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch-03 status: Network Function Service Density draft-mkonstan-nf-service-density (expired), revisit the overall problem space, explore tighter collaboration options status: sent e-mail with questions, set-up a chance to explore with many Orgs. Al: Will hold discussion of above on list. An Upgrade to Benchmarking Methodology for Network Interconnect Devices https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-rfc2544-bis-00 status: no updated text, but several exchanges on the list: https://mailarchive.ietf.org/arch/msg/bmwg/t3TKnwaPCqSAbox2Zdq61zrmZNA/ https://mailarchive.ietf.org/arch/msg/bmwg/sg28EfrZrs8J1mcWnrlBx-mpDHU/ https://mailarchive.ietf.org/arch/msg/bmwg/OZe3Gm2ik12C3KE_I-hcTOoSh_s/ Al: Will discuss above on list. Recent experiments with siitperf, the RFC 8219-compliant SIIT tester -=-=-=-=-=-=- EVPN work on hold -=-=-=-=-=- Benchmarking Methodology for EVPN VPWS https://tools.ietf.org/html/draft-kishjac-bmwg-evpnvpwstest-05 status: topic on-hold (new draft) Benchmarking Methodology for EVPN Multi-casting https://tools.ietf.org/html/draft-vikjac-bmwg-evpnmultest-05 status: topic on-hold (new draft) Benchmarks and Methods for Multihomed EVPN https://tools.ietf.org/html/draft-morton-bmwg-multihome-evpn-04 status: topic on hold (new draft) AOB: -=-=-=-=-=-=-=-=-