Benchmarking & Performance Measurements
charter-ietf-bpm-00-04

Scope

The Benchmarking & Performance Measurements Working Group (BPM WG) develops and maintains metrics, methodologies, and protocols that can be applied to the quality, performance, reliability, and scalability of data delivery services and applications running over public or private networks using IETF technologies. These metrics, methodologies, and protocols can be used by network operators and end users in production or independent testing groups in lab environments. Measurements in a lab environment aim to assess the benchmarks for network devices, systems, and services by using isolated and highly controlled stimuli, while measurements in a production environment seek to evaluate the network performance of services and applications under live and variable operational conditions.

The methodologies cover data, control, and management planes and they apply to hardware, virtualized, and containerized network infrastructure (such as Data Center network fabrics, controllers, routers, and firewalls), including Network Management software systems and endpoints in production and lab environments. In lab environments, the testing procedures ensure consistent, reproducible, and reportable benchmarking results including system calibration. In production environments, measurement and system calibration procedures ensure that production management, control, and data plane performance impact is minimized. The metrics defined by the BPM WG are intended to provide unbiased, quantitative performance measurements and will follow the guidance in BCP 170 (Guidelines for Considering New Performance Metric Development). The benchmarking recommendations produced by the BPM WG will describe the relevant performance characteristics, the test environment, and the requirements for the unambiguous reporting of the results but will not define acceptance criteria or performance requirements and shall strive to be vendor independent or otherwise have universal applicability to a given technology class.

The BPM WG fosters commonality and comparability of metrics and measurements across IETF protocols at different layers. The WG is also responsible for documenting guidance for methodologies and measurement protocols, including guidance to prevent that the methods do directly affect Internet security and privacy. In addition, the WG provides guidance on the interpretation of measurement results and the use of relevant operational context.

Within the IETF process, metrics, methodologies, and measurement protocols developed by the BPM WG will be subject to the same rigorous scrutiny for usefulness, clarity, and accuracy as other IETF specifications.

Work Items

The WG is responsible for the development and maintenance of Standards Track or Experimental documents specifying active, passive, and hybrid performance measurement protocols, methodologies, and metrics. The WG will also produce Informational or Best Current Practices (BCP) terminology documents and recommendations on key performance characteristics of internetworking technologies, on frameworks for assessing these characteristics, and on benchmarks for network devices, systems, and services. In addition, the WG will publish Standards Track or Experimental documents specifying performance-related manageability properties such as YANG data models for configuration and operation, and IPFIX entities for Network Telemetry data export.

The work scope is limited to protocols, methodologies, and metrics that are applicable to the IP and MPLS data plane and upper-layer protocols defined by the IETF. Work involving Layer 2 protocols is in scope when it applies IETF-defined benchmarking and performance measurement mechanisms to Layer 2 technologies. The BPM WG does not specify encapsulations required for measurements over non-IP and non-MPLS layers.

The WG is the successor of BMWG and IPPM WGs. As such, the BPM WG is responsible for the maintenance of RFCs published by the concluded BMWG and IPPM WGs (including updating those published as Proposed Standard to Internet Standard and Informational to BCP) and the maintenance of BCP 170.

Relationship With Existing WGs and Coordination

The WG will follow transport-related BCPs (mainly, BCP 133 on Specifying New Congestion Control Algorithms, BCP 145 on UDP Usage Guidelines, and BCP 208 on Network Transport Circuit Breakers) and will seek advice from the WIT area as needed.

The WG collaborates with other WGs such as MPLS, SPRING, INTAREA, and 6MAN, where performance measurement related data plane encapsulations are specified or where the technologies under benchmark evaluation are defined. Also, the WG will seek feedback on topics related to other OPS WGs such as V6OPS or SRV6OPS. The WG will involve the security area for advice and guidance on relevant issues (e.g., integrity protection) and BESS for service specifics (e.g., L2VPN, L3VPN). The WG closely collaborates with the Performance Metrics Directorate per the guidance in BCP 170.