Ballot for charter-ietf-bpm
Block
Yes
No Objection
No Record
Summary: Has 3 BLOCKs. Has enough positions to pass once BLOCK positions are resolved.
Ballot question: "Is this charter ready for external review?"
Thanks for putting this charter together. I support merging IPPM and BMWG into a single working group. The communities and work already overlap considerably, and maintaining two separate groups may no longer be useful. However, the proposed BPM charter is not simply a combination of the two existing charters. It broadens the scope in several areas while dropping some important constraints from both IPPM and BMWG. I think these points should be clarified before the charter goes to external review. The main distinction of BPM between BMWG & IPPM today is reasonably clear: * IPPM develops metrics and methods for measuring network paths and services, including operational networks, while BMWG develops repeatable benchmarks for implementations under controlled laboratory conditions. * BPM brings both activities together, which makes sense, but the charter should preserve the distinction between operational measurement and laboratory benchmarking. In particular, the BMWG charter contains several useful safeguards that are no longer present. It requires benchmarking methodologies to be vendor-independent or universally applicable, and it explicitly says that BMWG does not define acceptance criteria or performance requirements. Those constraints help prevent a benchmarking methodology from becoming a product-selection rule or a minimum performance specification. I suggest retaining both of them in the BPM charter. The BMWG charter also says that benchmarking uses controlled stimuli in a laboratory environment. The BPM text refers to both production and laboratory environments, but it no longer clearly states that benchmarking itself remains a controlled laboratory activity. If the intention is that BPM performs operational measurements in production and benchmarking in the lab, it would be helpful to say that directly. Some useful IPPM requirements have also disappeared. The current IPPM charter requires a new metric to explain how it improves upon an existing metric or measures something not already covered. It also requires metric definitions to follow the RFC 6390 template. BPM mentions maintaining BCP 170 and working with the Performance Metrics Directorate, but it does not clearly require its own metric work to follow that guidance. I suggest restoring that requirement, together with explicit responsibility for the Performance Metrics Registry defined by RFC 8911. The intended publication status of metric specifications is also unclear. The charter assigns Standards Track or Experimental status to measurement protocols and Informational or BCP status to terminology and benchmarking recommendations. It does not clearly say where standalone metric definitions and measurement methods fit. IPPM currently advances useful metrics along the Standards Track, so I assume that possibility is meant to remain. The proposed mechanism for Layer 2 work also seems problematic: "Layer 2 applications can be covered as well but require explicit approval from the Responsible Area Director." If the community agrees that Layer 2 work belongs in BPM, that should be stated in the charter. More serious, I am concerned that grabbing L2 ownership, without safeguarding, may overstep IETF boundary with IEEE. As prior mentioned by Roman, making the scope depend on approval by the serving AD creates a policy that may change when the AD changes. The scope around applications could become much broader than either existing WG. The charter covers the performance and scalability of applications running over networks using IETF technologies. I understand that this may be intended to include AI-related benchmarking proposals. If so, the charter should establish a clear networking boundary. BPM should be able to measure application performance attributable to IETF networking protocols or services, but generic application, compute or AI-model benchmarking should remain out of scope. The statement that BPM “coordinates performance measurement-related work within the IETF” also appears to give the WG a new IETF-wide role. It is not clear whether BPM is expected to review and advise other WGs, act as a dispatch point, or approve performance-related work elsewhere. The intended responsibility and process should be stated more precisely. Hope this helps, G/
Thanks for putting together this charter, I have a few concerns that I would like to discuss. 1) The WG is being formed as a merger of the IPPM and BMWG WGs; this merger brings certain implications that are not reflected in the charter text. a) I would like to understand the synergies and objectives behind this merger so it can get captured, as appropriate, in the text. Objectives that are administrative or related to management aspects (merging two small WGs, reducing administrative and IETF scheduling load, etc.) are not something that ought to be captured in the charter. So, I am looking for other synergies and how the WG would achieve them. Do the participant communities between the two groups overlap, and what is the measure of cross-reviews between them over a year since the two WGs have been having joint meetings? b) There are also several anti-synergies between the work done by the IPPM and BMWG WGs that a combined WG must manage carefully. Examples include: different operating environments (labs vs. production networks); different deliverables and ownership (benchmarking methodologies vs performance metrics and related protocols); different document tracks (Informational vs. Proposed Standard); different implementation policies; and different audiences and constituencies (as reflected in their current charters). How does one ensure that the clarity on all of these from the individual charters are carried over? Suggestion: Assuming that there is significant community interest in the merger of the WGs, I would have expected the charter to clearly call out these two distinct bodies of work as two separate tracks capturing their individual uniqueness (see (b) above), and then having a common part that lists their synergies and commonalities that the merger aims to achieve. Without a compartmentalization of these tracks, this charter is losing a lot of important scoping and operating constraints from the current charters of these two WG. Also, it is important for this charter to indicate that this WG is the successor of IPPM and BMWG WGs; I think this might be required for things like IANA registries and expert reviews - please cross-check. 2) The protocol ownership of IPPM WG is unclear. The charter refers to RFC7799, but that does not list any protocols. This is very important to determine which specific protocols are within the scope of the consolidated WG. I would expect this list to be: OWAMP (RFC 4656), TWAMP (RFC 5357), STAMP (RFC 8762, RFC 8972), IOAM (RFC 9197, RFC 9326), Alternate Marking (RFC 9341, RFC 9342), PDM (RFC 8250) and Hybrid Two-Step? Please also indicate which of these are being "actively" worked on and which are not in focus anymore. 3) Somewhat related to (2), there is work related to performance measurements and related OAM aspects that happens outside of the IPPM WG. It seems like the BPM charter is attempting to overreach into work being done in other WGs - notably MPLS (RFC 6374, RFC 7876, RFC 9714, etc.). Similarly, there is work going on in SPRING and 6MAN that is encapsulation related, but also involve extensions to protocols maintained by IPPM for those respective data planes, encapsulations, or use-cases. Has there been any discussion involving all these other WGs? CURRENT: "It specifically coordinates with other WGs such as MPLS, 6MAN, and SPRING, where data plane encapsulations are specified." SUGGEST: "Where measurement work bears on a data plane or an OAM framework chartered elsewhere, the responsible WG retains ownership and BPM coordinates with it. In particular, the MPLS WG is responsible for MPLS OAM mechanisms and the overall MPLS OAM framework; the SPRING WG is responsible for performance management and monitoring specific to the SR-MPLS and SRv6 data planes; and the 6MAN WG is responsible for IPv6 extension header and destination option encapsulations. BPM is responsible for the measurement protocols, metrics and methodologies themselves, and works with those WGs on their data-plane-specific encapsulation and extension aspects." Also related to this, is the text: "The WG coordinates performance measurement-related work within the IETF." - this is a new uber role that this is sought for this WG. What does it entail? Was this discussed with the other WGs? There is already the perfmetrdir that is designated to help with technical reviews of documents related to Performance metrics and measurements. 4) Related to the two-track point that I've made, we have the following text in the charter: "The work scope is limited to protocols, metrics, and methodologies that are applicable to the IP and MPLS data plane and its upper-layer protocols. Layer 2 applications can be covered as well but require explicit approval from the Responsible Area Director. This WG does not specify encapsulations required for measurements over non-IP and non-MPLS layers." MPLS was in scope for BMWG but not for IPPM. Layer-2 was not in scope for either and may have bearing on coordination and potential overlap with IEEE. As written in the charter, the Layer-2 aspect is pretty vague and the something of this magnitude does not seem to be appropriate to be left to an individual AD discretion. The part about MPLS seems like an error due to the two tracks not being called out distinctly? 5) Almost all of the current work on IPFIX is being done in the OPSAWG. What part of IPFIX is being sought to be introduced in this WG's charter? " In addition, the WG will specify related manageability aspects such as YANG data models for configuration, operation, and IPFIX entities for Network Telemetry data export (Standards Track or Experimental)." It is not clear, as written, if this is moving all of IPFIX work into this WG. If so, what is the synergy of that with the IPPM and BMWG work bodies? Or is it IPFIX extensions for the IPPM protocols alone?
Please confirm that all the milestones from both IPPM and BMWG are carried over and adopted work in both are mapped to be carried over.
** “The work scope is limited to protocols, metrics, and methodologies that are applicable to the IP and MPLS data plane and its upper-layer protocols. Layer 2 applications can be covered as well but require explicit approval from the Responsible Area Director.” This scope creates a situation that today’s AD for this WG may approved L2 work, but the next AD might not. This provides personality-based (dynamic) chartering. If the community consensus is to work on L2 topics why can't this be decided now? ** “In some cases an implementation proof might not be needed” -- (not blocking) Editorial. s/implementation proof/proof of implementation/ -- When would an implementation be needed? For what process(es)? How does this text impact the scope of the WG? ** “The WG … reaches out to other operator communities such as NANOG, RIPE, and APRICOT.” What formal role and associated processes does the WG use to “reach out” to operator communities? How does one know this is adequate? Being done?
** “The WG is also responsible for documenting guidance for methodologies and measurement protocols, including guidance on how to handle sensitive and vulnerable metrics, that methodologies are safe and cannot be easily abused, and that the results are trustworthy.” -- what are “sensitive and vulnerable” metrics? -- per “methodologies are safe”, “safe” from what? -- per “cannot be easily abused”, if the network operators are running these metrics, but abuse is being prevented (i.e., operators have full control) -- per “results are trustworthy”, what makes a result “trustworthy”? ** “The WG will follow transport-related BCPs (mainly, BCP 133 on Specifying New Congestion Control Algorithms, BCP 145 on UDP Usage Guidelines, and BCP 208 on Network Transport Circuit Breakers) and will seek advice from the WIT area as needed.” What is the alternative? Could this WG specify protocol behaviors that violate transport-related BCPs? Perhaps this text is not needed? ** “The WG coordinates performance measurement-related work within the IETF. It specifically coordinates with other WGs such as MPLS, 6MAN, and SPRING, where data plane encapsulations are specified.” What authorities are claimed in saying that this WG “coordinates performance measurement-related work within the IETF”? What standards process is being run here? Is this WG minting yet another specialized dispatch-style activity? ** “Also, the WG will seek feedback from other OPS WGs such as V6OPS or SRV6OPS as needed.” Seek feedback on what?
Merging IPPM and BMWG makes sense indeed. Some comments: s/The methodologies cover management, control, and forwarding planes/The methodologies cover *data*, control, and *management* planes/ to follow the usual wording and importance for performance. Noting that "data plane" is used later in the text. The charter will benefit of making `methodologies are safe` clear. Unsure to whom the 'its' relates to in `to the IP and MPLS data plane and its upper-layer protocols` ? s/Layer 2 applications/Layer-2 applications/ ? Suggest making `In general, the WG expects an implementation for Standards Track documents` more assertive. What is `its write-up` ? Be clear if this is about the shepherd's write-up ;-) Also add "INTAREA" to `such as MPLS, 6MAN, and SPRING,`.
I fully support the principle behind this Charter, and the opportunity this brings. I retain a concern that aligns with Gunter's, specifically "BPM brings both activities together, which makes sense, but the charter should preserve the distinction between operational measurement and laboratory benchmarking." This distinction is clear in most current RFCs. For me, benchmarking is generally scoped to a laboratory environment; IPPM tests Internet paths. The latter this brings extra considerations and requirements. The earlier charter had more distinction on the two and that might be helpful to scope new work also.
This is a merge of both IPPM and BMWG. The two WGs are having joint sessions since IETF#123 (07/25). Since some time now, messages are systematically cross-posted in ippm and bmwg. Some additional notes: * IPPM/BMWG will be concluded when BPM is approved. * A new mailing list will be created for BPM. That list will include both ippm and bmwg members. * Already adopted documents in ippm/bmwg will be inherited by BPM and will be resubmitted as draft-ietf-bpm-*. * The groups considered various options for the implementation policy before converging on the current one in the charter. * "Layer 2 applications can be covered as well but require explicit approval from the Responsible Area Director" is included in the charter because the group want to cover proposals such as draft-weis-ippm-ioam-eth. * The WG received several AI-related bench proposals. AI is not called as such in the charter as that is covered by "application".
Combining the WGs to form this new one seems a good idea to me. I have a few comments. - Clarify what is meant by "vulnerable metrics". - I am not sure of the intended meaning of, "YANG data models for configuration, operation, and IPFIX entities for Network Telemetry data export." Perhaps reword as, "YANG data models for configuration and operation, and IPFIX entities for network telemetry data export." - The "Submit RFC7799bis... (Best Current Practice)" milestone is the only one with no linked draft in the Associated Documents column. Is this because no draft exists yet, or is it an oversight? - +1 to Éric's comments Nits - s/designed as such that/designed such that - s/Where in lab/In lab - s/Where in production/In production
I support Gunter's block. Editorial suggestions: Scope, para 2, second to last sentence: Perhaps 'impact is a minimal as possible' could be 'impact is minimized. Scope, para 3, second sentence: The word choices here are confusing. To start, 'sensitive and vulnerable metrics', perhaps you mean metrics that contain sensitive data or metrics that expose vulnerabilities? If so, then the requirement would be to keep that information secure so it cannot be used to attack the systems being monitored. I'm not sure what 'results are trustworthy' means, but perhaps the results need integrity protection? Work Items, last para: Implementations are required except where they aren't. Is it possible to better specify when an implementation might not be required? Or better yet, just require implementations to exist. Surely one implementation isn't a hard bar to meet? Relationships: If there is a concern about exposure of data collected in metrics, is there a security area relationship required?