Precision Availability Metrics for Services Governed by Service Level Objectives (SLOs)
draft-ietf-ippm-pam-09

Received changes through RFC Editor sync (changed state to RFC, created became rfc relationship between draft-ietf-ippm-pam and RFC 9544, changed IESG state to RFC Published)

[Ballot comment]
- Thanks for this document

- One thing to note, because of the need to expand "RFC XXXX", this document effectively has to go into a cluster along with draft-ietf-teas-ietf-network-slices-25, even though the reference is only informative.

[Ballot comment]
- Thanks for this document

- One thing to note, because of the need to expand "RFC XXXX", this document effectively has to go into a cluster along with draft-ietf-teas-ietf-network-slices-25.

[Ballot comment]
Thanks for working on this specification.

Even though I get the SLO idea, but this specification says

  Precision Availability Metrics (PAM), aimed at capturing end-to-end service levels for a flow, specifically the degree to which flows comply with the SLOs that are in effect.

and I am not sure I get how PAM executes on the end-to-end service levels or what does the "end-to-end" mean in this context. I think it would good to clarify that aspect in this document.

[Ballot comment]
Thanks to the authors for the document.

Thanks as well to Benson Muitte for his internet directorate review at:
https://datatracker.ietf.org/doc/review-ietf-ippm-pam-08-intdir-telechat-muite-2023-11-22/ (it would be nice if the authors had replied to his review).

[Ballot comment]
** Section 3.1.  Consider defining or providing an explanation of “optimal level threshold” and “critical threshold” in the definitions of VI and SVI.  I’m assuming that these are two configured threshold levels.

** Section 3.1. 
  Beyond accounting for violated intervals, it is sometimes beneficial
  to maintain counts of packets for which a performance threshold is
  violated. 

No question of the utility of counting packets.  However, shouldn’t there be a count of flows too?  Section 1 said “This specification introduces a new set of metrics, Precision Availability Metrics (PAM), aimed at capturing end-to-end service levels for a flow, specifically the degree to which flows comply with the SLOs that are in effect.”

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

IANA has completed its review of draft-ietf-ippm-pam-07, which is currently in Last Call, and has the following comments:

We understand that this document doesn't require any registry actions.

While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, we do not object.

If this assessment is not accurate, please respond as soon as possible.

For definitions of IANA review states, please see:

https://datatracker.ietf.org/help/state/draft/iana-review

Thank you,

David Dong
IANA Services Sr. Specialist

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

IANA has completed its review of draft-ietf-ippm-pam-07, which is currently in Last Call, and has the following comments:

We understand that this document doesn't require any registry actions.

While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, we do not object.

If this assessment is not accurate, please respond as soon as possible.

For definitions of IANA review states, please see:

https://datatracker.ietf.org/help/state/draft/iana-review

Thank you,

David Dong
IANA Services Sr. Specialist

Request for Last Call review by GENART Completed: Ready with Issues. Reviewer: Peter Yee. Sent review to list. Submission of review completed at an earlier date.

The following Last Call announcement was sent out (ends 2023-10-20):

From: The IESG
To: IETF-Announce
CC: draft-ietf-ippm-pam@ietf.org, ippm-chairs@ietf.org, ippm@ietf.org, martin.h.duke@gmail.com, tpauly@apple.com
Reply-To: last-call@ietf.org
Sender:
Subject: Last Call:  (Precision Availability Metrics for Services Governed by Service Level Objectives (SLOs)) to Informational RFC


The IESG has received a request from the IP Performance Measurement WG (ippm)
to consider the following document: - 'Precision Availability Metrics for
Services Governed by Service Level
  Objectives (SLOs)'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2023-10-20. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


  This document defines a set of metrics for networking services with
  performance requirements expressed as Service Level Objectives (SLO).
  These metrics, referred to as Precision Availability Metrics (PAM),
  are useful for defining and monitoring SLOs.  For example, PAM can be
  used by providers and/or customers of an RFC XXXX Network Slice
  Service to assess whether the service is provided in compliance with
  its defined SLOs.

  Note to the RFC Editor: Please update "RFC XXXX Network Slice" with
  the RFC number assigned to draft-ietf-teas-ietf-network-slices.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-ippm-pam/



No IPR declarations have been submitted directly on this I-D.

The following Last Call announcement was sent out (ends 2023-10-12):

From: The IESG
To: IETF-Announce
CC: draft-ietf-ippm-pam@ietf.org, ippm-chairs@ietf.org, ippm@ietf.org, martin.h.duke@gmail.com, tpauly@apple.com
Reply-To: last-call@ietf.org
Sender:
Subject: Last Call:  (Precision Availability Metrics for Services Governed by Service Level Objectives (SLOs)) to Informational RFC


The IESG has received a request from the IP Performance Measurement WG (ippm)
to consider the following document: - 'Precision Availability Metrics for
Services Governed by Service Level
  Objectives (SLOs)'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2023-10-12. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


  This document defines a set of metrics for networking services with
  performance requirements expressed as Service Level Objectives (SLO).
  These metrics, referred to as Precision Availability Metrics (PAM),
  are useful for defining and monitoring SLOs.  For example, PAM can be
  used by providers and/or customers of an IETF Network Slice Service
  to assess whether the service is provided in compliance with its
  defined SLOs.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-ippm-pam/



No IPR declarations have been submitted directly on this I-D.

# Document Shepherd Write-Up for Group Documents

*This version is dated 4 July 2022.*

Thank you for your service as a document shepherd. Among the responsibilities is
answering the questions in this write-up to give helpful context to Last Call
and Internet Engineering Steering Group ([IESG][1]) reviewers, and your
diligence in completing it is appreciated. The full role of the shepherd is
further described in [RFC 4858][2]. You will need the cooperation of the authors
and editors to complete these checks.

Note that some numbered items contain multiple related questions; please be sure
to answer all of them.

## Document History

1. Does the working group (WG) consensus represent the strong concurrence of a
  few individuals, with others being silent, or did it reach broad agreement?

This document got good review across the WG, with positive input from WG
members who work in various areas. I believe that this had around the normal
amount of engagement for IPPM, and we reached good consensus on the document.

2. Was there controversy about particular points, or were there decisions where
  the consensus was particularly rough?

No controversies or rough consensus. The reviews and issues pointed out were
resolved with reviewers being happy with the changes.

3. Has anyone threatened an appeal or otherwise indicated extreme discontent? If
  so, please summarize the areas of conflict in separate email messages to the
  responsible Area Director. (It should be in a separate email because this
  questionnaire is publicly available.)

No appeals or extreme discontent.

4. For protocol documents, are there existing implementations of the contents of
  the document? Have a significant number of potential implementers indicated
  plans to implement? Are any existing implementations reported somewhere,
  either in the document itself (as [RFC 7942][3] recommends) or elsewhere
  (where)?

While the draft doesn't document implementations, I believe the authors have at
least one implementation that uses the metrics.

## Additional Reviews

5. Do the contents of this document closely interact with technologies in other
  IETF working groups or external organizations, and would it therefore benefit
  from their review? Have those reviews occurred? If yes, describe which
  reviews took place.

I don't think there is any notable overlap here that requires outside review
beyond the IPPM participants.

6. Describe how the document meets any required formal expert review criteria,
  such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

No formal expert review needed.

7. If the document contains a YANG module, has the final version of the module
  been checked with any of the [recommended validation tools][4] for syntax and
  formatting validation? If there are any resulting errors or warnings, what is
  the justification for not fixing them at this time? Does the YANG module
  comply with the Network Management Datastore Architecture (NMDA) as specified
  in [RFC 8342][5]?

No YANG model.

8. Describe reviews and automated checks performed to validate sections of the
  final version of the document written in a formal language, such as XML code,
  BNF rules, MIB definitions, CBOR's CDDL, etc.

No formal language used.

## Document Shepherd Checks

9. Based on the shepherd's review of the document, is it their opinion that this
  document is needed, clearly written, complete, correctly designed, and ready
  to be handed off to the responsible Area Director?

Yes, I think the document is ready to be handed off.

10. Several IETF Areas have assembled [lists of common issues that their
    reviewers encounter][6]. For which areas have such issues been identified
    and addressed? For which does this still need to happen in subsequent
    reviews?

No relevant issues identified yet.

11. What type of RFC publication is being requested on the IETF stream ([Best
    Current Practice][12], [Proposed Standard, Internet Standard][13],
    [Informational, Experimental or Historic][14])? Why is this the proper type
    of RFC? Do all Datatracker state attributes correctly reflect this intent?

The document is now listed as Informational, having previously been Proposed
Standard. This is due to it describing an overall technique without detailed
metric implementation advice.

12. Have reasonable efforts been made to remind all authors of the intellectual
    property rights (IPR) disclosure obligations described in [BCP 79][7]? To
    the best of your knowledge, have all required disclosures been filed? If
    not, explain why. If yes, summarize any relevant discussion, including links
    to publicly-available messages when applicable.

I reached out to all authors and confirmed that they were not aware of any
relevant IPR.

13. Has each author, editor, and contributor shown their willingness to be
    listed as such? If the total number of authors and editors on the front page
    is greater than five, please provide a justification.

Yes, all authors are willing to be listed. The number of authors is 6, slightly
above the normal 5 threshold. As shepherd and chair, I would be happy to see
this move to an editor model, but haven't gotten agreement from the authors on
that yet. I will note that this is a merged document (came from two other
documents with different author sets), which is why it has 6 authors.

14. Document any remaining I-D nits in this document. Simply running the [idnits
    tool][8] is not enough; please review the ["Content Guidelines" on
    authors.ietf.org][15]. (Also note that the current idnits tool generates
    some incorrect warnings; a rewrite is underway.)

No I-D nits found.

15. Should any informative references be normative or vice-versa? See the [IESG
    Statement on Normative and Informative References][16].

The document doesn't have any normative references (it doesn't offer specific
normative language). I don't see particular informative references that should
be elevated, unless ADs feel otherwise.

16. List any normative references that are not freely available to anyone. Did
    the community have sufficient access to review any such normative
    references?

None

17. Are there any normative downward references (see [RFC 3967][9] and [BCP
    97][10]) that are not already listed in the [DOWNREF registry][17]? If so,
    list them.

None

18. Are there normative references to documents that are not ready to be
    submitted to the IESG for publication or are otherwise in an unclear state?
    If so, what is the plan for their completion?

None

19. Will publication of this document change the status of any existing RFCs? If
    so, does the Datatracker metadata correctly reflect this and are those RFCs
    listed on the title page, in the abstract, and discussed in the
    introduction? If not, explain why and point to the part of the document
    where the relationship of this document to these other RFCs is discussed.

No updates or change of status.

20. Describe the document shepherd's review of the IANA considerations section,
    especially with regard to its consistency with the body of the document.
    Confirm that all aspects of the document requiring IANA assignments are
    associated with the appropriate reservations in IANA registries. Confirm
    that any referenced IANA registries have been clearly identified. Confirm
    that each newly created IANA registry specifies its initial contents,
    allocations procedures, and a reasonable name (see [RFC 8126][11]).

No IANA implications.

21. List any new IANA registries that require Designated Expert Review for
    future allocations. Are the instructions to the Designated Expert clear?
    Please include suggestions of designated experts, if appropriate.

No IANA implications.

[1]: https://www.ietf.org/about/groups/iesg/
[2]: https://www.rfc-editor.org/rfc/rfc4858.html
[3]: https://www.rfc-editor.org/rfc/rfc7942.html
[4]: https://wiki.ietf.org/group/ops/yang-review-tools
[5]: https://www.rfc-editor.org/rfc/rfc8342.html
[6]: https://wiki.ietf.org/group/iesg/ExpertTopics
[7]: https://www.rfc-editor.org/info/bcp79
[8]: https://www.ietf.org/tools/idnits/
[9]: https://www.rfc-editor.org/rfc/rfc3967.html
[10]: https://www.rfc-editor.org/info/bcp97
[11]: https://www.rfc-editor.org/rfc/rfc8126.html
[12]: https://www.rfc-editor.org/rfc/rfc2026.html#section-5
[13]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.1
[14]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.2
[15]: https://authors.ietf.org/en/content-guidelines-overview
[16]: https://www.ietf.org/about/groups/iesg/statements/normative-informative-references/
[17]: https://datatracker.ietf.org/doc/downref/

# Document Shepherd Write-Up for Group Documents

*This version is dated 4 July 2022.*

Thank you for your service as a document shepherd. Among the responsibilities is
answering the questions in this write-up to give helpful context to Last Call
and Internet Engineering Steering Group ([IESG][1]) reviewers, and your
diligence in completing it is appreciated. The full role of the shepherd is
further described in [RFC 4858][2]. You will need the cooperation of the authors
and editors to complete these checks.

Note that some numbered items contain multiple related questions; please be sure
to answer all of them.

## Document History

1. Does the working group (WG) consensus represent the strong concurrence of a
  few individuals, with others being silent, or did it reach broad agreement?

This document got good review across the WG, with positive input from WG
members who work in various areas. I believe that this had around the normal
amount of engagement for IPPM, and we reached good consensus on the document.

2. Was there controversy about particular points, or were there decisions where
  the consensus was particularly rough?

No controversies or rough consensus. The reviews and issues pointed out were
resolved with reviewers being happy with the changes.

3. Has anyone threatened an appeal or otherwise indicated extreme discontent? If
  so, please summarize the areas of conflict in separate email messages to the
  responsible Area Director. (It should be in a separate email because this
  questionnaire is publicly available.)

No appeals or extreme discontent.

4. For protocol documents, are there existing implementations of the contents of
  the document? Have a significant number of potential implementers indicated
  plans to implement? Are any existing implementations reported somewhere,
  either in the document itself (as [RFC 7942][3] recommends) or elsewhere
  (where)?

While the draft doesn't document implementations, I believe the authors have at
least one implementation that uses the metrics.

## Additional Reviews

5. Do the contents of this document closely interact with technologies in other
  IETF working groups or external organizations, and would it therefore benefit
  from their review? Have those reviews occurred? If yes, describe which
  reviews took place.

I don't think there is any notable overlap here that requires outside review
beyond the IPPM participants.

6. Describe how the document meets any required formal expert review criteria,
  such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

No formal expert review needed.

7. If the document contains a YANG module, has the final version of the module
  been checked with any of the [recommended validation tools][4] for syntax and
  formatting validation? If there are any resulting errors or warnings, what is
  the justification for not fixing them at this time? Does the YANG module
  comply with the Network Management Datastore Architecture (NMDA) as specified
  in [RFC 8342][5]?

No YANG model.

8. Describe reviews and automated checks performed to validate sections of the
  final version of the document written in a formal language, such as XML code,
  BNF rules, MIB definitions, CBOR's CDDL, etc.

No formal language used.

## Document Shepherd Checks

9. Based on the shepherd's review of the document, is it their opinion that this
  document is needed, clearly written, complete, correctly designed, and ready
  to be handed off to the responsible Area Director?

Yes, I think the document is ready to be handed off.

10. Several IETF Areas have assembled [lists of common issues that their
    reviewers encounter][6]. For which areas have such issues been identified
    and addressed? For which does this still need to happen in subsequent
    reviews?

No relevant issues identified yet.

11. What type of RFC publication is being requested on the IETF stream ([Best
    Current Practice][12], [Proposed Standard, Internet Standard][13],
    [Informational, Experimental or Historic][14])? Why is this the proper type
    of RFC? Do all Datatracker state attributes correctly reflect this intent?

The document is now listed as Informational, having previously been Proposed
Standard. This is due to it describing an overall technique without detailed
metric implementation advice.

12. Have reasonable efforts been made to remind all authors of the intellectual
    property rights (IPR) disclosure obligations described in [BCP 79][7]? To
    the best of your knowledge, have all required disclosures been filed? If
    not, explain why. If yes, summarize any relevant discussion, including links
    to publicly-available messages when applicable.

I reached out to all authors and confirmed that they were not aware of any
relevant IPR.

13. Has each author, editor, and contributor shown their willingness to be
    listed as such? If the total number of authors and editors on the front page
    is greater than five, please provide a justification.

Yes, all authors are willing to be listed. The number of authors is 6, slightly
above the normal 5 threshold. As shepherd and chair, I would be happy to see
this move to an editor model, but haven't gotten agreement from the authors on
that yet.

14. Document any remaining I-D nits in this document. Simply running the [idnits
    tool][8] is not enough; please review the ["Content Guidelines" on
    authors.ietf.org][15]. (Also note that the current idnits tool generates
    some incorrect warnings; a rewrite is underway.)

No I-D nits found.

15. Should any informative references be normative or vice-versa? See the [IESG
    Statement on Normative and Informative References][16].

The document doesn't have any normative references (it doesn't offer specific
normative language). I don't see particular informative references that should
be elevated, unless ADs feel otherwise.

16. List any normative references that are not freely available to anyone. Did
    the community have sufficient access to review any such normative
    references?

None

17. Are there any normative downward references (see [RFC 3967][9] and [BCP
    97][10]) that are not already listed in the [DOWNREF registry][17]? If so,
    list them.

None

18. Are there normative references to documents that are not ready to be
    submitted to the IESG for publication or are otherwise in an unclear state?
    If so, what is the plan for their completion?

None

19. Will publication of this document change the status of any existing RFCs? If
    so, does the Datatracker metadata correctly reflect this and are those RFCs
    listed on the title page, in the abstract, and discussed in the
    introduction? If not, explain why and point to the part of the document
    where the relationship of this document to these other RFCs is discussed.

No updates or change of status.

20. Describe the document shepherd's review of the IANA considerations section,
    especially with regard to its consistency with the body of the document.
    Confirm that all aspects of the document requiring IANA assignments are
    associated with the appropriate reservations in IANA registries. Confirm
    that any referenced IANA registries have been clearly identified. Confirm
    that each newly created IANA registry specifies its initial contents,
    allocations procedures, and a reasonable name (see [RFC 8126][11]).

No IANA implications.

21. List any new IANA registries that require Designated Expert Review for
    future allocations. Are the instructions to the Designated Expert clear?
    Please include suggestions of designated experts, if appropriate.

No IANA implications.

[1]: https://www.ietf.org/about/groups/iesg/
[2]: https://www.rfc-editor.org/rfc/rfc4858.html
[3]: https://www.rfc-editor.org/rfc/rfc7942.html
[4]: https://wiki.ietf.org/group/ops/yang-review-tools
[5]: https://www.rfc-editor.org/rfc/rfc8342.html
[6]: https://wiki.ietf.org/group/iesg/ExpertTopics
[7]: https://www.rfc-editor.org/info/bcp79
[8]: https://www.ietf.org/tools/idnits/
[9]: https://www.rfc-editor.org/rfc/rfc3967.html
[10]: https://www.rfc-editor.org/info/bcp97
[11]: https://www.rfc-editor.org/rfc/rfc8126.html
[12]: https://www.rfc-editor.org/rfc/rfc2026.html#section-5
[13]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.1
[14]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.2
[15]: https://authors.ietf.org/en/content-guidelines-overview
[16]: https://www.ietf.org/about/groups/iesg/statements/normative-informative-references/
[17]: https://datatracker.ietf.org/doc/downref/

# Document Shepherd Write-Up for Group Documents

*This version is dated 4 July 2022.*

Thank you for your service as a document shepherd. Among the responsibilities is
answering the questions in this write-up to give helpful context to Last Call
and Internet Engineering Steering Group ([IESG][1]) reviewers, and your
diligence in completing it is appreciated. The full role of the shepherd is
further described in [RFC 4858][2]. You will need the cooperation of the authors
and editors to complete these checks.

Note that some numbered items contain multiple related questions; please be sure
to answer all of them.

## Document History

1. Does the working group (WG) consensus represent the strong concurrence of a
  few individuals, with others being silent, or did it reach broad agreement?

This document got good review across the WG, with positive input from WG
members who work in various areas. I believe that this had around the normal
amount of engagement for IPPM, and we reached good consensus on the document.

2. Was there controversy about particular points, or were there decisions where
  the consensus was particularly rough?

No controversies or rough consensus. The reviews and issues pointed out were
resolved with reviewers being happy with the changes.

3. Has anyone threatened an appeal or otherwise indicated extreme discontent? If
  so, please summarize the areas of conflict in separate email messages to the
  responsible Area Director. (It should be in a separate email because this
  questionnaire is publicly available.)

No appeals or extreme discontent.

4. For protocol documents, are there existing implementations of the contents of
  the document? Have a significant number of potential implementers indicated
  plans to implement? Are any existing implementations reported somewhere,
  either in the document itself (as [RFC 7942][3] recommends) or elsewhere
  (where)?

While the draft doesn't document implementations, I believe the authors have at
least one implementation that uses the metrics.

## Additional Reviews

5. Do the contents of this document closely interact with technologies in other
  IETF working groups or external organizations, and would it therefore benefit
  from their review? Have those reviews occurred? If yes, describe which
  reviews took place.

I don't think there is any notable overlap here that requires outside review
beyond the IPPM participants.

6. Describe how the document meets any required formal expert review criteria,
  such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

No formal expert review needed.

7. If the document contains a YANG module, has the final version of the module
  been checked with any of the [recommended validation tools][4] for syntax and
  formatting validation? If there are any resulting errors or warnings, what is
  the justification for not fixing them at this time? Does the YANG module
  comply with the Network Management Datastore Architecture (NMDA) as specified
  in [RFC 8342][5]?

No YANG model.

8. Describe reviews and automated checks performed to validate sections of the
  final version of the document written in a formal language, such as XML code,
  BNF rules, MIB definitions, CBOR's CDDL, etc.

No formal language used.

## Document Shepherd Checks

9. Based on the shepherd's review of the document, is it their opinion that this
  document is needed, clearly written, complete, correctly designed, and ready
  to be handed off to the responsible Area Director?

Yes, I think the document is ready to be handed off.

10. Several IETF Areas have assembled [lists of common issues that their
    reviewers encounter][6]. For which areas have such issues been identified
    and addressed? For which does this still need to happen in subsequent
    reviews?

No relevant issues identified yet.

11. What type of RFC publication is being requested on the IETF stream ([Best
    Current Practice][12], [Proposed Standard, Internet Standard][13],
    [Informational, Experimental or Historic][14])? Why is this the proper type
    of RFC? Do all Datatracker state attributes correctly reflect this intent?

Proposed Standard. That is the common approach for IPPM to define new metrics
or definitions of metrics.

12. Have reasonable efforts been made to remind all authors of the intellectual
    property rights (IPR) disclosure obligations described in [BCP 79][7]? To
    the best of your knowledge, have all required disclosures been filed? If
    not, explain why. If yes, summarize any relevant discussion, including links
    to publicly-available messages when applicable.

I reached out to all authors and confirmed that they were not aware of any
relevant IPR.

13. Has each author, editor, and contributor shown their willingness to be
    listed as such? If the total number of authors and editors on the front page
    is greater than five, please provide a justification.

Yes, all authors are willing to be listed. The number of authors is 6, slightly
above the normal 5 threshold, but I don't think that's a major issue here.

14. Document any remaining I-D nits in this document. Simply running the [idnits
    tool][8] is not enough; please review the ["Content Guidelines" on
    authors.ietf.org][15]. (Also note that the current idnits tool generates
    some incorrect warnings; a rewrite is underway.)

No I-D nits found.

15. Should any informative references be normative or vice-versa? See the [IESG
    Statement on Normative and Informative References][16].

The document doesn't have any normative references (it doesn't offer specific
normative language). I don't see particular informative references that should
be elevated, unless ADs feel otherwise.

16. List any normative references that are not freely available to anyone. Did
    the community have sufficient access to review any such normative
    references?

None

17. Are there any normative downward references (see [RFC 3967][9] and [BCP
    97][10]) that are not already listed in the [DOWNREF registry][17]? If so,
    list them.

None

18. Are there normative references to documents that are not ready to be
    submitted to the IESG for publication or are otherwise in an unclear state?
    If so, what is the plan for their completion?

None

19. Will publication of this document change the status of any existing RFCs? If
    so, does the Datatracker metadata correctly reflect this and are those RFCs
    listed on the title page, in the abstract, and discussed in the
    introduction? If not, explain why and point to the part of the document
    where the relationship of this document to these other RFCs is discussed.

No updates or change of status.

20. Describe the document shepherd's review of the IANA considerations section,
    especially with regard to its consistency with the body of the document.
    Confirm that all aspects of the document requiring IANA assignments are
    associated with the appropriate reservations in IANA registries. Confirm
    that any referenced IANA registries have been clearly identified. Confirm
    that each newly created IANA registry specifies its initial contents,
    allocations procedures, and a reasonable name (see [RFC 8126][11]).

No IANA implications.

21. List any new IANA registries that require Designated Expert Review for
    future allocations. Are the instructions to the Designated Expert clear?
    Please include suggestions of designated experts, if appropriate.

No IANA implications.

[1]: https://www.ietf.org/about/groups/iesg/
[2]: https://www.rfc-editor.org/rfc/rfc4858.html
[3]: https://www.rfc-editor.org/rfc/rfc7942.html
[4]: https://wiki.ietf.org/group/ops/yang-review-tools
[5]: https://www.rfc-editor.org/rfc/rfc8342.html
[6]: https://wiki.ietf.org/group/iesg/ExpertTopics
[7]: https://www.rfc-editor.org/info/bcp79
[8]: https://www.ietf.org/tools/idnits/
[9]: https://www.rfc-editor.org/rfc/rfc3967.html
[10]: https://www.rfc-editor.org/info/bcp97
[11]: https://www.rfc-editor.org/rfc/rfc8126.html
[12]: https://www.rfc-editor.org/rfc/rfc2026.html#section-5
[13]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.1
[14]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.2
[15]: https://authors.ietf.org/en/content-guidelines-overview
[16]: https://www.ietf.org/about/groups/iesg/statements/normative-informative-references/
[17]: https://datatracker.ietf.org/doc/downref/

# Document Shepherd Write-Up for Group Documents

*This version is dated 4 July 2022.*

Thank you for your service as a document shepherd. Among the responsibilities is
answering the questions in this write-up to give helpful context to Last Call
and Internet Engineering Steering Group ([IESG][1]) reviewers, and your
diligence in completing it is appreciated. The full role of the shepherd is
further described in [RFC 4858][2]. You will need the cooperation of the authors
and editors to complete these checks.

Note that some numbered items contain multiple related questions; please be sure
to answer all of them.

## Document History

1. Does the working group (WG) consensus represent the strong concurrence of a
  few individuals, with others being silent, or did it reach broad agreement?

This document got good review across the WG, with positive input from WG
members who work in various areas. I believe that this had around the normal
amount of engagement for IPPM, and we reached good consensus on the document.

2. Was there controversy about particular points, or were there decisions where
  the consensus was particularly rough?

No controversies or rough consensus. The reviews and issues pointed out were
resolved with reviewers being happy with the changes.

3. Has anyone threatened an appeal or otherwise indicated extreme discontent? If
  so, please summarize the areas of conflict in separate email messages to the
  responsible Area Director. (It should be in a separate email because this
  questionnaire is publicly available.)

No appeals or extreme discontent.

4. For protocol documents, are there existing implementations of the contents of
  the document? Have a significant number of potential implementers indicated
  plans to implement? Are any existing implementations reported somewhere,
  either in the document itself (as [RFC 7942][3] recommends) or elsewhere
  (where)?

While the draft doesn't document implementations, I believe the authors have at
least one implementation that uses the metrics.

## Additional Reviews

5. Do the contents of this document closely interact with technologies in other
  IETF working groups or external organizations, and would it therefore benefit
  from their review? Have those reviews occurred? If yes, describe which
  reviews took place.

I don't think there is any notable overlap here that requires outside review
beyond the IPPM participants.

6. Describe how the document meets any required formal expert review criteria,
  such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

No formal expert review needed.

7. If the document contains a YANG module, has the final version of the module
  been checked with any of the [recommended validation tools][4] for syntax and
  formatting validation? If there are any resulting errors or warnings, what is
  the justification for not fixing them at this time? Does the YANG module
  comply with the Network Management Datastore Architecture (NMDA) as specified
  in [RFC 8342][5]?

No YANG model.

8. Describe reviews and automated checks performed to validate sections of the
  final version of the document written in a formal language, such as XML code,
  BNF rules, MIB definitions, CBOR's CDDL, etc.

No formal language used.

## Document Shepherd Checks

9. Based on the shepherd's review of the document, is it their opinion that this
  document is needed, clearly written, complete, correctly designed, and ready
  to be handed off to the responsible Area Director?

Yes, I think the document is ready to be handed off.

10. Several IETF Areas have assembled [lists of common issues that their
    reviewers encounter][6]. For which areas have such issues been identified
    and addressed? For which does this still need to happen in subsequent
    reviews?

No relevant issues identified yet.

11. What type of RFC publication is being requested on the IETF stream ([Best
    Current Practice][12], [Proposed Standard, Internet Standard][13],
    [Informational, Experimental or Historic][14])? Why is this the proper type
    of RFC? Do all Datatracker state attributes correctly reflect this intent?

Proposed Standard. That is the common approach for IPPM to define new metrics
or definitions of metrics.

12. Have reasonable efforts been made to remind all authors of the intellectual
    property rights (IPR) disclosure obligations described in [BCP 79][7]? To
    the best of your knowledge, have all required disclosures been filed? If
    not, explain why. If yes, summarize any relevant discussion, including links
    to publicly-available messages when applicable.

I reached out to all authors and confirmed that they were not aware of any
relevant IPR.

13. Has each author, editor, and contributor shown their willingness to be
    listed as such? If the total number of authors and editors on the front page
    is greater than five, please provide a justification.

Yes, all authors are willing to be listed. The number of authors is 6, slightly
above the normal 5 threshold, but I don't think that's a major issue here.

14. Document any remaining I-D nits in this document. Simply running the [idnits
    tool][8] is not enough; please review the ["Content Guidelines" on
    authors.ietf.org][15]. (Also note that the current idnits tool generates
    some incorrect warnings; a rewrite is underway.)

No I-D nits found.

15. Should any informative references be normative or vice-versa? See the [IESG
    Statement on Normative and Informative References][16].

The document doesn't have any normative references (it doesn't offer specific
normative language). I don't see particular informative references that should
be elevated, unless ADs feel otherwise.

16. List any normative references that are not freely available to anyone. Did
    the community have sufficient access to review any such normative
    references?

None

17. Are there any normative downward references (see [RFC 3967][9] and [BCP
    97][10]) that are not already listed in the [DOWNREF registry][17]? If so,
    list them.

None

18. Are there normative references to documents that are not ready to be
    submitted to the IESG for publication or are otherwise in an unclear state?
    If so, what is the plan for their completion?

None

19. Will publication of this document change the status of any existing RFCs? If
    so, does the Datatracker metadata correctly reflect this and are those RFCs
    listed on the title page, in the abstract, and discussed in the
    introduction? If not, explain why and point to the part of the document
    where the relationship of this document to these other RFCs is discussed.

No updates or change of status.

20. Describe the document shepherd's review of the IANA considerations section,
    especially with regard to its consistency with the body of the document.
    Confirm that all aspects of the document requiring IANA assignments are
    associated with the appropriate reservations in IANA registries. Confirm
    that any referenced IANA registries have been clearly identified. Confirm
    that each newly created IANA registry specifies its initial contents,
    allocations procedures, and a reasonable name (see [RFC 8126][11]).

No IANA implications.

21. List any new IANA registries that require Designated Expert Review for
    future allocations. Are the instructions to the Designated Expert clear?
    Please include suggestions of designated experts, if appropriate.

No IANA implications.

[1]: https://www.ietf.org/about/groups/iesg/
[2]: https://www.rfc-editor.org/rfc/rfc4858.html
[3]: https://www.rfc-editor.org/rfc/rfc7942.html
[4]: https://wiki.ietf.org/group/ops/yang-review-tools
[5]: https://www.rfc-editor.org/rfc/rfc8342.html
[6]: https://wiki.ietf.org/group/iesg/ExpertTopics
[7]: https://www.rfc-editor.org/info/bcp79
[8]: https://www.ietf.org/tools/idnits/
[9]: https://www.rfc-editor.org/rfc/rfc3967.html
[10]: https://www.rfc-editor.org/info/bcp97
[11]: https://www.rfc-editor.org/rfc/rfc8126.html
[12]: https://www.rfc-editor.org/rfc/rfc2026.html#section-5
[13]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.1
[14]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.2
[15]: https://authors.ietf.org/en/content-guidelines-overview
[16]: https://www.ietf.org/about/groups/iesg/statements/normative-informative-references/
[17]: https://datatracker.ietf.org/doc/downref/