Quality of Outcome (QoO)
draft-ietf-ippm-qoo-11

[Ballot comment]
Thank you producing revision -10 and the responses to my DISCUSS and COMMENT topics.

---

## Some non-blocking COMMENT points/nits:

## S1 - Comment
“This document defines a “minimum viable framework,” and later the “general framework” - please update the text to explain which is intended.
- Overall I found the introduction and section 2 were not clear and likely were far too verbose, which seemed to mask the salient points that were being introduced. A more concise introduction focussed on the key points would be easier to understand. There is already a detailed overview in section 2.

## S1 - Comment
There is some repeated discussion across sections 1-4. I found that this obscured, rather than provided background and these section would certainly benefit from more review and revision to condense the material. Please consider if they could be more focussed on the essential background points. TR-452.1 is mentioned, but this is not an IETF standard, please add a brief description of the key relevant details relating to the present specification.


## S1 - Comment
The statement: " Quality
  Attenuation meets most of the requirements for a meaningful network
  quality score set out in Section 4.1: it is composable, captures the
  ability of a network to satisfy application requirements, and can be
  compared to a variety of application needs."
- This text reads like a very strong statement for an IETF document, and more like a sales-pitch. If there was strong consensus within the WG on this item, please provide some reference - the Shepherd report suggests not - in which case, please revise this statement, to simple facts. Similarly, "owever, educated assessments of expected outcomes remain achievable". Similarly asserting optimality is a very strong statement, much more than stating that this can be used to "optimise", see: "one for optimal application performance". Similarly, this looks like a significant over-statement for a standards document: " If these requirements are met, network operators can monitor and test their network and understand where the true bottlenecks are, regardless of network technology."
“The novelty of Quality Attenuation lies in its unified treatment of latency and loss within a single distributional framework.“
Please remove the word novelty, this is not applicable to an SDO document. Please simply state:
“The Quality Attenuation framework provides unified treatment of latency and loss within a single distributional framework.“
Please consider changing:
“  Importantly, the framework does not enforce a minimum sample count.”
to “This framework does not enforce a minimum sample count.”
- Please do check the entire document carefully to avoid inadvertently making marketing statements, that might be acceptable in a published paper, but are inappropriate for an RFC.

## S4 - Comment
Sections 4 and 5 discuss topics around requirements but do not actually provide clear requirements. There are requirements in the related specification of TR-452.1, but the current text included in the document seems to motivate the idea rather than specify things.  At least this needs to be explained what are these requirements for? - specifying metrics? the framework itself? something else?

##  - Comment
The content of these paragraphs are motivational. Ought this to be omitted or moved to section 1 and reduced, it does not appear to be part of the defined framework.

## S4 Term: bandwidth - Comment
The term bandwidth, rather than “available capacity” seems wrong in the context of an end-to-end Internet path.

## S4.3 - Comment
I found the following statement rather odd: “Note that developers of similar applications may have arrived at different figures.” This may be true how does it impact the framework being defined and the need for standards?

##  S5 - Comment.
I was very surprised to see the statement on “novelty” in this sentence. The IETF does not normally make claims on the novelty of its protocols, rather specifications and other documents are normally based on their utility. Please re-consider this statement:
    “The novelty of Quality Attenuation lies in
  its unified treatment of latency and loss within a single
  distributional framework...”

## S 5.2 - Comment
I am unsure what to understand by “Still, it might be beneficial for future
  standardization activities to converge on a fixed set of general
  percentiles or for specific applications / application classes to
  make QoO measurements between different networks or providers more
  comparable.”
- Please explain in text.

## S 5.2 - Comment
Why is this not “MUST”:
“Therefore, the network requirements must include a minimum throughput requirement.”

## S 5.2 - Comment
Why is this not “MUST”:
“Whether the requirements are one-way or two-way must be specified.”

## S 5.2 - Comment
This text seems to require additional work to render it suitable for an RFC:
“To that aim, first recall the key design goal of establishing a
  quantifiable distance between optimal and unacceptable network
  conditions, thereby enabling an objective assessment of relative
  quality.  Accordingly, the requirements specification is extended to
  define both the network performance required for achieving optimal
  application performance and the lower network performance threshold
  below which the application performance is considered unacceptable.”
⁃ Please avoid the style of a technical paper, and simply state what is required, or must be clarified,

## S 5.2 - Comment
The following mixes the terms bandwidth and capacity. For an Internet path, I think the “available capacity” is the term that is needed:
“Applications do of course have throughput requirements, and thus a
  complete framework for application-level network quality must also
  take capacity into account.  Insufficient bandwidth may give
  unacceptable application outcomes without necessarily inducing a lot
  of latency or packet loss.  Therefore, the network requirements must
  include a minimum throughput requirement.”

## S5.2 - Comment
“A fully specified requirement can be thought of as specifying” - This is not good text for an RFC, please reword as either an example or a clear statement.

## S5.2 - Comment
Please reconsider this accademic argument and replace by specification text:
“ To that aim, first recall the key design goal of establishing..”

##
The requirement to specify if a measurement was “one-way” seems to be defined in two places. Please remove one.

## S6.1  - Comment
“Direct use of the approach described below in production scenarios is discouraged.”
- This needs to be explained.

## S6.1  - Comment
“application performance is optimal”
AND
“there are typically different levels of optimal performance”
- The IETF is not generally in a position to discuss optimality, please rephrase or justify how this optimum.

## S6.6 - Comment
“a study involving 25 participants tested the QoO
  framework in real-world settings [QoOUserStudy].  Participants used
  specially equipped routers in their homes for ten days, providing
  both network performance data and feedback through pre- and post-
  trial surveys.

  Participants found QoO scores more intuitive and actionable than
  traditional metrics (e.g., speed tests).  QoO directly aligned with
  their self-reported experiences, increasing trust and engagement.

  These results indicate that users find it easier to correlate QoO
  scores with real-world application performance than, for example, a
  speed test.  As such, QoO is expected to help... should be studied further, ”
It seems to me to be decidedly odd to make such a statement in a RFC based on a small sample size. I’d strongly urge removing this example and replacing this by more robust analysis (*or completely removing this text*)


## S7.1 - Comment
- Please provide a citable reference.

## S7.5  - Comment
- See earlier note that this is likely not a measure of bandwidth, but the path capacity.

## S8 - Comment
- Other IPPM documents have included detailed security considerations. Relevant security considerations need to be cited here.

## S8 - Comment
“If not properly rate-limited, this may inadvertently
  degrade services offered by a network or be exploited by malicious
  actors to launch DoS attacks.”
- This is very true, and the relevant security considerations need to be cited here.

## - Comment
Please provide supporting material to explain how the linear distance measure is to be used as a standard method.

## - Comment
How does your reporting work with continuous measurements? Which data points do you include in distributions or percentiles reported?  Last hour, day, week,
month, year, decade?

Best wishes,
Gorry Fairhurst

---

For completeness, I have included my previous DISCUSS topics here:

Thanks for providing draft-ietf-ippm-qoo and the many revisions to bring this document for consideration by the IESG.


Thanks to Joerg Ott for his TSVART review, which noted concerns with the document contents, readability, technical detail and precision. I would like to discuss how these comments can be addressed.

Please find below some blocking DISCUSS points (easy to address), some non-blocking COMMENT points/nits (replies would be appreciated even if only for my own education).

## DISCUSS (blocking)

As noted in https://datatracker.ietf.org/doc/statement-iesg-handling-ballot-positions-20220121/, a DISCUSS ballot is a request to have a discussion on the points below; I really think that the document would be improved with a change here, but can be convinced otherwise.

##  S5 - DISCUSS
- is it strictly necessary to make this normatively depend on TR-452, for instance this statement:
“However, in addition to measurement approaches
  standardized in the QED framework [TR-452.1], some relevant
  techniques are:”
- Please provide citable references for each technique!
- Can you instead make clear what the IETF-defined framework includes?

## S4 and S5 - DISCUSS
The document uses a lot of descriptive language to indicate requirements, but does not use any requirements language - RFC2119 language is acceptable in an Informational document, should the working group require this. At the moment I feel many of the requirements are just expressed as aspirations, and not sufficiently well-defined to allow someone to know whether a specific instance complies with the framework. I think this needs to be addressed, because without this the value of this being archived in the RFC series is doubtful.
For example, this seems a strong requirement: “  The framework is agnostic to traffic direction but mandates that measurements specify whether latency is one-way or round-trip.” There are further instances (see also my review comments).

## S5 - DISCUSS
“  When measurements are taken during periods of network load, the
  result naturally includes latency under load.  In scenarios such as
  passive monitoring of production traffic, capturing artificially
  loaded conditions may not always be feasible, whereas passively
  observing the actual network load may be possible.”
- This general statement could be true, but seems to lack any IETF statement around the safety of such measurements. Please update the text to provide this, for example by citing other IPPM work where this has been stated.
Similarly the later statement: “by generating artificial traffic to a level at least equivalent to the application traffic load“.

## S5.1 - DISCUSS
I suggest the essential “metadata” is under-specified (sometimes omitting units and details), please check carefully.
Section 5 discusses composable metrics. Please revise the text to address the TSVART review on the intention of this text.
For instance why is the following requirement not  at least a“SHOULD” :
“each measurement must be accompanied by the following metadata:”
⁃ In some cases. the stated metadata provides no units or expected accuracy for the listed elements. Please specify.
The section is followed  by text that says:
“These metadata elements are essential for interpreting the precision
  and reliability of the measurements.”
⁃ Is this really required, i.e. essential - making this a normative list of required meta data, or “useful”? - Please clarify the language.


##  S5.1 - DISCUSS
The requirement: “The repeatability of measurements under similar network conditions” does not seem to discuss path variation - either at L2, L3 (e.g. ECMP/Routing) and L4 - multi-path and cos mechanisms. Please clarify how this impact this requirement.

##  S5.1 - DISCUSS
Why is this not a requirement: “Implementers should document their precision assessment methodology”
- It would seem without this, the other requirements are moot, if I misunderstood, please help to understand and find better text.

## S6.1 - DISCUSS
Does a  measurement consider  load balancing along the forwarding path? (I think the referenced work in TR-452.1 is about a network segment, rather than an Internet Path, and hence this seems an important part of ensuring this is relevant as an Internet metric).

[Ballot discuss]
Thanks for providing draft-ietf-ippm-qoo and the many revisions to bring this document for consideration by the IESG.


Thanks to Joerg Ott for his TSVART review, which noted concerns with the document contents, readability, technical detail and precision. I would like to discuss how these comments can be addressed.

Please find below some blocking DISCUSS points (easy to address), some non-blocking COMMENT points/nits (replies would be appreciated even if only for my own education).

## DISCUSS (blocking)

As noted in https://datatracker.ietf.org/doc/statement-iesg-handling-ballot-positions-20220121/, a DISCUSS ballot is a request to have a discussion on the points below; I really think that the document would be improved with a change here, but can be convinced otherwise.

##  S5 - DISCUSS
- is it strictly necessary to make this normatively depend on TR-452, for instance this statement:
“However, in addition to measurement approaches
  standardized in the QED framework [TR-452.1], some relevant
  techniques are:”
- Please provide citable references for each technique!
- Can you instead make clear what the IETF-defined framework includes?

## S4 and S5 - DISCUSS
The document uses a lot of descriptive language to indicate requirements, but does not use any requirements language - RFC2119 language is acceptable in an Informational document, should the working group require this. At the moment I feel many of the requirements are just expressed as aspirations, and not sufficiently well-defined to allow someone to know whether a specific instance complies with the framework. I think this needs to be addressed, because without this the value of this being archived in the RFC series is doubtful.
For example, this seems a strong requirement: “  The framework is agnostic to traffic direction but mandates that measurements specify whether latency is one-way or round-trip.” There are further instances (see also my review comments).

## S5 - DISCUSS
“  When measurements are taken during periods of network load, the
  result naturally includes latency under load.  In scenarios such as
  passive monitoring of production traffic, capturing artificially
  loaded conditions may not always be feasible, whereas passively
  observing the actual network load may be possible.”
- This general statement could be true, but seems to lack any IETF statement around the safety of such measurements. Please update the text to provide this, for example by citing other IPPM work where this has been stated.
Similarly the later statement: “by generating artificial traffic to a level at least equivalent to the application traffic load“.

## S5.1 - DISCUSS
I suggest the essential “metadata” is under-specified (sometimes omitting units and details), please check carefully.
Section 5 discusses composable metrics. Please revise the text to address the TSVART review on the intention of this text.
For instance why is the following requirement not  at least a“SHOULD” :
“each measurement must be accompanied by the following metadata:”
⁃ In some cases. the stated metadata provides no units or expected accuracy for the listed elements. Please specify.
The section is followed  by text that says:
“These metadata elements are essential for interpreting the precision
  and reliability of the measurements.”
⁃ Is this really required, i.e. essential - making this a normative list of required meta data, or “useful”? - Please clarify the language.


##  S5.1 - DISCUSS
The requirement: “The repeatability of measurements under similar network conditions” does not seem to discuss path variation - either at L2, L3 (e.g. ECMP/Routing) and L4 - multi-path and cos mechanisms. Please clarify how this impact this requirement.

##  S5.1 - DISCUSS
Why is this not a requirement: “Implementers should document their precision assessment methodology”
- It would seem without this, the other requirements are moot, if I misunderstood, please help to understand and find better text.

## S6.1 - DISCUSS
Does a  measurement consider  load balancing along the forwarding path? (I think the referenced work in TR-452.1 is about a network segment, rather than an Internet Path, and hence this seems an important part of ensuring this is relevant as an Internet metric).

[Ballot comment]
## Some non-blocking COMMENT points/nits:

## S1 - Comment
“This document defines a “minimum viable framework,” and later the “general framework” - please update the text to explain which is intended.
- Overall I found the introduction and section 2 were not clear and likely were far too verbose, which seemed to mask the salient points that were being introduced. A more concise introduction focussed on the key points would be easier to understand. There is already a detailed overview in section 2.

## S1 - Comment
There is some repeated discussion across sections 1-4. I found that this obscured, rather than provided background and these section would certainly benefit from more review and revision to condense the material. Please consider if they could be more focussed on the essential background points. TR-452.1 is mentioned, but this is not an IETF standard, please add a brief description of the key relevant details relating to the present specification.


## S1 - Comment
The statement: " Quality
  Attenuation meets most of the requirements for a meaningful network
  quality score set out in Section 4.1: it is composable, captures the
  ability of a network to satisfy application requirements, and can be
  compared to a variety of application needs."
- This text reads like a very strong statement for an IETF document, and more like a sales-pitch. If there was strong consensus within the WG on this item, please provide some reference - the Shepherd report suggests not - in which case, please revise this statement, to simple facts. Similarly, "owever, educated assessments of expected outcomes remain achievable". Similarly asserting optimality is a very strong statement, much more than stating that this can be used to "optimise", see: "one for optimal application performance". Similarly, this looks like a significant over-statement for a standards document: " If these requirements are met, network operators can monitor and test their network and understand where the true bottlenecks are, regardless of network technology."
“The novelty of Quality Attenuation lies in its unified treatment of latency and loss within a single distributional framework.“
Please remove the word novelty, this is not applicable to an SDO document. Please simply state:
“The Quality Attenuation framework provides unified treatment of latency and loss within a single distributional framework.“
Please consider changing:
“  Importantly, the framework does not enforce a minimum sample count.”
to “This framework does not enforce a minimum sample count.”
- Please do check the entire document carefully to avoid inadvertently making marketing statements, that might be acceptable in a published paper, but are inappropriate for an RFC.

## S4 - Comment
Sections 4 and 5 discuss topics around requirements but do not actually provide clear requirements. There are requirements in the related specification of TR-452.1, but the current text included in the document seems to motivate the idea rather than specify things.  At least this needs to be explained what are these requirements for? - specifying metrics? the framework itself? something else?

##  - Comment
The content of these paragraphs are motivational. Ought this to be omitted or moved to section 1 and reduced, it does not appear to be part of the defined framework.

## S4 Term: bandwidth - Comment
The term bandwidth, rather than “available capacity” seems wrong in the context of an end-to-end Internet path.

## S4.3 - Comment
I found the following statement rather odd: “Note that developers of similar applications may have arrived at different figures.” This may be true how does it impact the framework being defined and the need for standards?

##  S5 - Comment.
I was very surprised to see the statement on “novelty” in this sentence. The IETF does not normally make claims on the novelty of its protocols, rather specifications and other documents are normally based on their utility. Please re-consider this statement:
    “The novelty of Quality Attenuation lies in
  its unified treatment of latency and loss within a single
  distributional framework...”

## S 5.2 - Comment
I am unsure what to understand by “Still, it might be beneficial for future
  standardization activities to converge on a fixed set of general
  percentiles or for specific applications / application classes to
  make QoO measurements between different networks or providers more
  comparable.”
- Please explain in text.

## S 5.2 - Comment
Why is this not “MUST”:
“Therefore, the network requirements must include a minimum throughput requirement.”

## S 5.2 - Comment
Why is this not “MUST”:
“Whether the requirements are one-way or two-way must be specified.”

## S 5.2 - Comment
This text seems to require additional work to render it suitable for an RFC:
“To that aim, first recall the key design goal of establishing a
  quantifiable distance between optimal and unacceptable network
  conditions, thereby enabling an objective assessment of relative
  quality.  Accordingly, the requirements specification is extended to
  define both the network performance required for achieving optimal
  application performance and the lower network performance threshold
  below which the application performance is considered unacceptable.”
⁃ Please avoid the style of a technical paper, and simply state what is required, or must be clarified,

## S 5.2 - Comment
The following mixes the terms bandwidth and capacity. For an Internet path, I think the “available capacity” is the term that is needed:
“Applications do of course have throughput requirements, and thus a
  complete framework for application-level network quality must also
  take capacity into account.  Insufficient bandwidth may give
  unacceptable application outcomes without necessarily inducing a lot
  of latency or packet loss.  Therefore, the network requirements must
  include a minimum throughput requirement.”

## S5.2 - Comment
“A fully specified requirement can be thought of as specifying” - This is not good text for an RFC, please reword as either an example or a clear statement.

## S5.2 - Comment
Please reconsider this accademic argument and replace by specification text:
“ To that aim, first recall the key design goal of establishing..”

##
The requirement to specify if a measurement was “one-way” seems to be defined in two places. Please remove one.

## S6.1  - Comment
“Direct use of the approach described below in production scenarios is discouraged.”
- This needs to be explained.

## S6.1  - Comment
“application performance is optimal”
AND
“there are typically different levels of optimal performance”
- The IETF is not generally in a position to discuss optimality, please rephrase or justify how this optimum.

## S6.6 - Comment
“a study involving 25 participants tested the QoO
  framework in real-world settings [QoOUserStudy].  Participants used
  specially equipped routers in their homes for ten days, providing
  both network performance data and feedback through pre- and post-
  trial surveys.

  Participants found QoO scores more intuitive and actionable than
  traditional metrics (e.g., speed tests).  QoO directly aligned with
  their self-reported experiences, increasing trust and engagement.

  These results indicate that users find it easier to correlate QoO
  scores with real-world application performance than, for example, a
  speed test.  As such, QoO is expected to help... should be studied further, ”
It seems to me to be decidedly odd to make such a statement in a RFC based on a small sample size. I’d strongly urge removing this example and replacing this by more robust analysis (*or completely removing this text*)


## S7.1 - Comment
- Please provide a citable reference.

## S7.5  - Comment
- See earlier note that this is likely not a measure of bandwidth, but the path capacity.

## S8 - Comment
- Other IPPM documents have included detailed security considerations. Relevant security considerations need to be cited here.

## S8 - Comment
“If not properly rate-limited, this may inadvertently
  degrade services offered by a network or be exploited by malicious
  actors to launch DoS attacks.”
- This is very true, and the relevant security considerations need to be cited here.

## - Comment
Please provide supporting material to explain how the linear distance measure is to be used as a standard method.

## - Comment
How does your reporting work with continuous measurements? Which data points do you include in distributions or percentiles reported?  Last hour, day, week,
month, year, decade?

Best wishes,
Gorry Fairhurst

[Ballot comment]
# Internet AD comments for draft-ietf-ippm-qoo-07
CC @ekline

* comment syntax:
  - https://github.com/mnot/ietf-comments/blob/main/format.md

* "Handling Ballot Positions":
  - https://ietf.org/about/groups/iesg/statements/handling-ballot-positions/

## Comments

* Thank you for S7.4.

* I wonder if (end-to-end/effective) MTU/MSS might be a factor to
  incorporate in future work.

Request for IETF Last Call review by ARTART Completed: Ready with Issues. Reviewer: Martin Thomson. Sent review to list. Submission of review completed at an earlier date.

The following Last Call announcement was sent out (ends 2026-02-13):

From: The IESG
To: IETF-Announce
CC: draft-ietf-ippm-qoo@ietf.org, ippm-chairs@ietf.org, ippm@ietf.org, marcus.ihlar@ericsson.com, mohamed.boucadair@orange.com
Reply-To: last-call@ietf.org
Sender:
Subject: Last Call:  (Quality of Outcome (QoO)) to Informational RFC


The IESG has received a request from the IP Performance Measurement WG (ippm)
to consider the following document: - 'Quality of Outcome (QoO)'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2026-02-13. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


  This document introduces the Quality of Outcome (QoO) network quality
  score and the corresponding QoO framework as an approach to network
  quality assessment designed to align with the needs of application
  developers, users, and operators.

  By leveraging the Quality Attenuation metric, QoO provides a method
  for defining and evaluating application-specific, quality-focused
  network performance requirements to enable insights for network
  optimization and simple Quality of Service scores for end-users.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-ippm-qoo/



No IPR declarations have been submitted directly on this I-D.

# Document Shepherd Write-Up for Group Documents

*This version is dated 4 July 2022.*

Thank you for your service as a document shepherd. Among the responsibilities is
answering the questions in this write-up to give helpful context to Last Call
and Internet Engineering Steering Group ([IESG][1]) reviewers, and your
diligence in completing it is appreciated. The full role of the shepherd is
further described in [RFC 4858][2]. You will need the cooperation of the authors
and editors to complete these checks.

Note that some numbered items contain multiple related questions; please be sure
to answer all of them.

## Document History

1. Does the working group (WG) consensus represent the strong concurrence of a
  few individuals, with others being silent, or did it reach broad agreement?

I would say that it's somewhere in between those two. There has been a moderate level of engagement throughout the process.
Overall, there has been constructive feedback from a decent number of individuals.

2. Was there controversy about particular points, or were there decisions where
  the consensus was particularly rough?

No, all the discussions and feedback has been constructive and neither rough nor controversial.
The draft does mention and acknowledge some limitations to the approach, such as using Binary throughput thresholds, for the sake of simplicity.
From what I've gathered, this has not resulted in difficult conversations.

3. Has anyone threatened an appeal or otherwise indicated extreme discontent? If
  so, please summarize the areas of conflict in separate email messages to the
  responsible Area Director. (It should be in a separate email because this
  questionnaire is publicly available.)

No.

4. For protocol documents, are there existing implementations of the contents of
  the document? Have a significant number of potential implementers indicated
  plans to implement? Are any existing implementations reported somewhere,
  either in the document itself (as [RFC 7942][3] recommends) or elsewhere
  (where)?

This is not strictly a protocol document, it's rather describing a metric / framework that can make use of a multitude of protocols or methods of data collection.
Nevertheless, there are two known implementations:

qoo-c:

https://github.com/getCUJO/qoo-c

And a PR to go-responsiveness:

https://github.com/network-quality/goresponsiveness
https://github.com/network-quality/goresponsiveness/pull/56



## Additional Reviews

5. Do the contents of this document closely interact with technologies in other
  IETF working groups or external organizations, and would it therefore benefit
  from their review? Have those reviews occurred? If yes, describe which
  reviews took place.

The document is largely based on “Quality Attenuation” defined in the Broadband Forum (TR-452.1).
It could be argued that a review from BBF would be beneficial, but since the document falls
so clearly within the scope of IPPM core competence it has not been deemed necessary.

A liaison was sent to BBF for review: https://datatracker.ietf.org/liaison/2079/
There was a response from BFF: “Overall, we see that this work builds appropriately on our work,
and we find no contradictions with our published or in-progress work on Quality Attenuation.”.
More at https://datatracker.ietf.org/liaison/2085/

An informal review from ITU-T SG12:
review from the WG chair + confirm that he is happy with the resolution in the latest draft.

6. Describe how the document meets any required formal expert review criteria,
  such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

N/A

7. If the document contains a YANG module, has the final version of the module
  been checked with any of the [recommended validation tools][4] for syntax and
  formatting validation? If there are any resulting errors or warnings, what is
  the justification for not fixing them at this time? Does the YANG module
  comply with the Network Management Datastore Architecture (NMDA) as specified
  in [RFC 8342][5]?

N/A

8. Describe reviews and automated checks performed to validate sections of the
  final version of the document written in a formal language, such as XML code,
  BNF rules, MIB definitions, CBOR's CDDL, etc.

N/A

## Document Shepherd Checks

9. Based on the shepherd's review of the document, is it their opinion that this
  document is needed, clearly written, complete, correctly designed, and ready
  to be handed off to the responsible Area Director?

Yes, except for a few minor nits that the authors can handle in parallel with progressing the document. 

10. Several IETF Areas have assembled [lists of common issues that their
    reviewers encounter][6]. For which areas have such issues been identified
    and addressed? For which does this still need to happen in subsequent
    reviews?

The document proposes a framework that can use a large number of methods and protocols to produce data.
Therefore the expert topics do not really apply to this document directly.

11. What type of RFC publication is being requested on the IETF stream ([Best
    Current Practice][12], [Proposed Standard, Internet Standard][13],
    [Informational, Experimental or Historic][14])? Why is this the proper type
    of RFC? Do all Datatracker state attributes correctly reflect this intent?

Informational. It is proper type imo, since this is a description of a framework for quality assessment
that does not contain any normative requirements for interoperability across the Internet.

12. Have reasonable efforts been made to remind all authors of the intellectual
    property rights (IPR) disclosure obligations described in [BCP 79][7]? To
    the best of your knowledge, have all required disclosures been filed? If
    not, explain why. If yes, summarize any relevant discussion, including links
    to publicly-available messages when applicable.

Yes.
No declarations filed, authors have confirmed that there are no pending declarations.

13. Has each author, editor, and contributor shown their willingness to be
    listed as such? If the total number of authors and editors on the front page
    is greater than five, please provide a justification.

Yes, both authors have indicated willingness to be listed as such.

14. Document any remaining I-D nits in this document. Simply running the [idnits
    tool][8] is not enough; please review the ["Content Guidelines" on
    authors.ietf.org][15]. (Also note that the current idnits tool generates
    some incorrect warnings; a rewrite is underway.)

There is quite a large use of the word (lower case) must throughout the document, but it's not really used in
a normative RFC 2119 way. I believe this is correct for the type of document. So while not strictly a nit in my view it's
something that might come up in reviews.

15. Should any informative references be normative or vice-versa? See the [IESG
    Statement on Normative and Informative References][16].

Since this is an informational document it could be argued that there is no need for normative references.
However, since TR-452.1 serves as a foundation for the framework it could arguably be considered a normative reference.

16. List any normative references that are not freely available to anyone. Did
    the community have sufficient access to review any such normative
    references?
N/A

17. Are there any normative downward references (see [RFC 3967][9] and [BCP
    97][10]) that are not already listed in the [DOWNREF registry][17]? If so,
    list them.

18. Are there normative references to documents that are not ready to be
    submitted to the IESG for publication or are otherwise in an unclear state?
    If so, what is the plan for their completion?

no

19. Will publication of this document change the status of any existing RFCs? If
    so, does the Datatracker metadata correctly reflect this and are those RFCs
    listed on the title page, in the abstract, and discussed in the
    introduction? If not, explain why and point to the part of the document
    where the relationship of this document to these other RFCs is discussed.
no
20. Describe the document shepherd's review of the IANA considerations section,
    especially with regard to its consistency with the body of the document.
    Confirm that all aspects of the document requiring IANA assignments are
    associated with the appropriate reservations in IANA registries. Confirm
    that any referenced IANA registries have been clearly identified. Confirm
    that each newly created IANA registry specifies its initial contents,
    allocations procedures, and a reasonable name (see [RFC 8126][11]).

N/A

21. List any new IANA registries that require Designated Expert Review for
    future allocations. Are the instructions to the Designated Expert clear?
    Please include suggestions of designated experts, if appropriate.

N/A

[1]: https://www.ietf.org/about/groups/iesg/
[2]: https://www.rfc-editor.org/rfc/rfc4858.html
[3]: https://www.rfc-editor.org/rfc/rfc7942.html
[4]: https://wiki.ietf.org/group/ops/yang-review-tools
[5]: https://www.rfc-editor.org/rfc/rfc8342.html
[6]: https://wiki.ietf.org/group/iesg/ExpertTopics
[7]: https://www.rfc-editor.org/info/bcp79
[8]: https://www.ietf.org/tools/idnits/
[9]: https://www.rfc-editor.org/rfc/rfc3967.html
[10]: https://www.rfc-editor.org/info/bcp97
[11]: https://www.rfc-editor.org/rfc/rfc8126.html
[12]: https://www.rfc-editor.org/rfc/rfc2026.html#section-5
[13]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.1
[14]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.2
[15]: https://authors.ietf.org/en/content-guidelines-overview
[16]: https://www.ietf.org/about/groups/iesg/statements/normative-informative-references/
[17]: https://datatracker.ietf.org/doc/downref/

# Document Shepherd Write-Up for Group Documents

*This version is dated 4 July 2022.*

Thank you for your service as a document shepherd. Among the responsibilities is
answering the questions in this write-up to give helpful context to Last Call
and Internet Engineering Steering Group ([IESG][1]) reviewers, and your
diligence in completing it is appreciated. The full role of the shepherd is
further described in [RFC 4858][2]. You will need the cooperation of the authors
and editors to complete these checks.

Note that some numbered items contain multiple related questions; please be sure
to answer all of them.

## Document History

1. Does the working group (WG) consensus represent the strong concurrence of a
  few individuals, with others being silent, or did it reach broad agreement?

I would say that it's somewhere in between those two. There has been a moderate level of engagement throughout the process.
Overall, there has been constructive feedback from a decent number of individuals.

2. Was there controversy about particular points, or were there decisions where
  the consensus was particularly rough?

No, all the discussions and feedback has been constructive and neither rough nor controversial.
The draft does mention and acknowledge some limitations to the approach, such as using Binary throughput thresholds, for the sake of simplicity.
From what I've gathered, this has not resulted in difficult conversations.

3. Has anyone threatened an appeal or otherwise indicated extreme discontent? If
  so, please summarize the areas of conflict in separate email messages to the
  responsible Area Director. (It should be in a separate email because this
  questionnaire is publicly available.)

No.

4. For protocol documents, are there existing implementations of the contents of
  the document? Have a significant number of potential implementers indicated
  plans to implement? Are any existing implementations reported somewhere,
  either in the document itself (as [RFC 7942][3] recommends) or elsewhere
  (where)?

This is not strictly a protocol document, it's rather describing a metric / framework that can make use of a multitude of protocols or methods of data collection.
Nevertheless, there are two known implementations:

qoo-c:

https://github.com/getCUJO/qoo-c

And a PR to go-responsiveness:

https://github.com/network-quality/goresponsiveness
https://github.com/network-quality/goresponsiveness/pull/56



## Additional Reviews

5. Do the contents of this document closely interact with technologies in other
  IETF working groups or external organizations, and would it therefore benefit
  from their review? Have those reviews occurred? If yes, describe which
  reviews took place.

The document is largely based on “Quality Attenuation” defined in the Broadband Forum (TR-452.1).
It could be argued that a review from BBF would be beneficial, but since the document falls
so clearly within the scope of IPPM core competence it has not been deemed necessary.

6. Describe how the document meets any required formal expert review criteria,
  such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

N/A

7. If the document contains a YANG module, has the final version of the module
  been checked with any of the [recommended validation tools][4] for syntax and
  formatting validation? If there are any resulting errors or warnings, what is
  the justification for not fixing them at this time? Does the YANG module
  comply with the Network Management Datastore Architecture (NMDA) as specified
  in [RFC 8342][5]?

N/A

8. Describe reviews and automated checks performed to validate sections of the
  final version of the document written in a formal language, such as XML code,
  BNF rules, MIB definitions, CBOR's CDDL, etc.

N/A

## Document Shepherd Checks

9. Based on the shepherd's review of the document, is it their opinion that this
  document is needed, clearly written, complete, correctly designed, and ready
  to be handed off to the responsible Area Director?

Yes, except for a few minor nits that the authors can handle in parallel with progressing the document. 

10. Several IETF Areas have assembled [lists of common issues that their
    reviewers encounter][6]. For which areas have such issues been identified
    and addressed? For which does this still need to happen in subsequent
    reviews?

The document proposes a framework that can use a large number of methods and protocols to produce data.
Therefore the expert topics do not really apply to this document directly.

11. What type of RFC publication is being requested on the IETF stream ([Best
    Current Practice][12], [Proposed Standard, Internet Standard][13],
    [Informational, Experimental or Historic][14])? Why is this the proper type
    of RFC? Do all Datatracker state attributes correctly reflect this intent?

Informational. It is proper type imo, since this is a description of a framework for quality assessment
that does not contain any normative requirements for interoperability across the Internet.

12. Have reasonable efforts been made to remind all authors of the intellectual
    property rights (IPR) disclosure obligations described in [BCP 79][7]? To
    the best of your knowledge, have all required disclosures been filed? If
    not, explain why. If yes, summarize any relevant discussion, including links
    to publicly-available messages when applicable.

Yes.
No declarations filed, authors have confirmed that there are no pending declarations.

13. Has each author, editor, and contributor shown their willingness to be
    listed as such? If the total number of authors and editors on the front page
    is greater than five, please provide a justification.

Yes, both authors have indicated willingness to be listed as such.

14. Document any remaining I-D nits in this document. Simply running the [idnits
    tool][8] is not enough; please review the ["Content Guidelines" on
    authors.ietf.org][15]. (Also note that the current idnits tool generates
    some incorrect warnings; a rewrite is underway.)

There is quite a large use of the word (lower case) must throughout the document, but it's not really used in
a normative RFC 2119 way. I believe this is correct for the type of document. So while not strictly a nit in my view it's
something that might come up in reviews.

15. Should any informative references be normative or vice-versa? See the [IESG
    Statement on Normative and Informative References][16].

Since this is an informational document it could be argued that there is no need for normative references.
However, since TR-452.1 serves as a foundation for the framework it could arguably be considered a normative reference.

16. List any normative references that are not freely available to anyone. Did
    the community have sufficient access to review any such normative
    references?
N/A

17. Are there any normative downward references (see [RFC 3967][9] and [BCP
    97][10]) that are not already listed in the [DOWNREF registry][17]? If so,
    list them.

18. Are there normative references to documents that are not ready to be
    submitted to the IESG for publication or are otherwise in an unclear state?
    If so, what is the plan for their completion?

no

19. Will publication of this document change the status of any existing RFCs? If
    so, does the Datatracker metadata correctly reflect this and are those RFCs
    listed on the title page, in the abstract, and discussed in the
    introduction? If not, explain why and point to the part of the document
    where the relationship of this document to these other RFCs is discussed.
no
20. Describe the document shepherd's review of the IANA considerations section,
    especially with regard to its consistency with the body of the document.
    Confirm that all aspects of the document requiring IANA assignments are
    associated with the appropriate reservations in IANA registries. Confirm
    that any referenced IANA registries have been clearly identified. Confirm
    that each newly created IANA registry specifies its initial contents,
    allocations procedures, and a reasonable name (see [RFC 8126][11]).

N/A

21. List any new IANA registries that require Designated Expert Review for
    future allocations. Are the instructions to the Designated Expert clear?
    Please include suggestions of designated experts, if appropriate.

N/A

[1]: https://www.ietf.org/about/groups/iesg/
[2]: https://www.rfc-editor.org/rfc/rfc4858.html
[3]: https://www.rfc-editor.org/rfc/rfc7942.html
[4]: https://wiki.ietf.org/group/ops/yang-review-tools
[5]: https://www.rfc-editor.org/rfc/rfc8342.html
[6]: https://wiki.ietf.org/group/iesg/ExpertTopics
[7]: https://www.rfc-editor.org/info/bcp79
[8]: https://www.ietf.org/tools/idnits/
[9]: https://www.rfc-editor.org/rfc/rfc3967.html
[10]: https://www.rfc-editor.org/info/bcp97
[11]: https://www.rfc-editor.org/rfc/rfc8126.html
[12]: https://www.rfc-editor.org/rfc/rfc2026.html#section-5
[13]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.1
[14]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.2
[15]: https://authors.ietf.org/en/content-guidelines-overview
[16]: https://www.ietf.org/about/groups/iesg/statements/normative-informative-references/
[17]: https://datatracker.ietf.org/doc/downref/

# Document Shepherd Write-Up for Group Documents

*This version is dated 4 July 2022.*

Thank you for your service as a document shepherd. Among the responsibilities is
answering the questions in this write-up to give helpful context to Last Call
and Internet Engineering Steering Group ([IESG][1]) reviewers, and your
diligence in completing it is appreciated. The full role of the shepherd is
further described in [RFC 4858][2]. You will need the cooperation of the authors
and editors to complete these checks.

Note that some numbered items contain multiple related questions; please be sure
to answer all of them.

## Document History

1. Does the working group (WG) consensus represent the strong concurrence of a
  few individuals, with others being silent, or did it reach broad agreement?

I would say that it's somewhere in between those two. There has been a moderate level of engagement throughout the process.
Overall, there has been constructive feedback from a decent number of individuals.

2. Was there controversy about particular points, or were there decisions where
  the consensus was particularly rough?

No, all the discussions and feedback has been constructive and neither rough nor controversial.
The draft does mention and acknowledge some limitations to the approach, such as using Binary throughput thresholds, for the sake of simplicity.
From what I've gathered, this has not resulted in difficult conversations.

3. Has anyone threatened an appeal or otherwise indicated extreme discontent? If
  so, please summarize the areas of conflict in separate email messages to the
  responsible Area Director. (It should be in a separate email because this
  questionnaire is publicly available.)

No.

4. For protocol documents, are there existing implementations of the contents of
  the document? Have a significant number of potential implementers indicated
  plans to implement? Are any existing implementations reported somewhere,
  either in the document itself (as [RFC 7942][3] recommends) or elsewhere
  (where)?

This is not strictly a protocol document, it's rather describing a metric / framework that can make use of a multitude of protocols or methods of data collection.
Nevertheless, there are two known implementations:

qoo-c:

https://github.com/getCUJO/qoo-c

And a PR to go-responsiveness:

https://github.com/network-quality/goresponsiveness
https://github.com/network-quality/goresponsiveness/pull/56



## Additional Reviews

5. Do the contents of this document closely interact with technologies in other
  IETF working groups or external organizations, and would it therefore benefit
  from their review? Have those reviews occurred? If yes, describe which
  reviews took place.

The document is largely based on “Quality Attenuation” defined in the Broadband Forum (TR-452.1).
It could be argued that a review from BBF would be beneficial, but since the document falls
so clearly within the scope of IPPM core competence it has not been deemed necessary.

6. Describe how the document meets any required formal expert review criteria,
  such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

N/A

7. If the document contains a YANG module, has the final version of the module
  been checked with any of the [recommended validation tools][4] for syntax and
  formatting validation? If there are any resulting errors or warnings, what is
  the justification for not fixing them at this time? Does the YANG module
  comply with the Network Management Datastore Architecture (NMDA) as specified
  in [RFC 8342][5]?

N/A

8. Describe reviews and automated checks performed to validate sections of the
  final version of the document written in a formal language, such as XML code,
  BNF rules, MIB definitions, CBOR's CDDL, etc.

N/A

## Document Shepherd Checks

9. Based on the shepherd's review of the document, is it their opinion that this
  document is needed, clearly written, complete, correctly designed, and ready
  to be handed off to the responsible Area Director?

Yes, except for a few minor nits that the authors can handle in parallel with progressing the document. 

10. Several IETF Areas have assembled [lists of common issues that their
    reviewers encounter][6]. For which areas have such issues been identified
    and addressed? For which does this still need to happen in subsequent
    reviews?

The document proposes a framework that can use a large number of methods and protocols to produce data.
Therefore the expert topics do not really apply to this document directly.

11. What type of RFC publication is being requested on the IETF stream ([Best
    Current Practice][12], [Proposed Standard, Internet Standard][13],
    [Informational, Experimental or Historic][14])? Why is this the proper type
    of RFC? Do all Datatracker state attributes correctly reflect this intent?

Informational. It is proper type imo, since this is a description of a framework for quality assessment
that does not contain any normative requirements for interoperability across the Internet.

12. Have reasonable efforts been made to remind all authors of the intellectual
    property rights (IPR) disclosure obligations described in [BCP 79][7]? To
    the best of your knowledge, have all required disclosures been filed? If
    not, explain why. If yes, summarize any relevant discussion, including links
    to publicly-available messages when applicable.

Yes.
No declarations filed, authors have confirmed that there are no pending declarations.

13. Has each author, editor, and contributor shown their willingness to be
    listed as such? If the total number of authors and editors on the front page
    is greater than five, please provide a justification.

Yes, both authors have indicated willingness to be listed as such.

14. Document any remaining I-D nits in this document. Simply running the [idnits
    tool][8] is not enough; please review the ["Content Guidelines" on
    authors.ietf.org][15]. (Also note that the current idnits tool generates
    some incorrect warnings; a rewrite is underway.)

There is quite a large use of the word (lower case) must throughout the document, but it's not really used in
a normative RFC 2119 way. I believe this is correct for the type of document. So while not strictly a nit in my view it's
something that might come up in reviews.

15. Should any informative references be normative or vice-versa? See the [IESG
    Statement on Normative and Informative References][16].

Since this is an informational document it could be argued that there is no need for normative references.
However, since TR-452.1 serves as a foundation for the framework it could arguably be considered a normative reference.

16. List any normative references that are not freely available to anyone. Did
    the community have sufficient access to review any such normative
    references?
N/A

17. Are there any normative downward references (see [RFC 3967][9] and [BCP
    97][10]) that are not already listed in the [DOWNREF registry][17]? If so,
    list them.

18. Are there normative references to documents that are not ready to be
    submitted to the IESG for publication or are otherwise in an unclear state?
    If so, what is the plan for their completion?

no

19. Will publication of this document change the status of any existing RFCs? If
    so, does the Datatracker metadata correctly reflect this and are those RFCs
    listed on the title page, in the abstract, and discussed in the
    introduction? If not, explain why and point to the part of the document
    where the relationship of this document to these other RFCs is discussed.
no
20. Describe the document shepherd's review of the IANA considerations section,
    especially with regard to its consistency with the body of the document.
    Confirm that all aspects of the document requiring IANA assignments are
    associated with the appropriate reservations in IANA registries. Confirm
    that any referenced IANA registries have been clearly identified. Confirm
    that each newly created IANA registry specifies its initial contents,
    allocations procedures, and a reasonable name (see [RFC 8126][11]).

N/A

21. List any new IANA registries that require Designated Expert Review for
    future allocations. Are the instructions to the Designated Expert clear?
    Please include suggestions of designated experts, if appropriate.

N/A

[1]: https://www.ietf.org/about/groups/iesg/
[2]: https://www.rfc-editor.org/rfc/rfc4858.html
[3]: https://www.rfc-editor.org/rfc/rfc7942.html
[4]: https://wiki.ietf.org/group/ops/yang-review-tools
[5]: https://www.rfc-editor.org/rfc/rfc8342.html
[6]: https://wiki.ietf.org/group/iesg/ExpertTopics
[7]: https://www.rfc-editor.org/info/bcp79
[8]: https://www.ietf.org/tools/idnits/
[9]: https://www.rfc-editor.org/rfc/rfc3967.html
[10]: https://www.rfc-editor.org/info/bcp97
[11]: https://www.rfc-editor.org/rfc/rfc8126.html
[12]: https://www.rfc-editor.org/rfc/rfc2026.html#section-5
[13]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.1
[14]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.2
[15]: https://authors.ietf.org/en/content-guidelines-overview
[16]: https://www.ietf.org/about/groups/iesg/statements/normative-informative-references/
[17]: https://datatracker.ietf.org/doc/downref/

# Document Shepherd Write-Up for Group Documents

*This version is dated 4 July 2022.*

Thank you for your service as a document shepherd. Among the responsibilities is
answering the questions in this write-up to give helpful context to Last Call
and Internet Engineering Steering Group ([IESG][1]) reviewers, and your
diligence in completing it is appreciated. The full role of the shepherd is
further described in [RFC 4858][2]. You will need the cooperation of the authors
and editors to complete these checks.

Note that some numbered items contain multiple related questions; please be sure
to answer all of them.

## Document History

1. Does the working group (WG) consensus represent the strong concurrence of a
  few individuals, with others being silent, or did it reach broad agreement?

I would say that it's somewhere in between those two. There has been a moderate level of engagement throughout the process.
Overall, there has been constructive feedback from a decent number of individuals.

2. Was there controversy about particular points, or were there decisions where
  the consensus was particularly rough?

No, all the discussions and feedback has been constructive and neither rough nor controversial.
The draft does mention and acknowledge some limitations to the approach, such as using Binary throughput thresholds, for the sake of simplicity.
From what I've gathered, this has not resulted in difficult conversations.

3. Has anyone threatened an appeal or otherwise indicated extreme discontent? If
  so, please summarize the areas of conflict in separate email messages to the
  responsible Area Director. (It should be in a separate email because this
  questionnaire is publicly available.)

No.

4. For protocol documents, are there existing implementations of the contents of
  the document? Have a significant number of potential implementers indicated
  plans to implement? Are any existing implementations reported somewhere,
  either in the document itself (as [RFC 7942][3] recommends) or elsewhere
  (where)?

This is not strictly a protocol document, it's rather describing a metric / framework that can make use of a multitude of protocols or methods of data collection.
Nevertheless, there are two known implementations:

qoo-c:

https://github.com/getCUJO/qoo-c

And a PR to go-responsiveness:

https://github.com/network-quality/goresponsiveness
https://github.com/network-quality/goresponsiveness/pull/56



## Additional Reviews

5. Do the contents of this document closely interact with technologies in other
  IETF working groups or external organizations, and would it therefore benefit
  from their review? Have those reviews occurred? If yes, describe which
  reviews took place.

The document is largely based on “Quality Attenuation” defined in the Broadband Forum (TR-452.1).
It could be argued that a review from BBF would be beneficial, but since the document falls
so clearly within the scope of IPPM core competence it has not been deemed necessary.

6. Describe how the document meets any required formal expert review criteria,
  such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

N/A

7. If the document contains a YANG module, has the final version of the module
  been checked with any of the [recommended validation tools][4] for syntax and
  formatting validation? If there are any resulting errors or warnings, what is
  the justification for not fixing them at this time? Does the YANG module
  comply with the Network Management Datastore Architecture (NMDA) as specified
  in [RFC 8342][5]?

N/A

8. Describe reviews and automated checks performed to validate sections of the
  final version of the document written in a formal language, such as XML code,
  BNF rules, MIB definitions, CBOR's CDDL, etc.

N/A

## Document Shepherd Checks

9. Based on the shepherd's review of the document, is it their opinion that this
  document is needed, clearly written, complete, correctly designed, and ready
  to be handed off to the responsible Area Director?

Yes, except for a few minor nits that the authors can handle in parallel with progressing the document. 

10. Several IETF Areas have assembled [lists of common issues that their
    reviewers encounter][6]. For which areas have such issues been identified
    and addressed? For which does this still need to happen in subsequent
    reviews?

The document proposes a framework that can use a large number of methods and protocols to produce data.
Therefore the expert topics do not really apply to this document directly.

11. What type of RFC publication is being requested on the IETF stream ([Best
    Current Practice][12], [Proposed Standard, Internet Standard][13],
    [Informational, Experimental or Historic][14])? Why is this the proper type
    of RFC? Do all Datatracker state attributes correctly reflect this intent?

Informational. It is proper type imo, since this is a description of a framework for quality assessment
that does not contain any normative requirements for interoperability across the Internet.

12. Have reasonable efforts been made to remind all authors of the intellectual
    property rights (IPR) disclosure obligations described in [BCP 79][7]? To
    the best of your knowledge, have all required disclosures been filed? If
    not, explain why. If yes, summarize any relevant discussion, including links
    to publicly-available messages when applicable.

Pending..

13. Has each author, editor, and contributor shown their willingness to be
    listed as such? If the total number of authors and editors on the front page
    is greater than five, please provide a justification.

Pending..

14. Document any remaining I-D nits in this document. Simply running the [idnits
    tool][8] is not enough; please review the ["Content Guidelines" on
    authors.ietf.org][15]. (Also note that the current idnits tool generates
    some incorrect warnings; a rewrite is underway.)

There is quite a large use of the word (lower case) must throughout the document, but it's not really used in
a normative RFC 2119 way. I believe this is correct for the type of document. So while not strictly a nit in my view it's
something that might come up in reviews.

15. Should any informative references be normative or vice-versa? See the [IESG
    Statement on Normative and Informative References][16].

Since this is an informational document it could be argued that there is no need for normative references.
However, since TR-452.1 serves as a foundation for the framework it could arguably be considered a normative reference.

16. List any normative references that are not freely available to anyone. Did
    the community have sufficient access to review any such normative
    references?
N/A

17. Are there any normative downward references (see [RFC 3967][9] and [BCP
    97][10]) that are not already listed in the [DOWNREF registry][17]? If so,
    list them.

18. Are there normative references to documents that are not ready to be
    submitted to the IESG for publication or are otherwise in an unclear state?
    If so, what is the plan for their completion?

no

19. Will publication of this document change the status of any existing RFCs? If
    so, does the Datatracker metadata correctly reflect this and are those RFCs
    listed on the title page, in the abstract, and discussed in the
    introduction? If not, explain why and point to the part of the document
    where the relationship of this document to these other RFCs is discussed.
no
20. Describe the document shepherd's review of the IANA considerations section,
    especially with regard to its consistency with the body of the document.
    Confirm that all aspects of the document requiring IANA assignments are
    associated with the appropriate reservations in IANA registries. Confirm
    that any referenced IANA registries have been clearly identified. Confirm
    that each newly created IANA registry specifies its initial contents,
    allocations procedures, and a reasonable name (see [RFC 8126][11]).

N/A

21. List any new IANA registries that require Designated Expert Review for
    future allocations. Are the instructions to the Designated Expert clear?
    Please include suggestions of designated experts, if appropriate.

N/A

[1]: https://www.ietf.org/about/groups/iesg/
[2]: https://www.rfc-editor.org/rfc/rfc4858.html
[3]: https://www.rfc-editor.org/rfc/rfc7942.html
[4]: https://wiki.ietf.org/group/ops/yang-review-tools
[5]: https://www.rfc-editor.org/rfc/rfc8342.html
[6]: https://wiki.ietf.org/group/iesg/ExpertTopics
[7]: https://www.rfc-editor.org/info/bcp79
[8]: https://www.ietf.org/tools/idnits/
[9]: https://www.rfc-editor.org/rfc/rfc3967.html
[10]: https://www.rfc-editor.org/info/bcp97
[11]: https://www.rfc-editor.org/rfc/rfc8126.html
[12]: https://www.rfc-editor.org/rfc/rfc2026.html#section-5
[13]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.1
[14]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.2
[15]: https://authors.ietf.org/en/content-guidelines-overview
[16]: https://www.ietf.org/about/groups/iesg/statements/normative-informative-references/
[17]: https://datatracker.ietf.org/doc/downref/