Quality of Outcome (QoO)
draft-ietf-ippm-qoo-11

Thanks to Martin Thomson for the ARTART review, which appears to have resulted in clarifying changes to this document.

Thank you producing revision -10 and the responses to my DISCUSS and COMMENT topics.

---

## Some non-blocking COMMENT points/nits: 

## S1 - Comment
“This document defines a “minimum viable framework,” and later the “general framework” - please update the text to explain which is intended.
- Overall I found the introduction and section 2 were not clear and likely were far too verbose, which seemed to mask the salient points that were being introduced. A more concise introduction focussed on the key points would be easier to understand. There is already a detailed overview in section 2. 

## S1 - Comment
There is some repeated discussion across sections 1-4. I found that this obscured, rather than provided background and these section would certainly benefit from more review and revision to condense the material. Please consider if they could be more focussed on the essential background points. TR-452.1 is mentioned, but this is not an IETF standard, please add a brief description of the key relevant details relating to the present specification.


## S1 - Comment
The statement: " Quality
   Attenuation meets most of the requirements for a meaningful network
   quality score set out in Section 4.1: it is composable, captures the
   ability of a network to satisfy application requirements, and can be
   compared to a variety of application needs."
- This text reads like a very strong statement for an IETF document, and more like a sales-pitch. If there was strong consensus within the WG on this item, please provide some reference - the Shepherd report suggests not - in which case, please revise this statement, to simple facts. Similarly, "owever, educated assessments of expected outcomes remain achievable". Similarly asserting optimality is a very strong statement, much more than stating that this can be used to "optimise", see: "one for optimal application performance". Similarly, this looks like a significant over-statement for a standards document: " If these requirements are met, network operators can monitor and test their network and understand where the true bottlenecks are, regardless of network technology." 
“The novelty of Quality Attenuation lies in its unified treatment of latency and loss within a single distributional framework.“
Please remove the word novelty, this is not applicable to an SDO document. Please simply state:
“The Quality Attenuation framework provides unified treatment of latency and loss within a single distributional framework.“
Please consider changing:
“   Importantly, the framework does not enforce a minimum sample count.”
to “This framework does not enforce a minimum sample count.”
- Please do check the entire document carefully to avoid inadvertently making marketing statements, that might be acceptable in a published paper, but are inappropriate for an RFC.

## S4 - Comment
Sections 4 and 5 discuss topics around requirements but do not actually provide clear requirements. There are requirements in the related specification of TR-452.1, but the current text included in the document seems to motivate the idea rather than specify things.  At least this needs to be explained what are these requirements for? - specifying metrics? the framework itself? something else?

##  - Comment
The content of these paragraphs are motivational. Ought this to be omitted or moved to section 1 and reduced, it does not appear to be part of the defined framework.

## S4 Term: bandwidth - Comment
The term bandwidth, rather than “available capacity” seems wrong in the context of an end-to-end Internet path.

## S4.3 - Comment
I found the following statement rather odd: “Note that developers of similar applications may have arrived at different figures.” This may be true how does it impact the framework being defined and the need for standards?

##  S5 - Comment.
I was very surprised to see the statement on “novelty” in this sentence. The IETF does not normally make claims on the novelty of its protocols, rather specifications and other documents are normally based on their utility. Please re-consider this statement:
     “The novelty of Quality Attenuation lies in
   its unified treatment of latency and loss within a single
   distributional framework...”

## S 5.2 - Comment
I am unsure what to understand by “Still, it might be beneficial for future
   standardization activities to converge on a fixed set of general
   percentiles or for specific applications / application classes to
   make QoO measurements between different networks or providers more
   comparable.”
- Please explain in text.

## S 5.2 - Comment
Why is this not “MUST”:
“Therefore, the network requirements must include a minimum throughput requirement.”

## S 5.2 - Comment
Why is this not “MUST”:
“Whether the requirements are one-way or two-way must be specified.”

## S 5.2 - Comment
This text seems to require additional work to render it suitable for an RFC:
“To that aim, first recall the key design goal of establishing a
   quantifiable distance between optimal and unacceptable network
   conditions, thereby enabling an objective assessment of relative
   quality.  Accordingly, the requirements specification is extended to
   define both the network performance required for achieving optimal
   application performance and the lower network performance threshold
   below which the application performance is considered unacceptable.”
	⁃	Please avoid the style of a technical paper, and simply state what is required, or must be clarified,

## S 5.2 - Comment
The following mixes the terms bandwidth and capacity. For an Internet path, I think the “available capacity” is the term that is needed:
“Applications do of course have throughput requirements, and thus a
   complete framework for application-level network quality must also
   take capacity into account.  Insufficient bandwidth may give
   unacceptable application outcomes without necessarily inducing a lot
   of latency or packet loss.  Therefore, the network requirements must
   include a minimum throughput requirement.”

## S5.2 - Comment
“A fully specified requirement can be thought of as specifying” - This is not good text for an RFC, please reword as either an example or a clear statement.

## S5.2 - Comment
Please reconsider this accademic argument and replace by specification text:
“ To that aim, first recall the key design goal of establishing..”

##
The requirement to specify if a measurement was “one-way” seems to be defined in two places. Please remove one.

## S6.1  - Comment
“Direct use of the approach described below in production scenarios is discouraged.”
- This needs to be explained.

## S6.1  - Comment
“application performance is optimal”
AND
“there are typically different levels of optimal performance”
- The IETF is not generally in a position to discuss optimality, please rephrase or justify how this optimum.

## S6.6 - Comment
“a study involving 25 participants tested the QoO
   framework in real-world settings [QoOUserStudy].  Participants used
   specially equipped routers in their homes for ten days, providing
   both network performance data and feedback through pre- and post-
   trial surveys.

   Participants found QoO scores more intuitive and actionable than
   traditional metrics (e.g., speed tests).  QoO directly aligned with
   their self-reported experiences, increasing trust and engagement.

   These results indicate that users find it easier to correlate QoO
   scores with real-world application performance than, for example, a
   speed test.  As such, QoO is expected to help... should be studied further, ”
It seems to me to be decidedly odd to make such a statement in a RFC based on a small sample size. I’d strongly urge removing this example and replacing this by more robust analysis (*or completely removing this text*)


## S7.1 - Comment
- Please provide a citable reference.

## S7.5  - Comment
- See earlier note that this is likely not a measure of bandwidth, but the path capacity.

## S8 - Comment
- Other IPPM documents have included detailed security considerations. Relevant security considerations need to be cited here.

## S8 - Comment
“If not properly rate-limited, this may inadvertently
   degrade services offered by a network or be exploited by malicious
   actors to launch DoS attacks.”
- This is very true, and the relevant security considerations need to be cited here.

## - Comment
Please provide supporting material to explain how the linear distance measure is to be used as a standard method.

## - Comment
How does your reporting work with continuous measurements? Which data points do you include in distributions or percentiles reported?  Last hour, day, week,
month, year, decade?

Best wishes,
Gorry Fairhurst

---

For completeness, I have included my previous DISCUSS topics here:

Thanks for providing draft-ietf-ippm-qoo and the many revisions to bring this document for consideration by the IESG. 


Thanks to Joerg Ott for his TSVART review, which noted concerns with the document contents, readability, technical detail and precision. I would like to discuss how these comments can be addressed.

Please find below some blocking DISCUSS points (easy to address), some non-blocking COMMENT points/nits (replies would be appreciated even if only for my own education).

## DISCUSS (blocking)

As noted in https://datatracker.ietf.org/doc/statement-iesg-handling-ballot-positions-20220121/, a DISCUSS ballot is a request to have a discussion on the points below; I really think that the document would be improved with a change here, but can be convinced otherwise. 

##  S5 - DISCUSS
- is it strictly necessary to make this normatively depend on TR-452, for instance this statement:
“However, in addition to measurement approaches
   standardized in the QED framework [TR-452.1], some relevant
   techniques are:”
- Please provide citable references for each technique!
- Can you instead make clear what the IETF-defined framework includes?

## S4 and S5 - DISCUSS
The document uses a lot of descriptive language to indicate requirements, but does not use any requirements language - RFC2119 language is acceptable in an Informational document, should the working group require this. At the moment I feel many of the requirements are just expressed as aspirations, and not sufficiently well-defined to allow someone to know whether a specific instance complies with the framework. I think this needs to be addressed, because without this the value of this being archived in the RFC series is doubtful.
For example, this seems a strong requirement: “   The framework is agnostic to traffic direction but mandates that measurements specify whether latency is one-way or round-trip.” There are further instances (see also my review comments).

## S5 - DISCUSS
“   When measurements are taken during periods of network load, the
   result naturally includes latency under load.  In scenarios such as
   passive monitoring of production traffic, capturing artificially
   loaded conditions may not always be feasible, whereas passively
   observing the actual network load may be possible.”
- This general statement could be true, but seems to lack any IETF statement around the safety of such measurements. Please update the text to provide this, for example by citing other IPPM work where this has been stated.
Similarly the later statement: “by generating artificial traffic to a level at least equivalent to the application traffic load“.

## S5.1 - DISCUSS
I suggest the essential “metadata” is under-specified (sometimes omitting units and details), please check carefully.
Section 5 discusses composable metrics. Please revise the text to address the TSVART review on the intention of this text.
For instance why is the following requirement not  at least a“SHOULD” :
“each measurement must be accompanied by the following metadata:”
	⁃	In some cases. the stated metadata provides no units or expected accuracy for the listed elements. Please specify.
The section is followed  by text that says:
“These metadata elements are essential for interpreting the precision
   and reliability of the measurements.”
	⁃	Is this really required, i.e. essential - making this a normative list of required meta data, or “useful”? - Please clarify the language.


##  S5.1 - DISCUSS
The requirement: “The repeatability of measurements under similar network conditions” does not seem to discuss path variation - either at L2, L3 (e.g. ECMP/Routing) and L4 - multi-path and cos mechanisms. Please clarify how this impact this requirement.

##  S5.1 - DISCUSS
Why is this not a requirement: “Implementers should document their precision assessment methodology”
- It would seem without this, the other requirements are moot, if I misunderstood, please help to understand and find better text.

## S6.1 - DISCUSS
Does a  measurement consider  load balancing along the forwarding path? (I think the referenced work in TR-452.1 is about a network segment, rather than an Internet Path, and hence this seems an important part of ensuring this is relevant as an Internet metric).

Thank you to Paul Kyzivat for the GENART review.

# Internet AD comments for draft-ietf-ippm-qoo-07
CC @ekline

* comment syntax:
  - https://github.com/mnot/ietf-comments/blob/main/format.md

* "Handling Ballot Positions":
  - https://ietf.org/about/groups/iesg/statements/handling-ballot-positions/

## Comments

* Thank you for S7.4.

* I wonder if (end-to-end/effective) MTU/MSS might be a factor to
  incorporate in future work.