Large-Scale Measurement of Broadband Performance
charter-ietf-lmap-01

I nearly balloted a "Block" on this charter because of the number of
little issues in the text. I hope that we can work to fix these 
issues and polish the text so that the WG can be formed soon and
meet in Berlin

I'm afraid that I find this charter text far too long and imprecise to 
be what I would call "safe". This is unfortunate because it will slow 
down the chartering of a WG that would do useful work.

Here are a number of comments on the text, although I would prefer that
half of the text was pruned.

> The Large-Scale Measurement of Broadband Performance (LMAP)
> working group standardizes the LMAP measurement system for
> performance measurements of broadband devices such as home 
> and enterprise edge routers, personal computers, mobile 
> devices, etc..., whether wired or wireless.

These examples are all broadband access devices. Is it your intention
to be limiting in this way? If so, you should say so explicitly. If
not, you should give some examples from a wider set.

"such as", "etc." and the ellipsis are three ways of indicating that 
the list is intentionally not complete. Drop two of them!


> Measuring portions of the Internet on a large scale is vital
> for accurate characterizations of performance over time and 
> geography, for network diagnostic investigations by providers
> and their users, and for collecting information to support 
> public policy development. The goal is to have the same
> measurements for a large number of points on the Internet, 
> and to have the results collected and stored in the same form.

It is not "vital". No-one will die. Maybe "critical" or "essential"?

Presumably not "the same measurements". Maybe "measurements made
using the same metrics and mechanisms"?

> Practically, the LMAP working group is chartered to determine
> the form of data models and select/extend one or more protocols
> for the secure communications from a Controller to instruct
> Measurement Agent what performance metrics to measure, when to
> measure them, how/when to report the measurement results to a
> Collector, and then for a Measurement Agent to report the results
> to the Collector. Data models should be extensible for new and
> additional measurements.

Why "Practically"? 
This paragraph is very hard to read because of the length of the
first sentence. Please consider rewording it.

> A key assumption constraining the initial work is that the 
> measurement system is under the control of a single organization
> (for example, an Internet Service Provider or a regulator). 
> However, the components of an LMAP measurement system can be 
> deployed in administrative domains that are not owned by the
> measuring organization. Thus, the system of functions deployed
> by a single organization constitutes a single LMAP domain which
> may span ownership or other administrative boundaries. 
>
> The LMAP architecture will allow for measurements that utilize
> either IPv4 or IPv6, or possibly both. Devices containing MAs 
> may have several interfaces using different link technologies.
> Multiple address families and interfaces must be considered in
> the Control and Report protocols.

What is an MA?
The term "Control and Report protocols" could usefully be introduced
as names for the work being done by the WG 2 paras earlier.

> It is assumed that different organization's LMAP measurement
> domains can overlap, and that active measurement packets appear
> along with normal user traffic when crossing another organization's
> network. In the initial chartering phase, there is no requirement
> to specify a mechanism for coordination between the LMAP
> measurements in overlapping domains (for instance a home network
> with MAs on the home hub, set top box and laptop). In principle,
> there are no restrictions on the type of device in which the MA 
> function resides. 

Not sure about an "organization's network". I thought that the point was
that a measuring organization (the only use of the word organization so
far) had an LMAP domain that spans ownership or other administrative
boundaries so that the organization does not have a network.

"In the initial chartering phase" means what? Delete it.

The final sentence seems to be an orphan. Probably important, but not
related to the previous text.

> Both active and passive measurements are in scope, although there
> may be differences in their applicability to specific use cases, 
> or in the security measures needed according to the threats 
> specific to each measurement category. At a high level, LMAP systems
> are agnostic to the measurements and results, and extensible to
> incorporate evolution in the measurement area, but the details
> such as the data models must be standardized to match the 
> measurements.

The second sentence has no meaning I can extract :-(
What does "At a high level" mean?
Doesn't an LMAP system exist to make measurements and record results? 
How then can it be agnostic to the measurements and results?
What is "evolution in the measurement area"?
What other details do you have in mind beyond the data models?
What does it mean to standardize a data model to match the
measurements?

> LMAP will, where possible, adopt the performance metrics of
> the IPPM WG, And bring any required new performance metrics
> to the IPPM WG for standardization.

s/And/and/

Does "where possible" apply to the final clause as well? If not then
you should not try to charter the IPPM WG here! How about you say

   "LMAP will not standardize performance metrics."

> The use case where an end user can independently perform 
> network diagnostic measurements (beyond their private network)
> is not directly in scope, recognizing that users have many 
> opportunities to do this today. However, end users can obtain
> an MA to run measurement tasks if desired and report their
> results to whomever they want, most likely the supplier of
> the MA. This provides for user-initiated on-demand measurement,
> which is an important component of the ISP use case.

"Not directly in scope" is a strange thing to say.

The paragraph is also odd because an end user could be a measuring
organization in its own right. I wonder whether you are trying to 
say something between the lines or whether this paragraph could be
safely deleted since "In principle, there are no restrictions on 
the type of device in which the MA function resides."

> Inter-organization communication of results is out of scope of 
> the LMAP charter.
>
> The management protocol to bootstrap the MAs in measurement 
> devices is out of scope of the LMAP charter, although a 
> bootstrapping process may be described and conducted in many
> ways, such as configuration during manufacture or with a local
> USB interface. 

What is the point of the text from "although" onwards? It is
out of scope, so don't talk about it!

> Service parameters, such as product category, can be useful
> to decide which measurements to run and how to interpret the
> results. These parameters are already gathered and stored by
> existing operations systems. Discovering the service parameters
> on the MAs or sharing the service parameters between MAs are 
> out of the scope. However, if the service parameters are
> available to the MAs, they could be combined with the 
> measurement results in the Report Protocol.

"combined with" or "reported"?

> Deciding the set of measurements to run is a business decision
> and is out of scope of the LMAP charter.
>
> Protection against the intentional or malicious insertion of 
> inaccuracies into the overall system or measurement process is
> outside the scope of work. However, the working group may design
> simple technical protection methods.

Hmmm. It is outside the scope of the charter, but the WG may work 
on it. That needs to be fixed.

> The LMAP working group will coordinate with other standards
> bodies working in this area (e.g., BBF, IEEE 802.16, ETSI) 
> regarding the information model, and other IETF working groups
> in the areas of data models, protocols, multiple interface
> management, and measurement of performance metrics.

s/and other/and with other/

> LMAP will consider re-use of existing protocols and data model
> languages in its efforts to produce the following work items:

Is this the full list of work items for the WG, or just the list
where the WG will consider re-use?

> 1. The LMAP Framework - provides common terminology and
> justifies the simplifying constraints

This is where the justification is. Where are the simplifying 
contraints defined? And what is a "simplifying constraint"?

> 2. The LMAP Use Cases - provides the motivating use cases as
> a basis for the work
> 
> 3. Information Model, the abstract definition of the information
> carried from the Controller to the MA and the information carried
> from the MA to the Collector. It includes

There seem to be some architectural terms being introduced here.
Where is this architecture documented?

>   * The metric(s) that can be measured and values for its 
>     parameters such as the Peer MA participating in the 
>     measurement and the desired environmental conditions (for
>     example, only conduct the measurement when there is no 
>     user traffic observed)
>   * The schedule: when the measurement should be run and how
>     the results should be reported (when and to which Collector)
>   * The report: the metric(s) measured and when, the actual
>     result, and supporting metadata such as location. Result
>     reports may be organized in batches or may be reported 
>     immediately, such as for an on-demand measurement.
>
> 4. The Control protocol and the associated data model: The 
> definition of how instructions are delivered from a Controller
> to a MA; this includes a Data Model consistent with the 
> Information Model plus a transport protocol.  This may be a
> simple instruction - response protocol, and LMAP will specify
> how it operates over an existing protocol (to be selected, 
> perhaps REST-style HTTP(s) or NETCONF).

I wonder whether the parenthetical "perhaps" suggestions might be
deemed to be guiding the WG ahead of its examination of the issues.

> 5. The Report protocol and the associated data model: The 
> definition of how the Report is delivered from a MA to a 
> Collector; this includes a Data Model consistent with the
> Information Model plus a transport protocol (to be selected,
> perhaps REST-style HTTP(s) or IPFIX).

Ditto

> The WG will decide later whether protocols and data models 
> (for Control, respectively Report) will be defined in one or
> separated documents. 

A fairly unnecessary final statement.

Whilst I support the objectives of this charter I am concerned about the length, complexity and verbosity of the charter text, which I found very difficult to understand at first.

Having spoken to Benoit the following would be two useful improvements:

Replace the first three paragraphs with the following:

The Large-Scale Measurement of Broadband Performance (LMAP) working group is chartered to standardize the mechanisms for performance measurements of broadband devices such as home and enterprise edge routers, personal computers, mobile devices and other networking devices, whether wired or wireless.

Measuring portions of the Internet on a large scale is vital for accurate characterizations of performance over time and geography, for network diagnostic investigations by providers and their users, and for collecting information to
support public policy development. The goal is to have the same measurements for a large number of points on the Internet, conducted according to the same characterization plan, and to have the results collected and stored
in the same form.

Many measurement systems that exist today use proprietary, custom-designed  mechanisms to coordinate their Measurement Agents (MAs) deployed across networks, to communicate between MAs and measurement Controllers, and to transfer results to measurement Collectors.  The LMAP working group is  chartered to determine the form of data models and select/extend one or more protocols for the communications between the Measurement Agents' (MAs) function and their Controller function and Collector function. These three functions comprise the LMAP measurement system

=====================================


OLD:
Many measurement aspects are already within the charter of IPPM. These include
standardized definitions of performance metrics, MA-to-MA measurement protocols,
and a registry of frequently-used metrics and parameter settings so they can be
identified in an efficient and consistent fashion. Neither the definition of the
new metrics and methods of measurement, nor the post-processing and analysis of
results falls within the remit of LMAP.

NEW:
LMAP will, where possible, adopt the performance metrics of the IPPM WG, and work with IPPM to develop any required new performance metrics.

===============================================

OLD:

Exhaustive protection against all possibilities of gaming the measurements -
where gaming is defined as intentionally (and perhaps maliciously) inserting
inaccuracy into the overall system or measurement process - is beyond the scope
of work. Some protections are lawyer problems, not engineering problems.
However, the working group may include protections that do not add significant
complexity, as determined by working group consensus.

NEW:

Protection against the intentional or malicious insertion of inaccuracies into the overall system or measurement process is outside the scope
of work. However, the working group may design simple technical protection methods.

===============================================