[Search] [txt|pdf|bibtex] [Tracker] [WG] [Email] [Diff1] [Diff2] [Nits]

Versions: 00 01 02 03                                                   
Network Working Group                                      Scott Bradner
Internet-Draft                                        Harvard University
                                                          Allison Mankin
                                                                 USC/ISI
                                                             Vern Paxson
                                                                   ACIRI
                                                              April 2007

   Advancement of metrics specifications on the IETF Standards Track

                   <draft-bradner-metricstest-01.txt>

1. Status of this Memo

   By submitting this Internet-Draft, each author represents that any
   applicable patent or other IPR claims of which he or she is aware
   have been or will be disclosed, and any of which he or she becomes
   aware will be disclosed, in accordance with Section 6 of BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html

   This Internet-Draft will expire on October 20, 2007.

Copyright Notice
   Copyright (C) The IETF Trust (2007).

2. Abstract

   The Internet Standards Process [RFC2026] requires that all IETF
   Standards Track specifications must have "multiple, independent, and
   interoperable implementations" before they can be advanced beyond
   Proposed Standard status. This document specifies the test which the
   IESG will use to determine if a metrics specification document meets
   these requirements.  It also discusses the rationale for this



Bradner, Mankin & Paxson                                        [Page 1]


Internet-Draft      Metrics Specification Advancement         April 2007


   process.

3. The Nature of the Problem

   The Internet Standards Process [RFC2026] requires that for a IETF
   specification to advance beyond the Proposed Standard level, at least
   two genetically  unrelated implementations must be shown to
   interoperate correctly  with all features and options. There are two
   distinct reasons for this requirement.

   The first reason is to verify that the text of the specification is
   adequately clear and accurate.  This is demonstrated by showing that
   multiple implementation efforts have used the specification to
   achieved interoperable implementations.

   The second reason is to discourage excessive options and "feature
   creep". This is accomplished by requiring interoperable
   implementation of all features, including options.  If an option is
   not included in at least two different interoperable implementations,
   it is safe to assume that it has not been deemed useful and must be
   removed before the specification can advance.

   In the case of a protocol specification which specifies the "bits on
   the wire" exchanged by executing state machines, the notion of
   "interoperability" is reasonably intuitive - the implementations must
   successfully "talk to each other", exchanging "bits on the wire",
   while exercising all features and options.

   In the case of a specification for a performance metric, network
   latency for example, exactly what constitutes "interoperation" is
   less obvious.  This document specifies how the IESG has decided to
   judge "metric specification interoperability" in the context of the
   IETF Standards Process.

   The aim is to ensure that the dual goals of specification clarity and
   feature evaluation have been met using an interpretation of the
   concept of metric specification interoperability that strikes a
   balance between testing complexity and practicality.

4. On The Nature of metric specifications

   Compared to "state machine" protocols which focus on procedural
   specifications, a metric specification describes a method of testing
   and a way to report the results of this testing. One example of such
   a metric would be a way to test and report the latency that data
   packets would incur while being sent from one network location to
   another.




Bradner, Mankin & Paxson                                        [Page 2]


Internet-Draft      Metrics Specification Advancement         April 2007


   Since implementations of testing metrics are by their nature stand-
   alone and do not interact with each other, the level of the
   interoperability called for in the IETF standards process cannot be
   simply determined by seeing that the implementations interact
   properly.


5. Discussion and Recommended Process

   In order to meet their obligations under the IETF Standards Process
   the IESG must be convinced that each metric specification advanced to
   Draft Standard or Internet Standard status is clearly written, that
   there are the required multiple interoperable implementations, and
   that all options have been implemented.  There may be multiple ways
   to achieve this goal; this memo documents the way that the IESG will
   use to determine if the requirements have been met.

   In the context of this memo, metrics are designed to measure some
   characteristic of a data network.  An aim of any metric definition
   should be that it should be specified in a way that can reliably
   measure the specific characteristic in a repeatable way.  Thus
   running the test a number of times on a stable network should produce
   essentially the same results.  In the same way, sequentially running
   different implementations of software that perform the tests
   described in the metric document on a stable network, or
   simultaneously on a network that may or may not be stable should
   produce essentially the same results.

   Following these assumptions any recommendation for the advancement of
   a metric specification must be accompanied by an implementation
   report, as is the case with all requests for the advancement of IETF
   specifications.  The implementation report must include reports of
   tests performed between sets of points on a network with two or more
   implementations of the software.  The tests MAY be sequential on the
   same or different hardware, with each implementation being run after
   the previous one finishes, or they MAY be run in parallel with
   multiple implementations each on different hardware.  It is
   RECOMMENDED that the tests be run not in deterministic order, but
   instead by executing a randomizing algorithm on the schedule, and
   interspersing tests from the several implementations at randomly
   selected times until all tests of all implementations are complete.
   This is a way of avoiding a biased result in relation to the network
   conditions and other factors while making the comparative tests.

   All of the tests for each set should be run in the same direction
   between the same two points on the same network.  The tests should be
   run simultaneously unless the network is stable enough to ensure that
   the path the data takes through the network will not change between



Bradner, Mankin & Paxson                                        [Page 3]


Internet-Draft      Metrics Specification Advancement         April 2007


   tests.  The tests should be run multiple times and the results
   compared and the mean and variance determined.  The results of the
   tests using different implementations of the testing software must be
   within the same variance as the results obtained from multiple
   executions of the same software.

   In particular, if the variance using Method A is sigma^2, then the
   value yielded by Method B should fall within 2*sigma of the mean
   value reported by Method A at least ### % of the time (The percentage
   value will be filled in during a discussion of statistics and metrics
   in the upcoming IPPM meeting).  Note that if Method A and Method B
   are identical, and the value they are estimating is distributed
   according to a Gaussian distribution, then we would expect that
   Method A's estimates would fall within 2*sigma of its mean value ###
   % of the time.  We allow more deviation in recognition of the many
   pragmatic (and sometimes systemic) difficulties in realizing
   consistent network measurement, and the fact that many quantities
   related to networking have distributions quite different from
   Gaussian.  In addition, we explicitly recognize the IESG's
   prerogative to relax the restriction of ### % within 2*sigma in light
   of the particular measurement factors and difficulties, providing
   these are expressed (in a communication with the relevant working
   group or document author) by the IESG.

   If the metric has options, all of the options must be tested in the
   same way.  For example, if the metric includes a list of packet sizes
   that can be used, all of the listed sizes should be tested.  If some
   of the options are not supported or tested they must be removed from
   the specification before the specification can be advanced on the
   standards track.

   As stated above, the measurements are made over a set of points in a
   network.  The Area Director(s), in consultation with the working
   group chairs (if applicable), will recommend to the IESG their
   judgment as to the adequacy of the set of measurement points in
   covering the range of relevant network conditions.

   An implementation report is required for both the advancement from
   Proposed Standard to Draft Standard and from Draft Standard to
   Internet Standard.  The implementation report for advancement from
   Draft Standard to Internet Standard can be an updated version of the
   one used for the advancement from Proposed Standard to Draft
   Standard.

   The prime concern of the IESG will be that the underlying reasons for
   the interoperable implementations are met, i.e. that the text of the
   specification is clear and unambiguous, and that features of the
   specification which have not garnered support have been removed.



Bradner, Mankin & Paxson                                        [Page 4]


Internet-Draft      Metrics Specification Advancement         April 2007


   The implementation report will be placed on the IETF web page along
   with the other pre-advancement implementation reports and will be
   specifically referred to in the IETF Last-Call.  As with all such
   implementation reports, the determination of adequacy is made by the
   IESG upon recommendation by the Area Director(s) of the relevant IETF
   Area. This determination of adequacy can be challenged during the
   Last-Call period.

6. Security Considerations

   Some may view this policy as possibly leading to a reduction in the
   level of confidence people can have in metrics specifications, but
   the IESG feels that it will adequately ensure a reasonable evaluation
   of the level of clarity and ensure that unused options can be
   identified and removed before the advancement of a specification.

   Good, clearly written metric specifications can be of great
   assistance in the measurement and management of the Internet and thus
   assist in the reduction of some types of security threats.


8. Acknowledgements
   The basic format and some of the text for this memo came from
   RFC2438], "Advancement of MIB specifications on the IETF Standards
   Track", which provides similar guidance for the advancement of MIBs.


8. References

   [RFC2026] "The Internet Standards Process -- Revision 3", Bradner,
   October 1996

   [RFC2438] "Advancement of MIB specifications on the IETF Standards
   Track", O'Dell, Alvestrand, Wijnen, & Bradner, October 1998

9. Author's Addresses

   Scott Bradner
   Harvard University
   29 Oxford St.
   Cambridge MA 02138

   Email: sob@harvard.edu
   Phone: +1-617-495-3864

   Allison Mankin
   NSF




Bradner, Mankin & Paxson                                        [Page 5]


Internet-Draft      Metrics Specification Advancement         April 2007


   Email: mankin@psg.com

   Vern Paxson
   ICRI
   1947 Center St., Suite 600,
   Berkeley, CA 94704-1198
   USA

   Email: vern@icir.org

Full Copyright Statement

   Copyright (C) The IETF Trust (2007).

   This document is subject to the rights, licenses and restrictions
   contained in BCP 78, and except as set forth therein, the authors
   retain all their rights.

   This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
   THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
   OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
   THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at
   http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at
   ietf-ipr@ietf.org.



Bradner, Mankin & Paxson                                        [Page 6]


Internet-Draft      Metrics Specification Advancement         April 2007


   Acknowledgment

   Funding for the RFC Editor function is provided by the IETF Administrative Support Activity (IASA).
















































Bradner, Mankin & Paxson                                        [Page 7]