Network Working Group                                      Scott Bradner
Internet-Draft                                        Harvard University
                                                             Vern Paxson
                                                                    ICSI
                                                               July 2007

   Advancement of metrics specifications on the IETF Standards Track

                   <draft-bradner-metricstest-03.txt>

1. Status of this Memo

   By submitting this Internet-Draft, each author represents that any
   applicable patent or other IPR claims of which he or she is aware
   have been or will be disclosed, and any of which he or she becomes
   aware will be disclosed, in accordance with Section 6 of BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html

   This Internet-Draft will expire on December 24, 2007.

Copyright Notice
   Copyright (C) The IETF Trust (2007).

2. Abstract

   The Internet Standards Process [RFC2026] requires that all IETF
   Standards Track specifications must have "multiple, independent, and
   interoperable implementations" before they can be advanced beyond
   Proposed Standard status. This document specifies the test which the
   IESG will use to determine if a metrics specification document meets
   these requirements.  It also discusses the rationale for this
   process.




Bradner & Paxson                                                [Page 1]


Internet-Draft      Metrics Specification Advancement          July 2007


3. The Nature of the Problem

   The Internet Standards Process [RFC2026] requires that for a IETF
   specification to advance beyond the Proposed Standard level, at least
   two genetically  unrelated implementations must be shown to
   interoperate correctly  with all features and options. There are two
   distinct reasons for this requirement.

   The first reason is to verify that the text of the specification is
   adequately clear and accurate.  This is demonstrated by showing that
   multiple implementation efforts have used the specification to
   achieved interoperable implementations.

   The second reason is to discourage excessive options and "feature
   creep". This is accomplished by requiring interoperable
   implementation of all features, including options.  If an option is
   not included in at least two different interoperable implementations,
   it is safe to assume that it has not been deemed useful and must be
   removed before the specification can advance.

   In the case of a protocol specification which specifies the "bits on
   the wire" exchanged by executing state machines, the notion of
   "interoperability" is reasonably intuitive - the implementations must
   successfully "talk to each other", exchanging "bits on the wire",
   while exercising all features and options.

   In the case of a specification for a performance metric, network
   latency for example, exactly what constitutes "interoperation" is
   less obvious.  This document specifies how the IESG has decided to
   judge "metric specification interoperability" in the context of the
   IETF Standards Process.

   The aim is to ensure that the dual goals of specification clarity and
   feature evaluation have been met using an interpretation of the
   concept of metric specification interoperability that strikes a
   balance between testing complexity and practicality.

4. On The Nature of metric specifications

   Compared to "state machine" protocols which focus on procedural
   specifications, a metric specification describes a method of testing
   and a way to report the results of this testing. One example of such
   a metric would be a way to test and report the latency that data
   packets would incur while being sent from one network location to
   another.

   Since implementations of testing metrics are by their nature stand-
   alone and do not interact with each other, the level of the



Bradner & Paxson                                                [Page 2]


Internet-Draft      Metrics Specification Advancement          July 2007


   interoperability called for in the IETF standards process cannot be
   simply determined by seeing that the implementations interact
   properly.  Instead, we can reasonably require that different
   implementations verifiably give statistically equivalent results.
   Verifiable equivalence takes the place of interoperability.


5. Discussion and Recommended Process
   In order to meet their obligations under the IETF Standards Process
   the IESG must be convinced that each metric specification advanced to
   Draft Standard or Internet Standard status is clearly written, that
   there are the required multiple verifiably equivalent
   implementations, and that all options have been implemented.  There
   may be multiple ways to achieve this goal; this memo documents the
   way that the IESG will use to determine if the requirements have been
   met.

   In the context of this memo, metrics are designed to measure some
   characteristic of a data network.  An aim of any metric definition
   should be that it should be specified in a way that can reliably
   measure the specific characteristic in a repeatable way.  Thus
   running the test a number of times on a stable network should produce
   essentially the same results.  In the same way, sequentially running
   different implementations of software that perform the tests
   described in the metric document on a stable network, or
   simultaneously on a network that may or may not be stable should
   produce essentially the same results.

   Following these assumptions any recommendation for the advancement of
   a metric specification must be accompanied by an implementation
   report, as is the case with all requests for the advancement of IETF
   specifications.  The implementation report must include a specific
   plan to test the specific metrics in the RFC in lab or real-world
   networks and reports of the tests performed with two or more
   implementations of the software.  The test plan should cover key
   parts of the specification, speak to the accuracy required for each
   aspect measured and thus define "statistically equivalent" for the
   specific metrics being tested.  Ideally, the test plan would co-
   evolve with the development of the metric, since that's when people
   have the most context in their thinking regarding the different
   subtleties that can arise.

   The tests MAY be sequential on the same or different hardware, with
   each implementation being run after the previous one finishes, or
   they MAY be run in parallel with multiple implementations each on
   different hardware.  It is RECOMMENDED that the tests be run not in
   deterministic order, but instead by executing a randomizing algorithm
   on the schedule, and interspersing tests from the several



Bradner & Paxson                                                [Page 3]


Internet-Draft      Metrics Specification Advancement          July 2007


   implementations at randomly selected times until all tests of all
   implementations are complete.  This is a way of avoiding a biased
   result in relation to the network conditions and other factors while
   making the comparative tests.

   All of the tests for each set should be run in the same direction
   between the same two points on the same network.  The tests should be
   run simultaneously unless the network is stable enough to ensure that
   the path the data takes through the network will not change between
   tests.  The tests should be run multiple times and the results
   compared and the mean and variance determined.  The results of the
   tests using different implementations of the testing software must be
   within the same variance as the results obtained from multiple
   executions of the same software.

   In particular, if the variance using Method A is sigma^2, then the
   value yielded by Method B should fall within 2*sigma of the mean
   value reported by Method A at least ### % of the time (The percentage
   value will be filled in during a discussion of statistics and metrics
   in the upcoming IPPM meeting).  Note that if Method A and Method B
   are identical, and the value they are estimating is distributed
   according to a Gaussian distribution, then we would expect that
   Method A's estimates would fall within 2*sigma of its mean value ###
   % of the time.  We allow more deviation in recognition of the many
   pragmatic (and sometimes systemic) difficulties in realizing
   consistent network measurement, and the fact that many quantities
   related to networking have distributions quite different from
   Gaussian.  In addition, we explicitly recognize the IESG's
   prerogative to relax the restriction of ### % within 2*sigma in light
   of the particular measurement factors and difficulties, providing
   these are expressed (in a communication with the relevant working
   group or document author) by the IESG.

   If the metric has options, all of the options must be tested in the
   same way.  For example, if the metric includes a list of packet sizes
   that can be used, all of the listed sizes should be tested.  If some
   of the options are not supported or tested they must be removed from
   the specification before the specification can be advanced on the
   standards track.

   As stated above, the measurements are made over a set of points in a
   lab or real world network.  The Area Director(s), in consultation
   with the working group chairs (if applicable), will recommend to the
   IESG their judgment as to the adequacy of the set of measurement
   points in covering the range of relevant network conditions.

   An implementation report is required for both the advancement from
   Proposed Standard to Draft Standard and from Draft Standard to



Bradner & Paxson                                                [Page 4]


Internet-Draft      Metrics Specification Advancement          July 2007


   Internet Standard.  The implementation report for advancement from
   Draft Standard to Internet Standard can be an updated version of the
   one used for the advancement from Proposed Standard to Draft
   Standard.

   The prime concern of the IESG will be that the underlying reasons for
   the verifiably equivalent implementations are met, i.e. that the text
   of the specification is clear and unambiguous, and that features of
   the specification which have not garnered support have been removed.

   The implementation report will be placed on the IETF web page along
   with the other pre-advancement implementation reports and will be
   specifically referred to in the IETF Last-Call.  As with all such
   implementation reports, the determination of adequacy is made by the
   IESG upon recommendation by the Area Director(s) of the relevant IETF
   Area. This determination of adequacy can be challenged during the
   Last-Call period.

6. Security Considerations

   Good, clearly written metric specifications can be of great
   assistance in the measurement and management of the Internet and thus
   assist in the reduction of some types of security threats.  Some may
   view this policy as possibly leading to a reduction in the level of
   confidence people can have in metrics specifications, but the IESG
   feels that it will adequately ensure a reasonable evaluation of the
   level of clarity and ensure that unused options can be identified and
   removed before the advancement of a specification.


7. Acknowledgements
   The basic format and some of the text for this memo came from
   RFC2438], "Advancement of MIB specifications on the IETF Standards
   Track", which provides similar guidance for the advancement of MIBs.

8. References

8.1 Normative References

   [RFC2026] "The Internet Standards Process -- Revision 3", Bradner,
   October 1996

8.2 Informative References

   [RFC2438] "Advancement of MIB specifications on the IETF Standards
   Track", O'Dell, Alvestrand, Wijnen, & Bradner, October 1998

9. Author's Addresses



Bradner & Paxson                                                [Page 5]


Internet-Draft      Metrics Specification Advancement          July 2007


   Scott Bradner
   Harvard University
   29 Oxford St.
   Cambridge MA 02138

   Email: sob@harvard.edu
   Phone: +1-617-495-3864

   Vern Paxson
   International Computer Science Institute
   1947 Center St., Suite 600,
   Berkeley, CA 94704-1198
   USA

   Email: vern@icir.org

Full Copyright Statement

   Copyright (C) The IETF Trust (2007).

   This document is subject to the rights, licenses and restrictions
   contained in BCP 78, and except as set forth therein, the authors
   retain all their rights.

   This document and the information contained herein are provided on an
   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
   THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
   OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
   THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at



Bradner & Paxson                                                [Page 6]


Internet-Draft      Metrics Specification Advancement          July 2007


   http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at ietf-
   ipr@ietf.org.

Acknowledgment

   Funding for the RFC Editor function is provided by the IETF
   Administrative Support Activity (IASA).

changes

   02->03
      correct author list


































Bradner & Paxson                                                [Page 7]