[Ballot comment]
Thanks for addressing my DISCUSS and COMMENT.

In Section 7.1.2, I would recommend the following change for clarity:

OLD
Spec: an immutable document, for RFCs, the RFC number and major section number that specifies this Registry entry in the form RFCXXXXsecY, such as RFC7799sec3.

NEW
Spec: an immutable document identifier combined with a document section identifier. For RFCs, this consists of the RFC number and major section number that specifies this Registry entry in the form RFCXXXXsecY, such as RFC7799sec3.

[Ballot comment]
The sometimes-conversational tone of the text here is at odds with my
mental preconception of what a BCP (or, for that matter, a
Standards-Track document) is, though I make no claim that my mental
preconception is relevant to anyone other than myself.

I agree with the comment about parts of the text assuing that metric
specifications will be RFCs being at odds with the lower
specification-required regisration policy.

I support Alissa's Discuss point.

Section 1

  applications transported over its protocols.  Performance metrics are
  such an important part of the operations of IETF protocols that
  [RFC6390] specifies guidelines for their development.

It's not entirely clear that the existence of RFC 6390 supports exactly
this conclusion.

  The definition and use of Performance Metrics in the IETF happens in
  various working groups (WG), most notably:

Is this going to age well?

      The "Performance Metrics for Other Layers" (PMOL) a concluded WG,

nits: s/a //, no comma

Section 2

      Examples of Performance Metrics are the FTP response time for a
      complete file download, the DNS response time to resolve the IP
      address, a database logging time, etc.  This definition is

nit: DNS responses are not limited to single IP addresses, so the
definite article seems wrong here.

      consistent with the definition of metric in [RFC2330] and broader
      than the definition of performance metric in [RFC6390].

Why are we using a different definition of performance metric than BCP
170?  [How can both be BCPs at the same time but be in conflict?]

  Parameter:  A Parameter is an input factor defined as a variable in
      the definition of a Performance Metric.  A Parameter is a
      numerical or other specified factor forming one of a set that
      defines a metric or sets the conditions of its operation.  All
      Parameters must be known to measure using a metric and interpret
      the results.  There are two types of Parameters: Fixed and Run-

nit: I suggest "all Parameters must be known in order to make a
measurement using a metric", as currently "known to measure" flows
oddly.

Section 3

  groups: IPPM, XRBLOCK, IPFIX, and BMWG.  This document analyzes an
  prior attempt to set up a Performance Metrics Registry, and the

nit: s/ an/ a/

  Based on [RFC8126] Section 4.3, this document is processed as Best
  Current Practice (BCP) [RFC2026].

As was already noted, the referenced section does not have any obvious
relevance to publication status for this document.

Section 4.1

  As any IETF registry, the primary use for a registry is to manage a

nit: I think this was intended to be "As for any IETF registry"

      particular example is the LMAP framework [RFC7594].  Using the
      LMAP terminology, the Performance Metrics Registry is used in the
      LMAP Control protocol to allow a Controller to request a
      measurement task to one or more Measurement Agents.  In order to

nit: please check the grammar of "request a measurement task to".

Section 4.2

  different (and incompatible) ways.  Having a registry would allow
  both the IETF community and external people to have a single list of

nit: I suggest rephrasing "external people" as needlessly divisive;
perhaps "other communities"?

Section 7

  form of registry extension.  The specification defining the new
  column(s) MUST give guidelines to populate the new column(s) for
  existing entries (in general).

Is "MUST (in general)" really still a "MUST"?

Section 7.1.2

        RTDelay (Round Trip Delay)

        RTDNS (Response Time Domain Name Service)

It's slightly unfortunate that "RT" expands to both "Round Trip" and
"Response Time".

  o  SubTypeMethod: One or more sub-types to further describe the
      features of the entry, such as:

        ICMP (Internet Control Message Protocol)

        IP (Internet Protocol)

        DSCPxx (where xx is replaced by a Diffserv code point)

I thought Diffserv code point interpretation was a local matter; is it
really appropriate in the metric definition?

Section 7.3.5

  devices.  For example, parameters that include an IPv4 address can be
  encoded as a 32 bit integer (i.e. binary base64 encoded value) or ip-
  address as defined in [RFC6991].  The actual encoding(s) used must be

nit: YANG types are specified distinctly from their possible encodings;
indicating an XML or JSON encoded version of the YANG ip-address would
be appropriate here.

  explicitly defined for each Run-time parameter.  IPv6 addresses and
  options MUST be accomodated, allowing Registered Metrics to be used
  in either address family.

nit: is "either" too limiting, here?

Section 8.2

  A change to a Registered Performance Metric SHALL be determined to be
  backward-compatible only when:

Is it better to give the Experts some leeway to do the right thing
rather than locking it down with rigid procedures?

Section 8.3

  The use of deprecated Registered Performance Metrics should result in
  a log entry or human-readable warning by the respective application.

Taken literally this would require any measurement agent to query IANA
before performing every measurement, to confirm that the metric being
used is not deprecated.  Presumably some less-frequent polling
frequencey would still be acceptable!

Section 10.2

  reviewed along with the metric entry.  New assignments for Registered
  Performance Metric Name Elements will be administered by IANA through
  Expert Review [RFC8126], i.e., review by one of a group of experts,
  the Performance Metric Experts, who are appointed by the IESG upon
  recommendation of the Transport Area Directors.

It's not entirely clear to me that we need to go into this much detail
when RFC 8126 covers it already.  (Also elsewhere.)

Section 11.1.3

Please use https:// URIs.

Section 13.2

One might argue that RFC 7799 should be normative, since we defer to it
for the allowed valuse for the Method column.

[Ballot comment]
I support Álvaro’s second DISCUSS point and the related issues he lists after it.

I also think we should resolve Álvaro’s first DISCUSS point by simply putting a note in the IANA Considerations in this document that refers to the other document as providing initial registrations, and not waste time and effort doing a formal expert review of registrations that the working group has already reviewed and accepted here.

[Ballot discuss]
I support Alvaro's DISCUSS point #2. I'm confused about what the registration policy is for metrics in the new registry. If it is Specification Required, then the places in the document that assume new metrics are defined in an RFC need to be generalized, because Specification Required need not involve any RFC at all.

I have an additional concern about this text:

"If the proposed registry entry is defined in an RFC but is not yet
  widely deployed, there SHOULD be a statement in the RFC that says the
  proposed registry entry is not ready for registration, and use SHOULD
  employ a private/experimental ID.  It is the responsibility of the
  document authors to submit the request to IANA when the proposed
  registry entry is ready for official registration."

This appears to put a requirement on RFCs to include language that is not timeless and may later become out of date. That is, if this guidance is followed but a metric is later widely deployed, the RFC would have to be updated just to remove the text about the metric not being ready for registration. It seems better to just give guidance about which identifier range registration requests should target, and to give guidance to the designated experts about how to evaluate requests in different ranges.

[Ballot comment]
Section 1:

"any other organization that wishes to create a Performance Metrics Registry"

It seems like performance metrics registry should not be capitalized here since the next section defines the capitalized version as the one maintained by IANA.

Section 6:

I would strongly recommend that this section be moved to an appendix. Some readers may find it useful, but it doesn't seem like it belongs in the body of the document.

Also, I would caution against a section heading entitled "Why this Attempt Will Succeed." I certainly hope it will, but the future is hard to predict and the IETF has a long history of being wrong about what will and will not succeed.

[Ballot discuss]
I have two separate issues that I would like to DISCUSS.

(1) Approval of the initial performance metrics entries (in draft-ietf-ippm-initial-registry).

This document describes the format of the registry, and the initial entries are defined in draft-ietf-ippm-initial-registry.  However, the registration policy of Specification Required would not be met if the entries in draft-ietf-ippm-initial-registry are approved without expert review.

As I mentioned in my ballot for draft-ietf-ippm-initial-registry, I believe that because both documents are being processed at the same time, and the new entries have been reviewed by the WG, IESG Approval [rfc8126] can be used. 

I can think of at least three ways to address this DISCUSS point (there may be others):

a. Designated Experts for this document can be assigned and the formal review can be done.

b. The text in this document can explicitly say that the entries in draft-ietf-ippm-initial-registry are to be approved using IESG Approval.

c. The Responsible AD can add a Management Item to the Telechat for the IESG to explicitly approve (beyond approval for the publication of draft-ietf-ippm-initial-registry) the new entries.

I am ok with either choice, but would prefer Option c because it would be faster and cause less churn.



(2)

§8.1 (Adding new Performance Metrics to the Performance Metrics Registry) defines the following process for entries that are "not yet widely deployed":

  If the proposed registry entry is defined in an RFC but is not yet
  widely deployed, there SHOULD be a statement in the RFC that says the
  proposed registry entry is not ready for registration, and use SHOULD
  employ a private/experimental ID.  It is the responsibility of the
  document authors to submit the request to IANA when the proposed
  registry entry is ready for official registration.

Considering the Specification Required policy and the fact that the RFC has already gone through all the reviews required for publication (including expert review, as mentioned in the same section), how will it work for the "authors to submit the request to IANA when the proposed registry entry is ready for official registration"?  What specification will be presented to IANA to satisfy the registration requirement?  It seems to me that the statement mentioned above would prevent the official registration in the first place, and that same statement (still present in the RFC) should prevent a second review of the same document from resulting in an official registration.

This process needs more discussion and clarity for it to work.


[Not part of this DISCUSS point, but related.] 

a. Section 8 talks about instructions about handling of the registry.  Perhaps it should be part of the IANA Considerations section.

b. §7.1.1 contains additional instructions for IANA, including the reservation of a private/experimental range of Identifiers.  This test should also be part of the IANA Considerations section.

[Ballot comment]
(1)  I don't understand why this document is a BCP and not on the Standards Track.  The Shepherd writeup says that the status "is best current practice...as explained in the draft", but the document only justifies it this way:

  Based on [RFC8126] Section 4.3, this document is processed as Best
  Current Practice (BCP) [RFC2026].

rfc8126/§4.3 talks about the Hierarchical Allocation registration policy.  I'm lost.

It does seem strange that this document is classified as a BCP when it is defining a registry...when many other documents serving the same function are simply on the Standards Track.  It would be ideal if there was an actual justification.

Related to being a BCP.  Should this document be part of BCP 170?


(2) Nits:

s/any other organization that wishes to create a Performance Metrics Registry MAY use the same formatting specifications/any other organization that wishes to create a Performance Metrics Registry may use the same formatting specifications      There is no Normative value, just stating a fact.

s/Expert Review [RFC8126]policy/Expert Review [RFC8126] policy

s/Submission to IANA MAY be during IESG review/Submission to IANA may be during IESG review    Just stating a fact...no Normative value.

[Ballot comment]
* Section 9.  Please consider adding something approximately along the lines of:

The aggregated results of the performance metrics described in this registry can reveal network topology information that might be considered sensitive.  If this is the case, access control mechanism should be applied.

* Editorial Nits:
- Section 7.1.2. Typo. s/obervations/ observations/
- Section 7.1.3. Typo. s/formated/formatted/
- Section 7.3.1.  Typo. s/unambigious/unambiguous/
- Section 7.3.2.  Typo. s/wether/whether/
- Section 7.3.5.  Typo. s/accomodated/accommodated/
- Section 7.4.4.  Typo. s/calbration/calibration/
- Section 8.1. Spacing. s/[RFC8126]policy/[RFC8126] policy/
- Section 8.2.  Typo. s/Regsirty/Registry/
- Section 8.2. Typo. s/unchamged/unchanged/
- Section 9.  Typo. s/secuity/security/

[Ballot comment]
Thank you for this document.

I only have a couple of comment/questions, and found few nits while reading.

  Performance Metrics Registry MAY use the same
I'm not sure 2119/8174 language is needed here.

I'm not sure to understand the Version (of Registry Format) column. All entries will have the same number there, right?
If this RFC-to-be is updated I guess we'll change to version 2. What shall happen to existing entries, will they keep Version=1 or adopt Version=2?


s/in order to included/in order to be included/

  As any IETF registry, the primary use for a registry is to manage a
  registry for its use within one or more protocols.
This sentence seems a bit hard to parse

s/other form of Performance Metric/other form of Performance Measurement/ ?

Document Writeup – draft-ietf-ippm-metric-registry-19

Summary

The document shepherd is Bill Cerveny. The responsible area director is Mirja Kühlewind.

The requested publication type is best current practice, based on recommendation of the authors and wg, as well as previous experience with metric registries as explained in the draft.

This document defines the format for the IANA Performance Metrics Registry.  This document also gives a set of guidelines for Registered Performance Metric requesters and reviewers.

Review and Consensus

This document has been discussed since 2014 (5 years), initially to support LMAP requirements. This document has broad working group support.

Mock-ups of the implementation of the registry itself have been prepared with IANA's help. A recent version is available here: http://encrypted.net/IETFMetricsRegistry-106.html

Intellectual Property

All authors have confirmed there is no intellectual property to report.

[Ballot comment]
Please be aware of the following note in the IANA section (which might simply reviewing):

"Mock-ups of the implementation of this set of requests
  have been prepared with IANA's help during development of this memo,
  and have been captured in the Proceedings of IPPM working group
  sessions.  IANA is currently preparing a mock-up.  A recent version
  is available here: http://encrypted.net/IETFMetricsRegistry-106.html"

[Ballot comment]
Thank you for the work put into this document. It is quite extensive (I even wonder whether it was useful to indicate the motivations for the registry).

There are a couple of COMMENTs below; feel free to ignore them but I would appreciate if you replied.

Regards,

-éric

== COMMENTS ==

-- Section 5 --
"interpretable by the user." who is the user in this case? (I have my guess of course but let's try to be clear)

Why specifying "implementable by the **software** designer," ? I.e., are HW designers out of scope ?

"accurate" is also quite vague

Also a couple of nits in the section about ',' or '.' and upper/lower case characters.

-- Section 6.1 --
s/will/should/ in "Why this Attempt Will Succeed" ? ;-)

-- Section 7 --
The table part below is quite unclear at first and second reading. Worth re-wording ?
"
Category
------------------
Column |  Column |
"
Or perhaps use a tree form (à la YANG module tree) ?

-- Section 7.1..2 --
Probably worth mentioning "such as and not limited to" rather than "such as" ?

It is also unclear how the MetricType, Method, ... can be extended.

-- Section 11.1.3. --
Should the URL be an https:// one ?

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Functions Operator has completed its review of draft-ietf-ippm-metric-registry-20. If any part of this review is inaccurate, please let us know.

IANA has noted the detailed instructions in this draft for the establishment of registries and the initial values created in the companion document [draft-ietf-ippm-initial-registry]. Upon publication of these two documents, IANA will work with the authors to establish and populate the registries.

Note:  The actions requested in this document will not be completed until the document has been approved for publication as an RFC. This message is meant only to confirm the list of actions that will be performed.

Thank you,

Sabrina Tanamal
Senior IANA Services Specialist

The following Last Call announcement was sent out (ends 2019-11-06):

From: The IESG <iesg-secretary@ietf.org>
To: IETF-Announce <ietf-announce@ietf.org>
CC: ietf@wjcerveny.com, draft-ietf-ippm-metric-registry@ietf.org, ippm-chairs@ietf.org, ietf@kuehlewind.net, ippm@ietf.org
Reply-To: last-call@ietf.org
Sender: <iesg-secretary@ietf.org>
Subject: Last Call: <draft-ietf-ippm-metric-registry-20.txt> (Registry for Performance Metrics) to Best Current Practice


The IESG has received a request from the IP Performance Measurement WG (ippm)
to consider the following document: - 'Registry for Performance Metrics'
  <draft-ietf-ippm-metric-registry-20.txt> as Best Current Practice

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2019-11-06. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


  This document defines the format for the IANA Performance Metrics
  Registry.  This document also gives a set of guidelines for
  Registered Performance Metric requesters and reviewers.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-ippm-metric-registry/

IESG discussion can be tracked via
https://datatracker.ietf.org/doc/draft-ietf-ippm-metric-registry/ballot/


No IPR declarations have been submitted directly on this I-D.

Document Writeup – draft-ietf-ippm-metric-registry-19

Summary

The document shepherd is Bill Cerveny. The responsible area director is Mirja Kühlewind.

The requested publication type is best current practice, based on recommendation of the authors and wg.

This document defines the format for the IANA Performance Metrics Registry.  This document also gives a set of guidelines for Registered Performance Metric requesters and reviewers.

Review and Consensus

This document has been discussed since 2014 (5 years), initially to support LMAP requirements. This document has broad working group support.

Intellectual Property

All authors have confirmed there is no intellectual property to report.

Document Writeup – draft-ietf-ippm-metric-registry-19

Summary

The document shepherd is Bill Cerveny. The responsible area director is Mirja Kühlewind.

The requested publication type is best current practice, based on recommendation of the authors and

This document defines the format for the IANA Performance Metrics Registry.  This document also gives a set of guidelines for Registered Performance Metric requesters and reviewers.

Review and Consensus

This document has been discussed since 2014 (5 years), initially to support LMAP requirements. This document has broad working group support.

Intellectual Property

All authors have confirmed there is no intellectual property to report.

Document Writeup – draft-ietf-ippm-metric-registry-19

Summary

The document shepherd is Bill Cerveny. The responsible area director is Mirja Kühlewind.

The requested publication type is best current practice, based on recommendation of the authors and

This document defines the format for the IANA Performance Metrics Registry.  This document also gives a set of guidelines for Registered Performance Metric requesters and reviewers.

Review and Consensus

This document has been discussed since 2014 (5 years), initially to support LDAP requirements. This document has broad working group support.

Intellectual Property

All authors have confirmed there is no intellectual property to report.

Document Writeup – draft-ietf-ippm-metric-registry-19

Summary

The document shepherd is Bill Cerveny. The responsible area director is Mirja Kühlewind.

The requested publication type is best current practice, based on recommendation of the authors and

This document defines the format for the IANA Performance Metrics Registry.  This document also gives a set of guidelines for Registered Performance Metric requesters and reviewers.

Review and Consensus

This document has been discussed since 2014 (5 years), initially to support LDAP requirements. This document has broad working group support.

Intellectual Property

All authors have confirmed there is no intellectual property to report.