Reporting IP Network Performance Metrics: Different Points of View
draft-ietf-ippm-reporting-metrics-09

[Ballot comment]
- Please address the OPS-DIR review by Bert Wijnen

    The one thing that concerns me a little bit is the fact that this
    document uses RFC2119 language. I think that is in-appropriate.
    Using lower case for the MUST, SHOULD and RECOMMEND in the document
    is perfectly fine I think.

- Support Adrian's comment regarding the title "Reporting IP Network Performance Metrics: Different Points of View"


-  Next to the question "How will the results be used?", it would have been nice to ask the question "Which audience will read the results"
Network Characterization = network operator
Application Performance Estimation = application designer, service developer, etc..
Actually, this is what you did, without clearly mentioning it, asking the question about "how", and answering with "two main audience categories"

-

  2.  Application Performance Estimation - describes the network
      conditions in a way that facilitates determining affects on user
      applications, and ultimately the users themselves.  This point-
      of-view looks outward, toward the user(s), accepting the network
      as-is.

What do you mean "accepting the network as-is."?
It's not because the results will be used for application performance estimation that you can't optimize your network.

- "The scope of this memo primarily covers the design and reporting of
  the loss and delay metrics [RFC2680] [RFC2679]."

What do you mean by design of metric? Do you mean choosing the measurement characteristics of a metric?
Note: multiple occurrences of "metric design" in the draft.

-  Section 2
  "These memos effectively describe two
  different categories of metrics,

  o  [RFC3148] includes restrictions of congestion control and the
      notion of unique data bits delivered, and

  o  [RFC5136] using a definition of raw capacity without the
      restrictions of data uniqueness or congestion-awareness.

  It might seem at first glance that each of these metrics has an
  obvious audience (Raw = Network Characterization, Restricted =
  Application Performance), but reality is more complex and consistent
  with the overall topic of capacity measurement and reporting.  For
  example, TCP is usually used in Restricted capacity measurement
  methods, while UDP appears in Raw capacity measurement. "

I was not sure what you meant by Raw and Restricted.

However, I saw a definition way down in the document, in section 6 and 7
  Raw capacity refers to the metrics defined in [RFC5136] which do not
  include restrictions such as data uniqueness or flow-control response
  to congestion...

  Restricted capacity refers to the metrics defined in [RFC3148] which
  include criteria of data uniqueness or flow-control response to
  congestion...

Please add those "definitions" in section 2.
It's specifically important since RFC5136 and RFC3148 don't mention Raw/Restricted

- I learned to avoid "we", "our", "us" in RFCs.
I double-checked if it's still the case with the RFC-editor. I will let you know the answer.

- I would add an extra point to "For these and other reasons, such as"
Something such as:

o the ability to drill down to a specific measurement interval for deeper analysis

Justification: most of the time, when checking SLA, we check with large measurement interval, but want to ability to do a postmortem analysis

- I don't understand
  "Fortunately, application performance estimation activities are not
  adversely affected by the estimated worst-case transfer time. 
  Although the designer's tendency might be to set the Loss Threshold
  at a value equivalent to a particular application's threshold, this
  specific threshold can be applied when post-processing the
  measurements. "

- "We can say that the Delay and Loss metrics are Orthogonal"
Orthogonal -> orthogonal?

- section 7.4.  Bulk Transfer Capacity Reporting

  When BTC of a link or path is estimated through some measurement
  technique, the following parameters SHOULD be reported:


Also transport type, link layer type, tunneling yes/no, etc...?

- Personal preference, no need to modify the document unless you feel like it.
All my customers are interested in delay, loss, and delay variation (jitter).
It would have been nice to have a clear pointer in the table of content, with a clear entry "Effect of POV on the Delay Variation Metric . . . . . . . . . . . . . ." instead of addressing delay variation in "delay metric" section 5.1.3

-

Section 4.1 of [RFC3393] describes this specification and its
  rationale (ipdv = inter-packet delay variation in the quote below).
Use IPDV (Remember you used Packet Delay Variation (PDV)) in the document, and refer to RFC5481
Several ipdv instances in the draft.

-

  "Network Characterization has traditionally used Poisson-distributed
  inter-packet spacing, as this provides an unbiased sample."

Is this correct? or Poisson-distributed start, with fixed inter-packet spacing, to match, for example, a voice/video application

[Ballot comment]
As per your reply to Eliot Lear's Apps Directorate review, please un-2119 the document. I don't think it's appropriate for this document.

You say "packets of type-P". Shouldn't that be "packets of type P" without the hyphen? Also, "type C"? With the hyphens, I'm not quite sure what you're talking about. Perhaps this is just unclear to someone outside the area.

[Ballot comment]
- 3.1: what does it mean to say the 51 seconds value
was "calculated above" when its (now, presumably)
done in 4.1.1. (Couldn't you have arranged that 42
seconds was the answer?)

- 8.2: might have been a nice thing to include some
reasonable representative sample sizes for some
statistics for some measurements. Definitely too
much to try add something with broad coverage, but
one good, and one bad, set of example numbers
would be a fine addition if someone had time.

[Ballot comment]
I have no objection to the publication of this document, but I think
it would be helpful if the document title reflected the fact that the
metrics being reported are IP network performance metrics.Perhaps...

  Reporting IP Network Performance Metrics: Different Points of View


I also have some small Comments as follows...

---

I think the document would benefit from a further read-through to fix
some of the English and readability issues.  Leaving these to the RFC
Editor risks errors of meaning being introduced during the edit
process.

---

Section 3.

  This section gives an overview of recommendations

And...

Section 3.1.

  This section gives an overview of reporting recommendations for the
  loss, delay, and delay variation metrics.

But...

Section 3.1

  The minimal report on measurements MUST include both Loss and Delay
  Metrics.

This "MUST" is not a recommendation. You need to decide whether you are
writing recommendations (which seems wholy appropriate since there are
no operational or interop implications of missing out some measurements)
or writing requirements.

Notwithstanding the resolution of the above point, I am not convinced
that you really need to use RFC 2119 language in this document.

---

Section 3.1

"We have calculated a waiting time" needs a forward reference to the
place in the document where this calculation is performed.

---

Section 3.1

"99.9%-ile" is really ugly!

---

A bit puzzled by Section 4.1.1 where you have

            n
          ---
          \
D = t  +  >  (t  +  q )
      0    /    i    i
          ---
          i = 1

Presume you decided to not consider queue at the source node because you
consider it as the generator of the packets and not subject to queuing.
This is slightly suspect in my opinion and depends on the nature of
- the source node
- the definition of the path.

Given this I wonder whether it is right to exclude q at the source or
to include q at the destination. In any case, it would be helpful to
explain your choices. But (of course) given the numbers being used to
arrive at D using this formula including or excluding one queue time is
not really significant.

It would also be nice to note that there are n+1 nodes on your path and
to clarify that q(i) is the delay due to queuing at node at the far end
of the ith link.

---

Not sure why section 4.3 is present in this document. It doesn't seem to
leverage or be leveraged by anything else in the document. What is more,
the concluding sentence ("After waiting sufficient time, packet loss can
probably be attributed to one of these causes.") is rather vague and out
of scope for the practice of measurement. Recall, the objective of ippm
isto takemeasurementsandprovide reports, not to make qualatative
assessments of the results.

---

Section 10

Are there no security considerations associated with running the
tests over longer periods of time? What if keys roll during the
measurement period? Don'tlong periods offer more chance of seeing an
attack?

[Ballot comment]
Substantive suggestions; please respond to these:

-- Section 5.2 --
  When both the sample mean and median are available, a comparison will
  sometimes be informative, because these two statistics are equal only
  when the delay distribution is perfectly symmetrical.

I'm not a statistician, but I don't think that's true.  For example, this has a symmetrical distribution with 5 as the mean and median:

1 1 4 4 5 6 6 9 9

But this also has mean and median of 5, and its distribution is not symmetrical:

1 2 3 4 5 6 6 9 9

Am I missing something?

========
Editorial suggestions.  No need to respond to these; take them or modify them as you please:

Throughout: there's no reason to hyphenate "point of view".

-- Introduction --
      in a way that facilitates determining affects on user
      applications,

"effects", not "affects".

-- Section 2 --
      [RFC5136] using a definition of raw capacity without the
      restrictions of data uniqueness or congestion-awareness.

Don't hyphenate "congestion awareness".

-- Section 3 --
Don't hyphenate "long term" here.  (The rule is that a compound modifier is hyphenated, but if it's not being used as a modifier (an adjective or adverb), it shouldn't be hyphenated.)

-- Section 3.1 --
  We have calculated a waiting time above that
  should be sufficient to differentiate between packets that are truly
  lost or have long finite delays under general measurement
  circumstances, 51 seconds.  Knowledge of specific conditions can help
  to reduce this threshold, but 51 seconds is considered to be
  manageable in practice.

"above"?  Does this need to be re-worded?  Maybe "above which it", or some such?  And 51 seconds seems oddly precise: does 50 seconds really not work, and is it really not appropriate to call it 55 or 60 ?  (Just asking; I have no idea of the answer here.)

  For example, the 99.9%-ile minus the minimum

I suggest just spelling out "percentile" here (and in 5.2); you're not tight on column-inches.  If you're worried, you can compensate by removing the extraneous "identified as" in the net paragraph.

-- Section 3.2 --

  The result would ideally appear in the same form as though a
  continuous measurement was conducted.

Needs subjunctive mood: "had been conducted."

  intervals it is possible to present the results as "metric A was less
  than or equal to objective X during Y% of time.

Missing closing quote.

  NOTE that numerical thresholds of acceptability are not set in IETF
  performance work and are explicitly excluded from the IPPM charter.

Once the RFC is published, its connection with the IPPM working group is not obvious.  I suggest just saying, "and are out of scope for this document," or some such.

-- Figure 2 --
I suggest moving "where j is the hop number where the loop begins" out of the figure, since you already have two other "wheres" out there.  You also don't say what "n" is, and should.  I see from below that it's the number of hops.  So make it, "where n is the total number of hops, j is the hop number where the loop begins, C is the number of times a packet circles the loop, and TTL is the packet's initial Time-to-Live value".

-- Section 4.3 --
In bullet 5, I would add a comma after "checking", to break up the length and to avoid confusion about what "and" conjoins.

-- Section 5.1.2 --

  As further evidence of overlap, consider the Cumulative Distribution
  Function (CDF) of Delay when the value positive infinity is assigned
  to all lost packets.

I suggest quoting "positive infinity" to set it off clearly.

  Although infinity is a familiar mathematical concept, it is somewhat
  disconcerting to see any time-related metric reported as infinity, in
  the opinion of the authors.

This is consensus of the WG, not opinion of authors, right?  I suggest just ending the sentence at the comma.  If you need to waffle, make it "it can be somewhat disconcerting".

-- Section 5.3 --
  the most efficient practice is to distinguish between truly lost and
  delayed packets with a sufficiently long waiting time, and to
  designate the delay of non-arriving packets as undefined.

Again, it's easy to misread what the "and" conjoins.  How about this way?:
NEW
  the most efficient practice is to distinguish between packets
  that are truly lost and those that are delayed with a sufficiently
  long waiting time, and to designate the delay of non-arriving
  packets as undefined.

-- Section 7.5 --
Last paragraph begins with a lower-case "w".

[Ballot comment]
Substantive suggestions; please respond to these:

-- Section 5.2 --
  When both the sample mean and median are available, a comparison will
  sometimes be informative, because these two statistics are equal only
  when the delay distribution is perfectly symmetrical.

I'm not a statistician, but I don't think that's true.  For example, this has a symmetrical distribution with 5 as the mean and median:
1 1 4 4 5 6 6 9 9
But this also has mean and median of 5, and its distribution is not symmetrical:
1 2 3 4 5 6 6 9 9
Am I missing something?

========
Editorial suggestions.  No need to respond to these; take them or modify them as you please:

Throughout: there's no reason to hyphenate "point of view".

-- Introduction --
      in a way that facilitates determining affects on user
      applications,

"effects", not "affects".

-- Section 2 --
      [RFC5136] using a definition of raw capacity without the
      restrictions of data uniqueness or congestion-awareness.

Don't hyphenate "congestion awareness".

-- Section 3 --
Don't hyphenate "long term" here.  (The rule is that a compound modifier is hyphenated, but if it's not being used as a modifier (an adjective or adverb), it shouldn't be hyphenated.)

-- Section 3.1 --
  We have calculated a waiting time above that
  should be sufficient to differentiate between packets that are truly
  lost or have long finite delays under general measurement
  circumstances, 51 seconds.  Knowledge of specific conditions can help
  to reduce this threshold, but 51 seconds is considered to be
  manageable in practice.

"above"?  Does this need to be re-worded?  Maybe "above which it", or some such?  And 51 seconds seems oddly precise: does 50 seconds really not work, and is it really not appropriate to call it 55 or 60 ?  (Just asking; I have no idea of the answer here.)

  For example, the 99.9%-ile minus the minimum

I suggest just spelling out "percentile" here (and in 5.2); you're not tight on column-inches.  If you're worried, you can compensate by removing the extraneous "identified as" in the net paragraph.

-- Section 3.2 --

  The result would ideally appear in the same form as though a
  continuous measurement was conducted.

Needs subjunctive mood: "had been conducted."

  intervals it is possible to present the results as "metric A was less
  than or equal to objective X during Y% of time.

Missing closing quote.

  NOTE that numerical thresholds of acceptability are not set in IETF
  performance work and are explicitly excluded from the IPPM charter.

Once the RFC is published, its connection with the IPPM working group is not obvious.  I suggest just saying, "and are out of scope for this document," or some such.

-- Figure 2 --
I suggest moving "where j is the hop number where the loop begins" out of the figure, since you already have two other "wheres" out there.  You also don't say what "n" is, and should.  I see from below that it's the number of hops.  So make it, "where n is the total number of hops, j is the hop number where the loop begins, C is the number of times a packet circles the loop, and TTL is the packet's initial Time-to-Live value".

-- Section 4.3 --
In bullet 5, I would add a comma after "checking", to break up the length and to avoid confusion about what "and" conjoins.

-- Section 5.1.2 --

  As further evidence of overlap, consider the Cumulative Distribution
  Function (CDF) of Delay when the value positive infinity is assigned
  to all lost packets.

I suggest quoting "positive infinity" to set it off clearly.

  Although infinity is a familiar mathematical concept, it is somewhat
  disconcerting to see any time-related metric reported as infinity, in
  the opinion of the authors.

This is consensus of the WG, not opinion of authors, right?  I suggest just ending the sentence at the comma.  If you need to waffle, make it "it can be somewhat disconcerting".

-- Section 5.3 --
  the most efficient practice is to distinguish between truly lost and
  delayed packets with a sufficiently long waiting time, and to
  designate the delay of non-arriving packets as undefined.

Again, it's easy to misread what the "and" conjoins.  How about this way?:
NEW
  the most efficient practice is to distinguish between packets
  that are truly lost and those that are delayed with a sufficiently
  long waiting time, and to designate the delay of non-arriving
  packets as undefined.

-- Section 7.5 --
Last paragraph begins with a lower-case "w".

IESG:

IANA has reviewed draft-ietf-ippm-reporting-metrics-08.txt, which is
currently in Last Call, and has the following comments:

We understand that this document doesn't require any IANA actions.

State changed to In Last Call from Last Call Requested

The following Last Call Announcement was sent out:

From: The IESG

To: IETF-Announce

CC:

Reply-To: ietf@ietf.org

Subject: Last Call:  (Reporting Metrics: Different Points of View) to Informational RFC





The IESG has received a request from the IP Performance Metrics WG (ippm)

to consider the following document:

- 'Reporting Metrics: Different Points of View'

  as an Informational RFC



The IESG plans to make a decision in the next few weeks, and solicits

final comments on this action. Please send substantive comments to the

ietf@ietf.org mailing lists by 2012-04-11. Exceptionally, comments may be

sent to iesg@ietf.org instead. In either case, please retain the

beginning of the Subject line to allow automated sorting.



Abstract





  Consumers of IP network performance metrics have many different uses

  in mind.  The memo provides "long-term" reporting considerations

  (e.g, days, weeks or months, as opposed to 10 seconds), based on

  analysis of the two key audience points-of-view.  It describes how

  the audience categories affect the selection of metric parameters and

  options when seeking info that serves their needs.











The file can be obtained via

http://datatracker.ietf.org/doc/draft-ietf-ippm-reporting-metrics/



IESG discussion can be tracked via

http://datatracker.ietf.org/doc/draft-ietf-ippm-reporting-metrics/ballot/





No IPR declarations have been submitted directly on this I-D.

      Reporting Metrics: Different Points of View
      draft-ietf-ippm-reporting-metrics-07

  (1.a) Who is the Document Shepherd for this document? Has the
        Document Shepherd personally reviewed this version of the
        document and, in particular, does he or she believe this
        version is ready for forwarding to the IESG for publication?

Document sheperd is Henk Uijterwaal.

  (1.b) Has the document had adequate review both from key WG members
        and from key non-WG members?

Yes, the document was reviewed by some members of the working group.

  (1.c) Does the Document Shepherd have concerns that the document
        needs more review from a particular or broader perspective,

No.

  (1.d) Does the Document Shepherd have any specific concerns or
        issues with this document that the Responsible Area Director
        and/or the IESG should be aware of?

No.

  (1.e) How solid is the WG consensus behind this document? Does it
        represent the strong concurrence of a few individuals, with
        others being silent, or does the WG as a whole understand and
        agree with it?

Good, the topic has been under discussion for years and the consensus
is that this document (and the related one on short term reports)
reflect current thinking about this topic.


  (1.f) Has anyone threatened an appeal or otherwise indicated extreme
        discontent?

No.

  (1.g) Has the Document Shepherd personally verified that the
        document satisfies all ID nits?

Yes, the pre RFC 5378 disclaimer is, afaik, necessary.

  (1.h) Has the document split its references into normative and
        informative?

Yes
        Are there normative references to documents that
        are not ready for advancement or are otherwise in an unclear
        state?

No.  I don't see an issue with the informative reference to
draft-ietf-ippm-reporting.

  (1.i) Has the Document Shepherd verified that the document IANA
        consideration section exists and is consistent with the body
        of the document?

Yes, it is empty.

  (1.j) Has the Document Shepherd verified that sections of the
        document that are written in a formal language, such as XML
        code, BNF rules, MIB definitions, etc., validate correctly in
        an automated checker?

N/A

  (1.k) The IESG approval announcement includes a Document
        Announcement Write-Up. Please provide such a Document
        Announcement Write-Up? Recent examples can be found in the
        "Action" announcements for approved documents. The approval
        announcement contains the following sections:

    Technical Summary

Consumers of IP network performance metrics have many different uses
  in mind.  The memo provides "long-term" reporting considerations
  (e.g, days, weeks or months, as opposed to 10 seconds), based on
  analysis of the two key audience points-of-view.  It describes how
  the audience categories affect the selection of metric parameters and
  options when seeking info that serves their needs.


    Working Group Summary
The normal WG process was followed and the document has been discussed for
several years.  The document as it is now, reflects WG consensus, with nothing
special worth noticing.

    Document Quality
Good