Application-Layer Traffic Optimization (ALTO) Performance Cost Metrics
draft-ietf-alto-performance-metrics-28

[Ballot comment]
Thanks for the updates from -21 through -26, and my apologies to have
taken so long to review the changes!

My previous discuss point about the "cur" operator of the "bw-utilized"
metric is no longer relevant after the removal of that metric from this
document.  I also appreciate the updated examples (including
Content-Length); I seem to now get values that are off by one from what's
currently in the -26, but given that the authors have consciously looked
into the topic and I have low confidence in my own methodology, I will
drop that as a Discuss point.

[Ballot comment]
Based of our email discussion and agreed changes ( working copy here : https://github.com/ietf-wg-alto/draft-ietf-alto-performance-metrics), I am balloting no Objection. Thanks for addressing my discuss and comments.

[Ballot comment]
Thank you for addressing my previous DISCUSS point, I have now updated my ballot position into a NO OBJECT.

Nevertheless, I would have appreciated a reaction on the IPv4-only examples and the focus on TCP only but this is obviously not blocking this document. I appreciate that my other comments were used to improve the document.

Regards

-éric

—— below is for archiving only —


Thank you for the work put into this document. Please bear with my lack of knowledge about ALTO in general.

Please find below one trivial blocking DISCUSS points (probably easy to address), some non-blocking COMMENT points (but replies would be appreciated even if only for my own education), and some nits.

Special thanks to Jan Seedorf for the shepherd's write-up about the WG consensus (even if not using the usual template).

I have appreciated the "operational considerations" section as it addresses many questions that popped up during reading the document; notably, how can the ALTO server measure any metric between the ALTO client and a resource.

I hope that this helps to improve the document,

Regards,

-éric


== COMMENTS ==

Minor regret about the examples as they are all about the IPv4 address family especially in a world of happy eyeballs where the IPv4 and IPv6 paths may still have different performance metrics.

-- Section 2.1 --
Should the figure 1 use "perf monitoring tools" rather than "management tool" ?

-- Section 4 --
This section title is about 'bandwidth' but the first sub-section is about 'throughput', while these concepts are related they are also distinct. How can the reader reconciliate them ?

-- Section 4.1 --
Is the intent of ALTO to only work for TCP and not for other transport protocols ? I.e., is QUIC out of scope ?


-- Section 4.2.3 --
Where are those 'tunnels' in "by subtracting tunnel reservations " coming from ? Probably about RSVP-TE but what is the link with ALTO ? (Again I am not familiar with ALTO so this may be an uneducated question).

== NITS ==


-- Section 3.1.3 --
Probably tedious to do but why not replacing "TBA" by the actual value in the examples for 'content-length' ?

[Ballot comment]
Thank you for the work on this document, and for addressing my previous DISCUSS points.

Many thanks to Christian Amsüss for his review: https://mailarchive.ietf.org/arch/msg/art/owYhcoFnl1vEipZ2D62cWiiE-LA/ , and thanks to the authors for addressing it.

Francesca

[Ballot discuss]
Thank you for the work on this document, and for addressing my previous DISCUSS points.

Many thanks to Christian Amsüss for his review: https://mailarchive.ietf.org/arch/msg/art/owYhcoFnl1vEipZ2D62cWiiE-LA/ , and thanks to the authors for addressing it.

Francesca

It seems clear that IESG evaluation will extend to the next IESG post IETF 113. Changes have been substantial; when existing DISCUSSes are cleared (including successor ADs given an opportunity to affirm/rescind DISCUSS from their predecessors), Martin will conduct a second AD review of the document and determine if there are additional steps needed before returning it to the ballot.

It seems clear that IESG evaluation will extend to the next IESG post IETF 113. Changes have been substantial; when existing DISCUSSes are cleared (including successor ADs given an opportunity to affirm/rescind DISCUSS from their predecessors), Martin will conduct a second AD review of the document and determine if there are additional steps needed before returning it to the ballot.

[Ballot discuss]
Thank you for the work on this document, and for addressing my previous DISCUSS points. I noticed two additional JSON issue, easy to fix, reported below.

Many thanks to Christian Amsüss for his review: https://mailarchive.ietf.org/arch/msg/art/owYhcoFnl1vEipZ2D62cWiiE-LA/ , and thanks to the authors for addressing it.

As noted in https://www.ietf.org/blog/handling-iesg-ballot-positions/, a
DISCUSS ballot is a request to have a discussion; I really think that the
document would be improved with a change here, but can be convinced otherwise.

Francesca

1. -----

Section 4.4.3

  {
    "cost-type" { "cost-mode":  "numerical",
                  "cost-metric": "bw-utilized"},
    "endpoints":  {
        "srcs": [ "ipv4 : 192.0.2.2" ],
        "dsts": [
          "ipv4:192.0.2.89",
          "ipv4:198.51.100.34"
        ]
    }
  }

FP: JSON doesn't validate: missing ":" after "cost-type".

2. -----

Section 4.3.3.

  {
    "cost-type" { "cost-mode":  "numerical",
                  "cost-metric": "bw-available"},
    "endpoints":  {
        "srcs": [ "ipv4 : 192.0.2.2" ],
        "dsts": [
          "ipv4:192.0.2.89",
          "ipv4:198.51.100.34"
        ]
    }
  }

FP: JSON doesn't validate: missing ":" after "cost-type". (Minor note - is there a reason why the "srcs" address has whitespaces while other addresses don't? 3 occurrences in the text).

[Ballot discuss]
Thank you for addressing my previous discuss points with the -21 (and my
apologies for the spurious one!); I'm glad to see that they were indeed
easy to address.

However, I have looked over the changes from -20 to -21 and seem to have
found a couple more issues that should be addressed:

(1) I can't replicate the Content-Length values in the examples (I only
looked at Examples 1 and 2).  Can you please share the methodology used
to generate the values?  My testing involved copy/paste from the
htmlized version of the draft to a file, manually editing that file to
remove the leading three spaces that come from the formatting of the
draft, and using Unix wc(1) on the resulting file.  It looks like the
numbers reported in the -21 are computed as the overall number of
characters in the file *minus* the number of lines in the file, but I
think it should be the number of characters *plus* the number of lines,
to accommodate the HTTP CRLF line endings.  (My local temporary files
contain standard Unix LF (0x0a) line endings, verified by hexdump(1).)

(2) We seem to be inconsistent about what the "cur" statistical operator
for the "bw-utilized" metric indicates -- in §4.4.3 it is "the current
instantaneous sample", but in §4.4.4 it is somehow repurposed as "The
current ("cur") utilized bandwidth of a path is the maximum of the
available bandwidth of all links on the path."

[Ballot comment]
I cannot currently provide a concise explanation of the nature of my
unease with the "bw-utilized" metric specification that is new in this
revision (so as to elevate it to a Discuss-level concern), but I
strongly urge the authors and WG to consider my comments on Section 4.4.3.

The new text in Section 1 explaining the origins of the metrics (e.g.,
from TE performance metrics) and why some other TE metrics are not
defined is nicely done.  I trust the responsible AD and WG chairs to
ensure that it, and the other places where we have added new exposition,
has gotten the appropriate level of review from the WG membership.

Section 3.1.2, 3.2.2

I see that the delay-ow and delay-rt semantics have been changed from
milliseconds to microseconds going from -20 to -21.  Either
representation seems fine, but it may be risky to make such a change so
late in the publication process, especially if there are already
implementations in place.  I also don't see any AD ballot comments that
seem to motivate the change, so I'm a bit curious how it arose -- is it
for consistency with the corresponding TE link metrics?

Section 3.3.3

  Intended Semantics: To specify temporal and spatial aggregated delay
  variation (also called delay jitter)) with respect to the minimum
  delay observed on the stream over the one-way delay from the
  specified source and destination, where the one-way delay is defined
  in Section 3.1.  A non-normative reference definition of end-to-end
  one-way delay variation is [RFC3393].  [...]

I note that RFC 3393 explicitly says that as part of the metric, several
parameters must be specified, most notably the selection function F that
unambiguously defines the two packets selected for the metric.  While
it's allowed for F to select as the "first" packet the one with the
smallest one-way delay, which maps up to the "with respect to the
minimum delay observed on the stream" here, it seems to me that it's
fairly important to call out that we are not allowing the full
flexibility of the RFC 3393 metric.  Assuming, of course, that we
specifically have that as the intent, versus allowing the full
generality of RFC 3393.  If there has been some research results since
RFC 3393 was published that indicate that it's preferred to use the
minimum delay for this purpose, that might be worth listing as a
reference in addition to RFC 3393.

Section 3.4.4

The estimation of end-to-end loss rate as the sum of per-link loss rates
is (1) only valid in the low-loss limit, and (2) assumes that each
link's loss events are uncorrelated with every other link's loss events.
The current text does mention (2) in the form of "should be cognizant of
correlated loss rates", but I don't think it touches on (1) at all.
(The general formula for aggregating loss assuming each link is
independent is to compute end-to-end loss as one minus the product of
the success rate for each link.)

Section 4.4.3

It seems like there may some subtlety in the interpretation of the
"bw-utilized" metric, which leads me to wonder if more caution is
advised prior to adding new metrics at this stage in the document
lifecycle.  In particular, it seems like it would be natural to attempt
to compare the "bw-utilized" value against the "bw-maxres" value and
"bw-residual" value, but it seems to me that the inferences that can be
made by such comparisons will depend on the topology in question.
Consider, for example,

Routers and link capacities between them:

      1Gbps            10Gbps            1Gbps
  +-----------------+=================+--------------+
  A                B                C              D

If there is a flow using 6GBps from B to C, that would show up when
querying "bw-utilized" between A and B, but that 6Gbps is obviously more
than both the maximum reservable and residual bandwidth end-to-end from
A to D; likewise, the 4GBps of residual bandwidth on the B-to-C link is
also more than the achievable bandwidth end-to-end from A to D.  So it
seems like the utilized bandwidth is potentially from totally unrelated
flows on paths that only have a minimal set of links in common with the
path being queried.  How do we expect someone to use the reported
"bw-utilized" values?

To put it differently, I don't think that the specification of "the
maximum utilized bandwidth among all links from the source to the
destination" will actually provide the desired "utilized bandwidth of
the path from the source to the destination", since the procedure as
stated can report a bandwidth that corresponds to a different path.


NITS

Section 1

s/"Semantics Base On" column/"Semantics Based On" column/ (in the prose,
first paragraph after the table).

Section 4.3

The section heading has a typo: s/Availlble/Available/

[Ballot comment]
I support Francesca's DISCUSS.

I suggest breaking Section 7 into separate subsections for each request.  Specifically, the registrations in the first paragraph should be clearly separated from the creation of the registry that starts below that.

Maybe I'm turning into a dinosaur, but since all of the syntaxes in these documents are constrained to US-ASCII, I wonder why listing Unicode characters individually, or ranges of them (e.g., Section 2.1), is preferred to using ABNF.

All of the SHOULDs in this document feel weak to me.  I can't tell, for example, why an implementer might do anything other than what follows the SHOULD, which suggests to me that they should be MUSTs.  If there's some reason to leave the option there, I think some supporting text would be warranted.

[Ballot discuss]
I perhaps understand the intention of extending the ALTO protocol so that the ALTO client and server have defined way of exchanging values for already defined metrics. However, I need to agree with my fellow AD colleagues that this document need to describe why those metrics are needed and describe the relationship with other RFCs those defines those metrics mostly for other contexts. To that end all the RFCs in the Table 1 in section 1 need to be normative references.

[Ballot comment]
Thanks for the work on this document and thanks to Brian Trammell for his TSVART early review.

I have following comments which I believe will improve the document quality -

1. In the abstract I read about "a better delay performance" and was hoping it will be clear to me what is "a better delay performance". Unfortunately, I was unable to get that. This comes to the point that this document needs to describe why purpose of using the defined metrics well.

2. Section 2.2 says

    The number MUST be a non-negative JSON integer in the range [0, 100] (i.e., greater than or equal to 0 and less than or equal to 100), followed by an optional decimal part, if a higher precision is needed.

  This should be a JSON number type not integer type.

3. There are number of broken JSON examples. for example, in section 4.2.3
    "ipv4:192.0.2.2" {
      "ipv4:192.0.2.89" :    0,
      "ipv4:198.51.100.34": 2000
    }
  missing ":" after  ipv4:192.0.2.2

4. Content-Length: TBA in the examples, I actually don't know how to interpret it.

[Ballot discuss]
These should all be trivial to resolve -- just some minor internal
inconsistencies that need to be fixed before publication.

The discussion of percentile statistical operator in §2.2 is internally
inconsistent -- if the percentile number must be an integer, then p99.9
is not valid.

Also, the listing of "cost-source" values introduced by this document
(in §5.1) does not include "nominal", but we do also introduce "nominal".

Similarly, in §3.1.3 we refer to the "-" component of a cost
metric string, that has been generalized to an arbitrary statistical
operator.

[Ballot comment]
All things considered, this is a pretty well-written document that was
easy to read.  That helped a lot as I reviewed it, especially so on a
week with a pretty full agenda for the IESG telechat.

Section 2.2

Should we say anything about how to handle a situation where a base
metric identifier is so long that the statistical operator string cannot
be appended while remaining under the 32-character limit?

  min:
      the minimal value of the observations.
  max:
      the maximal value of the observations.
  [...]

Should we say anything about what sampling period of observations is in
scope for these operators?

Section 3.x.4

If we're going to be recommending that implementations link to external
human-readable resources (e.g., for the SLA details of estimation
methodology), does the guidance from BCP 18 in indicating the language
of the text come into play?

It's also a bit surprising that we specify the new fields in the
"parameters" of a metric just in passing in the prose, without a more
prominent indication that we're defining a new field.

Section 3.1.4

  "nominal": Typically network one-way delay does not have a nominal
  value.

Does that mean that they MUST NOT be generated, or that they should be
ignored if received, or something else?  (Similarly for the other
sections where we say the same thing.)

  This description can be either free text for possible presentation to
  the user, or a formal specification; see [IANA-IPPM] for the
  specification on fields which should be included.  [...]

Is the IANA registry really the best reference for what fields to
include?  Tpically we would only refer to the registry when we care
about the current state of registered values, but the need here seems to
effectively be the column headings of the registry, which could be
obtained from the RFC defining the registry.

Section 3.3.3

  Intended Semantics: To specify spatial and temporal aggregated delay
  variation (also called delay jitter)) with respect to the minimum
  delay observed on the stream over the one-way delay from the
  specified source and destination.  The spatial aggregation level is
  specified in the query context (e.g., PID to PID, or endpoint to
  endpoint).

I do appreciate the note about how this is not the normal statistics
variation that follows this paragraph, but I also don't think this is a
particularly clear or precise specification for how to produce the
number that is to be reported.  It also doesn't seem to fully align with
the prior art in the IETF, e.g., RFC 3393.  It seems like it would be
highly preferrable to pick an existing RFC and refer to its
specification for computing a delay variation value.  (To be clear, such
a reference would then be a normative reference.)

Section 3.4.3

  Intended Semantics: To specify the number of hops in the path from
  the specified source to the specified destination.  The hop count is
  a basic measurement of distance in a network and can be exposed as
  the number of router hops computed from the routing protocols
  originating this information.  [...]

It seems like this could get a little messy if there are multiple
routing protocols in use (e.g., both normal IP routing and an overlay
network, as for service function chaining or other overlay schemes).
I don't have any suggestions for disambiguating things, though, and if
the usage is consistent within a given ALTO Server it may not have much
impact on the clients.

Section 3.4.4

  "sla": Typically hop count does not have an SLA value.

As for "nominal", earlier, is there any guidance to give on not
generating it or what to do if it is received?
(Also appears later, I suppose.)

Section 4.1.4

  "estimation": The exact estimation method is out of the scope of this
  document.  See [Prophet] for a method to estimate TCP throughput.  It
  is RECOMMENDED that the "parameters" field of an "estimation" TCP
  throughput metric provides two fields: (1) a congestion-control
  algorithm name (a field named "congestion-control-alg"); and (2) a
  link (a field named "link")to a description of the "estimation"
  method.  Note that as TCP congestion control algorithms evolve (e.g.,
  TCP Cubic Congestion Control [I-D.ietf-tcpm-rfc8312bis]), it helps to
  specify as many details as possible on the congestion control
  algorithm used.  This description can be either free text for
  possible presentation to the user, or a formal specification.  [...]

Do these specifics go into the "congestion-control-alg" name, or in the
linked content?

Section 5.3

  To address the backward-compatibility issue, if a "cost-metric" is
  "routingcost" and the metric contains a "cost-context" field, then it
  MUST be "estimation"; if it is not, the client SHOULD reject the
  information as invalid.

This seems like a sub-optimal route to backwards compatibility, as it
would (apparently) permanently lock the "routingcost" metric to only the
"estimation" source with no way to negotiate more flexibility.  Unless
we define a new "routingcost2" metric that differs only in the lack of
this restriction, of course.

Section 5.4.1

  the ALTO server may provide the client with two pieces of additional
  information: (1) when the metrics are last computed, and (2) when the
  metrics will be updated (i.e., the validity period of the exposed
  metric values).  The ALTO server can expose these two pieces of
  information by using the HTTP response headers Last-Modified and
  Expires.

While this seems like it would work okay in the usual case, it seems a
bit fragile, in that it may fail in boundary cases, such as when a
server is just starting up.  I would lean towards recommending use of
explicit data items to convey this sort of information (and also the
overall measurement interval over which statistics are computed, which
may not always go back to "the start of time").

Section 5.4.2

  often be link level.  For example, routing protocols often measure
  and report only per link loss, not end-to-end loss; similarly,
  routing protocols report link level available bandwidth, not end-to-
  end available bandwidth.  The ALTO server then needs to aggregate
  these data to provide an abstract and unified view that can be more
  useful to applications.  The server should consider that different
  metrics may use different aggregation computation.  For example, the
  end-to-end latency of a path is the sum of the latency of the links
  on the path; the end-to-end available bandwidth of a path is the
  minimum of the available bandwidth of the links on the path.

Some caution seems in order relating to aggregation of loss
measurements, as loss is not always uncorrolated across links in the
path.

Section 6

I thought that the outcome of the art-art review thread was that we
would add some mention of ordinal cost mode here as a means to mitigate
the risk of exposing sensitive numerical metric values, but I don't see
such test.

In light of the guidance in Section 7 for new cost source types to
document their security considerations, should we document the security
considerations for the "sla" type here?  The overall theme would be
similar to what RFC 7285 already describes, but we could mention that
knowledge specifically of provider SLA targets allow for attackers to
target the SLA, causing problems for the provider other than the typical
DoS attack class.  (I'm not coming up with anything new to say about
"nominal" or "estimation".)

I would also consider mentioning that the "origin" references in table 1
might have useful things to say about the individual metrics that we
use.

Giving an attacker the ability to receive the instantaneous loss rate on
a path could be useful in helping the attacker gauge the efficacy of an
ongoing attack targeting that path.  The RFCs from the DOTS WG (e.g.,
8783 and 9132) may have some useful text on this topic that could be
used as a model.

Section 9.1

It's not really clear to me that [IANA-IPPM] needs to be classified as
normative (or whatever it is replaced by, in light of my earlier comment
in §3.1.4).

RFC 2330 is cited only once, in a "for example" clause; this would
typically cause it to be classified as only an informative reference.

The mention of RFC 8895 is conditional on it being implemented, so that
could probably also be downgraded to an informative reference as well.

Section 9.2

Some kind of URL for [Prometheus] would be very helpful.
[Prophet], too, though at least that has the ACM/IEEE Transactions venue
to anchor the reference.

I'm not entirely sure why RFC 2818 is classified as normative but RFC
8446 only as informative, since they are part of the same (quoted)
requirement clause.

NITS

Section 2.1

  To make it possible to specify the source and the aforementioned
  parameters, this document introduces an optional "cost-context" field
  to the "cost-type" field defined by the ALTO base protocol
  (Section 10.7 of [RFC7285]) as the following:

I think s/"cost-type" field/CostType object/ would be slightly more
accurate.

  The "estimation" category indicates that the metric value is computed
  through an estimation process.  An ALTO server may compute
  "estimation" values by retrieving and/or aggregating information from
  routing protocols (e.g., [RFC8571]) and traffic measurement
  management tools (e.g., TWAMP [RFC5357]), with corresponding
  operational issues.  [...]

I'm not sure if "with corresponding operational issues" conveys the
intended phrasing -- to me, it seems to say "do [the previous things],
but expect that there will sometimes be operational issues that make the
data unavailable or inaccurate".

Section 2.2

  stddev:
      the standard deviation of the observations.


  stdvar:
      the standard variance of the observations.

Pedantically, we could say if these are sample or population standard
deviation/variance (a difference of one in the denominator), but it
seems very unlikely to matter for these purposes.

Section 3

  dropped before reaching the destination (pkt.dropped).  The semantics
  of the performance metrics defined in this section are that they are
  statistics (percentiles) computed from these measures; for example,

I suggest "e.g., percentiles" since stddev/variance are not percentiles
but are statistics.

  the x-percentile of the one-way delay is the x-percentile of the set
  of delays {pkt.delay} for the packets in the stream.

This phrasing presupposes that there is a definite stream under
consideration, but I don't think that much confusion is likely and am
not sure that there's a need to change anything.

Section 3.1.3

I'd perhaps make a note about the wrapping of the Accept: header field
line in the example (and all the other similarly affected examples).

Section 3.2.2

I suggest reusing the phrasing from §3.1.2 that mentions floating-point
values, for consistency..

Section 3.5.2

  The metric value type is a single 'JSONNumber' type value conforming
  to the number specification of [RFC8259] Section 6.  The number MUST
  be non-negative.  The value represents the percentage of packet
  losses.

I'd probably mention floating-point here as well.

Section 4.3.3

  Intended Semantics: To specify spatial and temporal maximum
  reservable bandwidth from the specified source to the specified
  destination.  The value corresponds to the maximum bandwidth that can
  be reserved (motivated from [RFC3630] Section 2.5.7).  The spatial

It's a little interesting to see an OSPF reference for max reservable
bandwidth when we used an IS-IS one for current residual bandwidth, but
it's hard to see much harm causing from the mixture of references (who
is going to follow the references anyway?).

Section 7

  IANA has created and now maintains the "ALTO Cost Metric Registry",
  listed in Section 14.2, Table 3 of [RFC7285].  This registry is
  located at .  This document requests to add the
  following entries to "ALTO Cost Metric Registry".

The live registry has a "reference" column, so I'd add ", with this
document as the reference", here.

  Registered ALTO address type identifiers MUST conform to the
  syntactical requirements specified in Section 2.1.  Identifiers are
  to be recorded and displayed as strings.

s/address type/cost source type/

[Ballot comment]
[S3.4.2; comment]

* I would recommend specifying a maximum value as well.  It's not clear
  how values greater than the 8-bit TTL/HopLimit fields can carry should
  be interpreted.

  I would recommend max 254, as 255 tends to have special interpretation.

[Ballot discuss]
Thank you for the work on this document.

Many thanks to Christian Amsüss for his review: https://mailarchive.ietf.org/arch/msg/art/owYhcoFnl1vEipZ2D62cWiiE-LA/ , and thanks to the authors for addressing it.

I am holding a DISCUSS to make sure the examples are fixed before publication. Additionally, I agree with Christian that the line "Content-Length: TBA" in all the examples is not really helpful to the reader, and I suggest to either remove it or replace TBA with the actual content length for each example.

As noted in https://www.ietf.org/blog/handling-iesg-ballot-positions/, a
DISCUSS ballot is a request to have a discussion; I really think that the
document would be improved with a change here, but can be convinced otherwise.

Francesca

1. -----

{
      "meta": {
                  "cost type": {
                "cost-mode": "numerical",
                "cost-metric":"hopcount"}
          }
      },
      "endpoint-cost-map": {
              "ipv4:192.0.2.2": {
              "ipv4:192.0.2.89"  : 5,
              "ipv4:198.51.100.34": 3
            }
      }
    }



FP: JSON doesn't validate. There is one "}" too many after "hopcount".

2. -----

  {
    "meta": {
        "cost type": {
          "cost-mode": "numerical",
          "cost-metric":"tput"
      }
    }
    "endpoint-cost-map": {
      "ipv4:192.0.2.2": {
        "ipv4:192.0.2.89"  : 256000,
        "ipv4:198.51.100.34": 128000
    }
  }

FP: JSON doesn't validate. I believe there is 2 errors: after the second "}" after "tput" there is a missing "," , and it is also missing a final "}" at the end.

3. -----

{
    "meta": {
      "cost-type" {
        "cost-mode": "numerical",
        "cost-metric": "bw-residual"
      }
    },
    "endpoint-cost-map" {
      "ipv4:192.0.2.2" {
        "ipv4:192.0.2.89" :    0,
        "ipv4:198.51.100.34": 2000
      }
    }
  }

FP: JSON doesn't validate - there is a bunch of missing ":" all over.

4. -----

{
      "cost-type" { "cost-mode":  "numerical",
                    "cost-metric": "bw-maxres"},
      "endpoints":  {
        "srcs": [ "ipv4 : 192.0.2.2" ],
        "dsts": [
          "ipv4:192.0.2.89",
          "ipv4:198.51.100.34"
        ]
      }
    }


FP: JSON doesn't validate: missing ":" after "cost-type"

5. -----

{
    "meta": {
      "cost-type": {
        "cost-mode":  "numerical",
        "cost-metric": "bw-maxres"
      }
    },
    "endpoint-cost-map": {
      "ipv4:192.0.2.2" {
        "ipv4:192.0.2.89" :    0,
        "ipv4:198.51.100.34": 2000
      }
    }
  }

FP: JSON doesn't validate: missing ":" after "ipv4:192.0.2.2"

[Ballot discuss]
Thank you for the work on this document.

Many thanks to Christian Amsüss for his review: https://mailarchive.ietf.org/arch/msg/art/owYhcoFnl1vEipZ2D62cWiiE-LA/ , and thanks to the authors for addressing it.

I am holding a DISCUSS to make sure the examples are fixed before publication. Additionally, I agree with Christian that the line "Content-Length: TBA" in all the examples is not really helpful to the reader, and I suggest to either remove it or replace TBA with the actual content length for each example.

Francesca

1. -----

{
      "meta": {
                  "cost type": {
                "cost-mode": "numerical",
                "cost-metric":"hopcount"}
          }
      },
      "endpoint-cost-map": {
              "ipv4:192.0.2.2": {
              "ipv4:192.0.2.89"  : 5,
              "ipv4:198.51.100.34": 3
            }
      }
    }



FP: JSON doesn't validate. There is one "}" too many after "hopcount".

2. -----

  {
    "meta": {
        "cost type": {
          "cost-mode": "numerical",
          "cost-metric":"tput"
      }
    }
    "endpoint-cost-map": {
      "ipv4:192.0.2.2": {
        "ipv4:192.0.2.89"  : 256000,
        "ipv4:198.51.100.34": 128000
    }
  }

FP: JSON doesn't validate. I believe there is 2 errors: after the second "}" after "tput" there is a missing "," , and it is also missing a final "}" at the end.

3. -----

{
    "meta": {
      "cost-type" {
        "cost-mode": "numerical",
        "cost-metric": "bw-residual"
      }
    },
    "endpoint-cost-map" {
      "ipv4:192.0.2.2" {
        "ipv4:192.0.2.89" :    0,
        "ipv4:198.51.100.34": 2000
      }
    }
  }

FP: JSON doesn't validate - there is a bunch of missing ":" all over.

4. -----

{
      "cost-type" { "cost-mode":  "numerical",
                    "cost-metric": "bw-maxres"},
      "endpoints":  {
        "srcs": [ "ipv4 : 192.0.2.2" ],
        "dsts": [
          "ipv4:192.0.2.89",
          "ipv4:198.51.100.34"
        ]
      }
    }


FP: JSON doesn't validate: missing ":" after "cost-type"

5. -----

{
    "meta": {
      "cost-type": {
        "cost-mode":  "numerical",
        "cost-metric": "bw-maxres"
      }
    },
    "endpoint-cost-map": {
      "ipv4:192.0.2.2" {
        "ipv4:192.0.2.89" :    0,
        "ipv4:198.51.100.34": 2000
      }
    }
  }

FP: JSON doesn't validate: missing ":" after "ipv4:192.0.2.2"

[Ballot comment]
Thanks to Vincent Roca for the SECDIR review.

** Section 1.
An ALTO server may provide only a subset of the metrics described in
  this document.  For example, those that are subject to privacy
  concerns should not be provided to unauthorized ALTO clients. 

Is this generic caution, or are any of the mentioned metrics considered privacy sensitive in some way?

** Section 2.1.  Editorial.  s/, and “sla”/, “sla”/

** Examples 1 – 7 all have a their “Content-Length” set to “TBA”.  Consider populating it with the real length of each of the examples.

** Nit.  Example 7 (in Section 4.2.4) comes earlier than Example 6 (in Section 4.3.3)

** Section 6.  In the spirit of inclusively, please rephrase “man-in-the-middle (MITM) attacks”

** Additional Security Considerations. It appears that in cases of an “sla” and certain “estimation” cost-estimates, it is recommended for a URI to be provided via the parameters field to point to additional information.

-- is there any further guidance that can be provided on how this URI can be secured.  Perhaps, requiring https?

-- additional, I would recommend guidance to this effect (please polish the language on what exact ALTO fields are in question):

When ALTO clients process the URIs in the “link” field provided in the “parameters” field of select “sla” and “estimation” cost-estimate metrics, they should heed the risks outlined in Section 7 of RFC3986.

[Ballot discuss]
This document needs to become much more formal about how it defines the
metrics it wishes to use with ALTO. This could either be done either by
identifying and normatively referencing existing metrics the IETF has defined,
or by defining them here. When normatively referencing existing IETF metrics, it
would need to explain why their use with ALTO makes sense.

At the moment, the document informatively points to a somewhat arbitrary
collection of prior IETF metrics (most of which are from IPPM, residual
bandwidth from IS-IS TE, but then reservable bandwidth from OSPF TE?). But it
only refers to them as "examples", without actually defining how exactly they
are to be used with ALTO, or - if not those - which actual metrics are supposed
to be used.

Defining a mechanism for exposing metric information to clients isn't really
useful unless the content of that information is much more clearly specified.

Section 4.1.3. , paragraph 2, discuss:
>    Intended Semantics: To give the throughput of a TCP congestion-
>    control conforming flow from the specified source to the specified
>    destination; see [RFC3649, Section 5.1 of RFC8312] on how TCP
>    throughput is estimated.  The spatial aggregation level is specified
>    in the query context (e.g., PID to PID, or endpoint to endpoint).

A TCP bandwidth estimate can only be meaningfully be derived for bulk TCP
transfers under a set of pretty strict and simplistic assumptions, making this
metric a meaningless at best and misleading at worst, given that the source of
this information doesn't know what workload, congestion controller and network
conditions the user of this information will use or see.

Also, RFC3649 is an Experimental RFC (from 2003!) and RFC8312 is an
Informational RFC. Since this document normatively refers to them, it needs to
cite them, and this will cause DOWNREFs for PS document. I would argue that
at least RFC3649 is certainly not an appropriate DOWNREF.

Why define this metric at all? The material you point to is the usual
model-based throughput calculation based on RTT and loss rates; a client that
intended to predict TCP performance could simply query ALTO for this and perform
their own computation, which will likely be more accurate, since the client will
hopefully know which congestion controller they will use for the given workload,
and what the characteristics of that workload are.

[Ballot comment]
Section 1. , paragraph 6, comment:
>    The purpose of this document is to ensure proper usage of the
>    performance metrics defined in Table 1; it does not claim novelty of
>    the metrics.  The "Origin Example" column of Table 1 gives an example
>    RFC that has defined each metric.

I don't understand what the purpose of the "origin example" column is. Most of
these point to IPPM metrics, which have a pretty clear and narrowly-defined area
of applicability. Since ALTO isn't performing IPPM-style network testing, it's
not clear why IPPM metrics are referenced here?

Section 2.2. , paragraph 23, comment:
>    If a cost metric string does not have the optional statical operator
>    string, the statistical operator SHOULD be interpreted as the default
>    statical operator in the definition of the base metric.  If the

What is a "statical" operator; I am not familiar with the term and it doesn't
seem to appear in other RFCs? (Also occurs elsewhere in this document.)

Section 3.1.4. , paragraph 4, comment:
>    link statistics.  Another example of a source to estimate the delay
>    is the IPPM framework [RFC2330].  It is RECOMMENDED that the

IPPM defines measurement metrics. How would they be a source for estimates?

Section 3.3. , paragraph 1, comment:
> 3.3.  Cost Metric: Delay Variation (delay-variation)

Is this supposed to apply to the one-way or bidirectional delay? Also, delay
variation is not independent from path utilization (c.f. bufferbloat), so why is
it being reported independently?

Section 3.5. , paragraph 1, comment:
> 3.5.  Cost Metric: Loss Rate (lossrate)

What is this metric supposed to capture? Loss is generally not independent from
network utilization (apart from random corruption loss). So it should be zero
for unloaded networks, and depends on utilization otherwise. Also, is this
unidirectional or bidirectional loss (wording below is unclear)?

Using lowercase "not" together with an uppercase RFC2119 keyword is not
acceptable usage. Found: "MUST not"

The document has 6 authors, which exceeds the recommended author limit. I
assume the sponsoring AD has agreed that this is appropriate?

No reference entries found for: [RFC3649] and [RFC8312].

Found terminology that should be reviewed for inclusivity; see
https://www.rfc-editor.org/part2/#inclusive_language for background and more
guidance:

* Term "man"; alternatives might be "individual", "people", "person".

-------------------------------------------------------------------------------
All comments below are about very minor potential issues that you may choose to
address in some way - or ignore - as you see fit. Some were flagged by
automated tools (via https://github.com/larseggert/ietf-reviewtool), so there
will likely be some false positives. There is no need to let me know what you
did with these suggestions.

"Abstract", paragraph 2, nit:
-    types of cost metric.  Since the ALTO base protocol (RFC 7285)
+    types of cost metrics.  Since the ALTO base protocol (RFC 7285)
+                        +

Section 1. , paragraph 2, nit:
> ] on registering ALTO cost metrics. Hence it specifies the identifier, the in
>                                    ^^^^^
A comma may be missing after the conjunctive/linking adverb "Hence".

Section 2.2. , paragraph 2, nit:
> of the observations. median: the mid point (i.e., p50) of the observations.
>                                  ^^^^^^^^^
This word is normally spelled with a hyphen.

"IPPM ", paragraph 2, nit:
>  Also, delay variation is not independent from path utilization (c.f. buffer
>                              ^^^^^^^^^^^^^^^^
The usual collocation for "independent" is "of", not "from". Did you mean
"independent of"?

Section 3.3.3. , paragraph 7, nit:
> apture? Loss is generally not independent from network utilization (apart fr
>                              ^^^^^^^^^^^^^^^^
The usual collocation for "independent" is "of", not "from". Did you mean
"independent of"?

Section 3.4.3. , paragraph 6, nit:
> imation" method. See Section 3.1.4 on on related discussions such as summing
>                                    ^^^^^
Possible typo: you repeated a word.

Section 3.5.4. , paragraph 3, nit:
>  [RFC8312]), it helps to specify as much details as possible on the the cong
>                                    ^^^^
Use "many" with countable plural nouns like "details".

Section 3.5.4. , paragraph 3, nit:
> ify as much details as possible on the the congestion control algorithm used
>                                    ^^^^^^^
Two determiners in a row. Choose either "the" or "the".

These URLs in the document can probably be converted to HTTPS:
* http://www.iana.org/assignments/alto-protocol/alto-protocol.xhtml#cost-metrics

[Ballot discuss]
Thank you for the work put into this document. Please bear with my lack of knowledge about ALTO in general.

Please find below one trivial blocking DISCUSS points (probably easy to address), some non-blocking COMMENT points (but replies would be appreciated even if only for my own education), and some nits.

Special thanks to Jan Seedorf for the shepherd's write-up about the WG consensus (even if not using the usual template).

I have appreciated the "operational considerations" section as it addresses many questions that popped up during reading the document; notably, how can the ALTO server measure any metric between the ALTO client and a resource.

I hope that this helps to improve the document,

Regards,

-éric


== DISCUSS ==


-- Section 4.1.3 --
A very trivial DISCUSS to fix: this document relies on RFC 8312 to specify how TCP throughput is estimated but RFC 8312 does not appear in the normative reference list (this will probably generate a down ref though).

[Ballot comment]
== COMMENTS ==

Minor regret about the examples as they are all about the IPv4 address family especially in a world of happy eyeballs where the IPv4 and IPv6 paths may still have different performance metrics.

-- Section 2.1 --
Should the figure 1 use "perf monitoring tools" rather than "management tool" ?

-- Section 4 --
This section title is about 'bandwidth' but the first sub-section is about 'throughput', while these concepts are related they are also distinct. How can the reader reconciliate them ?

-- Section 4.1 --
Is the intent of ALTO to only work for TCP and not for other transport protocols ? I.e., is QUIC out of scope ?


-- Section 4.2.3 --
Where are those 'tunnels' in "by subtracting tunnel reservations " coming from ? Probably about RSVP-TE but what is the link with ALTO ? (Again I am not familiar with ALTO so this may be an uneducated question).

== NITS ==


-- Section 3.1.3 --
Probably tedious to do but why not replacing "TBA" by the actual value in the examples for 'content-length' ?

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Functions Operator has completed its review of draft-ietf-alto-performance-metrics-16. If any part of this review is inaccurate, please let us know.

The IANA Functions Operator has a question about one of the actions requested in the IANA Considerations section of this document.

The IANA Functions Operator understands that, upon approval of this document, there are two actions which we must complete.

First, in the ALTO Cost Metric Registry on the Application-Layer Traffic Optimization (ALTO) Protocol registry page located at:

https://www.iana.org/assignments/alto-protocol/

eight new registrations are to be added to the registry as follows:

Identifier: delay-ow
Intended Semantics: See Section 3.1 of [ RFC-to-be ]
Reference: [ RFC-to-be ]

Identifier: delay-rt
Intended Semantics: See Section 3.2 of [ RFC-to-be ]
Reference: [ RFC-to-be ]

Identifier: delay-variation
Intended Semantics: See Section 3.3 of [ RFC-to-be ]
Reference: [ RFC-to-be ]

Identifier: hopcount
Intended Semantics: See Section 3.4 of [ RFC-to-be ]
Reference: [ RFC-to-be ]

Identifier: lossrate
Intended Semantics: See Section 3.5 of [ RFC-to-be ]
Reference: [ RFC-to-be ]

Identifier: tput
Intended Semantics: See Section 4.1 of [ RFC-to-be ]
Reference: [ RFC-to-be ]

Identifier: bw-residual
Intended Semantics: See Section 4.2 of [ RFC-to-be ]
Reference: [ RFC-to-be ]

Identifier: bw-maxres
Intended Semantics: See Section 4.3 of [ RFC-to-be ]
Reference: [ RFC-to-be ]

Second, a new registry is to be created called the ALTO Cost Source Registry.

IANA Question --> Where should this new registry be located? Should it be added to an existing registry page (for instance, the existing Alto page at https://www.iana.org/assignments/alto-protocol/? If not, does it belong in an existing category at http://www.iana.org/protocols?

IANA Question --> What is the registration policy for the new registry - please see RFC 8126 for details.

There will be initial registrations in the new registry as follows:

+------------+-----------------------------+--------------+
| Identifier | Intended Semantics | Reference |
+------------+-----------------------------+--------------+
| nominal | Values in nominal cases | [ RFC-to-be ]|
| sla | Values reflecting service | [ RFC-to-be ]|
| | level agreement | |
| estimation | Values by estimation | [ RFC-to-be ]|
+------------+-----------------------------+--------------+

The IANA Functions Operator understands that these are the only actions required to be completed upon approval of this document.

Note:  The actions requested in this document will not be completed until the document has been approved for publication as an RFC. This message is meant only to confirm the list of actions that will be performed.

Thank you,

Sabrina Tanamal
Senior IANA Services Specialist

The following Last Call announcement was sent out (ends 2021-08-02):

From: The IESG
To: IETF-Announce
CC: alto-chairs@ietf.org, alto@ietf.org, draft-ietf-alto-performance-metrics@ietf.org, ietf@j-f-s.de, martin.h.duke@gmail.com
Reply-To: last-call@ietf.org
Sender:
Subject: Last Call:  (ALTO Performance Cost Metrics) to Proposed Standard


The IESG has received a request from the Application-Layer Traffic
Optimization WG (alto) to consider the following document: - 'ALTO
Performance Cost Metrics'
  as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2021-08-02. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


  Cost metric is a basic concept in Application-Layer Traffic
  Optimization (ALTO), and different applications may use different
  cost metrics.  Since the ALTO base protocol (RFC 7285) defines only a
  single cost metric (i.e., the generic "routingcost" metric), if an
  application wants to issue a cost map or an endpoint cost request to
  determine the resource provider that offers better delay performance,
  the base protocol does not define the cost metric to be used.

  This document addresses the issue by introducing network performance
  metrics, including network delay, jitter, packet loss rate, hop
  count, and bandwidth.

  There are multiple sources (e.g., estimation based on measurements or
  service-level agreement) to derive a performance metric.  This
  document introduces an additional "cost-context" field to the ALTO
  "cost-type" field to convey the source of a performance metric.





The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-alto-performance-metrics/



No IPR declarations have been submitted directly on this I-D.

1. Summary

Jan Seedorf is the document shepherd for draft-ietf-alto-performance-metrics. Martin Duke is the responsible Area Director.

The ALTO base protocol (RFC7285) defines only a single cost metric, the generic “routing cost” metric. As new ALTO use cases are being envisioned (e.g. CDN, 5G, data-intensive science applications, flexible inter-domain routing, etc.), the demand for more concrete cost metrics to be conveyed via the ALTO protocol arises. This document defines a multitude of such concrete ALTO cost metrics, such as one-way delay, hop count, residue bandwidth, and several more.

This document is targeted as a Standards Track document (Proposed Standard). This designation is appropriate as the document contains normative behaviour and specifies several additions to the IANA "ALTO Cost Metric Registry" that should be adhered to by the communicating entities in order to realize the extension.


2. Review and Consensus
The document was introduced originally in 2013 and has been iterated and presented at IEFT meetings many times. It was adopted as WG item in 2016, showing the general consensus in the ALTO WG for adding more concrete costs metrics to the ALTO protocol.

The proposed metrics have been discussed extensively at IETF meetings and on the mailing list. Since some of these metrics must to be specified in an unambiguous fashion with clear semantics, external help from IETF experts was obtained: in 2018 a “Tsvart early review” was performed by Brian Trammell. The outcome has been discussed on the mailing list and has been addressed in newer versions of the document. Also, advice from IETF experts from the IPPM WG on the semantics of the proposed metrics was obtained, discussed, and incorporated into the document. All of these changes and semantics have been presented to the ALTO WG multiple times.

In summary, there is clear consensus for draft-ietf-alto-performance-metrics in the ALTO WG, and it provides very useful cost metric extensions needed for many of the currently envisioned (future) ALTO use cases. A WGLC has successfully been passed with no objections, and extensive reviews were provided by various members of the WG and have all been addressed.


3. Intellectual Property
The shepherd confirms that each author has stated to him that to the best of his/her (i.e. the author’s) knowledge, all IPR related to this document has been disclosed.


4. Other Points
Note any downward references (see RFC 3967) and whether they appear in the DOWNREF Registry (http://trac.tools.ietf.org/group/iesg/trac/wiki/DownrefRegistry), as these need to be announced during Last Call.

All normative references are ok (with respect to RFC 3967) as they are all towards documents with standards-level “Proposed Standards”, “Internet Standard”, or “BCP”.

1. Summary

Jan Seedorf is the document shepherd for draft-ietf-alto-performance-metrics. Martin Duke is the responsible Area Director.

The ALTO base protocol (RFC7285) defines only a single cost metric, the generic “routing cost” metric. As new ALTO use cases are being envisioned (e.g. CDN, 5G, data-intensive science applications, flexible inter-domain routing, etc.), the demand for more concrete cost metrics to be conveyed via the ALTO protocol arises. This document defines a multitude of such concrete ALTO cost metrics, such as one-way delay, hop count, residue bandwidth, and several more.

This document is targeted as a Standards Track document (Proposed Standard). This designation is appropriate as the document contains normative behaviour and specifies several additions to the IANA "ALTO Cost Metric Registry" that should be adhered to by the communicating entities in order to realize the extension.


2. Review and Consensus
The document was introduced originally in 2013 and has been iterated and presented at IEFT meetings many times. It was adopted as WG item in 2016, showing the general consensus in the ALTO WG for adding more concrete costs metrics to the ALTO protocol.

The proposed metrics have been discussed extensively at IETF meetings and on the mailing list. Since some of these metrics must to be specified in an unambiguous fashion with clear semantics, external help from IETF experts was obtained: in 2018 a “Tsvart early review” was performed by Brian Trammell. The outcome has been discussed on the mailing list and has been addressed in newer versions of the document. Also, advice from IETF experts from the IPPM WG on the semantics of the proposed metrics was obtained, discussed, and incorporated into the document. All of these changes and semantics have been presented to the ALTO WG multiple times.

In summary, there is clear consensus for draft-ietf-alto-performance-metrics in the ALTO WG, and it provides very useful cost metric extensions needed for many of the currently envisioned (future) ALTO use cases. A WGLC has successfully been passed with no objections, and extensive reviews were provided by various members of the WG and have all been addressed.


3. Intellectual Property
The shepherd confirms that each author has stated to him that to the best of his/her (i.e. the author’s) knowledge, all IPR related to this document has been disclosed.


4. Other Points
Note any downward references (see RFC 3967) and whether they appear in the DOWNREF Registry (http://trac.tools.ietf.org/group/iesg/trac/wiki/DownrefRegistry), as these need to be announced during Last Call.

All normative references are ok (with respect to RFC 3967) as they are all towards documents with standards-level “Proposed Standards”, “Internet Standard”, or “BCP”.

Request for posting confirmation emailed to previous authors: "Y. Yang" , Luis Contreras , Dhruv Dhody , Young Lee , Sabine Randriamasy , Qin WU , alto-chairs@ietf.org