Data Fields for In Situ Operations, Administration, and Maintenance (IOAM)
draft-ietf-ippm-ioam-data-17

[Ballot comment]
Many thanks for the many updates in the -16, and my apologies for
the delay in reballoting.

I'm happy to report that the changes all look good to me, but do
have one comment that seems to remain valid:  In Section 6.3 where
we discuss the POSIX-based timestamp format, I think we do need to
say that it is affected by leap seconds (analogously to how we do
for the NTP format), per the guidance in RFC 8877.  I see in the
email thread that there was some desire to defer to 8877 rather than
try to reproduce a lot of it here (and inevitably end up with an
incomplete treatment); in that case, perhaps the Section 6.2 discussion
should be trimmed so that we provide an analogous level of detail
on all three timestamp formats.

[Ballot discuss]
Thanks for the many updates and email discussion about the relationship
between limited (network) domains, IOAM domains, IOAM namespaces, and
the like -- I think I do now have a pretty clear picture of how they're
expected to interact!  However, I think there may still be a couple
places in the document that need to get updated in order to match that
vision.  One point here, and some (more minor) instances in the COMMENT
section...

Section 5.2 has:

  The role of an IOAM-encapsulating, IOAM-transit or IOAM-decapsulating
  node is always performed within a specific IOAM-Namespace.  This
  [...]
  described above, that is added in a future revision.  An IOAM
  decapsulating node situated at the edge of an IOAM domain MUST remove
  all IOAM-Option-Types and associated encapsulation headers for all
  IOAM-Namespaces from the packet.

The "MUST remove [...] for all IOAM-Namespaces" at the end seems to
conflict with the notion of the role of IOAM-decapsulating node being
performed within a specific IOAM-Namespace.  Indeed, later on in Section
5.3 we see that namespace identifiers "allow devices which are IOAM
capable to determine: [...] o  whether IOAM-Option-Type(s) have to be
removed from the packet, e.g., at a domain edge or domain boundary."  If
a decapsulating node always had to remove IOAM options from all
namespaces, then the namespace identifier is irrelevant to whether
option type(s) are removed from the packet.


[the following paragraph is retained unchanged from my ballot position
on the -12, since the topic seems to still be open.]

As foreshadowed in
https://mailarchive.ietf.org/arch/msg/last-call/Ak2NAIKQ7p4Rij9jfv123xeTXQY/
I think we need to have a discussion about the expectations and
provisions for cryptographic (e.g., integrity) protection of IOAM data.
From my perspective, IOAM is a new (class of) protocols that is seeking
publication on the IETF stream as Proposed Standard.  While we do make
exceptions for modifications to protocols that were developed before we
realized how important integrated security mechanisms are, it's
generally the case that new protocols are held to the current IETF
expectations for what security properties are provided; the default
expectation is that a protocol is expected to provide secure operation
in the internet threat model of RFC 3552.  This draft currently only
provides a brief note in the security considerations that there exists
an individual draft (draft-brockners-ippm-ioam-data-integrity) that
might provide ways to protect the integrity of IOAM data fields.
Shouldn't the security mechanisms be getting developed side-by-side by
the protocol mechanisms, to ensure that they are properly integrated and
fit for purpose?  (This does not necessarily have to be in the same
document and could be part of a cluster of related documents, but I
don't think that an informative reference to a non-WG draft really
qualifies.)

[new disucssion on this topic as of the -15]
The discussion on this topic was over a rather protracted timescale, for
which I share much of the blame.  I think that the latest message is
https://mailarchive.ietf.org/arch/msg/ippm/POycw2NpSl5cIruqSimTa_4WrwI/
where I make a proposal to have some text about how actual use of these
data fields in a protocol or encapsulation needs to provide some
(possibly optional) mechanism for cryptographic integrity protection,
which could be draft-brockners-ippm-ioam-data-integrity but could also
be native to the encapsulation format.  I think that such a construction
would allow this document to proceed to RFC without waiting for the
other one to be complete.

[Ballot comment]
All-new (okay, almost all) comments as of the -15.

Mentioning here for lack of a better place, I happened to follow the
reference to draft-brokners-opsawg-ioam-deployment, which seems to still
be using the old definition of transit node and should get updated to
match this document's definition.

Section 5.2

  IOAM is expected to be deployed in a specific domain.  The part of
  the network which employs IOAM is referred to as the "IOAM-Domain".

In light of the (new) text up in §5, we might consider rewording this
part as well, mostly to avoid mentioning "network" that risks confusion
about "network domain" (pre previous discussion).

  An "IOAM transit node" read and/or write and/or process one or more
  of the IOAM-Data-Fields.  If both the Pre-allocated and the
  Incremental Trace Option-Types are present in the packet, each IOAM
  transit node based on configuration and available implementation of
  IOAM populates IOAM trace data in either Pre-allocated or Incremental
  Trace Option-Type but not both.  [...]

Since we redefined transit nodes to include only reading, in addition to
modifying, it doesn't seem 100% accurate to say that "each transit node
populates [one or the other but not both]" -- it seems valid for a
transit node to populate zero of the trace option types.

Section 5.3

  An IOAM-Namespace can be associated to a subset or all of the the
  IOAM-Option-Types and their corresponding IOAM-Data-Fields.  IOAM-

This sentence seems confusing to me.  It talks about namespaces as if
they contain options, but if we go on to examine the actual data
structures each option has a field to indicate the associated namespace.
We even go on to say that the IOAM-Namespace field "MUST be included in
all future IOAM-Option-Types" (side note: might be worth calling that
out in the guidance in the IANA considerations, and also to specifically
say that it's the first field of the option).  So it's really not clear
to me what this sentence is adding -- would it be safe to just remove
it?

  o  IOAM-Namespaces can be used by an operator to distinguish
      different operational domains.  Devices at domain edges can filter
      on Namespace-IDs to provide for proper IOAM-Domain isolation.

I suggest tweaking the wording to clarify that the "different
operational domains" in the first sentence are precisely the
"IOAM-Domain"s of the second sentence.

      *  Assigning different IOAM Namespace-IDs to different sets of
        [...]

There are two instances of a bullet point that talks about assembling a
full trace from partial traces, but the text has substantial
differences.  I suspect that one is just an editing artifact and should
be removed, but am less sure which one has the intended text (I would
guess the latter, for what it's worth).

Section 5.4

  "IOAM tracing data" is expected to be collected at every IOAM transit
  node that a packet traverses to ensure visibility into the entire
  path a packet takes within an IOAM-Domain.  [...]

Since we redefined transit nodes to include only reading, this "is
expected to be collected" doesn't seem entirely representative anymore.

  [...].  If not all nodes within a domain support IOAM functionality
  as defined in this document, IOAM tracing information (i.e., node
  data, see below) will only be collected on those nodes which support
  IOAM functionality as defined in this document.  [...]

Similarly, I might s/will only/can only/ here for the same reason.


Section 5.4.2

  Some IOAM-Data-Fields defined below, such as interface identifiers or
  IOAM-Namespace specific data, are defined in both "short format" as
  well as "wide format".  "Short format" refers to an IOAM-Data-Field
  which comprises 4 octets.  "Wide format" refers to an IOAM-Data-Field
  which comprises 8 octets.  [...]

We have definition entries for short/wide format in Section 3, so these
clarification sentences may not be needed.

Section 5.5

draft-ietf-sfc-proof-of-transit might be undergoing significant changes
prompted by the results of its secdir review thread.  I don't object to
leaving this discusison in place but it may be prudent to think about
paring down what we say here.  In particular, there seems to be some
(admittely, small) chance that the SFC doc will not have exactly two POT
types.

Section 6.3

It seems that in going from the -12 to -15 we lost the paragraph about
leap seconds here.  My understanding is that posix timestamps *are*
affected by leap seconds, and so it is correct to include such a
statement.  My ballot comment on the -12 was conditioned on the use of
TAI for the epoch, and thus my comment on the -12 is irrelevant.

Section 10

We should probably say that the opaque snapshot, namespace specific
data, etc., will have security considerations corresponding to their
defined data contents that should be described where those formats are
defined. [ed. this remains unchanged from my comment on the -12; I'm not
sure if the intent to add some text just got overlooked, but don't
intend to rehash any discussions that were already made.]

Since we clarified the definition of transit nodes to include read-only
transit nodes, we might want to say something about how transit nodes
that only implement support for one or the other trace option types (as
is clearly permitted) will have an incomplete picture of the trace in
cases where both trace option types are used for the same packet.  In
many cases that is innocuous, of course, but it does not seem guaranteed
to always be so.

  allowing attackers to collect information about network paths,
  performance, queue states, buffer occupancy and other information.

One possible application of such reconiassance is to gauge the
effectiveness of an ongoing attack (e.g., if buffers and queues are
overflowing).  I don't know whether it's particularly useful to mention
that scenario here or not, though, and the lack of response the previous
time I made the comment suggests that it's not actually useful to
mention it.

                  Indeed, in order to limit the scope of threats
  mentioned above to within the current network domain the network
  operator is expected to enforce policies that prevent IOAM traffic
  from leaking outside of the IOAM domain, and prevent IOAM data from
  outside the domain to be processed and used within the domain.

On the -12, I said "it would be great if we could provide a bit more
detail on the scope of consequences if the operator fails to do so."
Some of the follow-up discussion suggested that
draft-brockners-opsawg-ioam-deployment would be a better home, which I
don't object to; I'm retaining this comment just in case there was
actual desire to put such content in this document.  No specific reply
is expected or required, either way.

NITS

Section 5.3

      controllers.  For example, the node identifier field (node_id, see
      below) does not need to be unique in a deployment (e.g., if an
      operator wishes to use different node identifiers for different
      IOAM layers, even within the same device; or node identifiers
      might not be unique for other organizational reasons, such as
      after a merger of two formerly separated organizations), the
      Namespace-ID can be used as a context identifier, such that the
      combination of node_id and Namespace-ID will always be unique.

This looks like a comma splice (maybe put a sentence break after the
long parenthetical?).

      *  Assigning different IOAM Namespace-IDs to different sets of
        nodes or network partitions and using the Namespace-ID as a
        selector at the IOAM encapsulating node, a full trace for a
        flow could be collected and constructed via partial traces in
        different packets of the same flow.  Example: An operator could

I think s/Assigning/By assigning/ (note the comment that indicates
there are "two copies" of this text).

Section 5.4

  o  Time of day when the packet was processed by the node as well as
      the transit delay.  Different definitions of processing time are
      feasible and expected, though it is important that all devices of
      an in-situ OAM domain follow the same definition.

I think we've been standardizing on the "IOAM domain" spelling.

[Ballot comment]
Thanks for cleaning up the problems I mentioned about Section 8.  It's much more complete now.

I support Lars' DISCUSS, since it reminds me of the painful problems SPF used to have with its DNS migration plan.

"SFC" appears in the glossary in Section 3, but nowhere else in the document.

[Ballot discuss]


EDIT(2021-03-26): I removed the question about IEEE1588v2 being normative. I look forward to discussion about 11., i.e. more details about IANA Expert guidelines.

----------
(2021-03-24)

Thank you for this document. I think that the discussion on point 5. about referencing normatively IEEE 1588, and 11. about IANA Expert guidelines are worth having, and hope we can get them cleared before the document moves forward.
Also, please find some minor comments below.

Francesca

[Ballot comment]
EDIT(2021-03-26) to add:  these are minor, except the DISCUSS point 11. and the (non-blocking) comment 8.

1. -----

  document are to be interpreted as described in [RFC2119].

FP: Please update the boilerplate as per RFC 8147.

2. -----

  The specification of how IOAM-Data-Fields
  are encapsulated into "parent" protocols, like e.g., NSH or IPv6 is
  outside the scope of this document.

FP: This sentence (or equivalent) appears several times - at least 2 times in section 4 and once more in 5.1 - please consider removing the repetition.

3. -----

      For example, if 3 IOAM-Trace-Type bits are set and none of them
      are wide, then NodeLen would be 3.  If 3 IOAM-Trace-Type bits are

FP: As this is the first time the term "wide" appears, it would be good to define it and add a reference. Alternatively, a sentence in the terminology might have helped too.

4. -----

      Bit 3    When set, indicates presence of timestamp subseconds in

FP: Same as for 3. - please add a reference to where "subsecond" is defined.

5. -----

  formats; based on either PTP [IEEE1588v2], NTP [RFC5905], or POSIX
  [POSIX].  The three timestamp formats are specified in Section 6.  In

FP: (minor) please consider updating to IEEE 1588-2019.

6. -----

        computation, indicates which POT-profile is used.  The two
        profiles are numbered 0, 1.

FP: (just a comment) I found strange at this point that although two profiles are mentioned, only one is defined in this document.

7. -----

  Namespace-ID:  16-bit identifier of an IOAM-Namespace.  The

FP: Each option-type starts with Namespace-ID: couldn't this be optimized, or what is the reasoning behind this?

8. -----

              IOAM domain.  Within the IOAM encapsulating node, the
              time that the timestamp is retrieved can depend on the
              implementation.  Some possibilities are: 1) the time at

  in one of three possible timestamp formats.  It is assumed that the
  management plane is responsible for determining which timestamp
  format is used.

FP: This is worrying for interoperability within different implementations. Maybe more details or guidelines about how the management plane deals with this (or references to relevant specification for which this is in scope) would help.

9. -----

  New registries in this group can be created via RFC Required process
  as per [RFC8126].

FP: (asking as this was not included in the shepherd review, and just want to make sure this was addressed) For this and following registries: what is the reasoning for not going for IETF Review?

10. -----

  The meaning for Bits 1 - 7 are available for assignment via RFC
  Required process as per [RFC8126].

FP: From section 5.5 the bits 1-7 are indicated as Reserved.

11. -----

  of the current document.  Registry entries for the values 0x8000 to
  0xFFFF are to be assigned via the "Expert Review" policy defined in
  [RFC8126].  Upon a new allocation request, the responsible AD will
  appoint a designated expert, who will review the allocation request.
  The expert will post the request on the mailing list of the IPPM
  working group in the IETF (ippm@ietf.org), and possibly on other
  relevant mailing lists, to allow for community feedback.  Based on
  the review, the expert will either approve or deny the request.  The
  intention is that any allocation will be accompanied by a published
  RFC.  But in order to allow for the allocation of values prior to the
  RFC being approved for publication, the designated expert can approve
  allocations once it seems clear that an RFC will be published.


FP: The text above indicates Expert review for the registry. According to RFC 8126:

  The required documentation and review criteria, giving clear guidance
  to the designated expert, should be provided when defining the
  registry.  It is particularly important to lay out what should be
  considered when performing an evaluation and reasons for rejecting a
  request. 

Although the paragraph above gives some indication of process for the experts, it does not give clear review criteria or guidance. I would suggest this to be expanded to give more indication to experts on what criteria to consider - these same criteria will be considered by the working group as well.

[Ballot comment]
I support Francesca's DISCUSS position on both points.  In particular on the second, I wonder why this registry isn't being created following the provisional/permanent model that has been successful elsewhere, such as with URI schemes.

I also support Lars' DISCUSS, since it reminds me of the painful problems SPF used to have with its DNS migration plan.

"SFC" appears in the glossary in Section 3, but nowhere else in the document.

In Section 8:

  New registries in this group can be created via RFC Required process
  as per [RFC8126].

Is there any other way to get IANA to create a sub-registry?

[Ballot comment]

I wanted to discuss about
- IOAM domain isolation and boundary
.- needed integrity. It good that there is already intention to work with integrity however, I felt the lack of integrity and impact of that needed to be discussed already for this document.

However, my fellow ADs have already put DISCUSSes on those. I support Roman Danyliw and Benjamin Kaduk's DISCUSSes.

Another major issue - the use of IOAM-Namespaces to provide domain isolation needs to be clarified. It is described that IOAM-Namespaces can be used to filter the IOAM-Domain. However, it is also described that the same node can have different roles ( encapsulating, dencapsulating and transit) for different Namespace. If both of the description is true then I don't think filtering based on IAOM-Namesapces works or I dont understand the filtering concept here.


* Section 3:
 
  Should use the updated RFC 8174 version of the BCP 14 boilerplate.

  Nit: 
        E2E        Edge to Edge
        PMTU      Path MTU
  Missing ":", where rest of the entries have them
     
      Geneve:    Generic Network Virtualization Encapsulation
              [I-D.ietf-nvo3-geneve]
  Should refer to RFC 8926.

* Section 4:
      The operator has to consider the
  potential operational impact of IOAM to mechanisms such as ECMP
  processing (e.g.  load-balancing schemes based on packet length could
  be impacted by the increased packet size due to IOAM)..
 
  is there any guideline available? I haven't seen any discussion in the security considerations on this matter either.

      IOAM control points: IOAM-Data-Fields are added to or removed from
  the live user traffic by the devices which form the edge of a domain.
 
  what is live user traffic here?  does this suppose to mean current traffic in transition within the IOAM domain?

      Devices which form an IOAM-Domain can add, update or remove IOAM-
  Data-Fields.  Edge devices of an IOAM-Domain can be hosts or network
  devices.

  "hosts" need to clarified more, host of what?

* Section 5.1 and 5.1.1 :
  I cannot find how a POT-profile is defined hence understanding of section 5.1 and section 5.1.1 is troublesome. It would be good have description of the POT-profile prior to talk about it. If this is defined somewhere else then need to refer to that.

* Section 5.4.1:

      NodeLen:  5-bit unsigned integer.  This field specifies the length of
      data added by each node in multiples of 4-octets, excluding the
      length of the "Opaque State Snapshot" field.
 
  I would suggest to add reference to the section where the"Opaque state snapshot" is defined.

* Section 6.3 :

      Epoch:

      The epoch is 1 January 1970 00:00:00 TAI, which is 31 December
      1969 23:59:51.999918 UTC.

  AFAIK, the UNIX epoch is 1 January 1970 (midnight UTC/GMT). is there any reference to the Epoch used here?

* Section 7:

  Needs a reference to IPFIX

[Ballot comment]
Section 1, paragraph 4, comment:
>    IOAM use cases and mechanisms have expanded as this document matured,
>    resulting in additional flags and options that could trigger creation
>    of additional packets dedicated to OAM.  The term IOAM continues to
>    be used for such mechanisms, in addition to the "in-situ" mechanisms
>    that motivated this terminology.

Suggest to rephrase this expanded view on IAM in a way that does not tie the
description to the time period during which this soon-to-be-archival document
was edited.

Section 5.2, paragraph 6, comment:
>    A transit node MUST ignore IOAM-Option-Types that it does not
>    understand.  A transit node MUST NOT add new IOAM-Option-Types to a
>    packet, MUST NOT remove IOAM-Option-Types from a packet, and MUST NOT
>    change the IOAM-Data-Fields of an IOAM Edge-to-Edge Option-Type.

I'm surprised that IOAM data isn't authenticated or even integrity-protected at
all. Relying on RFC2119 language alone seems a pretty weak protection.

Section 5.3, paragraph 9, comment:
>    Namespace identifiers allow devices which are IOAM capable to
>    determine:
>
>    o  whether IOAM-Option-Type(s) need to be processed by a device: If
>      the Namespace-ID contained in a packet does not match any
>      Namespace-ID the node is configured to operate on, then the node
>      MUST NOT change the contents of the IOAM-Data-Fields.
>
>    o  which IOAM-Option-Type needs to be processed/updated in case there
>      are multiple IOAM-Option-Types present in the packet.  Multiple
>      IOAM-Option-Types can be present in a packet in case of
>      overlapping IOAM-Domains or in case of a layered IOAM deployment.
>
>    o  whether IOAM-Option-Type(s) has to be removed from the packet,
>      e.g. at a domain edge or domain boundary.

I'll note that cryptographically authenticating IOM data would probably result
in a system that wouldn't need the concept of namespaces, because keys would
automatically serve that purpose. (A device can't update an IOAM data item if it
doesn't have the key to authenticate the update with.)

Section 5.4, paragraph 11, comment:
>    o  Time of day when the packet was processed by the node as well as
>      the transit delay.  Different definitions of processing time are
>      feasible and expected, though it is important that all devices of
>      an in-situ OAM domain follow the same definition.

I think "important" is an understatement, this seems required? Also, capturing
time-of-day seems to require synchronized clocks.

Section 5.4.2.12, paragraph 2, comment:
> 5.4.2.12.  buffer occupancy
>
>    The "buffer occupancy" field is a 4-octet unsigned integer field.
>    This field indicates the current status of the occupancy of the
>    common buffer pool used by a set of queues.  The units of this field
>    are implementation specific.  Hence, the units are interpreted within
>    the context of an IOAM-Namespace and/or node-id if used.  The authors
>    acknowledge that in some operational cases there is a need for the
>    units to be consistent across a packet path through the network,
>    hence it is RECOMMENDED for implementations to use standard units
>    such as Bytes.

There are other "standard units" here, such as packets. You'd need to recommend
a specific standard unit and not just give an example.

Section 5.5, paragraph 3, comment:
>    o  Random: Unique identifier for the packet (e.g., 64-bits allow for
>      the unique identification of 2^64 packets).

If this identifier is supposed to be unique, it can't be random. And if it's
random, it will only be able to statistically uniquely identify a much smaller
number of packets (birthday paradox).

Section 6.3, paragraph 11, comment:
>      Microseconds: specifies the fractional portion of the number of
>      seconds since the epoch.
>
>      + Size: 32 bits.
>
>      + Units: the unit is microseconds.  The value of this field is in
>      the range 0 to (10^6)-1.

Given that the max. value for microseconds is 999999, using a 32-bit field
leaves the top eight bits unused.

-------------------------------------------------------------------------------
All comments below are very minor change suggestions that you may choose to
incorporate in some way (or ignore), as you see fit. There is no need to let me
know what you did with these suggestions.

Dan Romascanu's Gen-ART review
(https://mailarchive.ietf.org/arch/msg/gen-art/vzngkYWy-W-f0PHqAPNRlyNSwnw/)
contains other nits that I wanted to make sure you were aware of.

"Abstract", paragraph 2, nit:
-    protocols such as NSH, Segment Routing, Geneve, IPv6 (via extension
-                                                        ^^^^^^^^^^^^^^^
-    header), or IPv4.  In-situ OAM can be used to complement OAM
-  ---------
-    mechanisms based on e.g.  ICMP or other types of probe packets.
-                            ^
+    protocols such as NSH, Segment Routing, Geneve, IPv6,
+                                                        ^
+    or IPv4.  In-situ OAM can be used to complement OAM
+    mechanisms based on, e.g., ICMP or other types of probe packets.
+                      +    ^

Section 1, paragraph 2, nit:
-    in [RFC7799] IOAM could be portrayed as Hybrid Type 1.  IOAM
-    mechanisms can be leveraged where mechanisms using e.g.  ICMP do not
-                                                          ^
+    in [RFC7799], IOAM could be portrayed as Hybrid Type 1.  IOAM
+                +
+    mechanisms can be leveraged where mechanisms using, e.g., ICMP, do not
+                                                      +    ^    +

Section 4, paragraph 3, nit:
-    expected that each such encapsulation will be defined in the relevant
-                                          ^ ^
+    expected that each such encapsulation would be defined in the relevant
+                                          ^^ ^

Section 4, paragraph 4, nit:
-    IOAM data does not leak beyond the edge of an IOAM domain using,for
+    IOAM data does not leak beyond the edge of an IOAM domain using, for
+                                                                    +

Section 4, paragraph 4, nit:
-    processing (e.g.  load-balancing schemes based on packet length could
-                    ^
-    be impacted by the increased packet size due to IOAM), path MTU (i.e.
+    processing (e.g., load-balancing schemes based on packet length could
+                    ^
+    be impacted by the increased packet size due to IOAM), path MTU (i.e.,
+                                                                        +

Section 4, paragraph 4, nit:
-    message handling (i.e. in case of IPv6, IOAM support for ICMPv6 Echo
+    message handling (i.e., in case of IPv6, IOAM support for ICMPv6 Echo
+                          +

Section 4, paragraph 7, nit:
-    are encapsulated into "parent" protocols, like e.g., NSH or IPv6 is
+    are encapsulated into "parent" protocols, like, e.g., NSH or IPv6 is
+                                                  +

Section 4, paragraph 8, nit:
-    the-night model, i.e. IOAM-Data-Fields in one layer are independent
+    the-night model, i.e., IOAM-Data-Fields in one layer are independent
+                        +

Section 5.1, paragraph 3, nit:
-    different categories.  In IOAM these categories are referred to as
+    different categories.  In IOAM, these categories are referred to as
+                                  +

Section 5.2, paragraph 2, nit:
-    transit nodes".  The role of a node (i.e. encapsulating, transit,
+    transit nodes".  The role of a node (i.e., encapsulating, transit,
+                                            +

Section 5.2, paragraph 3, nit:
-    called the "IOAM encapsulating node", whereas a device which removes
-          ^^^
-    an IOAM-Option-Type is referred to as the "IOAM decapsulating node".
-                                          ^^^
+    called an "IOAM encapsulating node", whereas a device which removes
+          ^^
+    an IOAM-Option-Type is referred to as an "IOAM decapsulating node".
+                                          ^^

Section 5.2, paragraph 7, nit:
-    means that an IOAM node which is e.g. an IOAM-decapsulating node for
+    means that an IOAM node which is, e.g., an IOAM-decapsulating node for
+                                    +    +

Section 5.3, paragraph 6, nit:
-    assigned range is intended to be domain specific, and managed by the
-                                          ^
+    assigned range is intended to be domain-specific, and managed by the
+                                          ^

Section 5.3, paragraph 9, nit:
-      e.g. at a domain edge or domain boundary.
+      e.g., at a domain edge or domain boundary.
+          +

Section 5.4, paragraph 2, nit:
-    deployment all nodes in an IOAM-Domain would participate in IOAM and
+    deployment, all nodes in an IOAM-Domain would participate in IOAM and
+              +

Section 5.4, paragraph 2, nit:
-    ways to deal with situations where the PMTU was underestimated, i.e.
+    ways to deal with situations where the PMTU was underestimated, i.e.,
+                                                                        +

Section 5.4, paragraph 10, nit:
-      i.e. ingress interface.
+      i.e., ingress interface.
+          +

Section 5.4, paragraph 11, nit:
-      i.e. egress interface.
+      i.e., egress interface.
+          +

Section 5.4.1, paragraph 2, nit:
-    i.e. an IOAM transit node MUST NOT modify the Namespace-ID, NodeLen,
+    i.e., an IOAM transit node MUST NOT modify the Namespace-ID, NodeLen,
+        +

Section 5.4.2.2, paragraph 6, nit:
-    with ingressing or egressing packets, i.e. ingress_if_id could
+    with ingressing or egressing packets, i.e., ingress_if_id could
+                                              +

Section 5.6, paragraph 9, nit:
-                encapsulating node e.g. by n-tuple based classification
+                encapsulating node, e.g., by n-tuple based classification
+                                  +    +

Section 5.6, paragraph 10, nit:
-                encapsulating node e.g. by n-tuple based classification
+                encapsulating node, e.g., by n-tuple based classification
+                                  +    +

[Ballot discuss]
Section 5.4, paragraph 6, discuss:
>    A particular implementation of IOAM MAY choose to support only one of
>    the two trace option types.  In the event that both options are

Not requiring at least one mandatory-to-implement trace option type is highly
problematic, since it creates two incompatible flavors of this standard.
Preventing bifurcation seems to trump the desire for allowing (minor?)
optimizations.

[Ballot comment]
Section 1, paragraph 4, comment:
>    IOAM use cases and mechanisms have expanded as this document matured,
>    resulting in additional flags and options that could trigger creation
>    of additional packets dedicated to OAM.  The term IOAM continues to
>    be used for such mechanisms, in addition to the "in-situ" mechanisms
>    that motivated this terminology.

Suggest to rephrase this expanded view on IAM in a way that does not tie the
description to the time period during which this soon-to-be-archival document
was edited.

Section 5.2, paragraph 6, comment:
>    A transit node MUST ignore IOAM-Option-Types that it does not
>    understand.  A transit node MUST NOT add new IOAM-Option-Types to a
>    packet, MUST NOT remove IOAM-Option-Types from a packet, and MUST NOT
>    change the IOAM-Data-Fields of an IOAM Edge-to-Edge Option-Type.

I'm surprised that IOAM data isn't authenticated or even integrity-protected at
all. Relying on RFC2119 language alone seems a pretty weak protection.

Section 5.3, paragraph 9, comment:
>    Namespace identifiers allow devices which are IOAM capable to
>    determine:
>
>    o  whether IOAM-Option-Type(s) need to be processed by a device: If
>      the Namespace-ID contained in a packet does not match any
>      Namespace-ID the node is configured to operate on, then the node
>      MUST NOT change the contents of the IOAM-Data-Fields.
>
>    o  which IOAM-Option-Type needs to be processed/updated in case there
>      are multiple IOAM-Option-Types present in the packet.  Multiple
>      IOAM-Option-Types can be present in a packet in case of
>      overlapping IOAM-Domains or in case of a layered IOAM deployment.
>
>    o  whether IOAM-Option-Type(s) has to be removed from the packet,
>      e.g. at a domain edge or domain boundary.

I'll note that cryptographically authenticating IOM data would probably result
in a system that wouldn't need the concept of namespaces, because keys would
automatically serve that purpose. (A device can't update an IOAM data item if it
doesn't have the key to authenticate the update with.)

Section 5.4, paragraph 11, comment:
>    o  Time of day when the packet was processed by the node as well as
>      the transit delay.  Different definitions of processing time are
>      feasible and expected, though it is important that all devices of
>      an in-situ OAM domain follow the same definition.

I think "important" is an understatement, this seems required? Also, capturing
time-of-day seems to require synchronized clocks.

Section 5.4.2.12, paragraph 2, comment:
> 5.4.2.12.  buffer occupancy
>
>    The "buffer occupancy" field is a 4-octet unsigned integer field.
>    This field indicates the current status of the occupancy of the
>    common buffer pool used by a set of queues.  The units of this field
>    are implementation specific.  Hence, the units are interpreted within
>    the context of an IOAM-Namespace and/or node-id if used.  The authors
>    acknowledge that in some operational cases there is a need for the
>    units to be consistent across a packet path through the network,
>    hence it is RECOMMENDED for implementations to use standard units
>    such as Bytes.

There are other "standard units" here, such as packets. You'd need to recommend
a specific standard unit and not just give an example.

Section 5.5, paragraph 3, comment:
>    o  Random: Unique identifier for the packet (e.g., 64-bits allow for
>      the unique identification of 2^64 packets).

If this identifier is supposed to be unique, it can't be random. And if it's
random, it will only be able to statistically uniquely identify a much smaller
number of packets (birthday paradox).

Section 6.3, paragraph 11, comment:
>      Microseconds: specifies the fractional portion of the number of
>      seconds since the epoch.
>
>      + Size: 32 bits.
>
>      + Units: the unit is microseconds.  The value of this field is in
>      the range 0 to (10^6)-1.

Given that the max. value for microseconds is 999999, using a 32-bit field
leaves the top eight bits unused.

-------------------------------------------------------------------------------
All comments below are very minor change suggestions that you may choose to
incorporate in some way (or ignore), as you see fit. There is no need to let me
know what you did with these suggestions.

Matt Joras' Gen-ART review
(https://mailarchive.ietf.org/arch/msg/gen-art/vzngkYWy-W-f0PHqAPNRlyNSwnw/)
contains several other nits that I wanted to make sure you were aware of.

"Abstract", paragraph 2, nit:
-    protocols such as NSH, Segment Routing, Geneve, IPv6 (via extension
-                                                        ^^^^^^^^^^^^^^^
-    header), or IPv4.  In-situ OAM can be used to complement OAM
-  ---------
-    mechanisms based on e.g.  ICMP or other types of probe packets.
-                            ^
+    protocols such as NSH, Segment Routing, Geneve, IPv6,
+                                                        ^
+    or IPv4.  In-situ OAM can be used to complement OAM
+    mechanisms based on, e.g., ICMP or other types of probe packets.
+                      +    ^

Section 1, paragraph 2, nit:
-    in [RFC7799] IOAM could be portrayed as Hybrid Type 1.  IOAM
-    mechanisms can be leveraged where mechanisms using e.g.  ICMP do not
-                                                          ^
+    in [RFC7799], IOAM could be portrayed as Hybrid Type 1.  IOAM
+                +
+    mechanisms can be leveraged where mechanisms using, e.g., ICMP, do not
+                                                      +    ^    +

Section 4, paragraph 3, nit:
-    expected that each such encapsulation will be defined in the relevant
-                                          ^ ^
+    expected that each such encapsulation would be defined in the relevant
+                                          ^^ ^

Section 4, paragraph 4, nit:
-    IOAM data does not leak beyond the edge of an IOAM domain using,for
+    IOAM data does not leak beyond the edge of an IOAM domain using, for
+                                                                    +

Section 4, paragraph 4, nit:
-    processing (e.g.  load-balancing schemes based on packet length could
-                    ^
-    be impacted by the increased packet size due to IOAM), path MTU (i.e.
+    processing (e.g., load-balancing schemes based on packet length could
+                    ^
+    be impacted by the increased packet size due to IOAM), path MTU (i.e.,
+                                                                        +

Section 4, paragraph 4, nit:
-    message handling (i.e. in case of IPv6, IOAM support for ICMPv6 Echo
+    message handling (i.e., in case of IPv6, IOAM support for ICMPv6 Echo
+                          +

Section 4, paragraph 7, nit:
-    are encapsulated into "parent" protocols, like e.g., NSH or IPv6 is
+    are encapsulated into "parent" protocols, like, e.g., NSH or IPv6 is
+                                                  +

Section 4, paragraph 8, nit:
-    the-night model, i.e. IOAM-Data-Fields in one layer are independent
+    the-night model, i.e., IOAM-Data-Fields in one layer are independent
+                        +

Section 5.1, paragraph 3, nit:
-    different categories.  In IOAM these categories are referred to as
+    different categories.  In IOAM, these categories are referred to as
+                                  +

Section 5.2, paragraph 2, nit:
-    transit nodes".  The role of a node (i.e. encapsulating, transit,
+    transit nodes".  The role of a node (i.e., encapsulating, transit,
+                                            +

Section 5.2, paragraph 3, nit:
-    called the "IOAM encapsulating node", whereas a device which removes
-          ^^^
-    an IOAM-Option-Type is referred to as the "IOAM decapsulating node".
-                                          ^^^
+    called an "IOAM encapsulating node", whereas a device which removes
+          ^^
+    an IOAM-Option-Type is referred to as an "IOAM decapsulating node".
+                                          ^^

Section 5.2, paragraph 7, nit:
-    means that an IOAM node which is e.g. an IOAM-decapsulating node for
+    means that an IOAM node which is, e.g., an IOAM-decapsulating node for
+                                    +    +

Section 5.3, paragraph 6, nit:
-    assigned range is intended to be domain specific, and managed by the
-                                          ^
+    assigned range is intended to be domain-specific, and managed by the
+                                          ^

Section 5.3, paragraph 9, nit:
-      e.g. at a domain edge or domain boundary.
+      e.g., at a domain edge or domain boundary.
+          +

Section 5.4, paragraph 2, nit:
-    deployment all nodes in an IOAM-Domain would participate in IOAM and
+    deployment, all nodes in an IOAM-Domain would participate in IOAM and
+              +

Section 5.4, paragraph 2, nit:
-    ways to deal with situations where the PMTU was underestimated, i.e.
+    ways to deal with situations where the PMTU was underestimated, i.e.,
+                                                                        +

Section 5.4, paragraph 10, nit:
-      i.e. ingress interface.
+      i.e., ingress interface.
+          +

Section 5.4, paragraph 11, nit:
-      i.e. egress interface.
+      i.e., egress interface.
+          +

Section 5.4.1, paragraph 2, nit:
-    i.e. an IOAM transit node MUST NOT modify the Namespace-ID, NodeLen,
+    i.e., an IOAM transit node MUST NOT modify the Namespace-ID, NodeLen,
+        +

Section 5.4.2.2, paragraph 6, nit:
-    with ingressing or egressing packets, i.e. ingress_if_id could
+    with ingressing or egressing packets, i.e., ingress_if_id could
+                                              +

Section 5.6, paragraph 9, nit:
-                encapsulating node e.g. by n-tuple based classification
+                encapsulating node, e.g., by n-tuple based classification
+                                  +    +

Section 5.6, paragraph 10, nit:
-                encapsulating node e.g. by n-tuple based classification
+                encapsulating node, e.g., by n-tuple based classification
+                                  +    +

[Ballot comment]
Thank you for the work put into this document and thank you for acknowledging my comments and advices (as they were minor, I am not recusing myself). Sometimes the text is difficult to read as paragraphs and sentences are long and somehow repetitive.

Please find below  some non-blocking COMMENT points (but replies would be appreciated), and some nits. Like Alvaro, I was hesitating to ballot a DISCUSS on one point about section 4 (insertion/deletion of data on the flight).

I hope that this helps to improve the document,

Regards,

-éric

== COMMENTS ==

-- Abstract --
"a path between two points in the network" does this mean that this document cannot be used with a multicast destination address ?

-- Section 2 --
Is the list about 'contributors' (as in the section title) or 'co-authors' (as in the text) ?

-- Section 3 --
Please use the BCP14 boilerplate

Please note that Geneve is now RFC 8926.

-- Section 4 --
Should the 'deployment domain' include a reference to RFC 8799 ?

I was about to ballot a DISCUSS on this one: While the actual encapsulation is out of scope, the definition of "IOAM control points" alludes to nodes at the edge and in the core being able to add/remove data fields. This behavior will obviously have some impacts on the PMTU discovery and possibly on the handling of ICMP. Did the authors think of writing a generic section in this document on how this can be done in a correct way?

What is "live user traffic" ? I guess that this is not about end-user real-time video ;-) but a better wording would be welcome.

I am afraid that "ships in the night" does not apply here as all ships are usually on the same layer ;-) (planes do flight at different flight levels) But, this is not that important. No need to reply.

-- Section 5.1 --
Is it expected to have additional IOAM data types than the 4 in this document? Text should clarify this.

-- Section 5.2 --
"A transit node MUST NOT add new IOAM-Option-Types" seems to contradict the "IOAM control points" definition of section 4.

-- Section 5.3 --
If 0x0000 is already reserved, then I would suggest to make it part of the IANA range (i.e., making IANA range 0x0000 t 0x7FFF).

-- Section 5.4.2.5 --
Should there be a precise definition (or reference) of "time in nanoseconds the packet spent in the transit node"? E.g., between first bit received and last bit sent ?

-- Section 5.4.2.7 --
Is there a reason why the queue depth is expressed in buffers and not in packets? (Both metrics have useful values imho)

-- Section 5.5 --
In "Random: Unique identifier for the packet" how are collisions resolved? Do they matter at all ?

How can a transit/decaps node can handle the PoT (and also the E2E of section 5.6) as the length is not specified in the header ?

== NITS ==

Usually "e.g." is enclosed between commas.

The sentence "The definition of how IOAM-Data-Fields are encapsulated into other protocols is outside the scope of this document." or a minor variation of it occurs multiple times in the document. Please consider avoiding those repetitions.

[Ballot discuss]
I'd also like to discuss what's going on in Section 8.

Section 8.1, for instance, says that the registry covers 128 code points.  The first seven entries are given, but it's not explicit that a registration comprises a code point and (apparently) a name.  It's more typical to include a template that a new registration needs to include.  You might require a requested code point number and a name at minimum, but also commonly included in such a template is a reference to the registering RFC.  If I were to register a code point here in some later RFC, it would be awfully convenient to have the registry include a reference to that defining document.  As it stands, the registry will only ever point to this one.

You might want to give RFC 8126 a once-over, at least Section 2.2.

[Ballot discuss]
I'd also like to discuss what's going on in Section 8.

Section 8.1, for instance, says that the registry covers 128 code points.  The first seven entries are given, but it's not explicit that a registration comprises a code point and (apparently) a name.  It's more typical to include a template that a new registration needs to include.  You might require a requested code point number and a name at minimum, but also commonly included in such a template is a reference to the registering RFC.  If I were to register a code point here in some later RFC, it would be awfully convenient to have the registry include a reference to that defining document.  As it stands, the registry will only ever point to this one.

[Ballot comment]
I support Francesca's DISCUSS position on both points.  In particular on the second, I wonder why this registry isn't being created following the provisional/permanent model that has been successful elsewhere, such as with URI schemes.

"SFC" appears in the glossary in Section 3, but nowhere else in the document.

In Section 8:

  New registries in this group can be created via RFC Required process
  as per [RFC8126].

Is there any other way to get IANA to create a sub-registry?

[Ballot discuss]
Thank you for this document. I think that the discussion on point 5. about referencing normatively IEEE 1588, and 11. about IANA Expert guidelines are worth having, and hope we can get them cleared before the document moves forward.
Also, please find some minor comments below.

Francesca

[Ballot comment]
1. -----

  document are to be interpreted as described in [RFC2119].

FP: Please update the boilerplate as per RFC 8147.

2. -----

  The specification of how IOAM-Data-Fields
  are encapsulated into "parent" protocols, like e.g., NSH or IPv6 is
  outside the scope of this document.

FP: This sentence (or equivalent) appears several times - at least 2 times in section 4 and once more in 5.1 - please consider removing the repetition.

3. -----

      For example, if 3 IOAM-Trace-Type bits are set and none of them
      are wide, then NodeLen would be 3.  If 3 IOAM-Trace-Type bits are

FP: As this is the first time the term "wide" appears, it would be good to define it and add a reference. Alternatively, a sentence in the terminology might have helped too.

4. -----

      Bit 3    When set, indicates presence of timestamp subseconds in

FP: Same as for 3. - please add a reference to where "subsecond" is defined.

5. -----

  formats; based on either PTP [IEEE1588v2], NTP [RFC5905], or POSIX
  [POSIX].  The three timestamp formats are specified in Section 6.  In

FP: Is the normative reference to IEEE 1588 (behind a paywall) absolutely necessary? Rob Wilton pointed out that maybe a normative reference to RFC 8877 would be enough, however that would create a downref since 8877 is informational.
Also (minor) if kept, please consider updating to IEEE 1588-2019.

6. -----

        computation, indicates which POT-profile is used.  The two
        profiles are numbered 0, 1.

FP: (just a comment) I found strange at this point that although two profiles are mentioned, only one is defined in this document.

7. -----

  Namespace-ID:  16-bit identifier of an IOAM-Namespace.  The

FP: Each option-type starts with Namespace-ID: couldn't this be optimized, or what is the reasoning behind this?

8. -----

              IOAM domain.  Within the IOAM encapsulating node, the
              time that the timestamp is retrieved can depend on the
              implementation.  Some possibilities are: 1) the time at

  in one of three possible timestamp formats.  It is assumed that the
  management plane is responsible for determining which timestamp
  format is used.

FP: This is worrying for interoperability within different implementations. Maybe more details or guidelines about how the management plane deals with this (or references to relevant specification for which this is in scope) would help.

9. -----

  New registries in this group can be created via RFC Required process
  as per [RFC8126].

FP: (asking as this was not included in the shepherd review, and just want to make sure this was addressed) For this and following registries: what is the reasoning for not going for IETF Review?

10. -----

  The meaning for Bits 1 - 7 are available for assignment via RFC
  Required process as per [RFC8126].

FP: From section 5.5 the bits 1-7 are indicated as Reserved.

11. -----

  of the current document.  Registry entries for the values 0x8000 to
  0xFFFF are to be assigned via the "Expert Review" policy defined in
  [RFC8126].  Upon a new allocation request, the responsible AD will
  appoint a designated expert, who will review the allocation request.
  The expert will post the request on the mailing list of the IPPM
  working group in the IETF (ippm@ietf.org), and possibly on other
  relevant mailing lists, to allow for community feedback.  Based on
  the review, the expert will either approve or deny the request.  The
  intention is that any allocation will be accompanied by a published
  RFC.  But in order to allow for the allocation of values prior to the
  RFC being approved for publication, the designated expert can approve
  allocations once it seems clear that an RFC will be published.


FP: The text above indicates Expert review for the registry. According to RFC 8126:

  The required documentation and review criteria, giving clear guidance
  to the designated expert, should be provided when defining the
  registry.  It is particularly important to lay out what should be
  considered when performing an evaluation and reasons for rejecting a
  request. 

Although the paragraph above gives some indication of process for the experts, it does not give clear review criteria or guidance. I would suggest this to be expanded to give more indication to experts on what criteria to consider - these same criteria will be considered by the working group as well.

[Ballot discuss]
Please clarify what constitutes the edge or boundary of the IOAM domain.  Consider:

(a) Section 4. 
IOAM is a
  network domain focused feature, with "network domain" being a set of
  network devices or entities within a single administration. 
…
Designers of
  protocol encapsulations for IOAM specify mechanisms to ensure that
  IOAM data stays within an IOAM domain.  In addition, the operator of
  such a domain is expected to put provisions in place to ensure that
  IOAM data does not leak beyond the edge of an IOAM domain.

(b) Section 5.3. 
Namespace identifiers allow devices which are IOAM capable to
  determine: …
whether IOAM-Option-Type(s) has to be removed from the packet,
      e.g. at a domain edge or domain boundary.

(a) suggests that the filtering occurs on the basis of the single administrative domain.  However, (b) suggests that namespace identifiers are part of the filtering decision; which suggests that sub-domains can be created in a given domain which should be partitioned from each other.

The Security Considerations should be clearer on who does the IOAM information filtering, on what criteria and on what boundary.

[Ballot comment]
Thank you to Shawn Emery for the SECDIR review.

I support Ben Kaduk’s DISCUSS position.

** Section 4.  Per the scope of “IOAM is a network domain focused feature, with ‘network domain’ being a set of network devices or entities within a single administration” and the implicit trust model, the more precise text seems to be a s/a set of network devices/a set of trusted network devices/.

** Section 10.  To the end of the first paragraph, “All nodes in the path of a IOAM carrying packet can perform such an attack”.

** Section 10.  It is not clear why the "Direct Exporting" mode, a reference an unadopted I-D, is being referenced here and then consideration for it is noted as out of scope.

** Section 10
  At the management plane, attacks can be set up by misconfiguring or
  by maliciously configuring IOAM-enabled nodes in a way that enables
  other attacks.  Thus, IOAM configuration has to be secured in a way
  that authenticates authorized users and verifies the integrity of
  configuration procedures.

The link to authenticating authorized users isn’t clear.  Perhaps the intent of the second sentence is that configurations should only managed by authorized processes or users?

** Section 10.  Please note that IOAM fields could introduce the possibility of a per-packet cover channel

** Editorial nits:
Section 4. Typo. s/using,for/using, for/

Section 10.  Editorial. s/Section Section 5.5/Section 5.5/

[Ballot discuss]
I think we need some greater clarity on the relationship between IOAM
"layers" and IOAM-Namespaces.  For example, in Section 4 there is a
principle of "Layering" that seems to indicate that different layers
operate entirely independently, such as might occur when traffic from
one operator that uses IOAM is conveyed in a tunnel over a different
operator's network and both operators use IOAM independently.  But in
Section 5.3 we seem to see some discussion that IOAM-Namespaces can be
used to enforce a separation of layers ("IOAM-Namespaces provide
additional context [...] e.g. if an operator wishes to use different
node identifiers for different IOAM layers"), and that namespace
identifiers allow for determination of which IOAM-Option-Types need to
be processed "in case of a layered IOAM deployment".

I think there is also some internal inconsistency relating to the role
of IOAM transit nodes.  This may be localised in Section 5.2 where we
see both that a transit node is one that "read and/or write or process
[the] IOAM data" and that a transit node "updates one or more of the
IOAM-Data-Fields" (i.e., always writes), but I did not attempt an
exhaustive check for other instances.

I don't think the definition of the POSIX epoch is correct -- it seems
to be copied (roughly) from the definition of the PTP epoch (i.e., using
midnight TAI as the reference) but all the references I consulted
indicate that the POSIX epoch started at midnight UTC.

As foreshadowed in
https://mailarchive.ietf.org/arch/msg/last-call/Ak2NAIKQ7p4Rij9jfv123xeTXQY/
I think we need to have a discussion about the expectations and
provisions for cryptographic (e.g., integrity) protection of IOAM data.
From my perspective, IOAM is a new (class of) protocols that is seeking
publication on the IETF stream as Proposed Standard.  While we do make
exceptions for modifications to protocols that were developed before we
realized how important integrated security mechanisms are, it's
generally the case that new protocols are held to the current IETF
expectations for what security properties are provided; the default
expectation is that a protocol is expected to provide secure operation
in the internet threat model of RFC 3552.  This draft currently only
provides a brief note in the security considerations that there exists
an individual draft (draft-brockners-ippm-ioam-data-integrity) that
might provide ways to protect the integrity of IOAM data fields.
Shouldn't the security mechanisms be getting developed side-by-side by
the protocol mechanisms, to ensure that they are properly integrated and
fit for purpose?  (This does not necessarily have to be in the same
document and could be part of a cluster of related documents, but I
don't think that an informative reference to a non-WG draft really
qualifies.)

[Ballot comment]
Thanks to Shawn Emery for the secdir review.

I have grave misgivings about the proposed behavior that provides for a
separate domain boundary at the granularity of IOAM-Namespace.  It does
not quite rise to a Discuss because it is technically possible to
implement successfully if done perfectly, but it seems incredibly risky
and very hard to get right, and there does not seem to be sufficient
motivation presented for the benefits of this behavior to justify the
risk.  Specifically, by partitioning all IOAM behaviors by
IOAM-Namespace, we require a node to know whether or not it is to behave
as ingress/egress for a domain on a per-namespace basis.  In effect, it
allows for a proliferation of as many distinct and partially overlapping
IOAM Domains as there are namespace values, and whether a node is
ingress (or egress) for any given domain has to be looked up on the
basis of the specific IOAM-Namespace value.  I find this concerning
because the security properties of the system rely on properly policing
the domain boundary, so that IOAM markings from outside the domain are
rejected at ingress, and IOAM data is not leaked out of the domain by
policing at egress.  While we have ample evidence to suggest that it is
feasible to maintain a single tightly controlled domain that properly
polices ingress and egress (e.g., MPLS), I don't know of evidence to
suggest that it is feasible to do so with a spate of partially
overlapping domains where domains and domain membership are malleable
based the interaction of configuration and in-band protocol elements.
It seems incredibly likely that some domain boundary somewhere will be
inadvertently permeable, and compromise the security properties of the
system.

Section 1

(side note) There's perhaps something of a philosophical question of
whether a mechanism really qualifies as "in situ" if it involves
encapsulating the packet in a full-fledged tunnel (as at least the IPv6
encapsulation does, as is needed to add the additional EHs that hold
IOAM data fields).  That said, I don't really have any alternative
suggestions, and I am sure that IOAM is a well-established terminology,
so this is just a side node.

Section 4

  Scope: This document defines the data fields and associated data
  types for in-situ OAM.  The in-situ OAM data field can be
  encapsulated in a variety of protocols, including NSH, Segment
  Routing, Geneve, IPv6, or IPv4.  Specification details for these

I found drafts (split between targeting ippm and "the relevant working
groups") that would cover Geneve and IPv6, but not anything for IPv4.
Are we fully confident that the aim is actually achievable for IPv4?

  Deployment domain (or scope) of in-situ OAM deployment: IOAM is a
  network domain focused feature, with "network domain" being a set of
  network devices or entities within a single administration.  For
  example, a network domain can include an enterprise campus using
  physical connections between devices or an overlay network using
  virtual connections / tunnels for connectivity between said devices.

If these virtual connections/tunnels do not provide cryptographic
confidentiality and integrity protection, then the security
considerations for IOAM need to include the full physical deployment
scope including the underaly over which the overlay is constructed, not
just the "administrative domain" as defined here.

Furthermore, there seems to be a very questionable interaction between
labeling the IOAM deployment domain an administrative domain yet
allowing for multiple partially overlapping IOAM domains as
distinguished by namespace ID.  Does the administrative domain have to
encompass the union of all the different IOAM domains (as identified by
namespace ID)?

Section 5.2

  An "IOAM transit node" updates one or more of the IOAM-Data-Fields.

(per the discuss)
The previous discussion suggested that even a node that only reads
IOAM-DataFields is still considered a "transit node", and that updating
the field contents is not necessary in order to justify that moniker.
That said, the notion that a transit node is writing (e.g., to update
trace option-types) seems to be prevailing throughout later portions of
the document, so perhaps it is the text a few paragraphs earlier that is
in error.

  If both the Pre-allocated and the Incremental Trace Option-Types are
  present in the packet, each IOAM transit node based on configuration
  and available implementation of IOAM populates IOAM trace data in
  either Pre-allocated or Incremental Trace Option-Type but not both.

(per the discuss)
Likewise, is this "populates" mandatory?

  A transit node MUST ignore IOAM-Option-Types that it does not
  understand.  A transit node MUST NOT add new IOAM-Option-Types to a
  packet, MUST NOT remove IOAM-Option-Types from a packet, and MUST NOT
  change the IOAM-Data-Fields of an IOAM Edge-to-Edge Option-Type.

Almost all of this (not the Edge-to-Edge stuff) seems fairly
tautological, in that if a node did that stuff it would be an
encapsulating and/or decapsulating node, possibly in addition to being a
transit node.

  The role of an IOAM-encapsulating, IOAM-transit or IOAM-decapsulating
  node is always performed within a specific IOAM-Namespace.  This
  means that an IOAM node which is e.g. an IOAM-decapsulating node for
  IOAM-Namespace "A" but not for IOAM-Namespace "B" will only remove
  the IOAM-Option-Types for IOAM-Namespace "A" from the packet.  Note
  that this applies even for IOAM-Option-Types that the node does not
  understand, for example an IOAM-Option-Type other than the four
  described above, that is added in a future revision.  An IOAM
  decapsulating node situated at the edge of an IOAM domain MUST remove
  all IOAM-Option-Types and associated encapsulation headers for all
  IOAM-Namespaces from the packet.

I can only make sense of this paragraph as a whole if the last sentence
instead says "MUST remove from the packet [...] for all IOAM-Namespaces
for which the node is a decapsulating node.  The current text says to
remove literally all IOAM information for literally all IOAM-Namespaces,
which seems to contradict the separation of namespaces depicted earlier
in the paragraph.

Section 5.4

  A particular implementation of IOAM MAY choose to support only one of
  the two trace option types.  In the event that both options are
  utilized at the same time, the Incremental Trace-Option MUST be
  placed before the Pre-allocated Trace-Option.  Deployments which mix
  devices with either the Incremental Trace-Option or the Pre-allocated
  Trace-Option could result in both Option-Types being present in a
  packet.  Given that the operator knows which equipment is deployed in
  a particular IOAM, the operator will decide by means of configuration
  which type(s) of trace options will be used for a particular domain.

Up in Section 5.2 we said that "each IOAM transit node based on
configuration and available implementation" populates exactly one trace
option type.  I think I can read the two statements as being consistent
with each other, but it might be useful to harmonize the specific
language used to make it very clear that these refer to the same
behavior.

Section 5.4.1

Just to check my understanding: when I first read the discussion of this
being an "array" and there being a potential "Opaque State Snapshot"
component, I assumed that the length of this opaque snapshot would need
to be fixed per array element (i.e., as part of the namespace
definition).  Having read a bit more, it seems that my initial
assumption is incorrect, since there is a length field in the opaque
state snapshot framing, and so a node parsing the array of elements will
be able to skip over the approprate (variable) length for each element
in the array.  Please confirm that my updated understanding is correct.

  Namespace-ID:  16-bit identifier of an IOAM-Namespace.  The
      Namespace-ID value of 0x0000 is defined as the "Default-Namespace-
      ID" (see Section 5.3) and MUST be known to all the nodes
      implementing IOAM.  For any other Namespace-ID value that does not
      match any Namespace-ID the node is configured to operate on, the
      node MUST NOT change the contents of the IOAM-Data-Fields.

Though it may seem banal to do so, explicitly listing "change, add, or
remove" might help avoid future questions of how to interpret this text.
E.g., there is some similar text in RFC 2460 about "examined or
processed" that proved to be highly controversial and was changed in RFC
8200.

      A node receiving an IOAM Pre-allocated or Incremental Trace-Option
      relies on the NodeLen value, or it can ignore the NodeLen value
      and calculate the node length from the IOAM-Trace-Type bits (see
      below).

Allowing an implementation to pick one or the other option like this
introduces fragility to the system as a whole and requires strict
controls that encapsulating nodes always set the value even if their
implementation does not use it.  I would prefer if we had a single
well-specified required behavior for all nodes, that could be easily
tested for conformance.

      Bit 0  "Overflow" (O-bit) (most significant bit).  If there are
        not enough octets left to record node data, the network element
        MUST NOT add any fields and MUST set the overflow "O-bit" to
        "1" in the IOAM-Trace-Option header.  This is useful for
        transit nodes to ignore further processing of the option.

This makes things awkward for a lot of the earlier statements that
require transit nodes to populate trace data.  How is the document
internally consistent in this regard, if transit nodes are both required
to populate trace data and required to not add trace data?

      NodeLen - sizeof(opaque snapshot) in 4 octet units.  If
      RemainingLen in a pre-allocated trace option exceeds the length of
      the option, as specified in the preceding header, then the node
      MUST NOT add any fields.

What is the "preceding header"?  This seems like perhaps it originated
in a description of a concrete protocol encoding of this data item and
does not quite apply to the abstract description thereof.

      Bit 12-21  Undefined.  An IOAM encapsulating node MUST set the
              value of each of these bits to 0.  If an IOAM transit
              node receives a packet with one or more of these bits set
              to 1, it MUST either:

              1.  Add corresponding node data filled with the reserved
                  value 0xFFFFFFFF, after the node data fields for the
                  IOAM-Trace-Type bits defined above, such that the

Just to confirm: this means that there cannot be any future allocations
of bits to "wide format" items and thus that only bits 8-10 will ever
consume 2 units each of four-octets?  (This does not preclude allocating
two adjacent bits to indicate a single logical data item, of course,
with appropriate error handling for when only one of the two is set.)

      Bit 23  Reserved: MUST be set to zero upon transmission and
              ignored upon receipt.

I note that this description for the "reserved" bit has a different
specified behavior than the unallocated bits 12-21 (that require a node
to fill the data with ones if the bit is set).  Do we have a sense for
any ways in which this reserved bit's semantics could be used for future
extensibility, vs being locked into these (basically useless) semantics
forever?

Section 5.4.2

  Some IOAM-Data-Fields defined below, such as interface identifiers or
  IOAM-Namespace specific data, are defined in both "short format" as
  well as "wide format".  Their use is not exclusive.  A deployment
  could choose to leverage both.  [...]

While the text goes on to clarify that, based on per-domain
configuration, they can hold qualitatively different types of
information, I still must ask if there is any entity that is or might be
tasked with enforcing that there is consistency between the two fields
(mostly for when they are different representations of the same
information, but not necessarily exclusively so).

Section 5.4.2.1

  Hop_Lim:  1-octet unsigned integer.  It is set to the Hop Limit value
      in the packet at the node that records this data.  Hop Limit

"In the packet" at ingress to, or egress from, the node?

  node_id:  3-octet unsigned integer.  Node identifier field to
      uniquely identify a node within the IOAM-Namespace and associated
      IOAM-Domain.  The procedure to allocate, manage and map the
      node_ids is beyond the scope of this document.

Even if we attempt to leave allocation/management of node_ids out of
scope for this document, I think we still need to talk about what goes
wrong and how to recover in case of collision.

Section 5.4.2.2

I kind of expected some note that the interpretation of the interface
identifier fields here is (or might be?) going to be specified by the
node they apply to and can only be interpreted in that context.  Or are
these expected to be allocated by a central entity?

Section 5.4.2.12

  The "buffer occupancy" field is a 4-octet unsigned integer field.
  This field indicates the current status of the occupancy of the
  common buffer pool used by a set of queues.  The units of this field
  are implementation specific.  Hence, the units are interpreted within
  the context of an IOAM-Namespace and/or node-id if used.  The authors
  acknowledge that in some operational cases there is a need for the
  units to be consistent across a packet path through the network,
  hence it is RECOMMENDED for implementations to use standard units
  such as Bytes.

I guess I'm not sure that I understand exactly what this field should be
indicating, even if my implementation does adhere to the recommendation
to "use standard units such as Bytes" (which, by the way, could probably
stand to be a stronger recommendation for a single distinguished unit
chosen by the authors).  That is, if I suppose that the node in question
has some common pool of buffers that's shared across queues.  Maybe all
the buffers are the same size, maybe not.  But in order to measure
"occupancy", is it more important to know how many bytes are occupied,
how many buffers are in use, or what proportion of the total buffers are
in use?  Just knowing the number of bytes or buffers in use does not
convey much information if the total capacity is unknown, and having 40
MB of buffers in use would mean something very different for a CPE
router vs "big iron".  Can we give a bit more clarity into at least what
portions of the semantics need to be set at a per-namespace level, even
if we can't nail them down more tightly as part of the protocol spec?

Section 5.4.13

  Schema ID:  3-octet unsigned integer identifying the schema of Opaque
      data.

Just to check my understanding: the interpretation of this schema ID is
per IOAM-Namespace, which is mostly going to be something maintained by
the individual operators.  So in some sense each operator will have to
maintain and publish (internally) a registry of these Schema IDs and
avoid collisions.  Is this the sort of thing that is already a common
practice, or is there a risk of operational fragility being introduced?

Furthermore, some of the Namespace IDs are not operator-managed, e.g.,
0x0000.  Is the opaque state snapshot functionality just not going to be
used for those well-known namespaces?  If so, we should say so
explicitly.

Section 5.5

  o  Random: Unique identifier for the packet (e.g., 64-bits allow for
      the unique identification of 2^64 packets).

We should probably say that this is generated using a
cryptographic-strength PRNG (i.e., not rand()).  BCP 106 covers
randomness requirements for security.

Also, due to the birthday paradox, an actual 64-bit random identifier will
produce collisions well before 2^64 packets.  Since these identifiers
(AFAICT) need to be assigned in an uncoordinated fashion, the random
allocation scheme may well be the best scheme (or "least bad"), but if
that's the case I don't think we should make bold statements like "allow
for the unique identification of 2^64 packets".

  IOAM POT flags:  8-bit.  Following flags are defined:

It's slightly surprising to me that the flags are defined as having
global semantics across all POT values, as opposed to being interpreted
in the context of the POT type they are used with, but that's not
inherently problematic.

Section 5.5.1

Should we just say that the Namespace-ID, POT Type, and flags are as
defined in Section 5 rather than repeating the definitions wholesale
(or, as we currently do, having to modify the definition text slightly)?
In particular (but not exclusively), it's pretty distracting to have
Section 5 refer to "Bit 0" and "Bit 1-7" but Section 5.5.1 refer to the
"P bit" and "R (7 bits)"

  P bit:  1-bit.  "Profile-to-use" (P-bit) (most significant bit).
      Indicates which POT-profile is used to generate the Cumulative.
      Any node participating in POT will have a maximum of 2 profiles
      configured that drive the computation of cumulative.  The two
      profiles are numbered 0, 1.  This bit conveys whether profile 0 or
      profile 1 is used to compute the Cumulative.

Is it worth saying a few more words about how the P bit is used as a
"generation count" for performing incremental/in-place updates of what
profile to use, and that profile 0 can be repurposed for something new
once all uses of its previous interpretation have been removed from the
network?  ("No" is a fine answer, but it might be worth proactively
allaying the concern that you can only have two profiles, ever.)

Section 5.6

I suggest giving some guidance as to whether the initial sequence number
should (not) start at zero, for the fields indicated by bits 0 and 1.
I note that draft-gont-numeric-ids-sec-considerations has some
discussion supporting starting at non-zero values.  It seems that the
"increment by one for each packet" behavior is needed, though, in order
to be able to use this value to detect loss.

      Bit 0    (Most significant bit) When set indicates presence of a
              64-bit sequence number added to a specific "packet group"
              which is used to detect packet loss, packet reordering,
              or packet duplication within the group.  The "packet
              group" is deployment dependent and defined at the IOAM
              encapsulating node e.g. by n-tuple based classification
              of packets.

      Bit 1    When set indicates presence of a 32-bit sequence number
              added to a specific "packet group" which is used to
              detect packet loss, packet reordering, or packet
              duplication within that group.  The "packet group" is
              deployment dependent and defined at the IOAM
              encapsulating node e.g. by n-tuple based classification
              of packets.

When both of these bits are set, are the contained values agglomerated
into a hybrid 96-bit sequence number?  If so, in which order?

      Bit 2    When set indicates presence of timestamp seconds,
              [...]
              packet is encapsulated into the tunnel.  Each
              implementation has to document when the E2E timestamp
              that is going to be put in the packet is retrieved.  This

It seems a little awkward that an operator is going to have to base
their processing logic on knowledge of what *implementation* the
encapsulating node is, especially on a heterogeneous network.  But I
suppose it would be an RFC 6919 "MUST (BUT WE KNOW YOU WON'T)" if we
said that implementations had to allow configuring the different
possibilities, based on the IOAM-Namespace...

Section 6.3 (POSIX-based Timestamp Format)

      The POSIX-based timestamp format is affected by leap seconds; it
      represents the number of seconds since the epoch minus the number
      of leap seconds that have occurred since the epoch.  The value of
      a timestamp during or slightly after a leap second could be
      temporarily inaccurate.

(If I'm wrong about the Discuss point, this description seems
inconsistent with the POSIX epoch being midnight TAI.)

Section 8

I note that the "RFC Required" policy lets ISE and IRTF stream documents
allocate values, in theory without any IETF review at all.  It seems
unlikely that this is what is intended in all of the cases where "RFC
Required" is specified, such asa the 4-bit IOAM Trace-Flags registry
that has only three unallocated codepoints.

Section 8.4

  0: 16 Octet POT data

I'd suggest a slightly more descriptive name, as there may well be other
POT formats that want to use 16 octets for their data.

Section 8.7

I suggest removing the sentence "Upon a new allocation request, the
responsible AD will appoint a designated expert, who will review the
allocation request." as it is not really accurate (the IESG appoints
DEs) and is not helpful for the potential registrant.  It's arguably
also needlessly restrictive, preventing the IESG from appointing an
expert before there is a request to review.

Section 10

"Direct Export" sounds like a self-induced DoS attack by traffic
amplification, but that's probably more a matter to be discussed in that
document, not this one.  (Though, do we need to mention direct export
here at all?)

We should probably say that the opaque snapshot, namespace specific
data, etc., will have security considerations corresponding to their
defined data contents that should be described where those formats are
defined.

  From a confidentiality perspective, although IOAM options do not
  contain user data, they can be used for network reconnaissance,

Given that we provide multiple fields that essentially carry opaque or
operator-defined data, the blanket "do not contain" may be too strong of
a statement.  (What if someone decides to put a subscriber identifier in
the namespace-specific data?)  So maybe "are not expected to" is more
appropriate".

  allowing attackers to collect information about network paths,
  performance, queue states, buffer occupancy and other information.

One possible application of such reconiassance is to gauge the
effectiveness of an ongoing attack (e.g., if buffers and queues are
overflowing).  I don't know whether it's particularly useful to mention
that scenario here or not, though.

  IOAM can be used as a means for implementing Denial of Service (DoS)
  attacks, or for amplifying them.  For example, a malicious attacker
  can add an IOAM header to packets in order to consume the resources
  of network devices that take part in IOAM or entities that receive,
  collect or analyze the IOAM data.  [...]

Messing up the POT data seems worth calling out here as well (though the
particular behavior when proof of transit fails is not defined in this
document, of course).

  Notably, in most cases IOAM is expected to be deployed in specific
  network domains, thus confining the potential attack vectors to

Where does the "most cases" come from?  I thought the definitions
restricted IOAM to the IOAM domain.

                  Indeed, in order to limit the scope of threats
  mentioned above to within the current network domain the network
  operator is expected to enforce policies that prevent IOAM traffic
  from leaking outside of the IOAM domain, and prevent IOAM data from
  outside the domain to be processed and used within the domain.

It would be great if we could provide a bit more detail on the scope of
consequences if the operator fails to do so.

Section 12.1

The 2008 POSIX reference has since been superseded by the 2017 version.





NITS

Abstract

nits: NSH is not listed as "well known" at the RFC Editor's abbreviation list,
so should probably be written out in full.  Also, please put commas
before and after "e.g." (which will presumably help with the spurious
double-space as well).

Section 1

  cannot be considered passive.  In terms of the classification given
  in [RFC7799] IOAM could be portrayed as Hybrid Type 1.  IOAM

RFC 7799 writes it with a majuscule 'I', not the numeral '1'.

Section 3

Please use the updated RFC 8174 version of the BCP 14 boilerplate.

Section 4

  Scope: This document defines the data fields and associated data
  types for in-situ OAM.  The in-situ OAM data field can be
  encapsulated in a variety of protocols, including NSH, Segment
  Routing, Geneve, IPv6, or IPv4.  [...]

s/field/fields/, and s/or/and/

Section 5.3

  A subset or all of the IOAM-Option-Types and their corresponding
  IOAM-Data-Fields can be associated to an IOAM-Namespace.  IOAM-

The way this is written seems to imply that any given IOAM-Option-Type
is associated with at most one IOAM-Namespace, which I think is not the
intent.

  Namespaces add further context to IOAM-Option-Types and associated
  IOAM-Data-Fields.  Any IOAM-Namespace MUST interpret the IOAM-Option-
  Types and associated IOAM-Data-Fields per the definition in this
  document.  IOAM-Namespaces group nodes to support different

Presumably this ("MUST interpret") only applies to the option-type and
data fields defined in this document?

  deployment approaches of IOAM (see a few example use-cases below) as

IIUC the meaning here is "IOAM-Namespaces provide a way to group nodes"
and would be easier to read if formulated in that manner.

The RFC Editor will probably have a hard time in this section with which
things that end in 's' are possessives (and thus benefit from an
apostrophe) and which are not, though it may not be an efficient use of
time to try to tidy up before the document gets to them.

  o  whether IOAM-Option-Type(s) has to be removed from the packet,
      e.g. at a domain edge or domain boundary.

there's a singular/plural mismatch here (irrespective of the "(s)" --
"whether an [option-type] has to be removed" vs "whether [option-types]
have to be removed"

Section 5.4.2.3

  The "timestamp seconds" field is a 4-octet unsigned integer field.
  Absolute timestamp in seconds that specifies the time at which the

s/Absolute timestamp/It contains the absolute timestamp/

Section 5.4.2.4

  The "timestamp subseconds" field is a 4-octet unsigned integer field.
  Absolute timestamp in subseconds that specifies the time at which the

s/Absolute timestamp/It contains the absolute timestamp/

Section 5.4.2.9

Please use the exact same wording in the description of the Hop_Lim
field that was used in Section 5.4.2.1 (or just incorporate that
definition by reference).  (The node_id descriptions properly differ
only in the 3-octet vs 7-octet phrase.)

Section 5.4.3

  An entry in the "node data list" array can have different formats,
  following the needs of the deployment.  [...]

This phrasing seems needlessly confusing.  Within a single "node data
list" (i.e., a single packet), all the list entries have the same
format.  What we want to be describing is that the per-entry format can
vary across packets and across deployments.  So perhaps just "the format
used for the entries in a packet's "node data list" array can vary from
packet to packet and deployment to deployment".
(Also, there's a singular/plural mismatch between "an entry" and
"different formats".)

Section 5.5, 5.6

When we talk about how the different Data-Fields "MUST be 4-octet
aligned", having the figure show a variable-height entry might be
helpful; the current formulation looks like it's exactly a 32-bit field.

Section 5.5

  IOAM Proof of Transit Option-Type is to support path or service
  function chain [RFC7665] verification use cases.  Proof-of-transit

s/is to/is used to/

Sections 6.1, 6.2, 6.3

      Seconds: specifies the integer portion of the number of seconds
      since the epoch.

I suggest writing out "the PTP epoch", "the NTP epoch", and "the Unix
epoch" in the respective sections, to avoid giving the impression (via
the definite article) that there is a single distinguished epoch, when
there is not.  (Similarly for the Fractions.)

[Ballot comment]
[ section 5.4 ]

* "deployed in a particular IOAM," -> "deployed in a particular IOAM-Domain,"
  perhaps

* I think perhaps it should be made clear, especially for the Incremental
  Trace Option, that dynamic insertion of data might need to be moderated
  "subject to any protocol constraints of the encapsulating layer", as it
  were.

[Ballot comment]
I have several significant issues to bring up.  Individually, none of them raise to the point of a DISCUSS, so I'm balloting No Objection.


(1) §5.3: "Any IOAM-Namespace MUST interpret the IOAM-Option-Types and associated IOAM-Data-Fields per the definition in this document."  This sentence seems to not say much beyond requiring compliance with this document, which is obvious since this document is the one defining the IOAM-Namespaces.


(2) §5.3: "IOAM-Namespace identifiers MUST be present and populated in all IOAM-Option-Types."  What does "populated" mean?  I assumed that it meant that the field has to have a value in it, but then I found out that the Default-Namespace-ID is 0x0000.  This seems to mean that the receiver can't tell the difference between the sender forgetting to populate the field and the default.  IOW, it seems to me that requiring that the field be populated doesn't serve a specific need and cannot be normatively enforced.


(3) §5.3: Given the example, I don't think this statement is true:

  "...node identifiers might not be unique for other organizational reasons,
  such as after a merger of two formerly separated organizations), the
  combination of node_id and Namespace-ID will always be unique."

It seems to me that for the same reason that merged organizations can end up with overlapping node_ids, they can also end up with overlapping Namespace-IDs.  If not deployed in an overlapping way (on the same set of nodes), it seems that it may be ok to have the same Namespace-ID in multiple places. 

This deployment consideration should be better explained in this document.  I took a quick look at draft-brockners-opsawg-ioam-deployment and this item is not covered there either. 


(4) I think that the encapsulation-independent pieces of draft-brockners-opsawg-ioam-deployment would be better placed in this document.


(5) §5.4: "Any deployment MAY choose to configure and support one or both of the following options."  This sentence looks like a statement of fact and not a normative statement. s/MAY/may


(6) §5.4.1: "The Namespace-ID value of 0x0000 is defined as the "Default-Namespace-ID" (see Section 5.3) and MUST be known to all the nodes implementing IOAM."  The second part ("MUST be known...") is not needed because this document is defining the default value...


(7) §5.4.1:

      An IOAM encapsulating node MUST set NodeLen.

      A node receiving an IOAM Pre-allocated or Incremental Trace-Option
      relies on the NodeLen value, or it can ignore the NodeLen value
      and calculate the node length from the IOAM-Trace-Type bits (see
      below).

Assuming that the NodeLen is not 0 (i.e. set), when can the receiver ignore it?  If NodeLen is 0 (not set), what should the receiver do?  The text above requires NodeLen to be set, but then it makes its use optional.


(8) §5.4.1: "RemainingLen: ...the sender MAY set the initial value of RemainingLen according to the number of node data bytes allowed before exceeding the MTU. ... When node data is added, the node MUST decrease RemainingLen by the amount of data added."

This text includes a normative inconsistency.  If the sender doesn't initialize the "value of RemainingLen according to the number of node data bytes allowed" (because it is optional!), and the transit node does "decrease RemainingLen by the amount of data added" (because it is required), then the value won't reflect the data space remaining and a downstream node may add enough bits to overflow -- or not add anything because it considered the overflow imminent.


(9) §5.4.1: "If RemainingLen in a pre-allocated trace option exceeds the length of the option, as specified in the preceding header, then the node MUST NOT add any fields."  I didn't find "the length of the option" defined anywhere.  Did I miss it?


(10) §5.4.1:

    Bit 12-21  Undefined.  An IOAM encapsulating node MUST set the
              value of each of these bits to 0.  If an IOAM transit
              node receives a packet with one or more of these bits set
              to 1, it MUST either:

              1.  Add corresponding node data filled with the reserved
                  value 0xFFFFFFFF, after the node data fields for the
                  IOAM-Trace-Type bits defined above, such that the
                  total node data added by this node in units of
                  4-octets is equal to NodeLen, or

This first option assumes that all undefined data fields will be 4-octets long.  But if future data fields are defined to be 8-octets (and the transit node doesn't understand them) then 0xFFFFFFFF won't be enough and there will be a misalignment.  IOW, I don't think it is possible to require this behavior -- unless it is also required that all future data fields have to be 4-octets long.


(11) Some of the values to be included in the data fields (§5.4.*) are not well defined and could result in nodes reporting inconsistent measurements.  Specifically, transit delay and queue depth.  It would be ideal if the document provided some guidance for the implementation/use.


(12) §8.7: "Upon a new allocation request, the responsible AD will appoint a designated expert, who will review the allocation request."  This sentence is not needed: the assignment happens only once and not when whenever "a new allocation request" comes up.

Note added 'In last call, the concern arose that there is no integrity protection for IOAM. While the normal use case is controlled networks where this is not a concern, the authors have created draft-brockners-ippm-ioam-data-integrity-01 to develop a design to enable integrity checks should IOAM be used over less secure networks.'

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Functions Operator has completed its review of draft-ietf-ippm-ioam-data-11. If any part of this review is inaccurate, please let us know.

The IANA Functions Operator understands that, upon approval of this document, there are eight actions which we must complete.

First, on the IANA Matrix located at:

https://www.iana.org/protocols

a new registry page is to be created called the "In-Situ OAM (IOAM)" registry page. The reference for the new registry page will be [ RFC-to-be ].

Second, a new registry is to be created called the IOAM Option-Type registry. The new registry is to be created on the "In-Situ OAM (IOAM)" registry page created in the first action above. The new registry will be managed via RFC Required as defined in RFC 8126. There are initial registrations in the new registry as follows:

Option Description Reference
------+-------------------------------------+--------------
0 IOAM Pre-allocated Trace Option-Type [ RFC-to-be ]
1 IOAM Incremental Trace Option-Type [ RFC-to-be ]
2 IOAM POT Option-Type [ RFC-to-be ]
3 IOAM E2E Option-Type [ RFC-to-be ]
4-127 Unassigned

Third, a new registry is to be created called the IOAM Trace-Type registry. The new registry is to be created on the "In-Situ OAM (IOAM)" registry page created in the first action above. The new registry will be managed via RFC Required as defined in RFC 8126. There are initial registrations in the new registry as follows:

Bit Description Reference
-----+----------------------------------------------+---------
0 hop_Lim and node_id in short format [ RFC-to-be ]
1 ingress_if_id and egress_if_id in short format [ RFC-to-be ]
2 timestamp seconds [ RFC-to-be ]
3 timestamp subseconds [ RFC-to-be ]
4 transit delay [ RFC-to-be ]
5 namespace specific data in short format [ RFC-to-be ]
6 queue depth [ RFC-to-be ]
7 checksum complement [ RFC-to-be ]
8 hop_Lim and node_id in wide format [ RFC-to-be ]
9 ingress_if_id and egress_if_id in wide format [ RFC-to-be ]
10 namespace specific data in wide format [ RFC-to-be ]
11 buffer occupancy [ RFC-to-be ]
12-21 unassigned
22 variable length Opaque State Snapshot [ RFC-to-be ]
23 reserved [ RFC-to-be ]

Fourth, a new registry is to be created called the IOAM Trace-Flags registry. The new registry is to be created on the "In-Situ OAM (IOAM)" registry page created in the first action above. The new registry will be managed via RFC Required as defined in RFC 8126. There are initial registrations in the new registry as follows:

Bit Description Reference
------+--------------------------+-------------
0 Overflow (0-bit) [ RFC-to-be ]
1-3 Unassigned

Fifth, a new registry is to be created called the IOAM POT-Type registry. The new registry is to be created on the "In-Situ OAM (IOAM)" registry page created in the first action above. The new registry will be managed via RFC Required as defined in RFC 8126. There are initial registrations in the new registry as follows:

POT
Type Description Reference
------+--------------------------+-------------
0 16 Octet POT data [ RFC-to-be ]
1-255 Unassigned

Sixth, a new registry is to be created called the IOAM POT-Flags registry. The new registry is to be created on the "In-Situ OAM (IOAM)" registry page created in the first action above. The new registry will be managed via RFC Required as defined in RFC 8126. There are initial registrations in the new registry as follows:

POT
Type Description Reference
------+--------------------------+-------------
0  "Profile-to-use" (P-bit) [ RFC-to-be ]
1-7 Unassigned

Seventh, a new registry is to be created called the IOAM E2E-Type registry. The new registry is to be created on the "In-Situ OAM (IOAM)" registry page created in the first action above. The new registry will be managed via RFC Required as defined in RFC 8126. There are initial registrations in the new registry as follows:

E2E
Type Bit Description Reference
--------+--------------------------+-------------
0 64-bit sequence number [ RFC-to-be ]
1 32-bit sequence number [ RFC-to-be ]
2 timestamp seconds [ RFC-to-be ]
3 timestamp subseconds [ RFC-to-be ]
4-15 Unassigned

Eighth, a new registry is to be created called the IOAM Namespace-ID registry. The new registry is to be created on the "In-Situ OAM (IOAM)" registry page created in the first action above.

The registration rules for the new registry are as follows:

0: registered by [ RFC-to-be ]
0x0001 - 0x7FFF: reserved for private use
0x8000 - 0xFFFF: unassigned

The initial registry will look like:

Namespace
ID Description Reference
----------+-------------------------------------------+-------------
0x0000 default namespace (known to all IOAM nodes) [ RFC-to-be ]
0x0001-
0xFFFF unassigned

There are initial registrations in the new registry as follows:

The IANA Functions Operator understands that these are the only actions required to be completed upon approval of this document.

Note:  The actions requested in this document will not be completed until the document has been approved for publication as an RFC. This message is meant only to confirm the list of actions that will be performed.

Thank you,

Sabrina Tanamal
Senior IANA Services Specialist

The following Last Call announcement was sent out (ends 2020-12-08):

From: The IESG
To: IETF-Announce
CC: ippm-chairs@ietf.org, ippm@ietf.org, martin.h.duke@gmail.com, draft-ietf-ippm-ioam-data@ietf.org, Al Morton , acm@research.att.com
Reply-To: last-call@ietf.org
Sender:
Subject: Last Call:  (Data Fields for In-situ OAM) to Proposed Standard


The IESG has received a request from the IP Performance Measurement WG (ippm)
to consider the following document: - 'Data Fields for In-situ OAM'
  as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2020-12-08. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


  In-situ Operations, Administration, and Maintenance (IOAM) records
  operational and telemetry information in the packet while the packet
  traverses a path between two points in the network.  This document
  discusses the data fields and associated data types for in-situ OAM.
  In-situ OAM data fields can be encapsulated into a variety of
  protocols such as NSH, Segment Routing, Geneve, IPv6 (via extension
  header), or IPv4.  In-situ OAM can be used to complement OAM
  mechanisms based on e.g.  ICMP or other types of probe packets.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-data/


The following IPR Declarations may be related to this I-D:

  https://datatracker.ietf.org/ipr/3526/

As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated 1 November 2019.

Document:
Data Fields for In-situ OAM
draft-ietf-ippm-ioam-data-09

Note: This write-up may not be final: see Shepherd's comments at the end.
Date May 30, 2020

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Standards Track, appropriate as a reference for additional RFCs that specify encapsulations in a variety of protocols, such as Segment Routing, Geneve, or IPv6.
Standards Track is indicated on the title page
>>>>>>(but the Datatracker needs update).

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

This memo describes a specific type of OAM capability intended to operate within a network domain, and complements traditional measurement tools for connectivity and route discovery (such as ping and traceroute). This form is called In-situ-OAM, and the techniques employed fall in the RFC 7799 category of Hybrid Type I (as a combination of Active measurement using synthetic traffic and pure Passive observations of user traffic, by adding measurement-specific information to user traffic).  Packets with In-situ OAM encapsulation record information as they traverse nodes within a specific network domain. This information includes timestamps, identification of interfaces, and other details that can assist with OAM activities which include new ones, such as proof of transit (exactly which nodes/interfaces were visited). The IOAM encapsulation will be added/removed at domain ingress/egress, may be added to all or a subset of packets, and updated at all or a subset of transit nodes. IOAM Namespaces provide another dimension of flexibility for actions. This memo will be used a as a reference for additional RFCs that specify encapsulations in a variety of protocols, such as Segment Routing, Geneve, or IPv6.

Working Group Summary:

Was there anything in WG process that is worth noting? For example, was there controversy about particular points or were there decisions where the consensus was particularly rough?

This work topic & proposal was bounced-around a bit before finding an appropriate home in Transport Area and IPPM WG.

Many of the details of IOAM data fields and operations were discussed at length on e-mail and debated at side meetings. Further the development used GitHub's capabilities to track issues and the discussion to resolve each item.
https://github.com/inband-oam/ietf/pulls?q=is%3Apr+is%3Aclosed

Note that at the present time (May 29), 2 relevant PRs are open:
https://github.com/inband-oam/ietf/pulls?q=is%3Aopen+is%3Apr


Document Quality:

Are there existing implementations of the protocol? Have a significant number of vendors indicated their plan to implement the specification? Are there any reviewers that merit special mention as having done a thorough review, e.g., one that resulted in important changes or a conclusion that the document had no substantive issues? If there was a MIB Doctor, YANG Doctor, Media Type or other expert review, what was its course (briefly)? In the case of a Media Type review, on what date was the request posted?

Personnel:

Who is the Document Shepherd?          Al Morton
Who is the Responsible Area Director?  Martin Duke

(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

The Shepherd has reviewed the present draft, and several previous version dating back to the original proposals. Some of the reviews were prompted by interaction between IOAM capabilities and the IPPM WG Draft on Route Metrics.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No concerns, the review in total has been extensive, with many detailed items precipitating active discussions.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

The usual Directorate Reviews should be sufficient.
OPS should ensure that Shepherd's comments appended to this form have been resolved.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

No concerns. There has been plenty of time for concerns to be expressed and for consensus to emerge (since July 2016).

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

F. Brockners
S. Bhandari
C. Pignataro
H. Gredler
J. Leddy
S. Youell
T. Mizrahi
D. Mozes
P. Lapukhov
R. Chang
D. Bernier
J. Lemon


(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

There is one IPR Disclosure:
OPERATIONS, ADMINISTRATION, AND MANAGEMENT FIELDS FOR PACKET TRANSPORT
https://datatracker.ietf.org/ipr/3526/

There appears to have been no discussion of this Disclosure when it appeared on the ippm-list (20 May 2019) to the present.
https://mailarchive.ietf.org/arch/msg/ippm/cYK91eFWvLpaBiLynPFXRh6sKkA/


(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Consensus in the end appears strong, after considerable review and negotiation.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No Appeals threatened.

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

No problems with nits, several comments on version 09 to check:

  == Unused Reference: 'I-D.lapukhov-dataplane-probe' is defined on line
    1815, but no explicit reference was found in the text

  -- Possible downref: Non-RFC (?) normative reference: ref. 'IEEE1588v2'

  -- Possible downref: Non-RFC (?) normative reference: ref. 'POSIX'

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

NA

(13) Have all references within this document been identified as either normative or informative?

Yes

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

All Normative References are complete/approved.

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

Possibly - see item (11) above.

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No change of status for other docs.

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 8126).

This memo creates a new family of registries, and the initial contents have been carefully provided.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

Most of the Registries are RFC Required, however the IOAM Namespace-ID Registry entries will be assigned after Expert Review.

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, YANG modules, etc.

None, NA.

(20) If the document contains a YANG module, has the module been checked with any of the recommended validation tools (https://trac.ietf.org/trac/ops/wiki/yang-review-tools) for syntax and formatting validation? If there are any resulting errors or warnings, what is the justification for not fixing them at this time? Does the YANG module comply with the Network Management Datastore Architecture (NMDA) as specified in RFC8342?

None, NA.

Doc Shepherd's Comments:

1. closed PR https://github.com/inband-oam/ietf/pull/96
Two Comments indicate the value of a Manageability Considerations section while resolving issues in the discussion. However, the -09 version still does not have this section a year later... The important topic discussed was congestion management, but there are no instances of "congest" in the -09 text.

Section 3, Scope, etc. contains topic:
Deployment domain (or scope) of in-situ OAM deployment:, in which many operational considerations are detailed that could be part of a Manageability Considerations: section.

4.4 Trace Option types
...
  ...The maximum
  number of hops and the minimum path MTU of the IOAM domain is assumed
  to be known.
What are the consequences when they are not known?
Looks like the Flag Bit 0 O-bit handles this case for number of hops.

Or, is this knowledge highly likely, and expected to be violated only under the most unexpected conditions (restoration from multiple failures)?
See point below on "minimum path MTU".

4.4.1 
RemainingLen:
...
      Given that the sender knows the minimum path MTU, the sender MAY
      set the initial value of RemainingLen according to the number of
      node data bytes allowed before exceeding the MTU.
"minimum path MTU" is the smallest Maximum Transmission Unit for all links in a path, or simply the Path MTU, PMTU, right?
https://en.wikipedia.org/wiki/Path_MTU_Discovery

4.5 Proof of Transit

Is there a Reference for "Shamir's Secret Sharing Schema (SSSS)" ?
Or, is it a secret?

7.  IANA Considerations
(apologies in advance for a long/recent/good experience with IANA, and the many other folks who try to help)
This section appears to define a set of related registries.
The Hierarchy could be named a bit more efficiently than:

7.1 In-Situ OAM Protocol Parameters Registry (IOAM) Protocol Parameters IANA registry

Suggest:
In-Situ OAM (IOAM) Protocol Parameters Group
    7.1  IOAM Protocol Parameters Registry
7.2  IOAM Option-Type Registry
7.3  IOAM Trace-Type Registry
  ...

Request for posting confirmation emailed to previous authors: David Mozes , " daniel.bernier@bell.ca" , Petr Lapukhov , Carlos Pignataro , Jennifer Lemon , John Leddy , Hannes Gredler , Tal Mizrahi , Frank Brockners , Shwetha Bhandari , Stephen Youell , " remy@barefootnetworks.com" , ippm-chairs@ietf.org

As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated 1 November 2019.

Document:
Data Fields for In-situ OAM
draft-ietf-ippm-ioam-data-09

Note: This write-up may not be final: see Shepherd's comments at the end.
Date May 30, 2020

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Standards Track, appropriate as a reference for additional RFCs that specify encapsulations in a variety of protocols, such as Segment Routing, Geneve, or IPv6.
Standards Track is indicated on the title page
>>>>>>(but the Datatracker needs update).

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

This memo describes a specific type of OAM capability intended to operate within a network domain, and complements traditional measurement tools for connectivity and route discovery (such as ping and traceroute). This form is called In-situ-OAM, and the techniques employed fall in the RFC 7799 category of Hybrid Type I (as a combination of Active measurement using synthetic traffic and pure Passive observations of user traffic, by adding measurement-specific information to user traffic).  Packets with In-situ OAM encapsulation record information as they traverse nodes within a specific network domain. This information includes timestamps, identification of interfaces, and other details that can assist with OAM activities which include new ones, such as proof of transit (exactly which nodes/interfaces were visited). The IOAM encapsulation will be added/removed at domain ingress/egress, may be added to all or a subset of packets, and updated at all or a subset of transit nodes. IOAM Namespaces provide another dimension of flexibility for actions. This memo will be used a as a reference for additional RFCs that specify encapsulations in a variety of protocols, such as Segment Routing, Geneve, or IPv6.

Working Group Summary:

Was there anything in WG process that is worth noting? For example, was there controversy about particular points or were there decisions where the consensus was particularly rough?

This work topic & proposal was bounced-around a bit before finding an appropriate home in Transport Area and IPPM WG.

Many of the details of IOAM data fields and operations were discussed at length on e-mail and debated at side meetings. Further the development used GitHub's capabilities to track issues and the discussion to resolve each item.
https://github.com/inband-oam/ietf/pulls?q=is%3Apr+is%3Aclosed

Note that at the present time (May 29), 2 relevant PRs are open:
https://github.com/inband-oam/ietf/pulls?q=is%3Aopen+is%3Apr


Document Quality:

Are there existing implementations of the protocol? Have a significant number of vendors indicated their plan to implement the specification? Are there any reviewers that merit special mention as having done a thorough review, e.g., one that resulted in important changes or a conclusion that the document had no substantive issues? If there was a MIB Doctor, YANG Doctor, Media Type or other expert review, what was its course (briefly)? In the case of a Media Type review, on what date was the request posted?

Personnel:

Who is the Document Shepherd?          Al Morton
Who is the Responsible Area Director?  Martin Duke

(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

The Shepherd has reviewed the present draft, and several previous version dating back to the original proposals. Some of the reviews were prompted by interaction between IOAM capabilities and the IPPM WG Draft on Route Metrics.

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No concerns, the review in total has been extensive, with many detailed items precipitating active discussions.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

The usual Directorate Reviews should be sufficient.
OPS should ensure that Shepherd's comments appended to this form have been resolved.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

No concerns. There has been plenty of time for concerns to be expressed and for consensus to emerge (since July 2016).

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

F. Brockners
S. Bhandari
C. Pignataro
H. Gredler
J. Leddy
S. Youell
T. Mizrahi
D. Mozes
P. Lapukhov
R. Chang
D. Bernier
J. Lemon


(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

There is one IPR Disclosure:
OPERATIONS, ADMINISTRATION, AND MANAGEMENT FIELDS FOR PACKET TRANSPORT
https://datatracker.ietf.org/ipr/3526/

There appears to have been no discussion of this Disclosure when it appeared on the ippm-list (20 May 2019) to the present.
https://mailarchive.ietf.org/arch/msg/ippm/cYK91eFWvLpaBiLynPFXRh6sKkA/


(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

Consensus in the end appears strong, after considerable review and negotiation.

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

No Appeals threatened.

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

No problems with nits, several comments on version 09 to check:

  == Unused Reference: 'I-D.lapukhov-dataplane-probe' is defined on line
    1815, but no explicit reference was found in the text

  -- Possible downref: Non-RFC (?) normative reference: ref. 'IEEE1588v2'

  -- Possible downref: Non-RFC (?) normative reference: ref. 'POSIX'

(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

NA

(13) Have all references within this document been identified as either normative or informative?

Yes

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

All Normative References are complete/approved.

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

Possibly - see item (11) above.

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No change of status for other docs.

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 8126).

This memo creates a new family of registries, and the initial contents have been carefully provided.

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

Most of the Registries are RFC Required, however the IOAM Namespace-ID Registry entries will be assigned after Expert Review.

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, YANG modules, etc.

None, NA.

(20) If the document contains a YANG module, has the module been checked with any of the recommended validation tools (https://trac.ietf.org/trac/ops/wiki/yang-review-tools) for syntax and formatting validation? If there are any resulting errors or warnings, what is the justification for not fixing them at this time? Does the YANG module comply with the Network Management Datastore Architecture (NMDA) as specified in RFC8342?

None, NA.

Doc Shepherd's Comments:

1. closed PR https://github.com/inband-oam/ietf/pull/96
Two Comments indicate the value of a Manageability Considerations section while resolving issues in the discussion. However, the -09 version still does not have this section a year later... The important topic discussed was congestion management, but there are no instances of "congest" in the -09 text.

Section 3, Scope, etc. contains topic:
Deployment domain (or scope) of in-situ OAM deployment:, in which many operational considerations are detailed that could be part of a Manageability Considerations: section.

4.4 Trace Option types
...
  ...The maximum
  number of hops and the minimum path MTU of the IOAM domain is assumed
  to be known.
What are the consequences when they are not known?
Looks like the Flag Bit 0 O-bit handles this case for number of hops.

Or, is this knowledge highly likely, and expected to be violated only under the most unexpected conditions (restoration from multiple failures)?
See point below on "minimum path MTU".

4.4.1 
RemainingLen:
...
      Given that the sender knows the minimum path MTU, the sender MAY
      set the initial value of RemainingLen according to the number of
      node data bytes allowed before exceeding the MTU.
"minimum path MTU" is the smallest Maximum Transmission Unit for all links in a path, or simply the Path MTU, PMTU, right?
https://en.wikipedia.org/wiki/Path_MTU_Discovery

4.5 Proof of Transit

Is there a Reference for "Shamir's Secret Sharing Schema (SSSS)" ?
Or, is it a secret?

7.  IANA Considerations
(apologies in advance for a long/recent/good experience with IANA, and the many other folks who try to help)
This section appears to define a set of related registries.
The Hierarchy could be named a bit more efficiently than:

7.1 In-Situ OAM Protocol Parameters Registry (IOAM) Protocol Parameters IANA registry

Suggest:
In-Situ OAM (IOAM) Protocol Parameters Group
    7.1  IOAM Protocol Parameters Registry
7.2  IOAM Option-Type Registry
7.3  IOAM Trace-Type Registry
  ...

Request for posting confirmation emailed to previous authors: Shwetha Bhandari , Jennifer Lemon , " remy@barefootnetworks.com" , " daniel.bernier@bell.ca" , Tal Mizrahi , Hannes Gredler , David Mozes , Petr Lapukhov , Carlos Pignataro , Stephen Youell , John Leddy , Frank Brockners

Request for posting confirmation emailed to previous authors: Frank Brockners , ippm-chairs@ietf.org, Tal Mizrahi , Shwetha Bhandari , Petr Lapukhov , Carlos Pignataro , John Leddy , " daniel.bernier@bell.ca" , Hannes Gredler , Stephen Youell , John Lemon , David Mozes , Remy Chang

Request for posting confirmation emailed to previous authors: Frank Brockners , ippm-chairs@ietf.org, Tal Mizrahi , Shwetha Bhandari , Petr Lapukhov , Carlos Pignataro , John Leddy , " daniel.bernier@bell.ca" , Hannes Gredler , Stephen Youell , John Lemon , David Mozes , Remy Chang

Request for posting confirmation emailed to previous authors: " daniel.bernier@bell.ca" , Frank Brockners , ippm-chairs@ietf.org, Tal Mizrahi , John Leddy , Petr Lapukhov , Carlos Pignataro , Shwetha Bhandari , Hannes Gredler , Stephen Youell , John Lemon , David Mozes , Remy Chang

Request for posting confirmation emailed to previous authors: " daniel.bernier@bell.ca" , Frank Brockners , ippm-chairs@ietf.org, Tal Mizrahi , John Leddy , Petr Lapukhov , Carlos Pignataro , Shwetha Bhandari , Hannes Gredler , Stephen Youell , John Lemon , David Mozes , Remy Chang

Request for posting confirmation emailed to previous authors: " daniel.bernier@bell.ca" , Frank Brockners , Tal Mizrahi , ippm-chairs@ietf.org, John Leddy , Petr Lapukhov , Carlos Pignataro , Shwetha Bhandari , Hannes Gredler , Stephen Youell , John Lemon , David Mozes , Remy Chang

Request for posting confirmation emailed to previous authors: " daniel.bernier@bell.ca" , Frank Brockners , Tal Mizrahi , ippm-chairs@ietf.org, John Leddy , Petr Lapukhov , Carlos Pignataro , Shwetha Bhandari , Hannes Gredler , Stephen Youell , John Lemon , David Mozes , Remy Chang

Request for posting confirmation emailed to previous authors: " daniel.bernier@bell.ca" , Frank Brockners , Tal Mizrahi , John Leddy , Petr Lapukhov , Carlos Pignataro , Shwetha Bhandari , Hannes Gredler , Stephen Youell , ippm-chairs@ietf.org, Remy Chang , David Mozes

Request for posting confirmation emailed to previous authors: " daniel.bernier@bell.ca" , Frank Brockners , Tal Mizrahi , John Leddy , Petr Lapukhov , Carlos Pignataro , Shwetha Bhandari , Hannes Gredler , Stephen Youell , ippm-chairs@ietf.org, Remy Chang , David Mozes