Network Telemetry Framework
draft-ietf-opsawg-ntf-13

[Ballot comment]
Thanks for making the applicability statement more prominent in the -12.

I think the document paints an exciting picture of a new mindset in
which to frame discussion of network monitoring and management (even if
it does stray too far into marketing language for my taste in places).
It doesn't do quite as well at convincing me that an entirely new
technology suite is merited (as opposed to just extending existing
protocols to align with the new mindset), but I am willing to admit the
possibility that the new technology suite is the right approach.

That said, I have strong misgivings about the current state of the
document, mostly relating to privacy considerations and the risk of
pervasive monitoring, so I am balloting Abstain.

While we do clearly say to not analyze individual users, we also have
guidance (e.g., in §2.1) that only says "no user packet content should
be collected".  However, packet contents are not the only things that
can be a threat to user privacy, and we've seen numerous instances where
just metadata about user flows are sufficient to make strong conclusions
about user behavior that impact user privacy.  But if we try to
strengthen the requirement to be not collecting any data about user
packets, the utility of the system decreases greatly, and I don't see a
clear way to reconcile the impasse.

(There are also a few lingering references to "user flows", "user
packets", "user traffic", etc. in the main body text, especially in
§2.3.  I'm not convinced that all of the instanaces of these phrases are
compatible with the applicability statement.)

Furthermore, the applicability statement seems to be a case of wishful
thinking.  I do not see any proposals for technical measures to enforce
that data is not collected from networks where endpoints represent
users, and I also don't see any mechansisms to disincentivize such use
in favor of other, more privacy-friendly, alternatives.  So even if we
consider such usage of the network telemetry framework to be an abuse
case rather than a use case, if we are going to honestly document the
implications of the technology, I can't escape the conclusion that we
need to consider these scenarios in our assessment of whether we are
defining the right technology.


Though I am balloting Abstain, I will also some specific comments on the
document that might help improve it, even if I may not be completely
happy with the resulting document (for the reasons described above).

It's pretty surprising to see a document that mentions autonomic
networking and aims to achieve self-managing networks make no reference
at all to the IETF ANIMA WG or its outputs, a group that is specifically
chartered to produce protocols and procedures for automated network
management.  In particular, it's my understanding that ANMIA has had
very little traction with network Intent thus far, and this document
references IRTF documents in many places (both for Intent and other
things).  Are we confident that these concepts are ready to move from
the IRTF into the engineering world?

Section 1

  Network visibility is the ability of management tools to see the
  state and behavior of a network, which is essential for successful

In the TLS WG we've sometimes seen participants use the term
"visibility" to include the plaintext of encrypted data flows.  While I
have no reason to believe that that's a universally held understanding
of the term, I mention it only to ask that clarification be provided if
the intent of the term here is to include such decryption capabilities.
If the intent is only to observe the normal visible wire image of the
protocol, I don't see particular need for clarification.

Section 2

  forward.  When a network's endpoints do not represent individual
  users (e.g. in industrial, datacenter, and infrastructure contexts),
  network operations can often benefit from large-scale data collection
  without breaching user privacy.

In the vein of my toplevel remarks, I don't think that just "a network's
endpoints do not represent individual users" is sufficient to ensure
that large-scale data collection does not breach user privacy.  It
covers first-order effects, I think, but we've seen a lot of research
indicating that second- and higher-order analyses can still extract
information that reduces user privacy.

Section 2.1

  To preserve the privacy of end-users, no user packet content should
  be collected.  Specifically, the data objects generated, exported,
  and collected by a network telemetry application should not include
  any packet payload from traffic associated with end-users systems.

Also in the vein of my toplevel remarks, while "do not include user
traffic payload" is a minimum requirement, and I'm happy to see it
stated clearly, it in and of itself is not sufficient to fully protect
end-user privacy.

Section 2.2

      visibility into networks.  The ultimate goal is to achieve the
      security with no, or only minimal, human intervention.

It's easy to achieve security without human intervention, if you're
willing to accept a high false positive rate and denial of legitimate
traffic.  Should we say something about tempering the security goal with
a need for not disrupting legitimate traffic flows?

Section 2.3

      Conventional OAM only covers a narrow range of data (e.g., SNMP
      only handles data from the Management Information Base (MIB)).

This argument feels a bit weak given that anyone with an OID arc (that
is, just about anyone) can add to the MIB.

Section 2.4

  Network telemetry has emerged as a mainstream technical term to refer

It's a little surprising to see network telemetry called a "mainstream"
term here, when up in §1 we said that it lacks an unambiguous
definition.

  *  Model-based: The telemetry data is modeled in advance which allows
      applications to configure and consume data with ease.
  [...]
  *  In-Network Customization: The data that is generated can be
      customized in network at run-time to cater to the specific need of
      applications.  This needs the support of a programmable data plane
      which allows probes with custom functions to be deployed at
      flexible locations.

I'm having a hard time seeing how data that's customized in-network at
runtime would be compatible with being modeled in advance.  Maybe the
disclaimer about "not expected to be held by every specific technique"
is intended to apply here, but it might be worth acknowledging the
tradeoff.

  *  In-band Data Collection: In addition to the passive and active
      data collection approaches, the new hybrid approach allows to
      directly collect data for any target flow on its entire forwarding
      path [I-D.song-opsawg-ifit-framework].

I'm pretty skeptical that the functionality that's claimed here (and in
the referenced draft) can be achieved while complying with the existing
requirements from current IETF RFCs.  I recognize that this is under the
"an ideal [solution] may also have" heading, but it still feels a little
premature to include.

Section 3.1

I'm having a really hard time seeing how figure 2 is internally
consistent if it lists "plain text" as the only option for data encoding
of data modelled using YANG (e.g., in the forwarding plane column).

Section 3.1.1

  network statistics and state data.  The management plane includes
  many protocols, including some that are considered "legacy", such as
  SNMP and syslog.  Regardless the protocol, management plane telemetry

It's not clear that we gain any real value from labeling SNMP and syslog
as "legacy".  Perhaps we should just skip the examples and avoid debate
on what is or isn't legacy (leaving each person to hold their own
opinion on that question)?

Section 3.1.2

      Then in case of an unusually poor UE KPI or a service
      disconnection, it is non-trivial to delimit and pinpoint the issue
      in the responsible protocol layer (e.g., the Transport Layer or
      the Network Layer), the responsible protocol (e.g., ISIS or BGP at
      the Network Layer), and finally the responsible device(s) with

I don't really follow the example of IS-IS or BGP "at the Network
Layer" -- in what sense do we use "network layer" here?

Section 3.3

I don't really understand the logic behind the direction of arrowheads
in Figure 4.  I'd be more inclined to just remove the figure than add
more explanatory text, though, as the relationships don't seem terribly
key to the core purpose of this document.

Section 5

  *  Authentication and signing of telemetry data to make data more
      trustworthy.

Signing is typically treated as a way to provide authentication; it
might make more sense to discuss "authentication and integrity
protection" in terms of the typical security properties we consider.

NITS

Section 1

  operations.  Based on the distinction of modules and function
  components, we can map the existing and emerging techniques and

It would be "distinction between" or "definition of", I think.

  protocols into the framework.  The framework can also simplify the
  designing, maintaining, and understanding a network telemetry system.

The "the" leading into "designing, maintaining, and understanding"
should be removed.

  The purpose of the framework and taxonomy is to set a common ground
  for the collection of related work and provide guidance for future
  technique and standard developments.  To the best of our knowledge,

s/technique/techniques/

Section 1.2

  AI:  Artificial Intelligence.  In network domain, AI refers to the
      machine-learning based technologies for automated network
      operation and other tasks.

"the network domain"

  SNMP:  Simple Network Management Protocol.  Version 1, 2, and 3 are
      specified in [RFC1157], [RFC3416], and [RFC3414], respectively.

RFC 3411 might be a better reference for SNMPv3, as it's the
architecture doc (rather than the user-based security model doc).

Section 2

  It is conceivable that an autonomic network [RFC7575] is the logical
  next step for network evolution following Software Defined Network

I think "Software Defined Networking" would fit better in this
situation.

  protocols are insufficient for these use cases.  The discussion
  underlines the need of new methods, techniques, and protocols, as
  well as the extensions of existing ones, which we assign under the

s/need of/need for/

Section 2.2

      Given increasingly sophisticated attack vector coupled with

"vectors" plural

      visibility into networks.  The ultimate goal is to achieve the
      security with no, or only minimal, human intervention.

s/the//

      visibility that is provided through network telemetry data.  Any
      violation must be notified immediately, potentially resulting in
      updates to how the policy or intent is applied in the network to

The subject of the verb "notified" is the target of the notification,
not the thing that the notification is about.  So "reported" might fit
better here.

      operators need to evaluate how they can deliver the services that
      can meet the SLA based on realtime network telemetry data,
      including data from network measurements.

s/deliver the services/deliver services/

Section 2.3

  *  Comprehensive data is needed from packet processing engines to
      traffic manager, from line cards to main control board, from user
      flows to control protocol packets, from device configurations to
      operations, and from physical layer to application layer.

It's possible to read this as a set of "from A to B" relations where A
is sending data to B".  I think that's not the intent, and this is just
intending to show a broad spread of scenarios across many different
axes; if that's the case, I'd suggest "... needed, ranging from"

  *  The conventional passive measurement techniques can either consume
      excessive network resources and render excessive redundant data,

Something seems awry around "render excessive redundant data", to the
extent that I can't extrct meaning and propose an alternative.

Section 2.4

      overall network automation needs.  Efforts are made to normalize
      the data representation and unify the protocols, so to simplify
      data analysis and provide integrated analysis across heterogeneous

"so as to"

Section 2.5

      network with a low data sampling rate.  Only when issues arise or
      critical trends emerge should telemetry data source be modified
      and telemetry data rates boosted as needed.

I think we need ""the telemetry data source".

Section 3.1.2

  *  An example of the control plane telemetry is the BGP monitoring
      protocol (BMP), it is currently used for monitoring the BGP routes

I'd end the sentence at this comma to avoid a comma splice.

Section 3.2

      responsible for configuring the desired data that might not be
      directly available form data sources.  The subscription data can

s/form/from/

Section 5

  *  Protocol transport used telemetry data and inherent security
      capabilities;

There seems to be a word or two missing here, maybe "used for" and "its
inherent".

Section A.3.6

  Various data planes raises unique OAM requirements.  IETF has

s/raises/raise/

[Ballot comment]
Thanks to Alexey Melnikov for the SECDIR review.

Thanks for addressing my DISCUSS point and some of my COMMENTs.

(Ballot note on -12: I wanted to quickly update my ballot before the telechat to reflect that -12 resolved my discuss.  I need to more carefully review the responses to the comments.  Where resolution could be quickly assessed from the diff, I have already updated my ballot accordingly.  -12 may in fact address more of the comments still noted below.)

(Ballot on -11):
I'm a bit of confusion on the framing of this document.  It seems to me to be suggesting that “OAM” is a tied to a series of static technologies and practices, and a set of new practices called “network telemetry” are needed.  I don’t disagree with the idea that network management practices need to evolve, and that the “networks of the future” will look different than today.  Relying on BCP 161 (RFC 6291), I took OAM to mean an evolving set of practices and technology.  Using Section 3 of BCP 161, O + A + M seemed like a contextual set of operations that would be done now and still required in networks of the future.  The document acknowledges that there is some ambiguity in “network telemetry”.  I think it needs to equally acknowledge that the same is true of OAM, and that RFC7276 is not OAM.  In the aggregate, I don’t think the text realizes the clarity that it set out to provide by defining “key characteristics of network telemetry which set a clear distinction from the conventional network OAM and show that some conventional OAM technologies can be considered a subset of the network telemetry technologies.”.  To be clear, I’m not raising an objection to many of the properties linked to network telemetry.  Instead, I think the clarity of message is getting diluted because a very particular distinction is trying to be made (OAM vs. network telemetry) and it isn’t clear.  See below for a specifics.

** Section 1
  Network telemetry extends beyond the historical network Operations,
  Administration, and Management (OAM) techniques and expects to
  support better flexibility, scalability, accuracy, coverage, and
  performance.

This seems hypothetical depending on the definition on which technologies are considered in scope of network telemetry and OAM.

** Section 2.

Today one can access advanced big data analytics capability through a
  plethora of commercial and open source platforms (e.g., Apache
  Hadoop), tools (e.g., Apache Spark), and techniques (e.g., machine
  learning).  Thanks to the advance of computing and storage
  technologies, network big data analytics gives network operators an
  opportunity to gain network insights and move towards network
  autonomy. 
In trying to contextual this observation, where is this capability relative to Figure 1?  In general, I would recommend that this reference architecture when assessing the ecosystem.

** Section 2.

However, while the data processing capability is improved and
  applications are hungry for more data ...

What does it mean and what applications are “hungry for more data”.  Is a reference possible here?

** Section 2.3
  For a long time, network operators have relied upon SNMP [RFC3416],
  Command-Line Interface (CLI), or Syslog to monitor the network.  Some
  other OAM techniques as described in [RFC7276] are also used to
  facilitate network troubleshooting.
...
  These challenges were addressed by newer standards and techniques
  (e.g., IPFIX/Netflow, PSAMP, IOAM, and YANG-Push) and more are
  emerging.  These standards and techniques need to be recognized and
  accommodated in a new framework.

This section is an exemplar of the disconnect I noted in the definitions of OAM.  The first paragraph presents a narrow view of currently used (albeit older) network monitoring technologies (SNMP, CLI Syslog).  However, in the closing paragraph, the text names more modern technologies I would also consider OAM, and these technologies could meet some of the challenges mentioned in this section.  Furthermore, some of these “newer standards” are framed as things that need to be “recognized”.  This is puzzling because my understanding was that technologies like IPFIX/Netflow have been very widely deployed for quite some time now.  What’s the new framework needed?

** Section 2.4
Network telemetry covers the conventional network OAM and
  has a wider scope. 

Can the text be more specific in what way network telemetry is wider.  I thought OAM was rather ambiguous.

** Section 2.4
Hence, the network telemetry can directly
  trigger the automated network operation, while in contrast some
  conventional OAM tools are designed and used to help human operators
  to monitor and diagnose the networks and guide manual network
  operations.

I’m not sure if this is a fair generalization.  Even “older technologies” like SNMP currently trigger automated responses based on the values they return.

** Section 2.4.  Per “data fusion,” which part of the Figure 1 is this happening?

** Section 2.5.

Network data analytics and machine-learning technologies are applied
  for network operation automation, relying on abundant and coherent
  data from networks. 

What is coherent data?

** Section 2.5
All the use cases and
      applications are better to be supported uniformly and coherently
      under a single intelligent agent

-- Editorial.  There is a missing word which leads to this sentence not parsing.

-- What’s the basis for asserting that a “single intelligent agent” is the  best approach?

-- Maybe the issue is of semantics, what is an “intelligent agent” in this context?

** Section 2.5. 

Network visibility presents multiple viewpoints

and

Efficient data fusion is critical for applications to reduce the
      overall quantity of data and improve the accuracy of analysis.

Are these generalizations expected to be true across the broad use cases?

** Figure 2.  For the management plane, the data model module has MIB and syslog listed, but the data encodings as GPB, JSON and XML.  These data models and encodings don’t line up (i.e., MIBs and syslog typically don’t rely on GPB, JSON or XML). 

** Section 3.1.  Where do network security applications such as WAFs, IDS/IPS/ NGF, DLP, web-proxies, and pDNS fit into this taxonomy?

** Section 3.1.* These sections inconsistently describe properties/requirements for an architectural element and their challenges (but no solutions or requirements for) a given elements.  As a result, I had trouble understanding what an implementer should understand these components.  It would have been clearer is the different modules had common and module specific requirements.

** Section 3.1.1.  Per the requirements of “Convenient Data Subscription”, “Structured Data”, etc. why wouldn’t those be desirable requirements for all four of the modules?

** Section 3.1.3.  Providing “timely data” and “structured data”, seem like the restatements of Section 4.1.1’s “structure data” and “high speed transport”.  Is this a common requirement?

** Section 3.1.3.  Why wouldn’t it be desirable for all of the modules to support incremental deployment note here?

[Ballot comment]
Thanks for working in the documents and thanks Michael Scharf for the TSVART review. I am happy with that fact that the table with selected technology is now lists examples.

[Ballot comment]
[S3.1; question]

* Figure 2: Is PCAP a possible Data Encoding for the Forwarding Plane?

* Figure 2: Should "plain text" be replaced by something like
  "proprietary"?

  I suspect the intent was to communicate that custom/bespoke encodings
  may still be quite common.

[Ballot comment]
Firstly, thank you for writing this, and thanks to Sheng Jiang for the OpsDir review.

I personally think that these sort of "overview" / framework documents are really useful; I've often had to get spun up on some new technology and reading an RFC which specifies the packet format without a good overview document (like this one ) is not helpful...

Anyway, with that soapbox rant over, I have 2 substantive comments and some nits to help make the document better / more readable. There is no need to reply to each comment (or at all) - these are non-blocking comments, but fixing them will help make the RFC Editor's job easier and help ensure that none sneak through...

1:S1 - Introduction
"All the modules are internally structured in the same way, including components that allow to configure data sources in ..."
P: "that allow for the configuration" or "that allow the  to configure what data to..."

"The framework can also simplify the tasks for designing, maintaining, and understanding a network telemetry system."
P:"the task of designing" (or just "simplify the design, maintenance and understanding of ..."

"At last, we outline the evolution stages of the network telemetry system and discuss the potential security concerns."
P: "Finally, ..." or "Lastly, ...". Actually, I think "In addition...." would be even better.

S 2.2 Use Cases
"Intent, as defined in [I-D.irtf-nmrg-ibn-concepts-definitions], is a set of operational goal that a network ..."
s/goal/goals/

"SLA Compliance: A Service-Level Agreement (SLA) defines the level of service a user expects from a network operator,"
I disagree with this -- an SLA is what a network operator has agreed to provide, usually with a contract and penalty to missing it. As a user I *expect* my network operator to always exceed the SLA - if the SLA specifies 95% uptime, I don't really expect 36 hours of downtime each month :-P. Also, in many cases I (sadly) *expect* much worse service than the SLA actually specifies... The Wikipedia https://en.wikipedia.org/wiki/Service-level_agreement page has some good text you could use...

"Root Cause Analysis: Any network failure can be the effect of a sequence of chained events."
The use of "Any" here seems odd / wrong -- perhaps "Network failures are often ..." or "Many network failures..." or just drop the first sentence and start with "Troubleshooting and recovery require ..."

S 2.3
"The poll-based low-frequency data collection is ill-suited..."
Drop the "The", or s/The/This/

"Subscription-based streaming data directly pushed from the data source (e.g., the forwarding chip) is preferred to provide enough data quantity and precision at scale."
Suggest s/enough/sufficient/ (just for flow)

"Comprehensive data is needed from packet processing engine to traffic manager, from line cards to main control board,..."
s/engine/engines/, or, better, "from the packet processing engine to the traffic manager..."

S 3.1. Top Level Modules
"Therefore, we categorize the network telemetry into four distinct modules with each having its own interface to Network Operation Applications."
It would be really helpful to list the "four distinct modules" here -- I spent quite a while looking at the figure (with 5 boxes :-)) trying to infer what you meant. Just adding them in a parenthetical would work well.

"For example, the forwarding chip has high throughput but limited capacity for processing complex data and maintaining states, while ..."
s/states/state/ (Yes, I get that the forwarding engine manages many different sets of state, but "maintaining states" still seems weird...)

Again, thank you for writing this, and also for considering / addressing these nits...

[Ballot discuss]
Thank you for being responsive to the SECDIR review thread to improve the security considerations text.  Specifically, https://mailarchive.ietf.org/arch/msg/secdir/GUvFWXP7n9IjXW8xlIdMS5ZE5u0/.

Even after these edits, there are a few straightforward ambiguities to clear up.

(a) Section 2.  “When a network's endpoints do not represent individual users (e.g. in industrial, datacenter, and infrastructure contexts), network operations can often benefit from large-scale data collection without breaching user privacy.”

Is network telemetry architecture being restricted to such a limited applicability?  To quote the original SECDIR thread, is this saying “The Network Telemetry Framework is not applicable to networks whose endpoints represent individual users, such as general-purpose access networks”?  If so, I’d recommend being that explicit.

(b) Section 2.1.  “To preserve user privacy, the user packet content should not be collected.”
This is a great principle, but extremely nuanced and potentially complicated to implement.  Is this saying (using the words of this framework), “To preserve the privacy of end-users, no user packet content should be collected.  Specifically, the data objects generated, exported, and collected by the Network Telemetry Framework should not include any packet payload from traffic associated with end-users systems”?

(c) Section 2.5.  Please use stronger and consistent language.

OLD
Disclaimer: large-scale network data collection is a major threat to
user privacy [RFC7258].  The network telemetry framework presented in
this document should not be applied to collect and retain individual
user data or any data that can identify end users without consent.
Any data collection or retention using the framework must be tightly
limited to protect user privacy.

NEW
Large-scale network data collection is a major threat to user privacy and may be indistinguishable from pervasive monitoring [RFC7258].  The network telemetry framework presented in this document must not be applied to generating, exporting, collecting, analyzing or retaining individual user data or any data that can identify end users or characterize their behavior without consent.

The principles described in (a), (b) and (c) seems sufficiently important they shouldn’t be scattered across the document.  Please either make an applicability statement section early in the document or a dedicated privacy consideration section.

[Ballot discuss]
Thank you for being responsive to the SECDIR review threat to improve the security considerations text.  Specifically, https://mailarchive.ietf.org/arch/msg/secdir/GUvFWXP7n9IjXW8xlIdMS5ZE5u0/.

Even after these edits, there are a few straightforward ambiguities to clear up.

(a) Section 2.  “When a network's endpoints do not represent individual users (e.g. in industrial, datacenter, and infrastructure contexts), network operations can often benefit from large-scale data collection without breaching user privacy.”

Is network telemetry architecture being restricted to such a limited applicability?  To quote the original SECDIR thread, is this saying “The Network Telemetry Framework is not applicable to networks whose endpoints represent individual users, such as general-purpose access networks”?  If so, I’d recommend being that explicit.

(b) Section 2.1.  “To preserve user privacy, the user packet content should not be collected.”
This is a great principle, but extremely nuanced and potentially complicated to implement.  Is this saying (using the words of this framework), “To preserve the privacy of end-users, no user packet content should be collected.  Specifically, the data objects generated, exported, and collected by the Network Telemetry Framework should not include any packet payload from traffic associated with end-users systems”?

(c) Section 2.5.  Please use stronger and consistent language.

OLD
Disclaimer: large-scale network data collection is a major threat to
user privacy [RFC7258].  The network telemetry framework presented in
this document should not be applied to collect and retain individual
user data or any data that can identify end users without consent.
Any data collection or retention using the framework must be tightly
limited to protect user privacy.

NEW
Large-scale network data collection is a major threat to user privacy and may be indistinguishable from pervasive monitoring [RFC7258].  The network telemetry framework presented in this document must not be applied to generating, exporting, collecting, analyzing or retaining individual user data or any data that can identify end users or characterize their behavior without consent.

The principles described in (a), (b) and (c) seems sufficiently important they shouldn’t be scattered across the document.  Please either make an applicability statement section early in the document or a dedicated privacy consideration section.

[Ballot comment]
(Apologize if any of the below section numbers are wrong.  I conducted most of my review on -10 and then -11 was published which renumbered the document)

Thanks to Alexey Melnikov for the SECDIR review.

I'm a bit of confusion on the framing of this document.  It seems to me to be suggesting that “OAM” is a tied to a series of static technologies and practices, and a set of new practices called “network telemetry” are needed.  I don’t disagree with the idea that network management practices need to evolve, and that the “networks of the future” will look different than today.  Relying on BCP 161 (RFC 6291), I took OAM to mean an evolving set of practices and technology.  Using Section 3 of BCP 161, O + A + M seemed like a contextual set of operations that would be done now and still required in networks of the future.  The document acknowledges that there is some ambiguity in “network telemetry”.  I think it needs to equally acknowledge that the same is true of OAM, and that RFC7276 is not OAM.  In the aggregate, I don’t think the text realizes the clarity that it set out to provide by defining “key characteristics of network telemetry which set a clear distinction from the conventional network OAM and show that some conventional OAM technologies can be considered a subset of the network telemetry technologies.”.  To be clear, I’m not raising an objection to many of the properties linked to network telemetry.  Instead, I think the clarity of message is getting diluted because a very particular distinction is trying to be made (OAM vs. network telemetry) and it isn’t clear.  See below for a specifics.

** Section 1.
… using a wide variety of techniques including machine learning, data analysis, and correlation.

ML, data analysis and correlation are unlike things.  ML is a particular AI technique, data analysis is a generic description of an activity, and is correlation intended to be a statistical technique?

** Section 1
  Network telemetry extends beyond the historical network Operations,
  Administration, and Management (OAM) techniques and expects to
  support better flexibility, scalability, accuracy, coverage, and
  performance.

This seems hypothetical depending on the definition on which technologies are considered in scope of network telemetry and OAM.

** Section 2.

Today one can access advanced big data analytics capability through a
  plethora of commercial and open source platforms (e.g., Apache
  Hadoop), tools (e.g., Apache Spark), and techniques (e.g., machine
  learning).  Thanks to the advance of computing and storage
  technologies, network big data analytics gives network operators an
  opportunity to gain network insights and move towards network
  autonomy. 
In trying to contextual this observation, where is this capability relative to Figure 1?  In general, I would recommend that this reference architecture when assessing the ecosystem.

** Section 2.

However, while the data processing capability is improved and
  applications are hungry for more data ...

What does it mean and what applications are “hungry for more data”.  Is a reference possible here?

** Section 2.  Editorial.  s/concerned in the context/relevant in this document/

** Section 2.1
Less but higher quality data are often better
  than lots of low quality data. 

This seems like a broad generalization that doesn’t consider the application and the cost of acquisition or processing.

** Section 2.2.

The ultimate goal is to achieve the
      ideal security with no, or only minimal, human intervention.

What is “ideal” security?

** Section 2.2.
While machine learning technologies can be used for
      root cause analysis, it up to the network to sense and provide the
      relevant diagnostic data which are either actively fed into, or
      passively retrieved by, machine learning applications.

This text is asymmetric with the others bullets since don’t discuss specific techniques.  Personally, it also seem odd to include this text as there are other ways  to do root cause analysis beyond ML (to include other AI approaches).

** Section 2.3
  For a long time, network operators have relied upon SNMP [RFC3416],
  Command-Line Interface (CLI), or Syslog to monitor the network.  Some
  other OAM techniques as described in [RFC7276] are also used to
  facilitate network troubleshooting.
...
  These challenges were addressed by newer standards and techniques
  (e.g., IPFIX/Netflow, PSAMP, IOAM, and YANG-Push) and more are
  emerging.  These standards and techniques need to be recognized and
  accommodated in a new framework.

This section is an exemplar of the disconnect I noted in the definitions of OAM.  The first paragraph presents a narrow view of currently used (albeit older) network monitoring technologies (SNMP, CLI Syslog).  However, in the closing paragraph, the text names more modern technologies I would also consider OAM, and these technologies could meet some of the challenges mentioned in this section.  Furthermore, some of these “newer standards” are framed as things that need to be “recognized”.  This is puzzling because my understanding was that technologies like IPFIX/Netflow have been very widely deployed for quite some time now.  What’s the new framework needed?

** Section 2.4
Network telemetry covers the conventional network OAM and
  has a wider scope. 

Can the text be more specific in what way network telemetry is wider.  I thought OAM was rather ambiguous.

** Section 2.4
Hence, the network telemetry can directly
  trigger the automated network operation, while in contrast some
  conventional OAM tools are designed and used to help human operators
  to monitor and diagnose the networks and guide manual network
  operations.

I’m not sure if this is a fair generalization.  Even “older technologies” like SNMP currently trigger automated responses based on the values they return.

** Section 2.4.  Per “data fusion,” which part of the Figure 1 is this happening?

** Section 2.5.

Network data analytics and machine-learning technologies are applied
  for network operation automation, relying on abundant and coherent
  data from networks. 

-- What is the difference between a network data analytics system and ML technologies?  Isn’t analytics a superset of ML?

-- What is coherent data?

** Section 2.5.
In detail, such a framework would benefit application
  development for the following reasons:

It might be helpful to level set what an application is in this context.  Is this the “network operations application” of Figure 1?

** Section 2.5
All the use cases and
      applications are better to be supported uniformly and coherently
      under a single intelligent agent

-- Editorial.  There is a missing word which leads to this sentence not parsing.

-- What’s the basis for asserting that a “single intelligent agent” is the  best approach?

-- Maybe the issue is of semantics, what is an “intelligent agent” in this context?

** Section 2.5. 

Network visibility presents multiple viewpoints

and

Efficient data fusion is critical for applications to reduce the
      overall quantity of data and improve the accuracy of analysis.

Are these generalizations expected to be true across the broad use cases?

** Figure 2.  For the management plane, the data model module has MIB and syslog listed, but the data encodings as GPB, JSON and XML.  These data models and encodings don’t line up (i.e., MIBs and syslog typically don’t rely on GPB, JSON or XML). 

** Section 3.1.  Where do network security applications such as WAFs, IDS/IPS/ NGF, DLP, web-proxies, and pDNS fit into this taxonomy?

** Section 3.1.* These sections inconsistently describe properties/requirements for an architectural element and their challenges (but no solutions or requirements for) a given elements.  As a result, I had trouble understanding what an implementer should understand these components.  It would have been clearer is the different modules had common and module specific requirements.

** Section 3.1.1.  Per the requirements of “Convenient Data Subscription”, “Structured Data”, etc. why wouldn’t those be desirable requirements for all four of the modules?

** Section 3.1.3.  Providing “timely data” and “structured data”, seem like the restatements of Section 4.1.1’s “structure data” and “high speed transport”.  Is this a common requirement?

** Section 3.1.3.  Why wouldn’t it be desirable for all of the modules to support incremental deployment note here?

** Section 3.2.
  *  Data Query, Analysis, and Storage: This component works at the
      application layer. 

I need a bit of topological orientation.  What is the application layer of say a “forwarding plane” or “external data” be?  What are the other layers?

** Section 5.  Recommend explicitly saying that this document doesn’t define specific technologies to shift the responsibility of specific considerations.

OLD
  Security considerations for networks that use telemetry methods may
  include:

NEW

This document proposes a conceptual architectural for collecting, transporting, and analyzing a wide variety of data sources in support of network applications.  The protocols, data formats, and configurations chosen to implement this framework will dictate the specific Security Considerations.  These considerations may include:

** Section 5.

OLD
  *  Telemetry data stores, storage encryption and methods of access;

NEW
  *  Telemetry data stores, storage encryption, methods of access, and retention practices.

[Ballot comment]
Found terminology that should be reviewed for inclusivity; see
https://www.rfc-editor.org/part2/#inclusive_language for background and more
guidance:

* Term "his"; alternatives might be "they", "them", "their".

* Term "traditional"; alternatives might be "classic", "classical",
  "common", "conventional", "customary", "fixed", "habitual", "historic",
  "long-established", "popular", "prescribed", "regular", "rooted",
  "time-honored", "universal", "widely used", "widespread".

Thanks to Gyan S. Mishra for their General Area Review Team (Gen-ART) review
(https://mailarchive.ietf.org/arch/msg/gen-art/ItkS25VOVfDg8eLogFDMCtpeoFM).

-------------------------------------------------------------------------------
All comments below are about very minor potential issues that you may choose to
address in some way - or ignore - as you see fit. Some were flagged by
automated tools (via https://github.com/larseggert/ietf-reviewtool), so there
will likely be some false positives. There is no need to let me know what you
did with these suggestions.

Section 3.2. , paragraph 4, nit:
> ic manipulation. In some cases, micro-bursts need to be detected in a very sh
>                                ^^^^^^^^^^^^
This word is normally spelled as one.

Section 3.2. , paragraph 6, nit:
> ant to be warned of issues in advance so proactive actions can be taken to av
>                                      ^^^
Use a comma before "so" if it connects two independent clauses (unless they are
closely connected and short).

Section 4.1. , paragraph 3, nit:
> lso be implemented over TCP and SCTP but that is not recommended for forward
>                                    ^^^^
Use a comma before "but" if it connects two independent clauses (unless they
are closely connected and short).

Section 10. , paragraph 19, nit:
> P/2 [RFC7540] based open-source micro-service communication framework. It pro
>                                ^^^^^^^^^^^^^
This word is normally spelled as one.

"A.3.1. ", paragraph 2, nit:
> pected movement makes it fascinating and many people connect to sites that a
>                                    ^^^^
Use a comma before "and" if it connects two independent clauses (unless they
are closely connected and short).

"A.3.1. ", paragraph 3, nit:
> ent can be affected by such situation so their management solutions should be
>                                      ^^^
Use a comma before "so" if it connects two independent clauses (unless they are
closely connected and short).

Document references draft-ietf-ippm-multipoint-alt-mark, but that has been
published as RFC8889.

Document references draft-song-ippm-postcard-based-telemetry-10, but -11 is the
latest available revision.

Document references draft-ietf-grow-bmp-adj-rib-out, but that has been
published as RFC8671.

[Ballot comment]
Thank you for the work put into this document. All in all, this is a very good introduction document to telemetry but I am unsure whether it is a 'framework' (and this is a real concern of mine). Due to the amount of documents on the next IESG telecast agenda, I did not review the appendixes.

Please find below some non-blocking COMMENT points (but replies would be appreciated even if only for my own education), and some nits.

Special thanks to Alexander Clemm for the shepherd's write-up about the WG consensus.

Thanks also to Jean-Michel Combes for his Internet directorate review at:
https://datatracker.ietf.org/doc/review-ietf-opsawg-ntf-10-intdir-telechat-combes-2021-11-24/

I believe that Jean-Michel spotted a serious issue in the security section but I trust the Security Area Directors on this specific topic and will not raise a block DISCUSS on this point especially as Haoyu Song has taken some actions.

I hope that this helps to improve the document,

Regards,

-éric

As a non-English speaker, the document title is a little ambiguous at first sight: is it about telemetry about network data or about using the network to get some telemetry ? Unsure whether this ambiguity can be removed.

BTW, I like the sections on use cases and challenges but I am ambivalent whether they are useful in this document.

A lot of the concepts in this framework could be used for other applications beyond network layer telemetry. Was this extension considered by the authors ?

There are several informative references to IRTF & IETF drafts, which is OK of course but those drafts may never be published. Perhaps, is it premature to publish this document ?

-- Section 2 --
Suggest to rely on https://www.rfc-editor.org/materials/abbrev.expansion.txt to avoid redefining well-known terms as JSON or MIB.

-- Section 3 --
No need to reply but I am unsure whether the paragraph about SDN, AN, ... brings any value to this document.

-- Section 3.1 --
Should there be some words about whether actual packet content is part of telemetry ?

-- Section 3.2 --
Should there be a mention of "big data" again when enumerating the 4 Vs ?

-- Section 3.3 --
Is it the forwarding chip that generate, package, and send the telemetry as hinted in the first bullet ? I would guess that the chip indeed collects the data but it is up to another component to do rest.

Please add a definition/reference for "sFlow".

-- Section 3.4 --
Should there also be a mention of common naming/ID if data needs to be merged among different sources ? I.e., having common keys with the same format or at least having some equivalence.

"In-network customisation" seems partly contradictory with unified models or did I misread this part ?

For an uneducated reader (like myself), I wonder whether actual packet content is included in "The data originated from the data plane forwarding chips"

-- Section 3.5 --
In "Efficient data fusion is critical for applications to reduce the overall quantity of data", is it about "fusion" or "aggregation" ?

-- Section 4.1 --
In figure 2, what is 'template' ? This does not seem a well-defined term, suggest to remove it.

Also in figure 2, suggest replace "HTTP" by "HTTPS"

In figure 2, what is the meaning of "plain" for data encoding ?

-- Section 4.1.3 --
Could the "quality" in the first paragraph be described as it is rather vague?

Later in this section "The data should be structured and labeled", what is meant by 'labeled' ? And later in the same §, I am unsure how to understand "data types"... (as I read "data types" as float vs. integer) or is it simply "data"

Suggest to make the taxonomy part a subsection.

Is there a reason why "AM" is not expanded in "AM [RFC8321]" ?

In "Big Data sources that analyse streams of information, such as Twitter messages", I am afraid that I fail to understand how sources can analyse data.

-- Section 4.2 --
I fail to understand why formatting is part of "Data Generation and Processing" and not of "Data Encoding and Export".

-- Section 4.4 --
Please excuse my lack of knowledge in this domain, but I have hard time to understand how MIB & YANG modules can help in data generation and processing in figure 5. Should there be some additional clarifications ? Again, possibly because I am not an expert in this field.

-- Section 5 --
It is unclear for me whether the 4 stages (BTW why starting at stage 0 rather than stage 1 ?) are about telemetry (as a means) or about use cases.



== NITS ==


-- Section 2 --
Missing ':' after YANG ECA

[Ballot comment]
A.3.4 and A.3.5. Note that IOAM has a direct export mode in an adopted draft with similar properties to PBT. https://datatracker.ietf.org/doc/draft-ietf-ippm-ioam-direct-export/

The reference to draft-ippm-multipoint-alt-mark is now RFC8889.

Thanks to Michael Scharf for the TSVART review.

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Functions Operator has reviewed draft-ietf-opsawg-ntf-09, which is currently in Last Call, and has the following comments:

We understand that this document doesn't require any registry actions.

While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, we do not object.

If this assessment is not accurate, please respond as soon as possible.

Thank you,

Sabrina Tanamal
Lead IANA Services Specialist

The following Last Call announcement was sent out (ends 2021-10-27):

From: The IESG
To: IETF-Announce
CC: draft-ietf-opsawg-ntf@ietf.org, ludwig@clemm.org, opsawg-chairs@ietf.org, opsawg@ietf.org, rwilton@cisco.com
Reply-To: last-call@ietf.org
Sender:
Subject: Last Call:  (Network Telemetry Framework) to Informational RFC


The IESG has received a request from the Operations and Management Area
Working Group WG (opsawg) to consider the following document: - 'Network
Telemetry Framework'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2021-10-27. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


  Network telemetry is a technology for gaining network insight and
  facilitating efficient and automated network management.  It
  encompasses various techniques for remote data generation,
  collection, correlation, and consumption.  This document describes an
  architectural framework for network telemetry, motivated by
  challenges that are encountered as part of the operation of networks
  and by the requirements that ensue.  This document clarifies the
  terminologies and classifies the modules and components of a network
  telemetry system from different perspectives.  The framework and
  taxonomy help to set a common ground for the collection of related
  work and provide guidance for related technique and standard
  developments.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-opsawg-ntf/



No IPR declarations have been submitted directly on this I-D.

As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated 1 November 2019.

Write-up filled out by Alexander Clemm.  Responses delimited with AC in pointy brackets.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational 

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

Relevant content can frequently be found in the abstract and/or introduction of the document. If not, this may be an indication that there are deficiencies in the abstract or introduction.

The document articulates a framework for the use of network telemetry for purposes of operations and management.  The framework accommodates various techniques for the generation of telemetry data by the network and for the collection and consumption of that data by applications.  It distinguishes between separate modules includig for control plane, forwarding plane, and management plane telemetry and lays out a common component structure for those modules. Interfaces are described and existing IETF techniques are mapped to the framework. The framework and the taxonomy that it contains is intended to provide a common ground and serve as guidance for the development and application of corresponding techniques and standards. 


Working Group Summary:

Was there anything in WG process that is worth noting? For example, was there controversy about particular points or were there decisions where the consensus was particularly rough?

There was no major controversy. The author accommodated the various points that were raised thoughout the process, as far as I can tell to everyone's satisfaction. There are no open issues.

Document Quality:

Are there existing implementations of the protocol? Have a significant number of vendors indicated their plan to implement the specification? Are there any reviewers that merit special mention as having done a thorough review, e.g., one that resulted in important changes or a conclusion that the document had no substantive issues? If there was a MIB Doctor, YANG Doctor, Media Type or other expert review, what was its course (briefly)? In the case of a Media Type review, on what date was the request posted?

The document does not describe a protocol.  What is described is a framework.  Similar frameworks will in fact already be deployed. However, the contribution of the document is that it articulates the "best practices" for such frameworks, that it provides a systemic mapping of how various existing techniques relate to it, that it provides a common taxonomy, that it helps to organize related efforts. There have also been contributions to the document by operators.  YANG Doctors etc were not applicable here. 

Personnel:

Who is the Document Shepherd? Who is the Responsible Area Director?

Document Shepherd: Alexander Clemm. Area Director: Robert Wilton.

Please note that the document shepherd is also a contributor to the document.  In case this is of concern, I am okay with having my name removed.

(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

Since having been appointed as document shepherd when the document was in -05, already after WG review.  I have put the document through two additional thorough rounds of reviews.  The current revision, -07, addresses all comments to my satisfaction. In my opinion the document is ready for publication. 

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No concerns.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

I do not think further review or additional perspectives are necessary.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

My concerns/comments have been addressed to my satisfaction.  I believe the document can can be advanced.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

Yes.  All authors and contributors have indicated that they are not aware of any IPR that applies to this draft.

(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

The consensus is solid and the WG agrees.  In addition, the document has also a broad set of contributors (from a broad set of organizations).

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

Not that I am aware of.

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

There are no issues.  However, there are a few references that need to be updated for publication:

  == Outdated reference: draft-ietf-grow-bmp-adj-rib-out has been published
    as RFC 8671

  == Outdated reference: A later version (-10) exists of
    draft-ietf-grow-bmp-local-rib-09

  == Outdated reference: A later version (-12) exists of
    draft-ietf-ippm-ioam-data-11

  == Outdated reference: A later version (-03) exists of
    draft-irtf-nmrg-ibn-concepts-definitions-02

  == Outdated reference: A later version (-09) exists of
    draft-song-ippm-postcard-based-telemetry-08
   


(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

None of these criteria apply. 

(13) Have all references within this document been identified as either normative or informative?

Yes.  All references are informative, also reflecting the "Informational" status of the draft.

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

No

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

No

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 8126).

This document contains no request to IANA and no registries are involved, hence this is not applicable

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

The document contains no request to IANA, hence this is not applicable

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, YANG modules, etc.

Not applicable

(20) If the document contains a YANG module, has the module been checked with any of the recommended validation tools (https://trac.ietf.org/trac/ops/wiki/yang-review-tools) for syntax and formatting validation? If there are any resulting errors or warnings, what is the justification for not fixing them at this time? Does the YANG module comply with the Network Management Datastore Architecture (NMDA) as specified in RFC8342?

Not applicable

As required by RFC 4858, this is the current template for the Document
Shepherd Write-Up. Changes are expected over time.

This version is dated 1 November 2019.

Write-up filled out by Alexander Clemm.  Responses delimited with AC in pointy brackets.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet Standard, Informational, Experimental, or Historic)? Why is this the proper type of RFC? Is this type of RFC indicated in the title page header?

Informational 

(2) The IESG approval announcement includes a Document Announcement Write-Up. Please provide such a Document Announcement Write-Up. Recent examples can be found in the "Action" announcements for approved documents. The approval announcement contains the following sections:

Technical Summary:

Relevant content can frequently be found in the abstract and/or introduction of the document. If not, this may be an indication that there are deficiencies in the abstract or introduction.

The document articulates a framework for the use of network telemetry for purposes of operations and management.  The framework accommodates various techniques for the generation of telemetry data by the network and for the collection and consumption of that data by applications.  It distinguishes between separate modules includig for control plane, forwarding plane, and management plane telemetry and lays out a common component structure for those modules. Interfaces are described and existing IETF techniques are mapped to the framework. The framework and the taxonomy that it contains is intended to provide a common ground and serve as guidance for the development and application of corresponding techniques and standards. 


Working Group Summary:

Was there anything in WG process that is worth noting? For example, was there controversy about particular points or were there decisions where the consensus was particularly rough?

There was no major controversy. The author accommodated the various points that were raised thoughout the process, as far as I can tell to everyone's satisfaction. There are no open issues.

Document Quality:

Are there existing implementations of the protocol? Have a significant number of vendors indicated their plan to implement the specification? Are there any reviewers that merit special mention as having done a thorough review, e.g., one that resulted in important changes or a conclusion that the document had no substantive issues? If there was a MIB Doctor, YANG Doctor, Media Type or other expert review, what was its course (briefly)? In the case of a Media Type review, on what date was the request posted?

The document does not describe a protocol.  What is described is a framework.  Similar frameworks will in fact already be deployed. However, the contribution of the document is that it articulates the "best practices" for such frameworks, that it provides a systemic mapping of how various existing techniques relate to it, that it provides a common taxonomy, that it helps to organize related efforts. There have also been contributions to the document by operators.  YANG Doctors etc were not applicable here. 

Personnel:

Who is the Document Shepherd? Who is the Responsible Area Director?

Document Shepherd: Alexander Clemm. Area Director: Robert Wilton.

Please note that the document shepherd is also a contributor to the document.  In case this is of concern, I am okay with having my name removed.

(3) Briefly describe the review of this document that was performed by the Document Shepherd. If this version of the document is not ready for publication, please explain why the document is being forwarded to the IESG.

Since having been appointed as document shepherd when the document was in -05, already after WG review.  I have put the document through two additional thorough rounds of reviews.  The current revision, -07, addresses all comments to my satisfaction. In my opinion the document is ready for publication. 

(4) Does the document Shepherd have any concerns about the depth or breadth of the reviews that have been performed?

No concerns.

(5) Do portions of the document need review from a particular or from broader perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML, or internationalization? If so, describe the review that took place.

I do not think further review or additional perspectives are necessary.

(6) Describe any specific concerns or issues that the Document Shepherd has with this document that the Responsible Area Director and/or the IESG should be aware of? For example, perhaps he or she is uncomfortable with certain parts of the document, or has concerns whether there really is a need for it. In any event, if the WG has discussed those issues and has indicated that it still wishes to advance the document, detail those concerns here.

My concerns/comments have been addressed to my satisfaction.  I believe the document can can be advanced.

(7) Has each author confirmed that any and all appropriate IPR disclosures required for full conformance with the provisions of BCP 78 and BCP 79 have already been filed. If not, explain why?

Yes.  All authors and contributors have indicated that they are not aware of any IPR that applies to this draft.

(8) Has an IPR disclosure been filed that references this document? If so, summarize any WG discussion and conclusion regarding the IPR disclosures.

No

(9) How solid is the WG consensus behind this document? Does it represent the strong concurrence of a few individuals, with others being silent, or does the WG as a whole understand and agree with it?

The consensus is solid and the WG agrees.  In addition, the document has also a broad set of contributors (from a broad set of organizations).

(10) Has anyone threatened an appeal or otherwise indicated extreme discontent? If so, please summarise the areas of conflict in separate email messages to the Responsible Area Director. (It should be in a separate email because this questionnaire is publicly available.)

Not that I am aware of.

(11) Identify any ID nits the Document Shepherd has found in this document. (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts Checklist). Boilerplate checks are not enough; this check needs to be thorough.

There are no issues.  However, there are a few references that need to be updated for publication:

  == Outdated reference: draft-ietf-grow-bmp-adj-rib-out has been published
    as RFC 8671

  == Outdated reference: A later version (-10) exists of
    draft-ietf-grow-bmp-local-rib-09

  == Outdated reference: A later version (-12) exists of
    draft-ietf-ippm-ioam-data-11

  == Outdated reference: A later version (-03) exists of
    draft-irtf-nmrg-ibn-concepts-definitions-02

  == Outdated reference: A later version (-09) exists of
    draft-song-ippm-postcard-based-telemetry-08
   


(12) Describe how the document meets any required formal review criteria, such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

None of these criteria apply. 

(13) Have all references within this document been identified as either normative or informative?

Yes.  All references are informative, also reflecting the "Informational" status of the draft.

(14) Are there normative references to documents that are not ready for advancement or are otherwise in an unclear state? If such normative references exist, what is the plan for their completion?

No

(15) Are there downward normative references references (see RFC 3967)? If so, list these downward references to support the Area Director in the Last Call procedure.

No

(16) Will publication of this document change the status of any existing RFCs? Are those RFCs listed on the title page header, listed in the abstract, and discussed in the introduction? If the RFCs are not listed in the Abstract and Introduction, explain why, and point to the part of the document where the relationship of this document to the other RFCs is discussed. If this information is not in the document, explain why the WG considers it unnecessary.

No

(17) Describe the Document Shepherd's review of the IANA considerations section, especially with regard to its consistency with the body of the document. Confirm that all protocol extensions that the document makes are associated with the appropriate reservations in IANA registries. Confirm that any referenced IANA registries have been clearly identified. Confirm that newly created IANA registries include a detailed specification of the initial contents for the registry, that allocations procedures for future registrations are defined, and a reasonable name for the new registry has been suggested (see RFC 8126).

This document contains no request to IANA and no registries are involved, hence this is not applicable

(18) List any new IANA registries that require Expert Review for future allocations. Provide any public guidance that the IESG would find useful in selecting the IANA Experts for these new registries.

The document contains no request to IANA, hence this is not applicable

(19) Describe reviews and automated checks performed by the Document Shepherd to validate sections of the document written in a formal language, such as XML code, BNF rules, MIB definitions, YANG modules, etc.

Not applicable

(20) If the document contains a YANG module, has the module been checked with any of the recommended validation tools (https://trac.ietf.org/trac/ops/wiki/yang-review-tools) for syntax and formatting validation? If there are any resulting errors or warnings, what is the justification for not fixing them at this time? Does the YANG module comply with the Network Management Datastore Architecture (NMDA) as specified in RFC8342?

Not applicable