A YANG Data Model for Network and VPN Service Performance Monitoring
draft-ietf-opsawg-yang-vpn-service-pm-15

[Ballot comment]
# GEN AD review of draft-ietf-opsawg-yang-vpn-service-pm-13

CC @larseggert

Thanks to Elwyn Davies for the General Area Review Team (Gen-ART) review
(https://mailarchive.ietf.org/arch/msg/gen-art/GZGiqBYW9PpHdVBNPE6f_f6JucU).

## Comments

## Nits

All comments below are about very minor potential issues that you may choose to
address in some way - or ignore - as you see fit. Some were flagged by
automated tools (via https://github.com/larseggert/ietf-reviewtool), so there
will likely be some false positives. There is no need to let me know what you
did with these suggestions.

### Typos

#### Section 4.3, paragraph 24
```
-    PM statistics per class ("one-way-pm-statistics-per-class"):  Lists p
-                                                                      --
```

#### Section 4.3, paragraph 24
```
-      erformance measurement statistics for the topology link or the
+      performance measurement statistics for the topology link or the
+      +
```

#### Section 4.3, paragraph 28
```
-    Last updatd time ("last-updated"):  Indicates the timestamp when the
-              ^
+    Last update time ("last-updated"):  Indicates the timestamp when the
+              ^
```

### Grammar/style

#### Section 5, paragraph 4
```
places, e.g. 10.00, 99.90 ,99.99 etc.. For example, for a given one-way del
                                    ^^
```
Two consecutive dots.

## Notes

This review is in the ["IETF Comments" Markdown format][ICMF], You can use the
[`ietf-comments` tool][ICT] to automatically convert this review into
individual GitHub issues. Review generated by the [`ietf-reviewtool`][IRT].

[ICMF]: https://github.com/mnot/ietf-comments/blob/main/format.md
[ICT]: https://github.com/mnot/ietf-comments
[IRT]: https://github.com/larseggert/ietf-reviewtool

[Ballot comment]
Thanks to Bob Briscoe for the tsvart review.

- the Percentile definition sets fraction-digits to 3 but the description still says it’s two decimal places.

- Why not IOAM as a data source?

[Ballot comment]
Thank you to Daniel Migault for the SECDIR.

** Section 4.4. s/Lists p  erformance/Lists performance/

** Section 4.4. s/Last updatd/Last updated/

** Section 6.  Recommend being more descriptive in the impact.
OLD
  Write operations (e.g., edit-config)
  to these data nodes without proper protection can have a negative
  effect on network operations. 

NEW
Write operations (e.g., edit-config) to these data nodes without proper protection can have a negative effect on network operations.  These write operates can lead to inaccurate or incomplete network measurements which case impact the visibility and decisions this data would be used to inform.

** Section 6.  Thanks for the detailed table describing the impact of write operations.  Since there is space to add a bit more descriptive language on the page, consider:

“impact reporting” --> “impacts reporting cadence”
“impact monitoring” --> “impacts monitoring fidelity”

I don’t understand “render invalid”?  Does this combine the idea of “turning it all off” and “making the data unusable”?  I’d recommend explaining the phrase or use something different.

[Ballot comment]

# Éric Vyncke, INT AD, comments for draft-ietf-opsawg-yang-vpn-service-pm-13
CC @evyncke

Thank you for the work put into this document.

Please find below some non-blocking COMMENT points (but replies would be appreciated even if only for my own education), and some nits.

Special thanks to Adrian Farrel for the shepherd's detailed write-up including the WG consensus *and* the justification of the intended status.

I hope that this review helps to improve the document,

Regards,

-éric


## COMMENTS

### Relationship with other OPSAWG documents

Just curious: it this work somehow related to draft-ietf-opsawg-service-assurance-architecture ? I.e., should this document add some reference to the service assurance I-Ds ?

### Section 2.1

As 'PE' is defined, I was also expecting to see 'CE' (notably used in section 4.1) and 'P' (notably used in section 4.3).

### Section 3

Suggest to explain why in `Models are key for automating network management operations.`.

### Section 4

Unsure about the value of figure 2, but feel free to keep it (this is a matter of taste).

### Section 4.3

Just curious about why `mac-num-limit` is used rather than `maximum-mac-entries` for consistency with the previous entries?

### Section 4.4

Would it be useful to split `inbound-non-unicast` into broadcast and multicast ?

## NITS

### Space after ':'

It would nicer if there was a space character after several ':'.

### Section 4.4

s/updatd/updated/


## Notes

This review is in the ["IETF Comments" Markdown format][ICMF], You can use the
[`ietf-comments` tool][ICT] to automatically convert this review into
individual GitHub issues.

[ICMF]: https://github.com/mnot/ietf-comments/blob/main/format.md
[ICT]: https://github.com/mnot/ietf-comments

[Ballot comment]
# Internet AD comments for {draft-ietf-opsawg-yang-vpn-service-pm-13}
CC @ekline

## Nits

### S4.4

* The description of "one-way-pm-statistics-per-class" at the bottom of
  page 14 has a strange word break: "performance" is split into "p" and
  "erformance" (page 15).

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Functions Operator has completed its review of draft-ietf-opsawg-yang-vpn-service-pm-12. If any part of this review is inaccurate, please let us know.

The IANA Services Operator understands that, upon approval of this document, there are two actions which we must complete.

First, in the ns registry on the IETF XML Registry page located at:

https://www.iana.org/assignments/xml-registry/

a single, new namespace will be registered as follows:

ID: yang:ietf-network-vpn-pm
URI: urn:ietf:params:xml:ns:yang:ietf-network-vpn-pm
Filename: [ TBD-at-Registration ]
Reference: [ RFC-to-be ]

As this document requests registrations in an Expert Review or Specification Required (see RFC 8126) registry, we will initiate the required Expert Review via a separate request.  Expert review will need to be completed before your document can be approved for publication as an RFC.

Second, in the YANG Module Names registry on the YANG Parameters registry page located at:

https://www.iana.org/assignments/yang-parameters/

a single, new YANG module will be registered as follows:

Name: ietf-network-vpn-pm
File: [ TBD-at-Registration ]
Maintained by IANA? N
Namespace: urn:ietf:params:xml:ns:yang:ietf-network-vpn-pm
Prefix: nvp
Module:
Reference: [ RFC-to-be ]

While the YANG module name will be registered after the IESG approves the document, the YANG module file will be posted after the RFC Editor notifies us that the document has been published.

The IANA Services Operator understands that these are the only actions required to be completed upon approval of this document.

Note:  The actions requested in this document will not be completed until the document has been approved for publication as an RFC. This message is meant only to confirm the list of actions that will be performed.

For definitions of IANA review states, please see:

https://datatracker.ietf.org/help/state/draft/iana-review

Thank you,

David Dong
IANA Services Specialist

The following Last Call announcement was sent out (ends 2022-10-04):

From: The IESG
To: IETF-Announce
CC: adrian@olddog.co.uk, draft-ietf-opsawg-yang-vpn-service-pm@ietf.org, opsawg-chairs@ietf.org, opsawg@ietf.org, rwilton@cisco.com
Reply-To: last-call@ietf.org
Sender:
Subject: Last Call:  (A YANG Model for Network and VPN Service Performance Monitoring) to Proposed Standard


The IESG has received a request from the Operations and Management Area
Working Group WG (opsawg) to consider the following document: - 'A YANG Model
for Network and VPN Service Performance Monitoring'
  as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2022-10-04. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


  The data model for network topologies defined in RFC 8345 introduces
  vertical layering relationships between networks that can be
  augmented to cover network and service topologies.  This document
  defines a YANG module for performance monitoring (PM) of both
  underlay networks and overlay VPN services that can be used to
  monitor and manage network performance on the topology of both
  layers.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-opsawg-yang-vpn-service-pm/



No IPR declarations have been submitted directly on this I-D.

Document shepherd write-up for draft-ietf-opsawg-yang-vpn-service-pm

> ## Document History
>
> 1. Does the working group (WG) consensus represent the strong concurrence of a
>    few individuals, with others being silent, or did it reach broad agreement?

The document has 5 co-authors, 4 contributors, and 7 reviewers
acknowledged in the document.

WG last call (which can be found in the archive at
https://mailarchive.ietf.org/arch/msg/opsawg/0UjzdhIAmLr_x0V6cqhgBUmGPic/)
attracted three detailed reviews and some additional comments
(https://mailarchive.ietf.org/arch/msg/opsawg/giJgyVBYg-7cDYNKNQR1HQUp1VM/)

The work was also presented at several OPSAWG meetings.

There was no objection to publication, so this represents relatively
broad consensus for publication.

> 2. Was there controversy about particular points, or were there decisions where
>    the consensus was particularly rough?

None observed.

> 3. Has anyone threatened an appeal or otherwise indicated extreme discontent? If
>    so, please summarize the areas of conflict in separate email messages to the
>    responsible Area Director. (It should be in a separate email because this
>    questionnaire is publicly available.)

No. There has been debate about various points, but these seem to have
been resolved through discussion and modification of the document.

> 4. For protocol documents, are there existing implementations of the contents of
>    the document? Have a significant number of potential implementers indicated
>    plans to implement? Are any existing implementations reported somewhere,
>    either in the document itself (as [RFC 7942][3] recommends) or elsewhere
>    (where)?

No implementations have been publicly reported. However, the document
shepherd is aware of plans to include this function in two separate
implementations.

> ### Additional Reviews
>
> 5. Does this document need review from other IETF working groups or external
>    organizations? Have those reviews occurred?

None needed.

It might have been useful to share this with IPPM and BESS. The shepherd
notified those two working groups to watch out for IETF last call, and
this resulted in an immediate additional review that has been addressed
in a new revision of the document.

A review by the Routing Directorate has been requested twice, but no
review has been received.

The ADs might reinvent the pm-dir to get some extra views on this
document.

> 6. Describe how the document meets any required formal expert review criteria,
>    such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

The document received YANG Doctor review from Radek Krejci at the time
of WG last call. It found nits that have been fixed.
https://datatracker.ietf.org/doc/review-ietf-opsawg-yang-vpn-service-pm-07-yangdoctors-lc-krejci-2022-04-27/

An additional review of -05 (updated after WG last call) was performed
by Ladislav Lhotka. It found nits that have been fixed.
https://datatracker.ietf.org/doc/review-ietf-opsawg-yang-vpn-service-pm-05-yangdoctors-early-lhotka-2022-04-08/

> 7. If the document contains a YANG module, has the final version of the module
>    been checked with any of the [recommended validation tools][4] for syntax and
>    formatting validation?

The Datatracker reports YANG validation passes with no errors and no
warnings.

>    Does the YANG module comply with the Network Management Datastore
>    Architecture (NMDA) as specified in [RFC 8342][5]?

The document makes no claims about conformance to NMDA, however to the
best of my knowledge, the model complies with the NMDA.

> 8. Describe reviews and automated checks performed to validate sections of the
>    final version of the document written in a formal language, such as XML code,
>    BNF rules, MIB definitions, CBOR's CDDL, etc.

As above:
- YANG Doctor reviews
- Datatracker automated checks

> ### Document Shepherd Checks

The document shepherd reviewed this document at adoption, during its
progress through the working group (-01), during WG last call (-03), and
in their role as document shepherd (-05). In each case, the authors made
updates and addressed all of the comments.

> 9. Based on the shepherd's review of the document, is it their opinion that this
>    document is needed, clearly written, complete, correctly designed, and ready
>    to be handed off to the responsible Area Director?

Yes

> 10. Several IETF Areas have assembled [lists of common issues that their
>    reviewers encounter][6]. Do any such issues remain that would merit specific
>    attention from subsequent reviews?

It would be easier to answer this question if a pointer was provided
to those lists.

No reviews or reviewers have pointed to any open issues that need
attention. The shepherd has some experience with getting documents
up to the right level, and considers this one good enough.

> 11. What type of RFC publication is being requested on the IETF stream (Best
>    Current Practice, Proposed Standard, Internet Standard, Informational,
>    Experimental, or Historic)? Why is this the proper type of RFC? Do all
>    Datatracker state attributes correctly reflect this intent?

This document is the product of the OPSAWG and is presented for
publication on the Standards Track as a Draft Standard. This is
appropriate for a YANG model that will be implemented and must
interoperate.

The status is properly indicated on the title page and in the
Datatracker. 

> 12. Has the interested community confirmed that any and all appropriate IPR
>    disclosures required by [BCP 78][7] and [BCP 79][8] have been filed? If not,
>    explain why. If yes, summarize any discussion and conclusion regarding the
>    intellectual property rights (IPR) disclosures, including links to relevant
>    emails.

The WG chairs requested an IPR response from all authors in a private
email at the time of WG last call. Responses from three of the authors
can be seen on the OPSAWG mailing list at
https://mailarchive.ietf.org/arch/msg/opsawg/a4E8SNuIzZtMrWtsJwCvCEiCO50/
Other responses are at
https://mailarchive.ietf.org/arch/msg/opsawg/vKW7Hpr2sSZNB3kLYWDR-aPKKoo/

No IPR has been disclosed, and all respondents declared no IP needed to
be disclosed.

> 13. Has each Author or Contributor confirmed their willingness to be listed as
>    such?

Well, no. But they have been listed there for a long time, and their
silence may be assumed to be consent. Note also that the IPR poll has
made all authors and contributors aware of their status on the document.

>    If the number of Authors/Editors on the front page is greater than 5,
>    please provide a justification.

It isn't.

> 14. Identify any remaining I-D nits in this document. (See [the idnits tool][9]
>    and the checkbox items found in Guidelines to Authors of Internet-Drafts).
>    Simply running the idnits tool is not enough; please review the entire
>    guidelines document.

id-nits is clean.
Manual check of guidance for YANG-based documents reveals no issues.

> 15. Should any informative references be normative or vice-versa?

The distribution seems to be right. Some fixes were made as a result of
various reviews.

> 16. List any normative references that are not freely available to anyone.

None

> 17. Are there any normative downward references (see [RFC 3967][10],
>    [BCP 97][11])? If so, list them.

No normative downrefs

> 18. Are there normative references to documents that are not ready for
>    advancement or are otherwise in an unclear state?

None

> 19. Will publication of this document change the status of any existing RFCs?

No

> 20. Describe the document shepherd's review of the IANA considerations section,
>    especially with regard to its consistency with the body of the document.

The document shepherd made themself a cup of tea with a slice of
lemon and sat down in their office. They then read the document
including the IANA considerations section, and thought about it
carefully, comparing it with recently published RFCs and the needs
of the rest of the document.

>    Confirm that all aspects of the document requiring IANA assignments are
>    associated with the appropriate reservations in IANA registries.

Confirmed

>    Confirm that any referenced IANA registries have been clearly identified.

Confirmed

>    Confirm that each newly created IANA registry specifies its initial contents,
>    allocations procedures, and a reasonable name (see [RFC 8126][12]).

Confirmed

> 21. List any new IANA registries that require Designated Expert Review for
>    future allocations. Are the instructions to the Designated Expert clear?
>    Please include suggestions of designated experts, if appropriate.

No new registries

Document shepherd write-up for draft-ietf-opsawg-yang-vpn-service-pm

> ## Document History
>
> 1. Does the working group (WG) consensus represent the strong concurrence of a
>    few individuals, with others being silent, or did it reach broad agreement?

The document has 5 co-authors, 4 contributors, and 7 reviewers
acknowledged in the document.

WG last call (which can be found in the archive at
https://mailarchive.ietf.org/arch/msg/opsawg/0UjzdhIAmLr_x0V6cqhgBUmGPic/)
attracted three detailed reviews and some additional comments
(https://mailarchive.ietf.org/arch/msg/opsawg/giJgyVBYg-7cDYNKNQR1HQUp1VM/)

The work was also presented at several OPSAWG meetings.

There was no objection to publication, so this represents relatively
broad consensus for publication.

> 2. Was there controversy about particular points, or were there decisions where
>    the consensus was particularly rough?

None observed.

> 3. Has anyone threatened an appeal or otherwise indicated extreme discontent? If
>    so, please summarize the areas of conflict in separate email messages to the
>    responsible Area Director. (It should be in a separate email because this
>    questionnaire is publicly available.)

No. There has been debate about various points, but these seem to have
been resolved through discussion and modification of the document.

> 4. For protocol documents, are there existing implementations of the contents of
>    the document? Have a significant number of potential implementers indicated
>    plans to implement? Are any existing implementations reported somewhere,
>    either in the document itself (as [RFC 7942][3] recommends) or elsewhere
>    (where)?

No implementations have been publicly reported. However, the document
shepherd is aware of plans to include this function in two separate
implementations.

> ### Additional Reviews
>
> 5. Does this document need review from other IETF working groups or external
>    organizations? Have those reviews occurred?

None needed.

It might have been useful to share this with IPPM and BESS. The shepherd
notified those two working groups to watch out for IETF last call, and
this resulted in an immediate additional review that has been addressed
in a new revision of the document.

A review by the Routing Directorate has been requested twice, but no
review has been received.

The ADs might reinvent the pm-dir to get some extra views on this
document.

> 6. Describe how the document meets any required formal expert review criteria,
>    such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

The document received YANG Doctor review from Radek Krejci at the time
of WG last call. It found nits that have been fixed.
https://datatracker.ietf.org/doc/review-ietf-opsawg-yang-vpn-service-pm-07-yangdoctors-lc-krejci-2022-04-27/

An additional review of -05 (updated after WG last call) was performed
by Ladislav Lhotka. It found nits that have been fixed.
https://datatracker.ietf.org/doc/review-ietf-opsawg-yang-vpn-service-pm-05-yangdoctors-early-lhotka-2022-04-08/

> 7. If the document contains a YANG module, has the final version of the module
>    been checked with any of the [recommended validation tools][4] for syntax and
>    formatting validation?

The Datatracker reports YANG validation passes with no errors and no
warnings.

>    Does the YANG module comply with the Network Management Datastore
>    Architecture (NMDA) as specified in [RFC 8342][5]?

The document makes no claims about conformance to NMDA, however to the
best of my knowledge, the model complies with the NMDA.

> 8. Describe reviews and automated checks performed to validate sections of the
>    final version of the document written in a formal language, such as XML code,
>    BNF rules, MIB definitions, CBOR's CDDL, etc.

As above:
- YANG Doctor reviews
- Datatracker automated checks

> ### Document Shepherd Checks

The document shepherd reviewed this document at adoption, during its
progress through the working group (-01), during WG last call (-03), and
in their role as document shepherd (-05). In each case, the authors made
updates and addressed all of the comments.

> 9. Based on the shepherd's review of the document, is it their opinion that this
>    document is needed, clearly written, complete, correctly designed, and ready
>    to be handed off to the responsible Area Director?

Yes

> 10. Several IETF Areas have assembled [lists of common issues that their
>    reviewers encounter][6]. Do any such issues remain that would merit specific
>    attention from subsequent reviews?

It would be easier to answer this question if a pointer was provided
to those lists.

No reviews or reviewers have pointed to any open issues that need
attention. The shepherd has some experience with getting documents
up to the right level, and considers this one good enough.

> 11. What type of RFC publication is being requested on the IETF stream (Best
>    Current Practice, Proposed Standard, Internet Standard, Informational,
>    Experimental, or Historic)? Why is this the proper type of RFC? Do all
>    Datatracker state attributes correctly reflect this intent?

This document is the product of the OPSAWG and is presented for
publication on the Standards Track as a Draft Standard. This is
appropriate for a YANG model that will be implemented and must
interoperate.

The status is properly indicated on the title page and in the
Datatracker. 

> 12. Has the interested community confirmed that any and all appropriate IPR
>    disclosures required by [BCP 78][7] and [BCP 79][8] have been filed? If not,
>    explain why. If yes, summarize any discussion and conclusion regarding the
>    intellectual property rights (IPR) disclosures, including links to relevant
>    emails.

The WG chairs requested an IPR response from all authors in a private
email at the time of WG last call. Responses from three of the authors
can be seen on the OPSAWG mailing list at
https://mailarchive.ietf.org/arch/msg/opsawg/a4E8SNuIzZtMrWtsJwCvCEiCO50/
Other responses are at
https://mailarchive.ietf.org/arch/msg/opsawg/vKW7Hpr2sSZNB3kLYWDR-aPKKoo/

No IPR has been disclosed, and all respondents declared no IP needed to
be disclosed.

> 13. Has each Author or Contributor confirmed their willingness to be listed as
>    such?

Well, no. But they have been listed there for a long time, and their
silence may be assumed to be consent. Note also that the IPR poll has
made all authors and contributors aware of their status on the document.

>    If the number of Authors/Editors on the front page is greater than 5,
>    please provide a justification.

It isn't.

> 14. Identify any remaining I-D nits in this document. (See [the idnits tool][9]
>    and the checkbox items found in Guidelines to Authors of Internet-Drafts).
>    Simply running the idnits tool is not enough; please review the entire
>    guidelines document.

id-nits is clean.
Manual check of guidance for YANG-based documents reveals no issues.

> 15. Should any informative references be normative or vice-versa?

The distribution seems to be right. Some fixes were made as a result of
various reviews.

> 16. List any normative references that are not freely available to anyone.

None

> 17. Are there any normative downward references (see [RFC 3967][10],
>    [BCP 97][11])? If so, list them.

No normative downrefs

> 18. Are there normative references to documents that are not ready for
>    advancement or are otherwise in an unclear state?

None

> 19. Will publication of this document change the status of any existing RFCs?

No

> 20. Describe the document shepherd's review of the IANA considerations section,
>    especially with regard to its consistency with the body of the document.

The document shepherd made themself a cup of tea with a slice of
lemon and sat down in their office. They then read the document
including the IANA considerations section, and thought about it
carefully, comparing it with recently published RFCs and the needs
of the rest of the document.

>    Confirm that all aspects of the document requiring IANA assignments are
>    associated with the appropriate reservations in IANA registries.

Confirmed

>    Confirm that any referenced IANA registries have been clearly identified.

Confirmed

>    Confirm that each newly created IANA registry specifies its initial contents,
>    allocations procedures, and a reasonable name (see [RFC 8126][12]).

Confirmed

> 21. List any new IANA registries that require Designated Expert Review for
>    future allocations. Are the instructions to the Designated Expert clear?
>    Please include suggestions of designated experts, if appropriate.

No new registries

Awaiting IPR confirmations from:
'ron.even.tlv@gmail.com' ;
'liuc131@chinaunicom.cn' ;
'xuhl.bri@chinatelecom.cn' ;  <<<<<<<<<<< Bouncing <<<<<<<<<<<<<<<<<<<<<<<
'oscar.gonzalezdedios@telefonica.com' ;
'bin_wen@comcast.com'

Awaiting update to draft for incorrect Security Considerations text.

Chairs to set the Intended Status in the Datatracker

Awaiting IPR confirmations from:
'wangzitao@huawei.com' ;
'ron.even.tlv@gmail.com' ;
'liuc131@chinaunicom.cn' ;
'xuhl.bri@chinatelecom.cn' ;  <<<<<<<<<<< Bouncing <<<<<<<<<<<<<<<<<<<<<<<
'oscar.gonzalezdedios@telefonica.com' ;
'bin_wen@comcast.com'

Awaiting update to draft for incorrect Security Considerations text.

Chairs to set the Intended Status in the Datatracker

Awaiting IPR confirmations from:
'wangzitao@huawei.com' ;
'ron.even.tlv@gmail.com' ;
'liuc131@chinaunicom.cn' ;
'xuhl.bri@chinatelecom.cn' ;  <<<<<<<<<<< Bouncing <<<<<<<<<<<<<<<<<<<<<<<
'oscar.gonzalezdedios@telefonica.com' ;
'bin_wen@comcast.com'

Awaiting update to draft for incorrect Security Considerations text.

Request for posting confirmation emailed to previous authors: Bin Wen , Bo Wu , Chang Liu , Honglei Xu , Mohamed Boucadair , Oscar de Dios , Qin WU , opsawg-chairs@ietf.org

Request for posting confirmation emailed to previous authors: Bin Wen , Bo Wu , Chang Liu , Honglei Xu , Mohamed Boucadair , Oscar de Dios , Qin WU