Network Virtualization Overlays (NVO3) Encapsulation Considerations
draft-ietf-nvo3-encap-12

I believe the latest addition addresses at least one of the abstains and various other issues - and hence I think this is ready to go.

[Ballot comment]
# Internet AD comments for draft-ietf-nvo3-encap-11
CC @ekline

* comment syntax:
  - https://github.com/mnot/ietf-comments/blob/main/format.md

* "Handling Ballot Positions":
  - https://ietf.org/about/groups/iesg/statements/handling-ballot-positions/

## Comments

* Adding an explanation for my Abstain: basically, I couldn't convince
  myself that this was a sufficiently wide and deep review of the topic,
  but I also couldn't come up with concrete suggestions to improve things
  (of the kind which I might have put into a DISCUSS).

  To be clear: there seems to be much good work here, which is also why I
  don't want to stand in the way of publication.

### S6.2.2

* Integrity can also be provided by an outer AH layer (RFC 4302), between the
  output IP and UDP headers, but maybe nobody wants to implement this.

### S6.4

* "minimum total available header length is 64 bytes"

  Is this the header after the outer UDP header an before any payload, or
  does "total available header" include outer IP and UDP headers?

### S6.6

* Is there anything interesting to be learned from the hardware offload
  support of something like GRE?  It has been extended a few times, but
  perhaps not as often as might be relevant here and obviously quite some
  time ago that it hardly matters.  Just curious.

[Ballot comment]
I also believe that a Design Team's output consists of only input to the WG. So I would be okay if the document mentions that work was done via a design team, but that the text should be updated to replace all current tense occurrences of the DT to either "WG" or some other kind of generalized phrasing. The shepherds review states there was strong WG consensus on this document, so why not make this WG document come from the WG properly, and perhaps have a single section mentioning/crediting the work(flow) of the Design Team?

I believe the content is worth publishing, but I feel this is a slippery slope to have design teams write RFCs

[Ballot comment]

# Éric Vyncke, INT AD, comments for draft-ietf-nvo3-encap-11

Thank you for the work put into this document.

I am balloting an ABSTAIN because this document sits between two chairs (French saying). It is:

- either the sheer output of the design team, then it has no WG/IETF consensus, meaning it must be an independent submission
- or the output of the NVO3 WG, meaning that it cannot really speak only about the design team (there are 16 occurrences of "DT" in the draft and I did not count "design team").

See also my COMMENT about section 7 about the differences with draft-dt-nvo3-encap-01 as recommendations have been added to the DT's ones.

It would have been more useful to focus only on section 6, which is indeed an interesting read. Thank you that section.

Please find below some non-blocking COMMENT points (but replies would be appreciated even if only for my own education).

Special thanks to Matthew Bocci for the shepherd's detailed write-up including the WG consensus *and* the justification of the intended status even if it lacks the justification for the *publication stream*.

Please note that Wassim Haddad is the Internet directorate reviewer (at my request) and you may want to consider this int-dir review as well when it will be available (no need to wait for it though):
https://datatracker.ietf.org/doc/draft-ietf-nvo3-encap/reviewrequest/18840/

I hope that this review helps to improve the document and/or change its stream,

Regards,

-éric

# COMMENTS (non-blocking)

## Section 5.2

`similar to those noted for Geneve above` would benefit of further explanation as the 'main' issue for Geneve is TLV encoding and this is not the case for GUE, or are the GUE extensions also TLV encoded ? Then this may be worth explaining.

## Section 6.1

Please explain why the absence of a Length field is important as it can (probably) be computed from the header (counting TLV, ...)

## Section 6.2

`Non-vendor specific TLVs` does it apply to all encapsulations or only to Geneve ?

## Section 6.5

Unsure what is meant by `The order of the extension headers should be hardware friendly` ? I.e., how can a designer check whether an order is HW friendly?

## Section 7

There are many additional recommendations that are *not* part of the original DT recommendations draft-dt-nvo3-encap-01. I.e., this seems more like a WG considerations rather than a design team considerations.

[Ballot comment]
Thanks for working with this document. I am fine with DT output to be published as RFCs, specially in this case as it is published as Informational, suits the purpose well.

[Ballot comment]
I think that some questions have been raised as to whether this sort of draft about a design team's reasoning and discussions should be published as an RFC.  To me, I read this document as the supporting justification as to the path taken vs the paths not taken, and I suspect that this reasoning could potentially be useful for other engineers considering similar issues.  Hence, writing it down and publishing it somewhere accessible is potentially useful, and as Informational RFC seems like a reasonable place, and better than, e.g., a wiki page that would likely suffer from bit rot.

Thanks for spending the time to write it and run it through the process.

Regards,
Rob

[Ballot comment]
# Internet AD comments for draft-ietf-nvo3-encap-11
CC @ekline

* comment syntax:
  - https://github.com/mnot/ietf-comments/blob/main/format.md

* "Handling Ballot Positions":
  - https://ietf.org/about/groups/iesg/statements/handling-ballot-positions/

## Comments

### S6.2.2

* Integrity can also be provided by an outer AH layer (RFC 4302), between the
  output IP and UDP headers, but maybe nobody wants to implement this.

### S6.4

* "minimum total available header length is 64 bytes"

  Is this the header after the outer UDP header an before any payload, or
  does "total available header" include outer IP and UDP headers?

### S6.6

* Is there anything interesting to be learned from the hardware offload
  support of something like GRE?  It has been extended a few times, but
  perhaps not as often as might be relevant here and obviously quite some
  time ago that it hardly matters.  Just curious.

Request for Last Call review by GENART Completed: Ready with Nits. Reviewer: Meral Shirazipour. Sent review to list. Submission of review completed at an earlier date.

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

IANA has completed its review of draft-ietf-nvo3-encap-10, which is currently in Last Call, and has the following comments:

We understand that this document doesn't require any registry actions.

While it's often helpful for a document's IANA Considerations section to remain in place upon publication even if there are no actions, if the authors strongly prefer to remove it, we do not object.

If this assessment is not accurate, please respond as soon as possible.

For definitions of IANA review states, please see:

https://datatracker.ietf.org/help/state/draft/iana-review

Thank you,

David Dong
IANA Services Sr. Specialist

The following Last Call announcement was sent out (ends 2023-11-18):

From: The IESG
To: IETF-Announce
CC: Matthew Bocci , andrew-ietf@liquid.tech, draft-ietf-nvo3-encap@ietf.org, matthew.bocci@nokia.com, nvo3-chairs@ietf.org, nvo3@ietf.org
Reply-To: last-call@ietf.org
Sender:
Subject: Last Call:  (Network Virtualization Overlays (NVO3) Encapsulation Considerations) to Informational RFC


The IESG has received a request from the Network Virtualization Overlays WG
(nvo3) to consider the following document: - 'Network Virtualization Overlays
(NVO3) Encapsulation Considerations'
  as Informational RFC

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2023-11-18. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


  The IETF Network Virtualization Overlays (NVO3) Working Group Chairs
  and Routing Area Director chartered a design team to take forward the
  encapsulation discussion and see if there was potential to design a
  common encapsulation that addresses the various technical concerns.
  This document provides a record, for the benefit of the IETF
  community, of the considerations arrived at by the NVO3 encapsulation
  design team, which may be helpful with future deliberations by
  working groups over the choice of encapsulation formats.

  There are implications of having different encapsulations in real
  environments consisting of both software and hardware implementations
  and within and spanning multiple data centers.  For example, OAM
  functions such as path MTU discovery become challenging with multiple
  encapsulations along the data path.

  The design team recommended Geneve with a few modifications as the
  common encapsulation. This document provides more details,
  particularly in Section 7.





The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-nvo3-encap/



No IPR declarations have been submitted directly on this I-D.

# Document Shepherd Write-Up for Group Documents

*This version is dated 4 July 2022.*

Thank you for your service as a document shepherd. Among the responsibilities is
answering the questions in this write-up to give helpful context to Last Call
and Internet Engineering Steering Group ([IESG][1]) reviewers, and your
diligence in completing it is appreciated. The full role of the shepherd is
further described in [RFC 4858][2]. You will need the cooperation of the authors
and editors to complete these checks.

Note that some numbered items contain multiple related questions; please be sure
to answer all of them.

## Document History

1. Does the working group (WG) consensus represent the strong concurrence of a
  few individuals, with others being silent, or did it reach broad agreement?

The draft represents strong consensus of the working group.

2. Was there controversy about particular points, or were there decisions where
  the consensus was particularly rough?

The draft describes the considerations of an encapsulation design team, and the objective of publishing the document as an RFC is to document these for the benefit of the IETF community.
There was strong consensus that this is a valuable exercise and that making this available to the broader community is helpful in future deliberations on encapsulations and other data-path critical aspects of new protocol design. Note that the draft has since been used as input to the deliberations of the MPLS Open Design Team on extensions to the MPLS architecture for MPLS network actions, as an example of how to quantify
the impact of various design choices on data plane hardware and software implementations. It was considered helpful in their deliberations.

3. Has anyone threatened an appeal or otherwise indicated extreme discontent? If
  so, please summarize the areas of conflict in separate email messages to the
  responsible Area Director. (It should be in a separate email because this
  questionnaire is publicly available.)

None indicated.

4. For protocol documents, are there existing implementations of the contents of
  the document? Have a significant number of potential implementers indicated
  plans to implement? Are any existing implementations reported somewhere,
  either in the document itself (as [RFC 7942][3] recommends) or elsewhere
  (where)?

Implementations of all of the encapsulations considered by the design team and described in the draft were claimed at the time.

## Additional Reviews

5. Do the contents of this document closely interact with technologies in other
  IETF working groups or external organizations, and would it therefore benefit
  from their review? Have those reviews occurred? If yes, describe which
  reviews took place.

A routing area directorate early review took place and comments were addressed as far as possible.
The draft does not directly overlap with technologies in other IETF WGs, although the impact of the choice to proceed with Geneve as a standards track
data plane encapsulation (RFC8926) has impacted the choices made by the BESS WG for which to support in their control planes. This did not require cross-review of this draft.

6. Describe how the document meets any required formal expert review criteria,
  such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

None required.

7. If the document contains a YANG module, has the final version of the module
  been checked with any of the [recommended validation tools][4] for syntax and
  formatting validation? If there are any resulting errors or warnings, what is
  the justification for not fixing them at this time? Does the YANG module
  comply with the Network Management Datastore Architecture (NMDA) as specified
  in [RFC 8342][5]?

N/A

8. Describe reviews and automated checks performed to validate sections of the
  final version of the document written in a formal language, such as XML code,
  BNF rules, MIB definitions, CBOR's CDDL, etc.

N/A

## Document Shepherd Checks

9. Based on the shepherd's review of the document, is it their opinion that this
  document is needed, clearly written, complete, correctly designed, and ready
  to be handed off to the responsible Area Director?

I have reviewed the draft, as well as being closely involved with original inception, chartering, and outcome of the design team whose work the draft describes.
I believe it is ready for handing off to the responsible AD.

10. Several IETF Areas have assembled [lists of common issues that their
    reviewers encounter][6]. For which areas have such issues been identified
    and addressed? For which does this still need to happen in subsequent
    reviews?

No issues identified.

11. What type of RFC publication is being requested on the IETF stream ([Best
    Current Practice][12], [Proposed Standard, Internet Standard][13],
    [Informational, Experimental or Historic][14])? Why is this the proper type
    of RFC? Do all Datatracker state attributes correctly reflect this intent?

Informational. The draft describes the decision process, and assessment of a number of protocol designs that were described in drafts that were intended to be standards track. This is done for the benefit of the
community. It does not specify any new protocols.

12. Have reasonable efforts been made to remind all authors of the intellectual
    property rights (IPR) disclosure obligations described in [BCP 79][7]? To
    the best of your knowledge, have all required disclosures been filed? If
    not, explain why. If yes, summarize any relevant discussion, including links
    to publicly-available messages when applicable.

Yes

13. Has each author, editor, and contributor shown their willingness to be
    listed as such? If the total number of authors and editors on the front page
    is greater than five, please provide a justification.

Yes.

14. Document any remaining I-D nits in this document. Simply running the [idnits
    tool][8] is not enough; please review the ["Content Guidelines" on
    authors.ietf.org][15]. (Also note that the current idnits tool generates
    some incorrect warnings; a rewrite is underway.)

None found.

15. Should any informative references be normative or vice-versa? See the [IESG
    Statement on Normative and Informative References][16].

The references appear to be correctly categorized.

16. List any normative references that are not freely available to anyone. Did
    the community have sufficient access to review any such normative
    references?

None

17. Are there any normative downward references (see [RFC 3967][9] and [BCP
    97][10]) that are not already listed in the [DOWNREF registry][17]? If so,
    list them.

None

18. Are there normative references to documents that are not ready to be
    submitted to the IESG for publication or are otherwise in an unclear state?
    If so, what is the plan for their completion?

None

19. Will publication of this document change the status of any existing RFCs? If
    so, does the Datatracker metadata correctly reflect this and are those RFCs
    listed on the title page, in the abstract, and discussed in the
    introduction? If not, explain why and point to the part of the document
    where the relationship of this document to these other RFCs is discussed.

Publication will not affect the status of any existing RFC.

20. Describe the document shepherd's review of the IANA considerations section,
    especially with regard to its consistency with the body of the document.
    Confirm that all aspects of the document requiring IANA assignments are
    associated with the appropriate reservations in IANA registries. Confirm
    that any referenced IANA registries have been clearly identified. Confirm
    that each newly created IANA registry specifies its initial contents,
    allocations procedures, and a reasonable name (see [RFC 8126][11]).

No IANA considerations.

21. List any new IANA registries that require Designated Expert Review for
    future allocations. Are the instructions to the Designated Expert clear?
    Please include suggestions of designated experts, if appropriate.

No new registries needed.

[1]: https://www.ietf.org/about/groups/iesg/
[2]: https://www.rfc-editor.org/rfc/rfc4858.html
[3]: https://www.rfc-editor.org/rfc/rfc7942.html
[4]: https://trac.ietf.org/trac/ops/wiki/yang-review-tools
[5]: https://www.rfc-editor.org/rfc/rfc8342.html
[6]: https://trac.ietf.org/trac/iesg/wiki/ExpertTopics
[7]: https://www.rfc-editor.org/info/bcp79
[8]: https://www.ietf.org/tools/idnits/
[9]: https://www.rfc-editor.org/rfc/rfc3967.html
[10]: https://www.rfc-editor.org/info/bcp97
[11]: https://www.rfc-editor.org/rfc/rfc8126.html
[12]: https://www.rfc-editor.org/rfc/rfc2026.html#section-5
[13]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.1
[14]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.2
[15]: https://authors.ietf.org/en/content-guidelines-overview
[16]: https://www.ietf.org/about/groups/iesg/statements/normative-informative-references/
[17]: https://datatracker.ietf.org/doc/downref/

# Document Shepherd Write-Up for Group Documents

*This version is dated 4 July 2022.*

Thank you for your service as a document shepherd. Among the responsibilities is
answering the questions in this write-up to give helpful context to Last Call
and Internet Engineering Steering Group ([IESG][1]) reviewers, and your
diligence in completing it is appreciated. The full role of the shepherd is
further described in [RFC 4858][2]. You will need the cooperation of the authors
and editors to complete these checks.

Note that some numbered items contain multiple related questions; please be sure
to answer all of them.

## Document History

1. Does the working group (WG) consensus represent the strong concurrence of a
  few individuals, with others being silent, or did it reach broad agreement?

The draft represents strong consensus of the working group.

2. Was there controversy about particular points, or were there decisions where
  the consensus was particularly rough?

The draft describes the considerations of an encapsulation design team, and the objective of publishing the document as an RFC is to document these for the benefit of the IETF community.
There was strong consensus that this is a valuable exercise and that making this available to the broader community is helpful in future deliberations on encapsulations and other data-path critical aspects of new protocol design. Note that the draft has since been used as input to the deliberations of the MPLS Open Design Team on extensions to the MPLS architecture for MPLS network actions, as an example of how to quantify
the impact of various design choices on data plane hardware and software implementations. It was considered helpful in their deliberations.

3. Has anyone threatened an appeal or otherwise indicated extreme discontent? If
  so, please summarize the areas of conflict in separate email messages to the
  responsible Area Director. (It should be in a separate email because this
  questionnaire is publicly available.)

None indicated.

4. For protocol documents, are there existing implementations of the contents of
  the document? Have a significant number of potential implementers indicated
  plans to implement? Are any existing implementations reported somewhere,
  either in the document itself (as [RFC 7942][3] recommends) or elsewhere
  (where)?

Implementations of all of the encapsulations considered by the design team and described in the draft were claimed at the time.

## Additional Reviews

5. Do the contents of this document closely interact with technologies in other
  IETF working groups or external organizations, and would it therefore benefit
  from their review? Have those reviews occurred? If yes, describe which
  reviews took place.

A routing area directorate early review took place and comments were addressed as far as possible.
The draft does not directly overlap with technologies in other IETF WGs, although the impact of the choice to proceed with Geneve as a standards track
data plane encapsulation (RFC8926) has impacted the choices made by the BESS WG for which to support in their control planes. This did not require cross-review of this draft.

6. Describe how the document meets any required formal expert review criteria,
  such as the MIB Doctor, YANG Doctor, media type, and URI type reviews.

None required.

7. If the document contains a YANG module, has the final version of the module
  been checked with any of the [recommended validation tools][4] for syntax and
  formatting validation? If there are any resulting errors or warnings, what is
  the justification for not fixing them at this time? Does the YANG module
  comply with the Network Management Datastore Architecture (NMDA) as specified
  in [RFC 8342][5]?

N/A

8. Describe reviews and automated checks performed to validate sections of the
  final version of the document written in a formal language, such as XML code,
  BNF rules, MIB definitions, CBOR's CDDL, etc.

N/A

## Document Shepherd Checks

9. Based on the shepherd's review of the document, is it their opinion that this
  document is needed, clearly written, complete, correctly designed, and ready
  to be handed off to the responsible Area Director?

I have reviewed the draft, as well as being closely involved with original inception, chartering, and outcome of the design team whose work the draft describes.
I believe it is ready for handing off to the responsible AD.

10. Several IETF Areas have assembled [lists of common issues that their
    reviewers encounter][6]. For which areas have such issues been identified
    and addressed? For which does this still need to happen in subsequent
    reviews?

No issues identified.

11. What type of RFC publication is being requested on the IETF stream ([Best
    Current Practice][12], [Proposed Standard, Internet Standard][13],
    [Informational, Experimental or Historic][14])? Why is this the proper type
    of RFC? Do all Datatracker state attributes correctly reflect this intent?

Informational. The draft describes the decision process, and assessment of a number of protocol designs that were described in drafts that were intended to be standards track. This is done for the benefit of the
community. It does not specify any new protocols.

12. Have reasonable efforts been made to remind all authors of the intellectual
    property rights (IPR) disclosure obligations described in [BCP 79][7]? To
    the best of your knowledge, have all required disclosures been filed? If
    not, explain why. If yes, summarize any relevant discussion, including links
    to publicly-available messages when applicable.

Yes

13. Has each author, editor, and contributor shown their willingness to be
    listed as such? If the total number of authors and editors on the front page
    is greater than five, please provide a justification.

Yes.

14. Document any remaining I-D nits in this document. Simply running the [idnits
    tool][8] is not enough; please review the ["Content Guidelines" on
    authors.ietf.org][15]. (Also note that the current idnits tool generates
    some incorrect warnings; a rewrite is underway.)

None found.

15. Should any informative references be normative or vice-versa? See the [IESG
    Statement on Normative and Informative References][16].

The references appear to be correctly categorized.

16. List any normative references that are not freely available to anyone. Did
    the community have sufficient access to review any such normative
    references?

None

17. Are there any normative downward references (see [RFC 3967][9] and [BCP
    97][10]) that are not already listed in the [DOWNREF registry][17]? If so,
    list them.

None

18. Are there normative references to documents that are not ready to be
    submitted to the IESG for publication or are otherwise in an unclear state?
    If so, what is the plan for their completion?

None

19. Will publication of this document change the status of any existing RFCs? If
    so, does the Datatracker metadata correctly reflect this and are those RFCs
    listed on the title page, in the abstract, and discussed in the
    introduction? If not, explain why and point to the part of the document
    where the relationship of this document to these other RFCs is discussed.

Publication will not affect the status of any existing RFC.

20. Describe the document shepherd's review of the IANA considerations section,
    especially with regard to its consistency with the body of the document.
    Confirm that all aspects of the document requiring IANA assignments are
    associated with the appropriate reservations in IANA registries. Confirm
    that any referenced IANA registries have been clearly identified. Confirm
    that each newly created IANA registry specifies its initial contents,
    allocations procedures, and a reasonable name (see [RFC 8126][11]).

No IANA considerations.

21. List any new IANA registries that require Designated Expert Review for
    future allocations. Are the instructions to the Designated Expert clear?
    Please include suggestions of designated experts, if appropriate.

No new registries needed.

[1]: https://www.ietf.org/about/groups/iesg/
[2]: https://www.rfc-editor.org/rfc/rfc4858.html
[3]: https://www.rfc-editor.org/rfc/rfc7942.html
[4]: https://trac.ietf.org/trac/ops/wiki/yang-review-tools
[5]: https://www.rfc-editor.org/rfc/rfc8342.html
[6]: https://trac.ietf.org/trac/iesg/wiki/ExpertTopics
[7]: https://www.rfc-editor.org/info/bcp79
[8]: https://www.ietf.org/tools/idnits/
[9]: https://www.rfc-editor.org/rfc/rfc3967.html
[10]: https://www.rfc-editor.org/info/bcp97
[11]: https://www.rfc-editor.org/rfc/rfc8126.html
[12]: https://www.rfc-editor.org/rfc/rfc2026.html#section-5
[13]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.1
[14]: https://www.rfc-editor.org/rfc/rfc2026.html#section-4.2
[15]: https://authors.ietf.org/en/content-guidelines-overview
[16]: https://www.ietf.org/about/groups/iesg/statements/normative-informative-references/
[17]: https://datatracker.ietf.org/doc/downref/

Request for posting confirmation emailed to previous authors: Sami Boutros , Tal Mizrahi , Sam Aldrin , Ignas Bagdonas , Erik Nordmark , Ilango Ganga , Pankaj Garg , Rajeev Manur , nvo3-chairs@ietf.org, Michael Smith , David Mozes

Request for posting confirmation emailed to previous authors: Sami Boutros , Tal Mizrahi , Erik Nordmark , Ilango Ganga , Pankaj Garg , Rajeev Manur , nvo3-chairs@ietf.org, David Mozes