Integrated Routing and Bridging in Ethernet VPN (EVPN)
draft-ietf-bess-evpn-inter-subnet-forwarding-15

[Ballot comment]
Thanks to the authors for their work in addressing my comments. Copying my (resolved) discuss points here for posterity.

----

I found this document difficult to review. Some of this might be due to the fact that I'm not an expert on EVPN, but I think some of the reason is that the document could be structured better and expressed more clearly. The only reason I'm not opposing progression of the document on the grounds that it's too unclear to implement is that I've been told, and accept on faith, that implementations *have* been successfully written starting from the spec, which implies it's implementable -- I guess by people who are expert in EVPN already, it wouldn't be implementable by me.

In any case, I do have some points I would like to discuss, that are more actionable.

1. I agree with Robert Wilton's comment on -09:

```
One question I have is whether it is possible to have a deployment where some devices support synchronous mode and others support asynchronous mode.  Am I right in presuming that this is not supported and if so is this capability signaled in any way? Or is the expectation that this would be controlled via deployment choice of network device, or though configuration management?
```

This issue still exists in -14. I think it should be addressed in the document. Similarly, I agree with Warren Kumari's comment, also on -09:

```
I would strongly recommend that the authors read the OpsDir review at: https://datatracker.ietf.org/doc/review-ietf-bess-evpn-inter-subnet-forwarding-09-opsdir-lc-jaeggli-2020-07-06/ , especially the: "it would be helpful if section 4 would be more explicit for non-implementors on when symetric or asymetric modules would be chosen, as it stands the variation basically reads like the enumeration of the features of various implementations." comment (which I fully agree with).
```

It seems both of these comments could -- and should! -- be addressed by adding a few paragraphs talking about these topics. This could be done either in §4, as Warren suggests, or in some other section (e.g. you could add an "operational considerations" section).

2. Section 7.1

I’m guessing this question isn’t unique to this document, but since this is where I encountered it, I’ll ask: it seems as though the described mobility procedures are vulnerable to a condition where a particular (IP, MAC) appears at two different NVEs at the same time. If this condition exists (either innocently, or maliciously) what prevents the source and target NVEs from continually attempting to claim the (IP, MAC) from one another, flooding the network with updates all the while?

(This applies to 7.2 as well.)

Since this seems like a potential security issue, I'm including it in my DISCUSS.

----

Below are a number of questions and comments that I hope might help improve the document. I haven't chosen to make them blocking by including them in my DISCUSS; nonetheless I would appreciate replies to them.

1. I agree with the comments by several of the other reviewers, that there are just too many gratuitous acronyms in this document. They aren't the only thing that makes it hard to read, but they certainly contribute. I'm disappointed to see this hasn't been addressed between versions -09 and -14. It would have been a small matter of search-and-replace to go through and expand most of the acronyms.

2. Section 2

```
  R1: The solution must allow for both inter-subnet and intra-subnet
  traffic belonging to the same tenant to be locally routed and bridged
  respectively.  The solution must provide IP routing for inter-subnet
  traffic and Ethernet Bridging for intra-subnet traffic.  It should be
  noted that if an IP-VRF in a NVE is configured for IPv6 and that NVE
  receives IPv4 traffic on the corresponding VLAN, then the IPv4
  traffic is treated as L2 traffic and it is bridged.  Also vise versa,
  if an IP-VRF in a NVE is configured for IPv4 and that NVE receives
  IPv6 traffic on the corresponding VLAN, then the IPv6 traffic is
  treated as L2 traffic and it is bridged.

  R2: The solution must support bridging for non-IP traffic.
```

R1 is a little tortured, where you add all the caveats about “treated as L2 traffic”. Seems to me like it would fall out more naturally if you had simply introduced the concepts of routable and non-routable traffic, where routable traffic is that for which a suitable IP-VRF exists. That would also have the pleasant effect of making R2 say “… must support bridging for non-routable traffic” instead of “non-IP traffic”, which is technically incorrect (since per R1 you might have non-routable IP traffic).

```
  R3: The solution must allow inter-subnet switching to be disabled on
  a per VLAN basis on PEs where the traffic needs to be backhauled to
  another node (i.e., for performing FW or DPI functionality).
```

What’s “switching”? The document is about routing vs. bridging, which do you mean? I think you mean “routing”. IMO you should get rid of the word “switching” and replace with something less ambiguous, e.g. “routing”. (Both here and the one other place in the doc where you use “switching”.)

Also, I think you don’t mean “i.e.”, I think you mean “e.g.”. The meaning of “i.e.” is “in other words”. The meaning of “e.g.” is “for example”. The best way to avoid these problems, IMO, is to simply write out what you mean, so in this case write “(for example, for performing FW or DPI functionality).” (And oh by the way, you haven’t defined or expanded FW or DPI, please do so.)

3. Section 4

```
  o  references to ARP table in the context of asymmetric IRB is a
      logical view of a forwarding table that maintains an IP to MAC
      binding entry on a layer 3 interface for both IPv4 and IPv6.
      These entries are not subject to ARP or ND protocol.
```

This passage shines a spotlight on the fact that “ARP table” as it’s used in this document is a misnomer, since it’s a table that is not (necessarily) populated by ARP. I don’t propose that you change the nomenclature, since it’s firmly established even though wrong — but it might be worth adding the first sentence or one like it to your Terminology section.

4. Section 4

Figure 2 depicts BT2 being present on the ingress PE, but the text makes it clear that in the symmetric mode that this figure depicts, BT2 doesn’t actually need to be there. Wouldn’t it be clearer if you didn’t show it?

5. Section 4

I have a hard time parsing this text:

```
                                              Each BT on a PE is
  associated with a unique VLAN (e.g., with a BD)
```

So, 1 VLAN —> at least 1 BT (1:many)

```
                                                  where in turn it is
  associated with a single MAC-VRF
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

```
                                    in the case of VLAN-Based mode or a
  number of BTs can be associated with a single MAC-VRF in the case of
  VLAN-Aware Bundle mode. 
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

Since this is stated as an exception I guess that means you meant the preceding two (that I parsed as 1:many) are actually supposed to be 1:1? If so I think this needs a rewrite (it probably does regardless, for clarity).

6. Section 4.1

When you write “Internet standard bit order“, do you mean “network byte order“? Although even network byte order appears to be non-applicable, since the values are shown with an explicit byte order.

I realize the definitions are merely pasted from RFC 5798 and that ship has sailed, but unless you can explain what “(in hex, in Internet standard bit-order)” is supposed to mean, I suggest removing it. (Alternately and less desirably, make it explicit that you’re providing a direct quotation of RFC 5798.)

7. Section 5.1

You say the Encapsulation Extended Community and Router’s MAC Extended Community have to be sent, but you say nothing about the required values. For Router's MAC, §8.1 specifies the required value, I suggest a forward reference to it. For Encapsulation, the closest I was able to find to a place where this is specified was section 9.1.1, but that's only an example. There really needs to be some place where it's spelled out. A bare minimum would be to cite RFC 9012 §4.1, but that just provides the syntax -- you really should say something more about how to decide what value to send. For that matter, it could be what valueS to send -- is it legal for a NVE to advertise multiple Encapsulation Extended Communities? You don't say it isn't, and there are potential reasons to do so.

8. Section 5.2

```
  o  Using MAC-VRF Route Target (and Ethernet Tag if different from
      zero), it identifies the corresponding MAC-VRF (and BT).  If the
      MAC- VRF (and BT) exists (e.g., it is locally configured) then it
```

You use “e.g.” so I presume there might be other reasons the MAC-VRF and BT might exist even if not locally configured?

```
      imports the MAC address into it.  Otherwise, it does not import
      the MAC address.

  o  Using IP-VRF route target, it identifies the corresponding IP-VRF
      and imports the IP address into it.
```

You don’t provide any conditional language in this bullet about “if the IP-VRF exists”. Why is that caveat required for MAC-VRF but not for IP-VRF?

9. Section 5.2

```
  The inclusion of MPLS label2 field in this route signals to the
  receiving PE that this route is for symmetric IRB mode and MPLS
  label2 needs to be installed in forwarding path to identify the
  corresponding IP-VRF.
```

I was unable to make head nor tail of this paragraph. I suppose §5.4 is where the behavior is actually specified, so in a way it doesn’t matter (although maybe a forward reference would help).

10. Section 5.2

```
  If the receiving PE receives this route with both the MAC-VRF and IP-
  VRF route targets and if the receiving PE does not support either
  asymmetric or symmetric IRB modes, then if it has the corresponding
  MAC-VRF, it only imports the MAC address.  Otherwise, if it doesn't
  have the corresponding MAC-VRF, it must not import this route.
```

If it doesn’t support either asymmetric or symmetric IRB modes, then doesn’t that mean it doesn’t implement this specification at all? In that circumstance, how do you expect your “must not” to be respected?

11. Section 5.3

```
                              If host B's (MAC, IP) has not yet been
  learnt either via a gratuitous ARP OR via a prior gleaning procedure,
  a new gleaning procedure MUST be triggered
```

Since you’ve used MUST here, you MUST provide a reference to where the “new gleaning procedure” is specified.

Also, has not been learnt by whom? The procedure must be triggered where?

12. Section 5.3

The second paragraph, that begins "Consider a subnet A", is tremendously confusing to a first-time reader (or at least to this first-time reader). I realize you probably think you're being helpful by providing a worked example, but as I read through it, it was the opposite of helpful. This is especially true because §5 and its subsections is about "Symmetric IRB Procedures" -- and the paragraph in question provides no procedures.

Some options to improve the situation --

- Remove the paragraph entirely.
- Preface the paragraph with "as an example to show why advertisement as RT-5 is required,"

13. Section 5.4

```
  o  global mode: VNI is set to the received label2 in the route which
      is domain-wide assigned.  This VNI value from received label2 MUST
      be the same as the locally configured VNI for the IP VRF as all
      PEs in the NVO MUST be configured with the same IP VRF VNI for
      this mode of operation.
```

What action is to be taken if this MUST is violated?

14. Section 6.1

```
  For asymmetric IRB mode, Router's MAC EC is not needed because
```

Please either expand “EC” or add it to your definitions section. (Also applies to 5.1)

15. Section 6.2

```
  o  If only MAC-VRF route target is used, then the receiving PE uses
      the MAC-VRF route target to identify the corresponding IP-VRF --
      i.e., many MAC-VRF route targets map to the same IP-VRF for a
      given tenant.  In this case, MAC-VRF may be used by the receiving
      PE to identify the corresponding IP VRF
```

Do you mean “in this case, the MAC-VRF *route target* may be used…”?

16. Section 6.2

```
  If the receiving PE receives the MAC/IP Advertisement route with MPLS
  label2 field and it uses symmetric IRB mode
```

This entire section is entitled “asymmetric IRB procedures“. Why is there specification language regarding symmetric procedures in it? (I’m pretty sure this is not the only place this kind of problem appears.)

17. Section 7.3

```
  On the source NVE, an age-out timer (for the silent host that has
  moved) is used to trigger an ARP probe.  This age-out timer can be
  either ARP timer or MAC age-out timer and this is an implementation
  choice.  The ARP request gets sent both locally to all the attached
  TSes on that subnet as well as it gets sent to all the remote NVEs
  (including the target NVE) participating in that subnet.  The source
  NVE also withdraw the EVPN MAC/IP Advertisement route with only the
  MAC address (if it has previously advertised such a route).
```

Wouldn’t the source NVE only withdraw the route after a timeout had expired? As you have written this paragraph, in case the silent TS has not moved, the following would happen:

```
Time t: age-out timer fires, ARP probe is sent
Time t: NVE withdraws route advertisement
Time u > t: TS receives ARP probe, sends ARP reply
Time v > u: NVE receives ARP reply
Time v: NVE re-advertises route
```

Presumably this churn isn’t what you intended.

18. Section 9.2

How does the NVE learn what subnets are behind its attached TS?

19. Section 9.2

What about if TS4 wants to reach SN1? How does it know where to send the packet? (I suppose the answer may be the same as for #18.)

[Ballot comment]
Thanks to the authors for their work in addressing my comments. Copying my (resolved) discuss points here for posterity.

--

I found this document difficult to review. Some of this might be due to the fact that I'm not an expert on EVPN, but I think some of the reason is that the document could be structured better and expressed more clearly. The only reason I'm not opposing progression of the document on the grounds that it's too unclear to implement is that I've been told, and accept on faith, that implementations *have* been successfully written starting from the spec, which implies it's implementable -- I guess by people who are expert in EVPN already, it wouldn't be implementable by me.

In any case, I do have some points I would like to discuss, that are more actionable.

1. I agree with Robert Wilton's comment on -09:

```
One question I have is whether it is possible to have a deployment where some devices support synchronous mode and others support asynchronous mode.  Am I right in presuming that this is not supported and if so is this capability signaled in any way? Or is the expectation that this would be controlled via deployment choice of network device, or though configuration management?
```

This issue still exists in -14. I think it should be addressed in the document. Similarly, I agree with Warren Kumari's comment, also on -09:

```
I would strongly recommend that the authors read the OpsDir review at: https://datatracker.ietf.org/doc/review-ietf-bess-evpn-inter-subnet-forwarding-09-opsdir-lc-jaeggli-2020-07-06/ , especially the: "it would be helpful if section 4 would be more explicit for non-implementors on when symetric or asymetric modules would be chosen, as it stands the variation basically reads like the enumeration of the features of various implementations." comment (which I fully agree with).
```

It seems both of these comments could -- and should! -- be addressed by adding a few paragraphs talking about these topics. This could be done either in §4, as Warren suggests, or in some other section (e.g. you could add an "operational considerations" section).

2. Section 7.1

I’m guessing this question isn’t unique to this document, but since this is where I encountered it, I’ll ask: it seems as though the described mobility procedures are vulnerable to a condition where a particular (IP, MAC) appears at two different NVEs at the same time. If this condition exists (either innocently, or maliciously) what prevents the source and target NVEs from continually attempting to claim the (IP, MAC) from one another, flooding the network with updates all the while?

(This applies to 7.2 as well.)

Since this seems like a potential security issue, I'm including it in my DISCUSS.
--

Below are a number of questions and comments that I hope might help improve the document. I haven't chosen to make them blocking by including them in my DISCUSS; nonetheless I would appreciate replies to them.

1. I agree with the comments by several of the other reviewers, that there are just too many gratuitous acronyms in this document. They aren't the only thing that makes it hard to read, but they certainly contribute. I'm disappointed to see this hasn't been addressed between versions -09 and -14. It would have been a small matter of search-and-replace to go through and expand most of the acronyms.

2. Section 2

```
  R1: The solution must allow for both inter-subnet and intra-subnet
  traffic belonging to the same tenant to be locally routed and bridged
  respectively.  The solution must provide IP routing for inter-subnet
  traffic and Ethernet Bridging for intra-subnet traffic.  It should be
  noted that if an IP-VRF in a NVE is configured for IPv6 and that NVE
  receives IPv4 traffic on the corresponding VLAN, then the IPv4
  traffic is treated as L2 traffic and it is bridged.  Also vise versa,
  if an IP-VRF in a NVE is configured for IPv4 and that NVE receives
  IPv6 traffic on the corresponding VLAN, then the IPv6 traffic is
  treated as L2 traffic and it is bridged.

  R2: The solution must support bridging for non-IP traffic.
```

R1 is a little tortured, where you add all the caveats about “treated as L2 traffic”. Seems to me like it would fall out more naturally if you had simply introduced the concepts of routable and non-routable traffic, where routable traffic is that for which a suitable IP-VRF exists. That would also have the pleasant effect of making R2 say “… must support bridging for non-routable traffic” instead of “non-IP traffic”, which is technically incorrect (since per R1 you might have non-routable IP traffic).

```
  R3: The solution must allow inter-subnet switching to be disabled on
  a per VLAN basis on PEs where the traffic needs to be backhauled to
  another node (i.e., for performing FW or DPI functionality).
```

What’s “switching”? The document is about routing vs. bridging, which do you mean? I think you mean “routing”. IMO you should get rid of the word “switching” and replace with something less ambiguous, e.g. “routing”. (Both here and the one other place in the doc where you use “switching”.)

Also, I think you don’t mean “i.e.”, I think you mean “e.g.”. The meaning of “i.e.” is “in other words”. The meaning of “e.g.” is “for example”. The best way to avoid these problems, IMO, is to simply write out what you mean, so in this case write “(for example, for performing FW or DPI functionality).” (And oh by the way, you haven’t defined or expanded FW or DPI, please do so.)

3. Section 4

```
  o  references to ARP table in the context of asymmetric IRB is a
      logical view of a forwarding table that maintains an IP to MAC
      binding entry on a layer 3 interface for both IPv4 and IPv6.
      These entries are not subject to ARP or ND protocol.
```

This passage shines a spotlight on the fact that “ARP table” as it’s used in this document is a misnomer, since it’s a table that is not (necessarily) populated by ARP. I don’t propose that you change the nomenclature, since it’s firmly established even though wrong — but it might be worth adding the first sentence or one like it to your Terminology section.

4. Section 4

Figure 2 depicts BT2 being present on the ingress PE, but the text makes it clear that in the symmetric mode that this figure depicts, BT2 doesn’t actually need to be there. Wouldn’t it be clearer if you didn’t show it?

5. Section 4

I have a hard time parsing this text:

```
                                              Each BT on a PE is
  associated with a unique VLAN (e.g., with a BD)
```

So, 1 VLAN —> at least 1 BT (1:many)

```
                                                  where in turn it is
  associated with a single MAC-VRF
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

```
                                    in the case of VLAN-Based mode or a
  number of BTs can be associated with a single MAC-VRF in the case of
  VLAN-Aware Bundle mode. 
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

Since this is stated as an exception I guess that means you meant the preceding two (that I parsed as 1:many) are actually supposed to be 1:1? If so I think this needs a rewrite (it probably does regardless, for clarity).

6. Section 4.1

When you write “Internet standard bit order“, do you mean “network byte order“? Although even network byte order appears to be non-applicable, since the values are shown with an explicit byte order.

I realize the definitions are merely pasted from RFC 5798 and that ship has sailed, but unless you can explain what “(in hex, in Internet standard bit-order)” is supposed to mean, I suggest removing it. (Alternately and less desirably, make it explicit that you’re providing a direct quotation of RFC 5798.)

7. Section 5.1

You say the Encapsulation Extended Community and Router’s MAC Extended Community have to be sent, but you say nothing about the required values. For Router's MAC, §8.1 specifies the required value, I suggest a forward reference to it. For Encapsulation, the closest I was able to find to a place where this is specified was section 9.1.1, but that's only an example. There really needs to be some place where it's spelled out. A bare minimum would be to cite RFC 9012 §4.1, but that just provides the syntax -- you really should say something more about how to decide what value to send. For that matter, it could be what valueS to send -- is it legal for a NVE to advertise multiple Encapsulation Extended Communities? You don't say it isn't, and there are potential reasons to do so.

8. Section 5.2

```
  o  Using MAC-VRF Route Target (and Ethernet Tag if different from
      zero), it identifies the corresponding MAC-VRF (and BT).  If the
      MAC- VRF (and BT) exists (e.g., it is locally configured) then it
```

You use “e.g.” so I presume there might be other reasons the MAC-VRF and BT might exist even if not locally configured?

```
      imports the MAC address into it.  Otherwise, it does not import
      the MAC address.

  o  Using IP-VRF route target, it identifies the corresponding IP-VRF
      and imports the IP address into it.
```

You don’t provide any conditional language in this bullet about “if the IP-VRF exists”. Why is that caveat required for MAC-VRF but not for IP-VRF?

9. Section 5.2

```
  The inclusion of MPLS label2 field in this route signals to the
  receiving PE that this route is for symmetric IRB mode and MPLS
  label2 needs to be installed in forwarding path to identify the
  corresponding IP-VRF.
```

I was unable to make head nor tail of this paragraph. I suppose §5.4 is where the behavior is actually specified, so in a way it doesn’t matter (although maybe a forward reference would help).

10. Section 5.2

```
  If the receiving PE receives this route with both the MAC-VRF and IP-
  VRF route targets and if the receiving PE does not support either
  asymmetric or symmetric IRB modes, then if it has the corresponding
  MAC-VRF, it only imports the MAC address.  Otherwise, if it doesn't
  have the corresponding MAC-VRF, it must not import this route.
```

If it doesn’t support either asymmetric or symmetric IRB modes, then doesn’t that mean it doesn’t implement this specification at all? In that circumstance, how do you expect your “must not” to be respected?

11. Section 5.3

```
                              If host B's (MAC, IP) has not yet been
  learnt either via a gratuitous ARP OR via a prior gleaning procedure,
  a new gleaning procedure MUST be triggered
```

Since you’ve used MUST here, you MUST provide a reference to where the “new gleaning procedure” is specified.

Also, has not been learnt by whom? The procedure must be triggered where?

12. Section 5.3

The second paragraph, that begins "Consider a subnet A", is tremendously confusing to a first-time reader (or at least to this first-time reader). I realize you probably think you're being helpful by providing a worked example, but as I read through it, it was the opposite of helpful. This is especially true because §5 and its subsections is about "Symmetric IRB Procedures" -- and the paragraph in question provides no procedures.

Some options to improve the situation --

- Remove the paragraph entirely.
- Preface the paragraph with "as an example to show why advertisement as RT-5 is required,"

13. Section 5.4

```
  o  global mode: VNI is set to the received label2 in the route which
      is domain-wide assigned.  This VNI value from received label2 MUST
      be the same as the locally configured VNI for the IP VRF as all
      PEs in the NVO MUST be configured with the same IP VRF VNI for
      this mode of operation.
```

What action is to be taken if this MUST is violated?

14. Section 6.1

```
  For asymmetric IRB mode, Router's MAC EC is not needed because
```

Please either expand “EC” or add it to your definitions section. (Also applies to 5.1)

15. Section 6.2

```
  o  If only MAC-VRF route target is used, then the receiving PE uses
      the MAC-VRF route target to identify the corresponding IP-VRF --
      i.e., many MAC-VRF route targets map to the same IP-VRF for a
      given tenant.  In this case, MAC-VRF may be used by the receiving
      PE to identify the corresponding IP VRF
```

Do you mean “in this case, the MAC-VRF *route target* may be used…”?

16. Section 6.2

```
  If the receiving PE receives the MAC/IP Advertisement route with MPLS
  label2 field and it uses symmetric IRB mode
```

This entire section is entitled “asymmetric IRB procedures“. Why is there specification language regarding symmetric procedures in it? (I’m pretty sure this is not the only place this kind of problem appears.)

17. Section 7.3

```
  On the source NVE, an age-out timer (for the silent host that has
  moved) is used to trigger an ARP probe.  This age-out timer can be
  either ARP timer or MAC age-out timer and this is an implementation
  choice.  The ARP request gets sent both locally to all the attached
  TSes on that subnet as well as it gets sent to all the remote NVEs
  (including the target NVE) participating in that subnet.  The source
  NVE also withdraw the EVPN MAC/IP Advertisement route with only the
  MAC address (if it has previously advertised such a route).
```

Wouldn’t the source NVE only withdraw the route after a timeout had expired? As you have written this paragraph, in case the silent TS has not moved, the following would happen:

```
Time t: age-out timer fires, ARP probe is sent
Time t: NVE withdraws route advertisement
Time u > t: TS receives ARP probe, sends ARP reply
Time v > u: NVE receives ARP reply
Time v: NVE re-advertises route
```

Presumably this churn isn’t what you intended.

18. Section 9.2

How does the NVE learn what subnets are behind its attached TS?

19. Section 9.2

What about if TS4 wants to reach SN1? How does it know where to send the packet? (I suppose the answer may be the same as for #18.)

[Ballot comment]
Thanks to the authors for their work in addressing my comments. Copying my (resolved) discuss points here for posterity.

--
I found this document difficult to review. Some of this might be due to the fact that I'm not an expert on EVPN, but I think some of the reason is that the document could be structured better and expressed more clearly. The only reason I'm not opposing progression of the document on the grounds that it's too unclear to implement is that I've been told, and accept on faith, that implementations *have* been successfully written starting from the spec, which implies it's implementable -- I guess by people who are expert in EVPN already, it wouldn't be implementable by me.

In any case, I do have some points I would like to discuss, that are more actionable.

1. I agree with Robert Wilton's comment on -09:

```
One question I have is whether it is possible to have a deployment where some devices support synchronous mode and others support asynchronous mode.  Am I right in presuming that this is not supported and if so is this capability signaled in any way? Or is the expectation that this would be controlled via deployment choice of network device, or though configuration management?
```

This issue still exists in -14. I think it should be addressed in the document. Similarly, I agree with Warren Kumari's comment, also on -09:

```
I would strongly recommend that the authors read the OpsDir review at: https://datatracker.ietf.org/doc/review-ietf-bess-evpn-inter-subnet-forwarding-09-opsdir-lc-jaeggli-2020-07-06/ , especially the: "it would be helpful if section 4 would be more explicit for non-implementors on when symetric or asymetric modules would be chosen, as it stands the variation basically reads like the enumeration of the features of various implementations." comment (which I fully agree with).
```

It seems both of these comments could -- and should! -- be addressed by adding a few paragraphs talking about these topics. This could be done either in §4, as Warren suggests, or in some other section (e.g. you could add an "operational considerations" section).

2. Section 7.1

I’m guessing this question isn’t unique to this document, but since this is where I encountered it, I’ll ask: it seems as though the described mobility procedures are vulnerable to a condition where a particular (IP, MAC) appears at two different NVEs at the same time. If this condition exists (either innocently, or maliciously) what prevents the source and target NVEs from continually attempting to claim the (IP, MAC) from one another, flooding the network with updates all the while?

(This applies to 7.2 as well.)

Since this seems like a potential security issue, I'm including it in my DISCUSS.
--

Below are a number of questions and comments that I hope might help improve the document. I haven't chosen to make them blocking by including them in my DISCUSS; nonetheless I would appreciate replies to them.

1. I agree with the comments by several of the other reviewers, that there are just too many gratuitous acronyms in this document. They aren't the only thing that makes it hard to read, but they certainly contribute. I'm disappointed to see this hasn't been addressed between versions -09 and -14. It would have been a small matter of search-and-replace to go through and expand most of the acronyms.

2. Section 2

```
  R1: The solution must allow for both inter-subnet and intra-subnet
  traffic belonging to the same tenant to be locally routed and bridged
  respectively.  The solution must provide IP routing for inter-subnet
  traffic and Ethernet Bridging for intra-subnet traffic.  It should be
  noted that if an IP-VRF in a NVE is configured for IPv6 and that NVE
  receives IPv4 traffic on the corresponding VLAN, then the IPv4
  traffic is treated as L2 traffic and it is bridged.  Also vise versa,
  if an IP-VRF in a NVE is configured for IPv4 and that NVE receives
  IPv6 traffic on the corresponding VLAN, then the IPv6 traffic is
  treated as L2 traffic and it is bridged.

  R2: The solution must support bridging for non-IP traffic.
```

R1 is a little tortured, where you add all the caveats about “treated as L2 traffic”. Seems to me like it would fall out more naturally if you had simply introduced the concepts of routable and non-routable traffic, where routable traffic is that for which a suitable IP-VRF exists. That would also have the pleasant effect of making R2 say “… must support bridging for non-routable traffic” instead of “non-IP traffic”, which is technically incorrect (since per R1 you might have non-routable IP traffic).

```
  R3: The solution must allow inter-subnet switching to be disabled on
  a per VLAN basis on PEs where the traffic needs to be backhauled to
  another node (i.e., for performing FW or DPI functionality).
```

What’s “switching”? The document is about routing vs. bridging, which do you mean? I think you mean “routing”. IMO you should get rid of the word “switching” and replace with something less ambiguous, e.g. “routing”. (Both here and the one other place in the doc where you use “switching”.)

Also, I think you don’t mean “i.e.”, I think you mean “e.g.”. The meaning of “i.e.” is “in other words”. The meaning of “e.g.” is “for example”. The best way to avoid these problems, IMO, is to simply write out what you mean, so in this case write “(for example, for performing FW or DPI functionality).” (And oh by the way, you haven’t defined or expanded FW or DPI, please do so.)

3. Section 4

```
  o  references to ARP table in the context of asymmetric IRB is a
      logical view of a forwarding table that maintains an IP to MAC
      binding entry on a layer 3 interface for both IPv4 and IPv6.
      These entries are not subject to ARP or ND protocol.
```

This passage shines a spotlight on the fact that “ARP table” as it’s used in this document is a misnomer, since it’s a table that is not (necessarily) populated by ARP. I don’t propose that you change the nomenclature, since it’s firmly established even though wrong — but it might be worth adding the first sentence or one like it to your Terminology section.

4. Section 4

Figure 2 depicts BT2 being present on the ingress PE, but the text makes it clear that in the symmetric mode that this figure depicts, BT2 doesn’t actually need to be there. Wouldn’t it be clearer if you didn’t show it?

5. Section 4

I have a hard time parsing this text:

```
                                              Each BT on a PE is
  associated with a unique VLAN (e.g., with a BD)
```

So, 1 VLAN —> at least 1 BT (1:many)

```
                                                  where in turn it is
  associated with a single MAC-VRF
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

```
                                    in the case of VLAN-Based mode or a
  number of BTs can be associated with a single MAC-VRF in the case of
  VLAN-Aware Bundle mode. 
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

Since this is stated as an exception I guess that means you meant the preceding two (that I parsed as 1:many) are actually supposed to be 1:1? If so I think this needs a rewrite (it probably does regardless, for clarity).

6. Section 4.1

When you write “Internet standard bit order“, do you mean “network byte order“? Although even network byte order appears to be non-applicable, since the values are shown with an explicit byte order.

I realize the definitions are merely pasted from RFC 5798 and that ship has sailed, but unless you can explain what “(in hex, in Internet standard bit-order)” is supposed to mean, I suggest removing it. (Alternately and less desirably, make it explicit that you’re providing a direct quotation of RFC 5798.)

7. Section 5.1

You say the Encapsulation Extended Community and Router’s MAC Extended Community have to be sent, but you say nothing about the required values. For Router's MAC, §8.1 specifies the required value, I suggest a forward reference to it. For Encapsulation, the closest I was able to find to a place where this is specified was section 9.1.1, but that's only an example. There really needs to be some place where it's spelled out. A bare minimum would be to cite RFC 9012 §4.1, but that just provides the syntax -- you really should say something more about how to decide what value to send. For that matter, it could be what valueS to send -- is it legal for a NVE to advertise multiple Encapsulation Extended Communities? You don't say it isn't, and there are potential reasons to do so.

8. Section 5.2

```
  o  Using MAC-VRF Route Target (and Ethernet Tag if different from
      zero), it identifies the corresponding MAC-VRF (and BT).  If the
      MAC- VRF (and BT) exists (e.g., it is locally configured) then it
```

You use “e.g.” so I presume there might be other reasons the MAC-VRF and BT might exist even if not locally configured?

```
      imports the MAC address into it.  Otherwise, it does not import
      the MAC address.

  o  Using IP-VRF route target, it identifies the corresponding IP-VRF
      and imports the IP address into it.
```

You don’t provide any conditional language in this bullet about “if the IP-VRF exists”. Why is that caveat required for MAC-VRF but not for IP-VRF?

9. Section 5.2

```
  The inclusion of MPLS label2 field in this route signals to the
  receiving PE that this route is for symmetric IRB mode and MPLS
  label2 needs to be installed in forwarding path to identify the
  corresponding IP-VRF.
```

I was unable to make head nor tail of this paragraph. I suppose §5.4 is where the behavior is actually specified, so in a way it doesn’t matter (although maybe a forward reference would help).

10. Section 5.2

```
  If the receiving PE receives this route with both the MAC-VRF and IP-
  VRF route targets and if the receiving PE does not support either
  asymmetric or symmetric IRB modes, then if it has the corresponding
  MAC-VRF, it only imports the MAC address.  Otherwise, if it doesn't
  have the corresponding MAC-VRF, it must not import this route.
```

If it doesn’t support either asymmetric or symmetric IRB modes, then doesn’t that mean it doesn’t implement this specification at all? In that circumstance, how do you expect your “must not” to be respected?

11. Section 5.3

```
                              If host B's (MAC, IP) has not yet been
  learnt either via a gratuitous ARP OR via a prior gleaning procedure,
  a new gleaning procedure MUST be triggered
```

Since you’ve used MUST here, you MUST provide a reference to where the “new gleaning procedure” is specified.

Also, has not been learnt by whom? The procedure must be triggered where?

12. Section 5.3

The second paragraph, that begins "Consider a subnet A", is tremendously confusing to a first-time reader (or at least to this first-time reader). I realize you probably think you're being helpful by providing a worked example, but as I read through it, it was the opposite of helpful. This is especially true because §5 and its subsections is about "Symmetric IRB Procedures" -- and the paragraph in question provides no procedures.

Some options to improve the situation --

- Remove the paragraph entirely.
- Preface the paragraph with "as an example to show why advertisement as RT-5 is required,"

13. Section 5.4

```
  o  global mode: VNI is set to the received label2 in the route which
      is domain-wide assigned.  This VNI value from received label2 MUST
      be the same as the locally configured VNI for the IP VRF as all
      PEs in the NVO MUST be configured with the same IP VRF VNI for
      this mode of operation.
```

What action is to be taken if this MUST is violated?

14. Section 6.1

```
  For asymmetric IRB mode, Router's MAC EC is not needed because
```

Please either expand “EC” or add it to your definitions section. (Also applies to 5.1)

15. Section 6.2

```
  o  If only MAC-VRF route target is used, then the receiving PE uses
      the MAC-VRF route target to identify the corresponding IP-VRF --
      i.e., many MAC-VRF route targets map to the same IP-VRF for a
      given tenant.  In this case, MAC-VRF may be used by the receiving
      PE to identify the corresponding IP VRF
```

Do you mean “in this case, the MAC-VRF *route target* may be used…”?

16. Section 6.2

```
  If the receiving PE receives the MAC/IP Advertisement route with MPLS
  label2 field and it uses symmetric IRB mode
```

This entire section is entitled “asymmetric IRB procedures“. Why is there specification language regarding symmetric procedures in it? (I’m pretty sure this is not the only place this kind of problem appears.)

17. Section 7.3

```
  On the source NVE, an age-out timer (for the silent host that has
  moved) is used to trigger an ARP probe.  This age-out timer can be
  either ARP timer or MAC age-out timer and this is an implementation
  choice.  The ARP request gets sent both locally to all the attached
  TSes on that subnet as well as it gets sent to all the remote NVEs
  (including the target NVE) participating in that subnet.  The source
  NVE also withdraw the EVPN MAC/IP Advertisement route with only the
  MAC address (if it has previously advertised such a route).
```

Wouldn’t the source NVE only withdraw the route after a timeout had expired? As you have written this paragraph, in case the silent TS has not moved, the following would happen:

```
Time t: age-out timer fires, ARP probe is sent
Time t: NVE withdraws route advertisement
Time u > t: TS receives ARP probe, sends ARP reply
Time v > u: NVE receives ARP reply
Time v: NVE re-advertises route
```

Presumably this churn isn’t what you intended.

18. Section 9.2

How does the NVE learn what subnets are behind its attached TS?

19. Section 9.2

What about if TS4 wants to reach SN1? How does it know where to send the packet? (I suppose the answer may be the same as for #18.)

[Ballot comment]
I agree with John's point that the presentation of the material in this document
is very dense and likely doesn't lend itself to easily writing interoperable
implementations.

This document uses RFC2119 keywords, but does not contain the recommended
RFC8174 boilerplate. (It contains some text with a similar beginning.)

Found terminology that should be reviewed for inclusivity; see
https://www.rfc-editor.org/part2/#inclusive_language for background and more
guidance:

* Term "traditionally"; alternatives might be "classic", "classical",
  "common", "conventional", "customary", "fixed", "habitual", "historic",
  "long-established", "popular", "prescribed", "regular", "rooted",
  "time-honored", "universal", "widely used", "widespread".

-------------------------------------------------------------------------------
All comments below are about very minor potential issues that you may choose to
address in some way - or ignore - as you see fit. Some were flagged by
automated tools (via https://github.com/larseggert/ietf-reviewtool), so there
will likely be some false positives. There is no need to let me know what you
did with these suggestions.

Section 2. , paragraph 6, nit:
-    traffic is treated as L2 traffic and it is bridged.  Also vise versa,
-                                                                ^
+    traffic is treated as L2 traffic and it is bridged.  Also vice versa,
+                                                                ^

Section 7. , paragraph 6, nit:
-    Depending on the expexted TS's behavior, an NVE needs to handle at
-                        ^
+    Depending on the expected TS's behavior, an NVE needs to handle at
+                        ^

Section 7. , paragraph 7, nit:
- 7.1.  Initiating a gratutious ARP upon a Move
-                          -
+ 7.1.  Initiating a gratuitous ARP upon a Move
+                        +

Section 5.5. , paragraph 2, nit:
> ese subnets. The reason for this is because ingress PE needs to do forwarding
>                                  ^^^^^^^^^^
The word "because" means "for the reason that" and thus introduces redundancy.

Section 9.1. , paragraph 5, nit:
> e actual implementation may differ. Lets consider data-plane operation when T
>                                    ^^^^
Did you mean "Let's" (let's = let us; lets = 3rd person singular of "let")?

Section 9.1. , paragraph 6, nit:
> n the egress NVE, if the packet arrives on Ethernet NVO tunnel (e.g., it is
>                                ^^^^^^^^^^
The usual preposition after "arrives" is "at", not "on". Did you mean "arrives
at"?

Section 9.2. , paragraph 2, nit:
> e actual implementation may differ. Lets consider data-plane operation when a
>                                    ^^^^
Did you mean "Let's" (let's = let us; lets = 3rd person singular of "let")?

Document references draft-ietf-idr-tunnel-encaps, but that has been published
as RFC9012.

Document references draft-ietf-bess-evpn-irb-extended-mobility-03, but -05 is
the latest available revision.

Document references draft-ietf-nvo3-vxlan-gpe-10, but -11 is the latest
available revision.

[Ballot comment]
Below are a number of questions and comments that I hope might help improve the document. I haven't chosen to make them blocking by including them in my DISCUSS; nonetheless I would appreciate replies to them.

1. I agree with the comments by several of the other reviewers, that there are just too many gratuitous acronyms in this document. They aren't the only thing that makes it hard to read, but they certainly contribute. I'm disappointed to see this hasn't been addressed between versions -09 and -14. It would have been a small matter of search-and-replace to go through and expand most of the acronyms.

2. Section 2

```
  R1: The solution must allow for both inter-subnet and intra-subnet
  traffic belonging to the same tenant to be locally routed and bridged
  respectively.  The solution must provide IP routing for inter-subnet
  traffic and Ethernet Bridging for intra-subnet traffic.  It should be
  noted that if an IP-VRF in a NVE is configured for IPv6 and that NVE
  receives IPv4 traffic on the corresponding VLAN, then the IPv4
  traffic is treated as L2 traffic and it is bridged.  Also vise versa,
  if an IP-VRF in a NVE is configured for IPv4 and that NVE receives
  IPv6 traffic on the corresponding VLAN, then the IPv6 traffic is
  treated as L2 traffic and it is bridged.

  R2: The solution must support bridging for non-IP traffic.
```

R1 is a little tortured, where you add all the caveats about “treated as L2 traffic”. Seems to me like it would fall out more naturally if you had simply introduced the concepts of routable and non-routable traffic, where routable traffic is that for which a suitable IP-VRF exists. That would also have the pleasant effect of making R2 say “… must support bridging for non-routable traffic” instead of “non-IP traffic”, which is technically incorrect (since per R1 you might have non-routable IP traffic).

```
  R3: The solution must allow inter-subnet switching to be disabled on
  a per VLAN basis on PEs where the traffic needs to be backhauled to
  another node (i.e., for performing FW or DPI functionality).
```

What’s “switching”? The document is about routing vs. bridging, which do you mean? I think you mean “routing”. IMO you should get rid of the word “switching” and replace with something less ambiguous, e.g. “routing”. (Both here and the one other place in the doc where you use “switching”.)

Also, I think you don’t mean “i.e.”, I think you mean “e.g.”. The meaning of “i.e.” is “in other words”. The meaning of “e.g.” is “for example”. The best way to avoid these problems, IMO, is to simply write out what you mean, so in this case write “(for example, for performing FW or DPI functionality).” (And oh by the way, you haven’t defined or expanded FW or DPI, please do so.)

3. Section 4

```
  o  references to ARP table in the context of asymmetric IRB is a
      logical view of a forwarding table that maintains an IP to MAC
      binding entry on a layer 3 interface for both IPv4 and IPv6.
      These entries are not subject to ARP or ND protocol.
```

This passage shines a spotlight on the fact that “ARP table” as it’s used in this document is a misnomer, since it’s a table that is not (necessarily) populated by ARP. I don’t propose that you change the nomenclature, since it’s firmly established even though wrong — but it might be worth adding the first sentence or one like it to your Terminology section.

4. Section 4

Figure 2 depicts BT2 being present on the ingress PE, but the text makes it clear that in the symmetric mode that this figure depicts, BT2 doesn’t actually need to be there. Wouldn’t it be clearer if you didn’t show it?

5. Section 4

I have a hard time parsing this text:

```
                                              Each BT on a PE is
  associated with a unique VLAN (e.g., with a BD)
```

So, 1 VLAN —> at least 1 BT (1:many)

```
                                                  where in turn it is
  associated with a single MAC-VRF
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

```
                                    in the case of VLAN-Based mode or a
  number of BTs can be associated with a single MAC-VRF in the case of
  VLAN-Aware Bundle mode. 
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

Since this is stated as an exception I guess that means you meant the preceding two (that I parsed as 1:many) are actually supposed to be 1:1? If so I think this needs a rewrite (it probably does regardless, for clarity).

6. Section 4.1

When you write “Internet standard bit order“, do you mean “network byte order“? Although even network byte order appears to be non-applicable, since the values are shown with an explicit byte order.

I realize the definitions are merely pasted from RFC 5798 and that ship has sailed, but unless you can explain what “(in hex, in Internet standard bit-order)” is supposed to mean, I suggest removing it. (Alternately and less desirably, make it explicit that you’re providing a direct quotation of RFC 5798.)

7. Section 5.1

You say the Encapsulation Extended Community and Router’s MAC Extended Community have to be sent, but you say nothing about the required values. For Router's MAC, §8.1 specifies the required value, I suggest a forward reference to it. For Encapsulation, the closest I was able to find to a place where this is specified was section 9.1.1, but that's only an example. There really needs to be some place where it's spelled out. A bare minimum would be to cite RFC 9012 §4.1, but that just provides the syntax -- you really should say something more about how to decide what value to send. For that matter, it could be what valueS to send -- is it legal for a NVE to advertise multiple Encapsulation Extended Communities? You don't say it isn't, and there are potential reasons to do so.

8. Section 5.2

```
  o  Using MAC-VRF Route Target (and Ethernet Tag if different from
      zero), it identifies the corresponding MAC-VRF (and BT).  If the
      MAC- VRF (and BT) exists (e.g., it is locally configured) then it
```

You use “e.g.” so I presume there might be other reasons the MAC-VRF and BT might exist even if not locally configured?

```
      imports the MAC address into it.  Otherwise, it does not import
      the MAC address.

  o  Using IP-VRF route target, it identifies the corresponding IP-VRF
      and imports the IP address into it.
```

You don’t provide any conditional language in this bullet about “if the IP-VRF exists”. Why is that caveat required for MAC-VRF but not for IP-VRF?

9. Section 5.2

```
  The inclusion of MPLS label2 field in this route signals to the
  receiving PE that this route is for symmetric IRB mode and MPLS
  label2 needs to be installed in forwarding path to identify the
  corresponding IP-VRF.
```

I was unable to make head nor tail of this paragraph. I suppose §5.4 is where the behavior is actually specified, so in a way it doesn’t matter (although maybe a forward reference would help).

10. Section 5.2

```
  If the receiving PE receives this route with both the MAC-VRF and IP-
  VRF route targets and if the receiving PE does not support either
  asymmetric or symmetric IRB modes, then if it has the corresponding
  MAC-VRF, it only imports the MAC address.  Otherwise, if it doesn't
  have the corresponding MAC-VRF, it must not import this route.
```

If it doesn’t support either asymmetric or symmetric IRB modes, then doesn’t that mean it doesn’t implement this specification at all? In that circumstance, how do you expect your “must not” to be respected?

11. Section 5.3

```
                              If host B's (MAC, IP) has not yet been
  learnt either via a gratuitous ARP OR via a prior gleaning procedure,
  a new gleaning procedure MUST be triggered
```

Since you’ve used MUST here, you MUST provide a reference to where the “new gleaning procedure” is specified.

Also, has not been learnt by whom? The procedure must be triggered where?

12. Section 5.3

The second paragraph, that begins "Consider a subnet A", is tremendously confusing to a first-time reader (or at least to this first-time reader). I realize you probably think you're being helpful by providing a worked example, but as I read through it, it was the opposite of helpful. This is especially true because §5 and its subsections is about "Symmetric IRB Procedures" -- and the paragraph in question provides no procedures.

Some options to improve the situation --

- Remove the paragraph entirely.
- Preface the paragraph with "as an example to show why advertisement as RT-5 is required,"

13. Section 5.4

```
  o  global mode: VNI is set to the received label2 in the route which
      is domain-wide assigned.  This VNI value from received label2 MUST
      be the same as the locally configured VNI for the IP VRF as all
      PEs in the NVO MUST be configured with the same IP VRF VNI for
      this mode of operation.
```

What action is to be taken if this MUST is violated?

14. Section 6.1

```
  For asymmetric IRB mode, Router's MAC EC is not needed because
```

Please either expand “EC” or add it to your definitions section. (Also applies to 5.1)

15. Section 6.2

```
  o  If only MAC-VRF route target is used, then the receiving PE uses
      the MAC-VRF route target to identify the corresponding IP-VRF --
      i.e., many MAC-VRF route targets map to the same IP-VRF for a
      given tenant.  In this case, MAC-VRF may be used by the receiving
      PE to identify the corresponding IP VRF
```

Do you mean “in this case, the MAC-VRF *route target* may be used…”?

16. Section 6.2

```
  If the receiving PE receives the MAC/IP Advertisement route with MPLS
  label2 field and it uses symmetric IRB mode
```

This entire section is entitled “asymmetric IRB procedures“. Why is there specification language regarding symmetric procedures in it? (I’m pretty sure this is not the only place this kind of problem appears.)

17. Section 7.3

```
  On the source NVE, an age-out timer (for the silent host that has
  moved) is used to trigger an ARP probe.  This age-out timer can be
  either ARP timer or MAC age-out timer and this is an implementation
  choice.  The ARP request gets sent both locally to all the attached
  TSes on that subnet as well as it gets sent to all the remote NVEs
  (including the target NVE) participating in that subnet.  The source
  NVE also withdraw the EVPN MAC/IP Advertisement route with only the
  MAC address (if it has previously advertised such a route).
```

Wouldn’t the source NVE only withdraw the route after a timeout had expired? As you have written this paragraph, in case the silent TS has not moved, the following would happen:

```
Time t: age-out timer fires, ARP probe is sent
Time t: NVE withdraws route advertisement
Time u > t: TS receives ARP probe, sends ARP reply
Time v > u: NVE receives ARP reply
Time v: NVE re-advertises route
```

Presumably this churn isn’t what you intended.

18. Section 9.2

How does the NVE learn what subnets are behind its attached TS?

19. Section 9.2

What about if TS4 wants to reach SN1? How does it know where to send the packet? (I suppose the answer may be the same as for #18.)

[Ballot comment]
Below are a number of questions and comments that I hope might help improve the document. I haven't chosen to make them blocking by including them in my DISCUSS; nonetheless I would appreciate replies to them.

1. I agree with the comments by several of the other reviewers, that there are just too many gratuitous acronyms in this document. They aren't the only thing that makes it hard to read, but they certainly contribute. I'm disappointed to see this hasn't been addressed between versions -09 and -14. It would have been a small matter of search-and-replace to go through and expand most of the acronyms.

2. Section 2

```
  R1: The solution must allow for both inter-subnet and intra-subnet
  traffic belonging to the same tenant to be locally routed and bridged
  respectively.  The solution must provide IP routing for inter-subnet
  traffic and Ethernet Bridging for intra-subnet traffic.  It should be
  noted that if an IP-VRF in a NVE is configured for IPv6 and that NVE
  receives IPv4 traffic on the corresponding VLAN, then the IPv4
  traffic is treated as L2 traffic and it is bridged.  Also vise versa,
  if an IP-VRF in a NVE is configured for IPv4 and that NVE receives
  IPv6 traffic on the corresponding VLAN, then the IPv6 traffic is
  treated as L2 traffic and it is bridged.

  R2: The solution must support bridging for non-IP traffic.
```

R1 is a little tortured, where you add all the caveats about “treated as L2 traffic”. Seems to me like it would fall out more naturally if you had simply introduced the concepts of routable and non-routable traffic, where routable traffic is that for which a suitable IP-VRF exists. That would also have the pleasant effect of making R2 say “… must support bridging for non-routable traffic” instead of “non-IP traffic”, which is technically incorrect (since per R1 you might have non-routable IP traffic).

```
  R3: The solution must allow inter-subnet switching to be disabled on
  a per VLAN basis on PEs where the traffic needs to be backhauled to
  another node (i.e., for performing FW or DPI functionality).
```

What’s “switching”? The document is about routing vs. bridging, which do you mean? I think you mean “routing”. IMO you should get rid of the word “switching” and replace with something less ambiguous, e.g. “routing”. (Both here and the one other place in the doc where you use “switching”.)

Also, I think you don’t mean “i.e.”, I think you mean “e.g.”. The meaning of “i.e.” is “in other words”. The meaning of “e.g.” is “for example”. The best way to avoid these problems, IMO, is to simply write out what you mean, so in this case write “(for example, for performing FW or DPI functionality).” (And oh by the way, you haven’t defined or expanded FW or DPI, please do so.)

3. Section 4

```
  o  references to ARP table in the context of asymmetric IRB is a
      logical view of a forwarding table that maintains an IP to MAC
      binding entry on a layer 3 interface for both IPv4 and IPv6.
      These entries are not subject to ARP or ND protocol.
```

This passage shines a spotlight on the fact that “ARP table” as it’s used in this document is a misnomer, since it’s a table that is not (necessarily) populated by ARP. I don’t propose that you change the nomenclature, since it’s firmly established even though wrong — but it might be worth adding the first sentence or one like it to your Terminology section.

4. Section 4

Figure 2 depicts BT2 being present on the ingress PE, but the text makes it clear that in the symmetric mode that this figure depicts, BT2 doesn’t actually need to be there. Wouldn’t it be clearer if you didn’t show it?

5. Section 4

I have a hard time parsing this text:

```
                                              Each BT on a PE is
  associated with a unique VLAN (e.g., with a BD)
```

So, 1 VLAN —> at least 1 BT (1:many)

```
                                                  where in turn it is
  associated with a single MAC-VRF
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

```
                                    in the case of VLAN-Based mode or a
  number of BTs can be associated with a single MAC-VRF in the case of
  VLAN-Aware Bundle mode. 
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

Since this is stated as an exception I guess that means you meant the preceding two (that I parsed as 1:many) are actually supposed to be 1:1? If so I think this needs a rewrite (it probably does regardless, for clarity).

6. Section 4.1

When you write “Internet standard bit order“, do you mean “network byte order“? Although even network byte order appears to be non-applicable, since the values are shown with an explicit byte order.

I realize the definitions are merely pasted from RFC 5798 and that ship has sailed, but unless you can explain what “(in hex, in Internet standard bit-order)” is supposed to mean, I suggest removing it. (Alternately and less desirably, make it explicit that you’re providing a direct quotation of RFC 5798.

7. Section 5.1

You say the Encapsulation Extended Community and Router’s MAC Extended Community have to be sent, but you say nothing about the required values. For Router's MAC, §8.1 specifies the required value, I suggest a forward reference to it. For Encapsulation, the closest I was able to find to a place where this is specified was section 9.1.1, but that's only an example. There really needs to be some place where it's spelled out. A bare minimum would be to cite RFC 9012 §4.1, but that just provides the syntax -- you really should say something more about how to decide what value to send. For that matter, it could be what valueS to send -- is it legal for a NVE to advertise multiple Encapsulation Extended Communities? You don't say it isn't, and there are potential reasons to do so.

8. Section 5.2

```
  o  Using MAC-VRF Route Target (and Ethernet Tag if different from
      zero), it identifies the corresponding MAC-VRF (and BT).  If the
      MAC- VRF (and BT) exists (e.g., it is locally configured) then it
```

You use “e.g.” so I presume there might be other reasons the MAC-VRF and BT might exist even if not locally configured?

```
      imports the MAC address into it.  Otherwise, it does not import
      the MAC address.

  o  Using IP-VRF route target, it identifies the corresponding IP-VRF
      and imports the IP address into it.
```

You don’t provide any conditional language in this bullet about “if the IP-VRF exists”. Why is that caveat required for MAC-VRF but not for IP-VRF?

9. Section 5.2

```
  The inclusion of MPLS label2 field in this route signals to the
  receiving PE that this route is for symmetric IRB mode and MPLS
  label2 needs to be installed in forwarding path to identify the
  corresponding IP-VRF.
```

I was unable to make head nor tail of this paragraph. I suppose §5.4 is where the behavior is actually specified, so in a way it doesn’t matter (although maybe a forward reference would help).

10. Section 5.2

```
  If the receiving PE receives this route with both the MAC-VRF and IP-
  VRF route targets and if the receiving PE does not support either
  asymmetric or symmetric IRB modes, then if it has the corresponding
  MAC-VRF, it only imports the MAC address.  Otherwise, if it doesn't
  have the corresponding MAC-VRF, it must not import this route.
```

If it doesn’t support either asymmetric or symmetric IRB modes, then doesn’t that mean it doesn’t implement this specification at all? In that circumstance, how do you expect your “must not” to be respected?

11. Section 5.3

```
                              If host B's (MAC, IP) has not yet been
  learnt either via a gratuitous ARP OR via a prior gleaning procedure,
  a new gleaning procedure MUST be triggered
```

Since you’ve used MUST here, you MUST provide a reference to where the “new gleaning procedure” is specified.

Also, has not been learnt by whom? The procedure must be triggered where?

12. Section 5.3

The second paragraph, that begins "Consider a subnet A", is tremendously confusing to a first-time reader (or at least to this first-time reader). I realize you probably think you're being helpful by providing a worked example, but as I read through it, it was the opposite of helpful. This is especially true because §5 and its subsections is about "Symmetric IRB Procedures" -- and the paragraph in question provides no procedures.

Some options to improve the situation --

- Remove the paragraph entirely.
- Preface the paragraph with "as an example to show why advertisement as RT-5 is required,"

13. Section 5.4

```
  o  global mode: VNI is set to the received label2 in the route which
      is domain-wide assigned.  This VNI value from received label2 MUST
      be the same as the locally configured VNI for the IP VRF as all
      PEs in the NVO MUST be configured with the same IP VRF VNI for
      this mode of operation.
```

What action is to be taken if this MUST is violated?

14. Section 6.1

```
  For asymmetric IRB mode, Router's MAC EC is not needed because
```

Please either expand “EC” or add it to your definitions section. (Also applies to 5.1)

15. Section 6.2

```
  o  If only MAC-VRF route target is used, then the receiving PE uses
      the MAC-VRF route target to identify the corresponding IP-VRF --
      i.e., many MAC-VRF route targets map to the same IP-VRF for a
      given tenant.  In this case, MAC-VRF may be used by the receiving
      PE to identify the corresponding IP VRF
```

Do you mean “in this case, the MAC-VRF *route target* may be used…”?

16. Section 6.2

```
  If the receiving PE receives the MAC/IP Advertisement route with MPLS
  label2 field and it uses symmetric IRB mode
```

This entire section is entitled “asymmetric IRB procedures“. Why is there specification language regarding symmetric procedures in it? (I’m pretty sure this is not the only place this kind of problem appears.)

17. Section 7.3

```
  On the source NVE, an age-out timer (for the silent host that has
  moved) is used to trigger an ARP probe.  This age-out timer can be
  either ARP timer or MAC age-out timer and this is an implementation
  choice.  The ARP request gets sent both locally to all the attached
  TSes on that subnet as well as it gets sent to all the remote NVEs
  (including the target NVE) participating in that subnet.  The source
  NVE also withdraw the EVPN MAC/IP Advertisement route with only the
  MAC address (if it has previously advertised such a route).
```

Wouldn’t the source NVE only withdraw the route after a timeout had expired? As you have written this paragraph, in case the silent TS has not moved, the following would happen:

```
Time t: age-out timer fires, ARP probe is sent
Time t: NVE withdraws route advertisement
Time u > t: TS receives ARP probe, sends ARP reply
Time v > u: NVE receives ARP reply
Time v: NVE re-advertises route
```

Presumably this churn isn’t what you intended.

18. Section 9.2

How does the NVE learn what subnets are behind its attached TS?

19. Section 9.2

What about if TS4 wants to reach SN1? How does it know where to send the packet? (I suppose the answer may be the same as for #18.)

[Ballot comment]
Below are a number of questions and comments that I hope might help improve the document. I haven't chosen to make them blocking by including them in my DISCUSS; nonetheless I would appreciate replies to them.

1. I agree with the comments by several of the other reviewers, that there are just too many gratuitous acronyms in this document. They aren't the only thing that makes it hard to read, but they certainly contribute. I'm disappointed to see this hasn't been addressed between version -09 and -14. It would have been a small matter of search-and-replace to go through and expand most of the acronyms.

2. Section 2

```
  R1: The solution must allow for both inter-subnet and intra-subnet
  traffic belonging to the same tenant to be locally routed and bridged
  respectively.  The solution must provide IP routing for inter-subnet
  traffic and Ethernet Bridging for intra-subnet traffic.  It should be
  noted that if an IP-VRF in a NVE is configured for IPv6 and that NVE
  receives IPv4 traffic on the corresponding VLAN, then the IPv4
  traffic is treated as L2 traffic and it is bridged.  Also vise versa,
  if an IP-VRF in a NVE is configured for IPv4 and that NVE receives
  IPv6 traffic on the corresponding VLAN, then the IPv6 traffic is
  treated as L2 traffic and it is bridged.

  R2: The solution must support bridging for non-IP traffic.
```

R1 is a little tortured, where you add all the caveats about “treated as L2 traffic”. Seems to me like it would fall out more naturally if you had simply introduced the concepts of routable and non-routable traffic, where routable traffic is that for which a suitable IP-VRF exists. That would also have the pleasant effect of making R2 say “… must support bridging for non-routable traffic” instead of “non-IP traffic”, which is technically incorrect (since per R1 you might have non-routable IP traffic).

```
  R3: The solution must allow inter-subnet switching to be disabled on
  a per VLAN basis on PEs where the traffic needs to be backhauled to
  another node (i.e., for performing FW or DPI functionality).
```

What’s “switching”? The document is about routing vs. bridging, which do you mean? I think you mean “routing”. IMO you should get rid of the word “switching” and replace with something less ambiguous, e.g. “routing”. (Both here and the one other place in the doc where you use “switching”.)

Also, I think you don’t mean “i.e.”, I think you mean “e.g.”. The meaning of “i.e.” is “in other words”. The meaning of “e.g.” is “for example”. The best way to avoid these problems, IMO, is to simply write out what you mean, so in this case write “(for example, for performing FW or DPI functionality).” (And oh by the way, you haven’t defined or expanded FW or DPI, please do so.)

3. Section 4

```
  o  references to ARP table in the context of asymmetric IRB is a
      logical view of a forwarding table that maintains an IP to MAC
      binding entry on a layer 3 interface for both IPv4 and IPv6.
      These entries are not subject to ARP or ND protocol.
```

This passage shines a spotlight on the fact that “ARP table” as it’s used in this document is a misnomer, since it’s a table that is not (necessarily) populated by ARP. I don’t propose that you change the nomenclature, since it’s firmly established even though wrong — but it might be worth adding the first sentence or one like it to your Terminology section.

4. Section 4

Figure 2 depicts BT2 being present on the ingress PE, but the text makes it clear that in the symmetric mode that this figure depicts, BT2 doesn’t actually need to be there. Wouldn’t it be clearer if you didn’t show it?

5. Section 4

I have a hard time parsing this text:

```
                                              Each BT on a PE is
  associated with a unique VLAN (e.g., with a BD)
```

So, 1 VLAN —> at least 1 BT (1:many)

```
                                                  where in turn it is
  associated with a single MAC-VRF
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

```
                                    in the case of VLAN-Based mode or a
  number of BTs can be associated with a single MAC-VRF in the case of
  VLAN-Aware Bundle mode. 
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

Since this is stated as an exception I guess that means you meant the preceding two (that I parsed as 1:many) are actually supposed to be 1:1? If so I think this needs a rewrite (it probably does regardless, for clarity).

6. Section 4.1

When you write “Internet standard bit order“, do you mean “network byte order“? Although even network byte order appears to be non-applicable, since the values are shown with an explicit byte order.

I realize the definitions are merely pasted from RFC 5798 and that ship has sailed, but unless you can explain what “(in hex, in Internet standard bit-order)” is supposed to mean, I suggest removing it. (Alternately and less desirably, make it explicit that you’re providing a direct quotation of RFC 5798.

7. Section 5.1

You say the Encapsulation Extended Community and Router’s MAC Extended Community have to be sent, but you say nothing about the required values. For Router's MAC, §8.1 specifies the required value, I suggest a forward reference to it. For Encapsulation, the closest I was able to find to a place where this is specified was section 9.1.1, but that's only an example. There really needs to be some place where it's spelled out. A bare minimum would be to cite RFC 9012 §4.1, but that just provides the syntax -- you really should say something more about how to decide what value to send. For that matter, it could be what valueS to send -- is it legal for a NVE to advertise multiple Encapsulation Extended Communities? You don't say it isn't, and there are potential reasons to do so.

8. Section 5.2

```
  o  Using MAC-VRF Route Target (and Ethernet Tag if different from
      zero), it identifies the corresponding MAC-VRF (and BT).  If the
      MAC- VRF (and BT) exists (e.g., it is locally configured) then it
```

You use “e.g.” so I presume there might be other reasons the MAC-VRF and BT might exist even if not locally configured?

```
      imports the MAC address into it.  Otherwise, it does not import
      the MAC address.

  o  Using IP-VRF route target, it identifies the corresponding IP-VRF
      and imports the IP address into it.
```

You don’t provide any conditional language in this bullet about “if the IP-VRF exists”. Why is that caveat required for MAC-VRF but not for IP-VRF?

9. Section 5.2

```
  The inclusion of MPLS label2 field in this route signals to the
  receiving PE that this route is for symmetric IRB mode and MPLS
  label2 needs to be installed in forwarding path to identify the
  corresponding IP-VRF.
```

I was unable to make head nor tail of this paragraph. I suppose §5.4 is where the behavior is actually specified, so in a way it doesn’t matter (although maybe a forward reference would help).

10. Section 5.2

```
  If the receiving PE receives this route with both the MAC-VRF and IP-
  VRF route targets and if the receiving PE does not support either
  asymmetric or symmetric IRB modes, then if it has the corresponding
  MAC-VRF, it only imports the MAC address.  Otherwise, if it doesn't
  have the corresponding MAC-VRF, it must not import this route.
```

If it doesn’t support either asymmetric or symmetric IRB modes, then doesn’t that mean it doesn’t implement this specification at all? In that circumstance, how do you expect your “must not” to be respected?

11. Section 5.3

```
                              If host B's (MAC, IP) has not yet been
  learnt either via a gratuitous ARP OR via a prior gleaning procedure,
  a new gleaning procedure MUST be triggered
```

Since you’ve used MUST here, you MUST provide a reference to where the “new gleaning procedure” is specified.

Also, has not been learnt by whom? The procedure must be triggered where?

12. Section 5.3

The second paragraph, that begins "Consider a subnet A", is tremendously confusing to a first-time reader (or at least to this first-time reader). I realize you probably think you're being helpful by providing a worked example, but as I read through it, it was the opposite of helpful. This is especially true because §5 and its subsections is about "Symmetric IRB Procedures" -- and the paragraph in question provides no procedures.

Some options to improve the situation --

- Remove the paragraph entirely.
- Preface the paragraph with "as an example to show why advertisement as RT-5 is required,"

13. Section 5.4

```
  o  global mode: VNI is set to the received label2 in the route which
      is domain-wide assigned.  This VNI value from received label2 MUST
      be the same as the locally configured VNI for the IP VRF as all
      PEs in the NVO MUST be configured with the same IP VRF VNI for
      this mode of operation.
```

What action is to be taken if this MUST is violated?

14. Section 6.1

```
  For asymmetric IRB mode, Router's MAC EC is not needed because
```

Please either expand “EC” or add it to your definitions section. (Also applies to 5.1)

15. Section 6.2

```
  o  If only MAC-VRF route target is used, then the receiving PE uses
      the MAC-VRF route target to identify the corresponding IP-VRF --
      i.e., many MAC-VRF route targets map to the same IP-VRF for a
      given tenant.  In this case, MAC-VRF may be used by the receiving
      PE to identify the corresponding IP VRF
```

Do you mean “in this case, the MAC-VRF *route target* may be used…”?

16. Section 6.2

```
  If the receiving PE receives the MAC/IP Advertisement route with MPLS
  label2 field and it uses symmetric IRB mode
```

This entire section is entitled “asymmetric IRB procedures“. Why is there specification language regarding symmetric procedures in it? (I’m pretty sure this is not the only place this kind of problem appears.)

17. Section 7.3

```
  On the source NVE, an age-out timer (for the silent host that has
  moved) is used to trigger an ARP probe.  This age-out timer can be
  either ARP timer or MAC age-out timer and this is an implementation
  choice.  The ARP request gets sent both locally to all the attached
  TSes on that subnet as well as it gets sent to all the remote NVEs
  (including the target NVE) participating in that subnet.  The source
  NVE also withdraw the EVPN MAC/IP Advertisement route with only the
  MAC address (if it has previously advertised such a route).
```

Wouldn’t the source NVE only withdraw the route after a timeout had expired? As you have written this paragraph, in case the silent TS has not moved, the following would happen:

```
Time t: age-out timer fires, ARP probe is sent
Time t: NVE withdraws route advertisement
Time u > t: TS receives ARP probe, sends ARP reply
Time v > u: NVE receives ARP reply
Time v: NVE re-advertises route
```

Presumably this churn isn’t what you intended.

18. Section 9.2

How does the NVE learn what subnets are behind its attached TS?

19. Section 9.2

What about if TS4 wants to reach SN1? How does it know where to send the packet? (I suppose the answer may be the same as for #18.)

[Ballot comment]
Below are a number of questions and comments that I hope might help improve the document. I haven't chosen to make them blocking by including them in my DISCUSS; nonetheless I would appreciate replies to them.

1. I agree with the comments by several of the other reviewers, that there are just too many gratuitous acronyms in this document. They aren't the only thing that makes it hard to read, but they certainly contribute. I'm disappointed to see this hasn't been addressed between version -09 and -14. It would have been a small matter of search-and-replace to go through and expand most of the acronyms.
2. Section 2

```
  R1: The solution must allow for both inter-subnet and intra-subnet
  traffic belonging to the same tenant to be locally routed and bridged
  respectively.  The solution must provide IP routing for inter-subnet
  traffic and Ethernet Bridging for intra-subnet traffic.  It should be
  noted that if an IP-VRF in a NVE is configured for IPv6 and that NVE
  receives IPv4 traffic on the corresponding VLAN, then the IPv4
  traffic is treated as L2 traffic and it is bridged.  Also vise versa,
  if an IP-VRF in a NVE is configured for IPv4 and that NVE receives
  IPv6 traffic on the corresponding VLAN, then the IPv6 traffic is
  treated as L2 traffic and it is bridged.

  R2: The solution must support bridging for non-IP traffic.
```

R1 is a little tortured, where you add all the caveats about “treated as L2 traffic”. Seems to me like it would fall out more naturally if you had simply introduced the concepts of routable and non-routable traffic, where routable traffic is that for which a suitable IP-VRF exists. That would also have the pleasant effect of making R2 say “… must support bridging for non-routable traffic” instead of “non-IP traffic”, which is technically incorrect (since per R1 you might have non-routable IP traffic).

```
  R3: The solution must allow inter-subnet switching to be disabled on
  a per VLAN basis on PEs where the traffic needs to be backhauled to
  another node (i.e., for performing FW or DPI functionality).
```

What’s “switching”? The document is about routing vs. bridging, which do you mean? I think you mean “routing”. IMO you should get rid of the word “switching” and replace with something less ambiguous, e.g. “routing”. (Both here and the one other place in the doc where you use “switching”.)

Also, I think you don’t mean “i.e.”, I think you mean “e.g.”. The meaning of “i.e.” is “in other words”. The meaning of “e.g.” is “for example”. The best way to avoid these problems, IMO, is to simply write out what you mean, so in this case write “(for example, for performing FW or DPI functionality).” (And oh by the way, you haven’t defined or expanded FW or DPI, please do so.)

3. Section 4

```
  o  references to ARP table in the context of asymmetric IRB is a
      logical view of a forwarding table that maintains an IP to MAC
      binding entry on a layer 3 interface for both IPv4 and IPv6.
      These entries are not subject to ARP or ND protocol.
```

This passage shines a spotlight on the fact that “ARP table” as it’s used in this document is a misnomer, since it’s a table that is not (necessarily) populated by ARP. I don’t propose that you change the nomenclature, since it’s firmly established even though wrong — but it might be worth adding the first sentence or one like it to your Terminology section.

4. Section 4

Figure 2 depicts BT2 being present on the ingress PE, but the text makes it clear that in the symmetric mode that this figure depicts, BT2 doesn’t actually need to be there. Wouldn’t it be clearer if you didn’t show it?

5. Section 4

I have a hard time parsing this text:

```
                                              Each BT on a PE is
  associated with a unique VLAN (e.g., with a BD)
```

So, 1 VLAN —> at least 1 BT (1:many)

```
                                                  where in turn it is
  associated with a single MAC-VRF
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

```
                                    in the case of VLAN-Based mode or a
  number of BTs can be associated with a single MAC-VRF in the case of
  VLAN-Aware Bundle mode. 
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

Since this is stated as an exception I guess that means you meant the preceding two (that I parsed as 1:many) are actually supposed to be 1:1? If so I think this needs a rewrite (it probably does regardless, for clarity).

6. Section 4.1

When you write “Internet standard bit order“, do you mean “network byte order“? Although even network byte order appears to be non-applicable, since the values are shown with an explicit byte order.

I realize the definitions are merely pasted from RFC 5798 and that ship has sailed, but unless you can explain what “(in hex, in Internet standard bit-order)” is supposed to mean, I suggest removing it. (Alternately and less desirably, make it explicit that you’re providing a direct quotation of RFC 5798.

7. Section 5.1

You say the Encapsulation Extended Community and Router’s MAC Extended Community have to be sent, but you say nothing about the required values. For Router's MAC, §8.1 specifies the required value, I suggest a forward reference to it. For Encapsulation, the closest I was able to find to a place where this is specified was section 9.1.1, but that's only an example. There really needs to be some place where it's spelled out. A bare minimum would be to cite RFC 9012 §4.1, but that just provides the syntax -- you really should say something more about how to decide what value to send. For that matter, it could be what valueS to send -- is it legal for a NVE to advertise multiple Encapsulation Extended Communities? You don't say it isn't, and there are potential reasons to do so.

8. Section 5.2

```
  o  Using MAC-VRF Route Target (and Ethernet Tag if different from
      zero), it identifies the corresponding MAC-VRF (and BT).  If the
      MAC- VRF (and BT) exists (e.g., it is locally configured) then it
```

You use “e.g.” so I presume there might be other reasons the MAC-VRF and BT might exist even if not locally configured?

```
      imports the MAC address into it.  Otherwise, it does not import
      the MAC address.

  o  Using IP-VRF route target, it identifies the corresponding IP-VRF
      and imports the IP address into it.
```

You don’t provide any conditional language in this bullet about “if the IP-VRF exists”. Why is that caveat required for MAC-VRF but not for IP-VRF?

9. Section 5.2

```
  The inclusion of MPLS label2 field in this route signals to the
  receiving PE that this route is for symmetric IRB mode and MPLS
  label2 needs to be installed in forwarding path to identify the
  corresponding IP-VRF.
```

I was unable to make head nor tail of this paragraph. I suppose §5.4 is where the behavior is actually specified, so in a way it doesn’t matter (although maybe a forward reference would help).

10. Section 5.2

```
  If the receiving PE receives this route with both the MAC-VRF and IP-
  VRF route targets and if the receiving PE does not support either
  asymmetric or symmetric IRB modes, then if it has the corresponding
  MAC-VRF, it only imports the MAC address.  Otherwise, if it doesn't
  have the corresponding MAC-VRF, it must not import this route.
```

If it doesn’t support either asymmetric or symmetric IRB modes, then doesn’t that mean it doesn’t implement this specification at all? In that circumstance, how do you expect your “must not” to be respected?

11. Section 5.3

```
                              If host B's (MAC, IP) has not yet been
  learnt either via a gratuitous ARP OR via a prior gleaning procedure,
  a new gleaning procedure MUST be triggered
```

Since you’ve used MUST here, you MUST provide a reference to where the “new gleaning procedure” is specified.

Also, has not been learnt by whom? The procedure must be triggered where?

12. Section 5.3

The second paragraph, that begins "Consider a subnet A", is tremendously confusing to a first-time reader (or at least to this first-time reader). I realize you probably think you're being helpful by providing a worked example, but as I read through it, it was the opposite of helpful. This is especially true because §5 and its subsections is about "Symmetric IRB Procedures" -- and the paragraph in question provides no procedures.

Some options to improve the situation --

- Remove the paragraph entirely.
- Preface the paragraph with "as an example to show why advertisement as RT-5 is required,"

13. Section 5.4

```
  o  global mode: VNI is set to the received label2 in the route which
      is domain-wide assigned.  This VNI value from received label2 MUST
      be the same as the locally configured VNI for the IP VRF as all
      PEs in the NVO MUST be configured with the same IP VRF VNI for
      this mode of operation.
```

What action is to be taken if this MUST is violated?

14. Section 6.1

```
  For asymmetric IRB mode, Router's MAC EC is not needed because
```

Please either expand “EC” or add it to your definitions section. (Also applies to 5.1)

15. Section 6.2

```
  o  If only MAC-VRF route target is used, then the receiving PE uses
      the MAC-VRF route target to identify the corresponding IP-VRF --
      i.e., many MAC-VRF route targets map to the same IP-VRF for a
      given tenant.  In this case, MAC-VRF may be used by the receiving
      PE to identify the corresponding IP VRF
```

Do you mean “in this case, the MAC-VRF *route target* may be used…”?

16. Section 6.2

```
  If the receiving PE receives the MAC/IP Advertisement route with MPLS
  label2 field and it uses symmetric IRB mode
```

This entire section is entitled “asymmetric IRB procedures“. Why is there specification language regarding symmetric procedures in it? (I’m pretty sure this is not the only place this kind of problem appears.)

17. Section 7.3

```
  On the source NVE, an age-out timer (for the silent host that has
  moved) is used to trigger an ARP probe.  This age-out timer can be
  either ARP timer or MAC age-out timer and this is an implementation
  choice.  The ARP request gets sent both locally to all the attached
  TSes on that subnet as well as it gets sent to all the remote NVEs
  (including the target NVE) participating in that subnet.  The source
  NVE also withdraw the EVPN MAC/IP Advertisement route with only the
  MAC address (if it has previously advertised such a route).
```

Wouldn’t the source NVE only withdraw the route after a timeout had expired? As you have written this paragraph, in case the silent TS has not moved, the following would happen:

```
Time t: age-out timer fires, ARP probe is sent
Time t: NVE withdraws route advertisement
Time u > t: TS receives ARP probe, sends ARP reply
Time v > u: NVE receives ARP reply
Time v: NVE re-advertises route
```

Presumably this churn isn’t what you intended.

18. Section 9.2

How does the NVE learn what subnets are behind its attached TS?

19. Section 9.2

What about if TS4 wants to reach SN1? How does it know where to send the packet? (I suppose the answer may be the same as for #18.)

[Ballot discuss]
I found this document difficult to review. Some of this might be due to the fact that I'm not an expert on EVPN, but I think some of the reason is that the document could be structured better and expressed more clearly. The only reason I'm not opposing progression of the document on the grounds that it's too unclear to implement is that I've been told, and accept on faith, that implementations *have* been successfully written starting from the spec, which implies it's implementable -- I guess by people who are expert in EVPN already, it wouldn't be implementable by me.

In any case, I do have some points I would like to discuss, that are more actionable.

1. I agree with Robert Wilton's comment on -09:

```
One question I have is whether it is possible to have a deployment where some devices support synchronous mode and others support asynchronous mode.  Am I right in presuming that this is not supported and if so is this capability signaled in any way? Or is the expectation that this would be controlled via deployment choice of network device, or though configuration management?
```

This issue still exists in -14. I think it should be addressed in the document. Similarly, I agree with Warren Kumari's comment, also on -09:

```
I would strongly recommend that the authors read the OpsDir review at: https://datatracker.ietf.org/doc/review-ietf-bess-evpn-inter-subnet-forwarding-09-opsdir-lc-jaeggli-2020-07-06/ , especially the: "it would be helpful if section 4 would be more explicit for non-implementors on when symetric or asymetric modules would be chosen, as it stands the variation basically reads like the enumeration of the features of various implementations." comment (which I fully agree with).
```

It seems both of these comments could -- and should! -- be addressed by adding a few paragraphs talking about these topics. This could be done either in §4, as Warren suggests, or in some other section (e.g. you could add an "operational considerations" section).

2. Section 7.1

I’m guessing this question isn’t unique to this document, but since this is where I encountered it, I’ll ask: it seems as though the described mobility procedures are vulnerable to a condition where a particular (IP, MAC) appears at two different NVEs at the same time. If this condition exists (either innocently, or maliciously) what prevents the source and target NVEs from continually attempting to claim the (IP, MAC) from one another, flooding the network with updates all the while?

(This applies to 7.2 as well.)

Since this seems like a potential security issue, I'm including it in my DISCUSS.

[Ballot discuss]
I found this document difficult to review. Some of this might be due to the fact that I'm not an expert on EVPN, but I think some of the reason is that the document could be structured better and expressed more clearly. The only reason I'm not opposing the progression of the document on the grounds that it's too unclear to implement is that I've been told, and accept on faith, that implementations *have* been successfully written starting from the spec, which implies it's implementable -- I guess by people who are expert in EVPN already, it wouldn't be implementable by me.

In any case, I do have some points I would like to discuss, that are more actionable.

1. I agree with Robert Wilton's comment on -09:

```
One question I have is whether it is possible to have a deployment where some devices support synchronous mode and others support asynchronous mode.  Am I right in presuming that this is not supported and if so is this capability signaled in any way? Or is the expectation that this would be controlled via deployment choice of network device, or though configuration management?
```

This issue still exists in -14. I think it should be addressed in the document. Similarly, I agree with Warren Kumari's comment, also on -09:

```
I would strongly recommend that the authors read the OpsDir review at: https://datatracker.ietf.org/doc/review-ietf-bess-evpn-inter-subnet-forwarding-09-opsdir-lc-jaeggli-2020-07-06/ , especially the: "it would be helpful if section 4 would be more explicit for non-implementors on when symetric or asymetric modules would be chosen, as it stands the variation basically reads like the enumeration of the features of various implementations." comment (which I fully agree with).
```

It seems both of these comments could -- and should! -- be addressed by adding a few paragraphs talking about these topics. This could be done either in §4, as Warren suggests, or in some other section (e.g. you could add an "operational considerations" section).

2. Section 7.1

I’m guessing this question isn’t unique to this document, but since this is where I encountered it, I’ll ask: it seems as though the described mobility procedures are vulnerable to a condition where a particular (IP, MAC) appears at two different NVEs at the same time. If this condition exists (either innocently, or maliciously) what prevents the source and target NVEs from continually attempting to claim the (IP, MAC) from one another, flooding the network with updates all the while?

(This applies to 7.2 as well.)

Since this seems like a potential security issue, I'm including it in my DISCUSS.

[Ballot comment]
Below are a number of questions and comments that I hope might help improve the document. I haven't chosen to make them blocking by including them in my DISCUSS; nonetheless I would appreciate replies to them.

1. I agree with the comments by several of the other reviewers, that there are just too many gratuitous acronyms in this document. They aren't the only thing that makes it hard to read, but they certainly contribute. I'm disappointed to see this hasn't been addressed between version -09 and -14. It would have been a small matter of search-and-replace to go through and expand most of the acronyms.
2. Section 2

```
  R1: The solution must allow for both inter-subnet and intra-subnet
  traffic belonging to the same tenant to be locally routed and bridged
  respectively.  The solution must provide IP routing for inter-subnet
  traffic and Ethernet Bridging for intra-subnet traffic.  It should be
  noted that if an IP-VRF in a NVE is configured for IPv6 and that NVE
  receives IPv4 traffic on the corresponding VLAN, then the IPv4
  traffic is treated as L2 traffic and it is bridged.  Also vise versa,
  if an IP-VRF in a NVE is configured for IPv4 and that NVE receives
  IPv6 traffic on the corresponding VLAN, then the IPv6 traffic is
  treated as L2 traffic and it is bridged.

  R2: The solution must support bridging for non-IP traffic.
```

R1 is a little tortured, where you add all the caveats about “treated as L2 traffic”. Seems to me like it would fall out more naturally if you had simply introduced the concepts of routable and non-routable traffic, where routable traffic is that for which a suitable IP-VRF exists. That would also have the pleasant effect of making R2 say “… must support bridging for non-routable traffic” instead of “non-IP traffic”, which is technically incorrect (since per R1 you might have non-routable IP traffic).

```
  R3: The solution must allow inter-subnet switching to be disabled on
  a per VLAN basis on PEs where the traffic needs to be backhauled to
  another node (i.e., for performing FW or DPI functionality).
```

What’s “switching”? The document is about routing vs. bridging, which do you mean? I think you mean “routing”. IMO you should get rid of the word “switching” and replace with something less ambiguous, e.g. “routing”. (Both here and the one other place in the doc where you use “switching”.)

Also, I think you don’t mean “i.e.”, I think you mean “e.g.”. The meaning of “i.e.” is “in other words”. The meaning of “e.g.” is “for example”. The best way to avoid these problems, IMO, is to simply write out what you mean, so in this case write “(for example, for performing FW or DPI functionality).” (And oh by the way, you haven’t defined or expanded FW or DPI, please do so.)

2. Section 4

```
  o  references to ARP table in the context of asymmetric IRB is a
      logical view of a forwarding table that maintains an IP to MAC
      binding entry on a layer 3 interface for both IPv4 and IPv6.
      These entries are not subject to ARP or ND protocol.
```

This passage shines a spotlight on the fact that “ARP table” as it’s used in this document is a misnomer, since it’s a table that is not (necessarily) populated by ARP. I don’t propose that you change the nomenclature, since it’s firmly established even though wrong — but it might be worth adding the first sentence or one like it to your Terminology section.

3. Section 4

Figure 2 depicts BT2 being present on the ingress PE, but the text makes it clear that in the symmetric mode that this figure depicts, BT2 doesn’t actually need to be there. Wouldn’t it be clearer if you didn’t show it?

4. Section 4

I have a hard time parsing this text:

```
                                              Each BT on a PE is
  associated with a unique VLAN (e.g., with a BD)
```

So, 1 VLAN —> at least 1 BT (1:many)

```
                                                  where in turn it is
  associated with a single MAC-VRF
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

```
                                    in the case of VLAN-Based mode or a
  number of BTs can be associated with a single MAC-VRF in the case of
  VLAN-Aware Bundle mode. 
```

So, 1 MAC-VRF —> at least 1 BT (1:many)

Since this is stated as an exception I guess that means you meant the preceding two (that I parsed as 1:many) are actually supposed to be 1:1? If so I think this needs a rewrite (it probably does regardless, for clarity).

5. Section 4.1

When you write “Internet standard bit order“, do you mean “network byte order“? Although even network byte order appears to be non-applicable, since the values are shown with an explicit byte order.

I realize the definitions are merely pasted from RFC 5798 and that ship has sailed, but unless you can explain what “(in hex, in Internet standard bit-order)” is supposed to mean, I suggest removing it. (Alternately and less desirably, make it explicit that you’re providing a direct quotation of RFC 5798.)

6. Section 5.1

You say the Encapsulation Extended Community and Router’s MAC Extended Community have to be sent, but you say nothing about the required values. For Router's MAC, §8.1 specifies the required value, I suggest a forward reference to it. For Encapsulation, the closest I was able to find to a place where this is specified was section 9.1.1, but that's only an example. There really needs to be some place where it's spelled out. A bare minimum would be to cite RFC 9012 §4.1, but that just provides the syntax -- you really should say something more about how to decide what value to send. For that matter, it could be what valueS to send -- is it legal for a NVE to advertise multiple Encapsulation Extended Communities? You don't say it isn't, and there are potential reasons to do so.

7. Section 5.2

```
  o  Using MAC-VRF Route Target (and Ethernet Tag if different from
      zero), it identifies the corresponding MAC-VRF (and BT).  If the
      MAC- VRF (and BT) exists (e.g., it is locally configured) then it
```

You use “e.g.” so I presume there might be other reasons the MAC-VRF and BT might exist even if not locally configured?

```
      imports the MAC address into it.  Otherwise, it does not import
      the MAC address.

  o  Using IP-VRF route target, it identifies the corresponding IP-VRF
      and imports the IP address into it.
```

You don’t provide any conditional language in this bullet about “if the IP-VRF exists”. Why is that caveat required for MAC-VRF but not for IP-VRF?

8. Section 5.2

```
  The inclusion of MPLS label2 field in this route signals to the
  receiving PE that this route is for symmetric IRB mode and MPLS
  label2 needs to be installed in forwarding path to identify the
  corresponding IP-VRF.
```

I was unable to make head nor tail of this paragraph. I suppose §5.4 is where the behavior is actually specified, so in a way it doesn’t matter (although maybe a forward reference would help).

9. Section 5.2

```
  If the receiving PE receives this route with both the MAC-VRF and IP-
  VRF route targets and if the receiving PE does not support either
  asymmetric or symmetric IRB modes, then if it has the corresponding
  MAC-VRF, it only imports the MAC address.  Otherwise, if it doesn't
  have the corresponding MAC-VRF, it must not import this route.
```

If it doesn’t support either asymmetric or symmetric IRB modes, then doesn’t that mean it doesn’t implement this specification at all? In that circumstance, how do you expect your “must not” to be respected?

10. Section 5.3

```
                              If host B's (MAC, IP) has not yet been
  learnt either via a gratuitous ARP OR via a prior gleaning procedure,
  a new gleaning procedure MUST be triggered
```

Since you’ve used MUST here, you MUST provide a reference to where the “new gleaning procedure” is specified.

Also, has not been learnt by whom? The procedure must be triggered where?

11. Section 5.3

The second paragraph, that begins "Consider a subnet A", is tremendously confusing to a first-time reader (or at least to this first-time reader). I realize you probably think you're being helpful by providing a worked example, but as I read through it, it was the opposite of helpful. This is especially true because §5 and its subsections is about "Symmetric IRB Procedures" -- and the paragraph in question provides no procedures.

Some options to improve the situation --

- Remove the paragraph entirely.
- Preface the paragraph with "as an example to show why advertisement as RT-5 is required,"

12. Section 5.4

```
  o  global mode: VNI is set to the received label2 in the route which
      is domain-wide assigned.  This VNI value from received label2 MUST
      be the same as the locally configured VNI for the IP VRF as all
      PEs in the NVO MUST be configured with the same IP VRF VNI for
      this mode of operation.
```

What action is to be taken if this MUST is violated?

13. Section 6.1

```
  For asymmetric IRB mode, Router's MAC EC is not needed because
```

Please either expand “EC” or add it to your definitions section. (Also applies to 5.1)

14. Section 6.2

```
  o  If only MAC-VRF route target is used, then the receiving PE uses
      the MAC-VRF route target to identify the corresponding IP-VRF --
      i.e., many MAC-VRF route targets map to the same IP-VRF for a
      given tenant.  In this case, MAC-VRF may be used by the receiving
      PE to identify the corresponding IP VRF
```

Do you mean “in this case, the MAC-VRF *route target* may be used…”?

15. Section 6.2

```
  If the receiving PE receives the MAC/IP Advertisement route with MPLS
  label2 field and it uses symmetric IRB mode
```

This entire section is entitled “asymmetric IRB procedures“. Why is there specification language regarding symmetric procedures in it? (I’m pretty sure this is not the only place this kind of problem appears.)

16. Section 7.3

```
  On the source NVE, an age-out timer (for the silent host that has
  moved) is used to trigger an ARP probe.  This age-out timer can be
  either ARP timer or MAC age-out timer and this is an implementation
  choice.  The ARP request gets sent both locally to all the attached
  TSes on that subnet as well as it gets sent to all the remote NVEs
  (including the target NVE) participating in that subnet.  The source
  NVE also withdraw the EVPN MAC/IP Advertisement route with only the
  MAC address (if it has previously advertised such a route).
```

Wouldn’t the source NVE only withdraw the route after a timeout had expired? As you have written this paragraph, in case the silent TS has not moved, the following would happen:

```
Time t: age-out timer fires, ARP probe is sent
Time t: NVE withdraws route advertisement
Time u > t: TS receives ARP probe, sends ARP reply
Time v > u: NVE receives ARP reply
Time v: NVE re-advertises route
```

Presumably this churn isn’t intended.

18. Section 9.2

How does the NVE learn what subnets are behind its attached TS?

19. Section 9.2

What about if TS4 wants to reach SN1? How does it know where to send the packet? (I suppose the answer may be the same as for #18.)

[Ballot comment]
Thank you for the work on this document.

I have scanned the document for ART issues and found none. I only have two editorial comments.

Francesca


1. -----

FP: Please expand PE, DPI, FW (which maybe should be just replaced by forwarding), TTL, RD, on first use. PE also seem to belong in the terminology section, in my opinion.

2. -----

FP: nit - s/gratutious/gratuitous

[Ballot comment]
No objection as I didn't notice any transport related issues.

However, idnits check returned some outdated references see below -

  == Outdated reference: draft-ietf-idr-tunnel-encaps has been published as
    RFC 9012

  == Outdated reference: A later version (-05) exists of
    draft-ietf-bess-evpn-irb-extended-mobility-03

  == Outdated reference: A later version (-11) exists of
    draft-ietf-nvo3-vxlan-gpe-10

and downref

  ** Downref: Normative reference to an Informational RFC: RFC 7348

  ** Downref: Normative reference to an Informational RFC: RFC 7637

Some more nits I found as I read through:

-- Section 7 : s/expexted/expected, s/moblity/mobility

[Ballot comment]
No objection as I didn't notice any transport related issues.

However, idnits check returned some outdated references see below -

  == Outdated reference: draft-ietf-idr-tunnel-encaps has been published as
    RFC 9012

  == Outdated reference: A later version (-05) exists of
    draft-ietf-bess-evpn-irb-extended-mobility-03

  == Outdated reference: A later version (-11) exists of
    draft-ietf-nvo3-vxlan-gpe-10

and downref

  ** Downref: Normative reference to an Informational RFC: RFC 7348

  ** Downref: Normative reference to an Informational RFC: RFC 7637

and some more nits:

-- Section 7 : s/expexted/expected, s/moblity/mobility

[Ballot discuss]
This document provides semantics for the EVPN RT-2 "MPLS Label2"
field allocated in but undocumented by RFC 7432.  I believe that we need
to provide some "trail of breadcrumbs" from the IANA registry to this document
so that the semantics of the protocol field are discoverable.  This could take the
form of a direct reference from the IANA registry to this document, or by marking
this document as Updating RFC 7432, or some other mechanism that provides
the needed discoverability.

[Ballot comment]
I think there might still be complications for TSes with multiple IPv6
GUAs (which can be very normal is in line with BCP 204), but I guess
time will tell and other docs / future work will have a chance to
resolve any issues encountered.

[Ballot discuss]
I think draft-ietf-bess-evpn-irb-extended mobility needs to be a
normative reference, since "the procedures in [it] must be exercised"
incorporates its procedures by reference.  Similarly,
draft-ietf-idr-tunnel-encaps seems like a normative reference since we
require the RT-2 route used by this document to be advertised along with
the EC that indicates the tunnel type.

I still think we need to discuss whether this document Updates: 7432.
RFC 7432 specifies the layout and interpretation of the RT-2 (MAC-IP
Advertisement Route) EVPN NRLI, but we extend it in several ways (e.g.,
the Label1 and Label2 (which we spell "Label-1" and "Label-2",
unfortunately) are only MPLS labels in 7432, but we expand that to also
allow VNI values in those fields; likewise, 7432 gives no semantics at
all for Label2, but we define some).  I understand that 7432 only covers
the L2 case but this document adds mixed L2/L3 (IRB), and furthermore
that the IRB case can be inferred based on the contets of the fields in
the advertisement, *if you know how to look for them*.  But I still
can't shake the feeling that this document should either be added as an
additional reference for EVPN Route Type 2, or listed as Updating 7432,
in order to indicate that we provide more details for the
interpretation and contents of the RT-2 NLRI.

[Ballot comment]
I support the DISCUSS ballot position of Erik Kline

I support the DISCUSS ballot position of Alvaro Retana

I support the DISCUSS ballot position of Ben Kaduk

Not much to add to the feedback of my peer ADs.

** Please respond to the SECDIR feedback (and thank you Chris Lonvick for doing it!)

====
Thanks for addressing my COMMENT.

[Ballot comment]
I agree with the other ADs that this document seemed terse in places and won't repeat those same comments here.

One question I have is whether it is possible to have a deployment where some devices support synchronous mode and others support asynchronous mode.  Am I right in presuming that this is not supported and if so is this capability signaled in any way? Or is the expectation that this would be controlled via deployment choice of network device, or though configuration management?

Regards,
Rob

[Ballot comment]
I support the DISCUSS ballot position of Erik Kline

I support the DISCUSS ballot position of Alvaro Retana

I support the DISCUSS ballot position of Ben Kaduk

Not much to add to the feedback of my peer ADs.

** Please respond to the SECDIR feedback (and thank you Chris Lonvick for doing it!)

** Section 11.  As there is nothing documented to prevent this approach from being used across administrative domains with different policies (i.e., there is no applicability statement or normative language providing caution from being used outside of the commonly reference data center use case) or any security assumptions made about the elements involved, please reiterate that there are no inherent security services being provided to protect the traffic.  If this is desired it should provide through other means.

[Ballot discuss]
[ general ]

* Can you give an example of what happens to non-IPv4/IPv6 Ethernet packets
  received at the NVE/PE?  Do they get bridged, and if so how far?  What
  happens if a host in BT1 ARPs for IPv4 address associated with a TS in
  a different BT?

* Where there are multiple prefixes in an IP-VRF, is there a constraint that
  any other IP-VRF that contains one of the prefixes must contain all of them?
  Perhaps that's in 7432...?

[ sections 4, 5.4, 5.4, 6.3, 6.4, 9.1.2, 9.2.2 ]

* Please document what happens to IPv4 TTL/IPv6 Hop Limit values as they
  cross various NVEs/PEs.

[ section 7 ]

* Is there a reference for IRB-EXT-MOBILITY?

* The two statements:

  (1) "Although the language used in this section is for IPv4 ARP,
      it equally applies to IPv6 ND."

  (2) "If there is [a] many-to-one relationship such that there are many host
      IP addresses correspond[ing] to a single host MAC address ..., then to
      detect host mobility, the procedures in [IRB-EXT-MOBILITY] must be
      exercised..."

  are in direct conflict.  All IPv6 hosts having at least one non-link-local
  unicast address will have more than one IP address per MAC and this section,
  or even this document, would not apply?

[Ballot comment]
[ general ]

* I believe this is true, but can you just state here (not in the doc) that
  there are no multi-link subnets that can be created with this model as
  defined in RFC 4903? It seems like everything is as it should be, but just
  to double-check.

[ section 1 ]

* IP-VRF definition: s/VPN Routing.../Virtual Routing/?

[ section 3 ]

* 2nd to last paragraph: Is any of this still true for a setup that
  involves multiple IPv6 prefixes per BD, maybe I misunderstood
  (or maybe this assumed single prefix per broadcast domain w/ IPv4-only?).

  Perhaps avoid suggesting there's a 1:1 relationship and use phrases
  likes "at least as many X as there are Y" or something?

[ section 4 ]

* ARP table: a less IPv4-specific name, even though it's define to hold both
  IPv4 and IPv6 associations, would be "neighbor table".  That might be
  overloaded in routing contexts so no need to change it.

[Ballot comment]
I'm balloting NoObj - initially I had this as a DISCUSS, but others already have my issues in their positions, and so I will let them carry it :-)

Like other, I found the document a challenging read - it is very full of acronyms, run on sentences and misplaced commas. While one can figure out the meaning, it's hard to keep the big picture in mind when having to re-read sections.

I would strongly recommend that the authors read the OpsDir review at: https://datatracker.ietf.org/doc/review-ietf-bess-evpn-inter-subnet-forwarding-09-opsdir-lc-jaeggli-2020-07-06/ , especially the: "it would be helpful if section 4 would be more explicit for non-implementors on when symetric or asymetric modules would be chosen, as it stands the variation basically reads like the enumeration of the features of various implementations." comment (which I fully agree with).

[Ballot discuss]
(1) Possibly a "discuss discuss", but ...
if I'm understanding correctly, the symmetric IRB case over an Ethernet
NVO tunnel (not MPLS or IP NVO) described in this document is
introducing a new scenario where traffic using router (PE) MAC addresses
as source and destination is comingled on the same tunnel with traffic
using tenant system MAC addresses as source and destination.  This
places an obligation on the tunnel endpoints to properly isolate and
process such "internal" tunnel traffic without hampering the ability of
tenans systems to communicate.  In a world where tenant systems can
appear at any time, using previously unknown MAC addresses, this
represents a rather challenging problem: how will the PEs be able to
pick (and avertise) MAC addresses that they know will not conflict with
any present or future customer systems?  (A similar dilemma led to quite
a delay in the processing of draft-ietf-bfd-vxlan, which in that case
was resolved by limiting the BFD operation to just the "management VNI"
which is not subject to MAC address conflict with customer systems.)  In
this docuement's case, we seem to be using a "well-known"/reserved MAC
address range from RFC 5798; in principle, this should be enough to
avoid conflicts, if customer systems are known to not squat on this
reserved range.  However, this document has some text in Section 4.1
that indicates that there must be external knowledge that auto-derived
MAC addresses in the RFC 5798 ranges "[do] not collide with any customer
MAC address".  So I'm left uncertain whether or not this potential
problem is addressed or not.  (Also, I don't know if the limit on 255
distinct such MAC addresses presents a scaling issue.)

Also, is there any risk that the EVPN IRB setup might also want to use
VRRPv3, and thus collide on the MAC addresses in a different manner?

(1.1) I'm less sure whether there's a similar collision risk for IP
addresses -- we assign IP addresses to NVEs (e.g., for use as BGP Next
Hop addresses) and these are used as VTEP addresses when encapsulating
packets that are going inter-subnet.  I think that at this point in the
packet processing we may already know that we're in the the "IRB tunnel"
scope and that may be enough to de-conflict any potential IP address
collision between NVE and customer addresses.

(2) Section 7 appears to reference (in a normative fashion)
[IRB-EXT-MOBILITY] but there is no such reference.
Similarly (as Murray notes), there are a couple apparent references to
[TUNNEL-ENCAP] that are also arguably normative, but the target of the
reference is not forthcoming (and the IANA registry does not show a "BGP
Encapsulation Extended Community" that is supposedly defined by
[TUNNEL-ENCAP]).

There is also not a listed reference for [EVPN-PREFIX], though one could
perhaps surmise that it is intended to be
[I-D.ietf-bess-evpn-prefix-advertisement] (given that the latter does
allocate EVPN route type 5 for carrying IP Prefix routes, etx.).
However, given that this document has text like "MUST advertise local
subnet routes as RT-5", this needs to be a normative (not informative)
reference.  (We may also want to explicitly reference [EVPN-PREFIX]
where those normative requirements are made.)

(3) I'm not sure whether we are modifying the error-handling semantics
for RT-2 from what is specified in RFC 7432 (and, furthermore, whether
the changes are backwards-compatible).  If so, it seems like we may need
an Updates: relationship.  The text in question is in Section 9.1.1
(which itself seems problematic, since this section is advertised as an
(example) operational model/scenario):

  If the receiving NVE receives a RT-2 with only Label-1 and only a
  single Route Target corresponding to IP-VRF, or if it receives a RT-2
  with only a single Route Target corresponding to MAC-VRF but with
  both Label-1 and Label-2, or if it receives a RT-2 with MAC Address
  Length of zero, then it MUST treat the route as withdraw [RFC7606]
  and log an error message.

Are all of these treat-as-withdraw behaviors specified in RFC 7432?

(4) Let's discuss whether we need more generally a "backwards
compatibility" section (I mention a couple other places where this might
come into play, in the COMMENT).

[Ballot comment]
The shepherd writeup links to the IPR search page, which finds a
disclosure, yet also claims "no discussions".  In the absence of
explicit discussion, it's not clear that the WG actively concluded that
it's appropriate to publish despite the potential encumberance, though
I'm told that this is fairly common in the routing area (hence only a COMMENT).

Please use the BCP 14 boilerplate as given in RFC 8174 (i.e., including
"NOT RECOMMENDED" as a keyword).

I'm not entirely sure that the mobility procedures are fully specified
for producing correct operation, but I'm not confident enough in my
understanding of the setup to ballot Discuss over it.  I have lots of
inline comments for Section 7 (and subsections).

Section 2

  EVPN [RFC7432] provides an extensible and flexible multi-homing VPN
  solution over an MPLS/IP network for intra-subnet connectivity among
  Tenant Systems (TSes) and End Devices that can be physical or
  virtual; where an IP subnet is represented by an EVI for a VLAN-based
  service or by an (EVI, VLAN) for a VLAN-aware bundle service.

In the terminology section (Broadcast Domain definition) we considered
three classes of model: VLAN-bundle, VLAN-based, and VLAN-aware.  Should
we mention VLAN-bundle here as well?

  The inter-subnet communication is traditionally achieved at
  centralized L3 Gateway (L3GW) devices where all the inter-subnet
  forwarding are performed and all the inter-subnet communication
  policies are enforced.  When two TSes belonging to two different
[...]
  In order to overcome the drawback of centralized layer-3 GW approach,
  IRB functionality is needed on the PEs (also referred to as EVPN
  NVEs) attached to TSes in order to avoid inefficient forwarding of
  tenant traffic (i.e., avoid back-hauling and hair-pinning).  When a

It feels like there's a bit of jargon in here that wasn't captured in
the terminology section (or references).  None of it seems particularly
inscrutable to me, personally, but perhaps it's worth another thought.

  PE with IRB capability receives tenant traffic over an Attachment
  Circuit (AC), it can not only locally bridge the tenant intra-subnet
  traffic but also can locally route the tenant inter-subnet traffic on
  a packet by packet basis thus meeting the requirements for both intra
  and inter-subnet forwarding and avoiding non-optimum traffic
  forwarding associated with centralized layer-3 GW approach.

nit(?): mybe "avoiding the non-optimal traffic forwarding"?

Section 3

  BD).  If service interfaces for an EVPN PE are configured in VLAN-
  Based mode (i.e., section 6.1 of RFC7432), then there is only a
  single BT per MAC-VRF (per EVI) - i.e., there is only one tenant VLAN
  per EVI.  However, if service interfaces for an EVPN PE are
  configured in VLAN-Aware Bundle mode (i.e., section 6.3 of RFC7432),
  then there are several BTs per MAC-VRF (per EVI) - i.e., there are
  several tenant VLANs per EVI.

[Is VLAN-bundle excluded again here?  No need to repeat the response,
just noting it as similar.]

Section 4

I think Figures 2 and 3 would benefit from some visual indication that
the "top half" is in "IP space" but the "bottom half" is in "Ethernet
space".

  In symmetric IRB as shown in figure-2, the inter-subnet forwarding
  between two PEs is done between their associated IP-VRFs.  Therefore,
  the tunnel connecting these IP-VRFs can be either IP-only tunnel (in
  case of MPLS or GENEVE encapsulation) or Ethernet NVO tunnel (in case
  of VxLAN encapsulation).  [...]

[editorial]: there's nothing forbidding an Ethernet tunnel for this case
when using MPLS or GENEVE, right, it's just inefficient?  This text
could be read as saying that it's forbidden.

  to each BT via its associated IRB interface.  Each BT on a PE is
  associated with a unique VLAN (e.g., with a BD) where in turn is
  associated with a single MAC-VRF in case of VLAN-Based mode or a
  number of BTs can be associated with a single MAC-VRF in case of
  VLAN-Aware Bundle mode.  Whether the service interface on a PE is

nit: missing "it", probably for "where in turn it is associated".

  VLAN-Based or VLAN-Aware Bundle mode does not impact the IRB
  operation and procedures.  It mainly impacts the setting of Ethernet

[noting another non-mention of VLAN-bundle mode]

  tag field in EVPN BGP routes as described in [RFC7432].

Perhaps this is clear to most people, but I think I would benefit from a
section reference within 7432 or another sentence indicating how this
works.

Figure 4 has a line for "Mx/IPx" in PE1 but no corresponding line in
PE2.  Since the IP address of the VRF is anycase, we can presume that
PE2 also uses IPx, but I'm not entirely sure what the "Mx" represents
for PE1 and what the corresponding behavior for PE2 would be.

Section 4.1

  It is worth noting that if the applications that are running on the
  TSes are employing or relying on any form of MAC security, then
  either the first model (i.e. using anycast MAC address) should be
  used to ensure that the applications receive traffic from the same
  IRB interface MAC address that they are sending to, or if the second
  model is used, then the IRB interface MAC address MUST be the one
  used in the initial ARP reply or ND Neighbor Advertisement (NA)for
  that TS.

nit: the "either" here seems redundant.  There are only two choices, so
to say "either do (1) or, if you do (2), also do this" is equivalent to
just saying "if you do (2), also do this", optionally with a note that
(1) also works and is simpler.  E.g.,

% It is worth noting that if the applications that are running on the
% TSes are employing or relying on any form of MAC security, then when
% using the second model (using per-PE MAC addresses), the IRB interface
% MAC address MUST be the one used in the initial ARP reply or ND
% Neighbor Advertisement (NA) for that TS.

  Although both of these options are equally applicable to both
  [...]
  the VLANs for that tenant.  Furthermore, it simplifies the operation
  as there is no need for Default Gateway extended community
  advertisement and its associated MAC aliasing procedure.  Yet another
  advantage is that following host mobility, the host does not need to
  refresh the default GW ARP/ND entry.

We're really promoting option-1 here, almost to the extent that we
should ask if option-2 needs to be mentioned at all.  The only benefit I
can think of so far is that for option-1 you have to share MACsec keys
all over the place, which option-2 avoids (to be fair, this is a
significant advantage in some cases); are there other benefits?

  Where the last octet is generated based on a configurable Virtual
  Router ID (VRID, range 1-255)).  If not explicitly configured, the
  default value for the VRID octet is '01'.  Auto-derivation of the

We could perhaps not mix decimal and (presumed) hex in the same
paragraph.

  anycast MAC can only be used if there is certainty that the auto-
  derived MAC does not collide with any customer MAC address.

Per the Discuss point, how might such certainty be obtained?  I note
that there does not seem to be anything preventing new customer MAC
addresses from appearing at any time.  (It also seems like this risk of
collision would hold however the PE MAC addresses are selected?)

  Irrespective of using only the anycast address or both anycast and
  non-anycast addresses on the same IRB, when a TS sends an ARP request
  or ND Neighbor Solicitation (NS) to the PE that is attached to, the
  request is sent for the anycast IP address of the IRB interface
  associated with the TS's subnet and the reply will use anycast MAC
  address (in both Source MAC in the Ethernet header and Sender
  hardware address in the payload).  [...]

nit(?): I don't really see how the declarative "the [ARP or NS] request
is sent for the anycast IP address can place such a restriction on the
TS behavior; are we intending to instead say that "when a [ARP or NS]
request is sent for the anycast IP address of [the IRB interface], the
reply will use the anycase MAC address", with a more clear cause/effect
relationship?

  Traffic routed from IP-VRF1 to TS1 SHOULD use the anycast MAC address
  as source MAC address.

When would this SHOULD be violated?

Section 5.1

  When a PE (e.g., PE1 in figure 4 above) learns MAC and IP address of
  a TS (via an ARP request), it adds the MAC address to the

nit: in Section 6.1 this parenthetical is "e.g., via an ARP request".

  o  Route Distinguisher (RD), Ethernet Segment Identifier, Ethernet
      Tag ID, MAC Address Length, MAC Address, IP Address Length, IP
      Address, and MPLS Label1 fields MUST be set per [RFC7432] and
      [RFC8365].

(side note) I would have an easier time understanding what is needed if
we phrased this in terms of "the same values as used for [some existing
advertisement]", but I'm also not in the target audience for this
document, so maybe that is not a good idea.

  o  The MPLS Label2 field is set to either an MPLS label or a VNI
      corresponding to the tenant's IP-VRF.  In case of an MPLS label,
      this field is encoded as 3 octets, where the high-order 20 bits
      contain the label value.

How do I choose between MPLS label and VNI?

Section 5.2

  The inclusion of MPLS label2 field in this route signals to the
  receiving PE that this route is for symmetric IRB mode and MPLS
  label2 needs to be installed in forwarding path to identify the
  corresponding IP-VRF.

How does the receiving PE know whether the label2 field is a label or a
VNI?  (Maybe a section reference to RFC 8365 would be useful, not
necessarily right here, but perhaps from earlier in this document.)

  If the receiving PE receives this route with both the MAC-VRF and IP-
  VRF route targets but the MAC/IP Advertisement route does not include
  MPLS label2 field and if the receiving PE supports asymmetric IRB
  mode, then the receiving PE installs the MAC address in the
  corresponding MAC-VRF and (IP, MAC) association in the ARP table for
  that tenant (identified by the corresponding IP-VRF route target).

[a little surprising to see this given that Section 5 is about
"Symmetric IRB Procedures", but I guess it makes sense to say something,
so I shouldn't quibble about what form that something takes.]

  If the receiving PE receives this route with both the MAC-VRF and IP-
  VRF route targets and if the receiving PE does not support either
  asymmetric or symmetric IRB modes, then if it has the corresponding
  MAC-VRF, it only imports the MAC address; otherwise, if it doesn't
  have the corresponding MAC-VRF, it MUST not import this route.

I can't quite tell if this MUST NOT (nit: "NOT" should be capital, too)
is just re-expressing the preexisting behavior for non-IRB EVPN behavior
or if it's a new normative requirement.

Section 5.3

  only if that PE has locally attached hosts in that subnet.  In order
  to enable inter-subnet routing across PEs in a deployment where all
  subnets are not provisioned at all PEs participating in an EVPN IRB
  instance, PEs MUST advertise local subnet routes as RT-5.  These

nit: "where all subnets are not provisioned at all PEs participating"
has a slightly different meaning than "where not all subnets are
provisioned at all PEs participating".  I suspect the latter is what's
intended, though I am not 100% sure.

  Consider a subnet A that is locally attached to PE1 and subnet B that
  is locally attached to PE2 and to PE3.  Host A in subnet A, that is
  attached to PE1 initiates a data packet destined to host B in subnet
  B that is attached to PE3.  If host B's (MAC, IP) has not yet been

Could we perhaps use different names/symbols for subnets and hosts
within them?

  subnet B locally attached.  Once the packet is received at PE2 OR
  PE3, and the route lookup yields a glean result, an ARP request is
  triggered and flooded across the layer-2 overlay.  This ARP request
  would be received and replied to by host B, resulting in host B (MAC,
  IP) learning at PE3, and its advertisement across the EVPN network.

(side note) When I first read this, the "layer-2 overlay" (vs. "layer-3
overlay") confused me, but I think I figured it out.  To check: the ARP
request on the layer-2 overlay is needed in order to cover all of subnet
B (and thus, to meet the criterion that the packet from PE1 has to be
able to be processed whether it arrives at PE2 or PE3).  The ARP request
triggers the PE to which host B is attached in order to advertise the
EVPN route over BGP, so that PE1 learns about host B's (MAC, IP) and can
send directly to PE3 in the future.

Section 5.4

  routed.  Hence, the ingress PE performs an IP lookup in the
  associated IP-VRF table.  The lookup identifies BGP next hop of

(This is a lookup of the destination IP address, right?)

Also, nit: "the BGP next hop" (and "the egress PE" going into the next
line).

  label is set to the received label2 in the route.  In case of
  Ethernet NVO tunnel type, VNI may be set one of two ways:
  [...]
  PE's may be configured to operate in one of these two modes depending
  on the administrative domain boundaries across PEs participating in
  the NVO, and PE's capability to support downstream VNI mode.

Is it safe to mix these two ways in a single deployment, or does the
operator need to be careful to provide homogeneous configuration?

Section 6.2

  o  Using MAC-VRF route target, it identifies the corresponding MAC-

Can a non-zero Ethernet Tag also come into play here (as was the case
for Symmetric operation)?

      An implementation may choose to consolidate the lookup at the
      ingress PE's IP-VRF with the lookup at the ingress PE's
      destination subnet MAC-VRF.  Consideration for such consolidation
      of lookups is an implementation exercise and thus its
      specification is outside the scope of this document.

(This feels a little redundant with the earlier text in Section 4 (where
we described the operation with "collapsed" vs. "consolidate" here).)

  If the receiving PE receives the MAC/IP Advertisement route with MPLS
  label2 field and it can support symmetric IRB mode, then it should
  use the MAC-VRF route target to identify its corresponding MAC-VRF
  table and import the MAC address.  It should use the IP-VRF route
  target to identify the corresponding IP-VRF table and import the IP
  address, as specified in symmetric IRB handling.  It MUST NOT import
  (IP, MAC) association into its ARP table.

I don't think I understand why this is a MUST NOT, especially since we
do store the (IP, MAC) association in the ARP table in the previous
paragraph's case.

Section 7

The mobility procedures in Section 15 of RFC 7432 on paper don't have
very great security properties -- at any remote signal indicating a
mobility event for a given MAC address, a PE advertising reachability to
that MAC will withdraw the route.  (Though, this is probably something
of a "the entire system is trusted" effect.)  Perhaps local learning
would cause it to be reintrodced if the MAC in question had not actually
moved away, but there seems to be some risk of flapping and triggering
the MAC Duplication detection.  The concrete advice we give in Section
7.1 to send a local ARP probe is good, but how rigid does the sequencing
need to be amongst (receive EVPN MAC/IP Advertisement, send local ARP
probe/wait for response, and withdraw EVPN Mac/IP Advertisement)?  If
there was a way to avoid the need to withdraw+readvertise step, it seems
like that might be preferable.

Can we confirm that the backwards- and forwards-compatibility story is
okay for mixed deployments with "stock EVPN" vs. IRB-enhanced EVPN (this
document)?  It looks like we are using the MAC Mobility Extended
Community to signal both MAC and IP mobility events, but I am not sure
that I can reason through all the cases and confirm that the right thing
happens.

Also, do all NVEs need to be prepared to cope with all three classes of
TS behavior at all times?  That seems like something worth stating more
clearly.

Section 7.2

  If EVPN-IRB NVEs are configured to advertise MAC-only routes in
  addition to MAC-and-IP EVPN routes, then the following steps are
  taken:

Does this configuration need to be globally consistent across all PEs?
If so, where do we state this requirement?

  o  The target NVE upon learning this MAC address in data-plane,
      updates this MAC address entry in the corresponding MAC-VRF with
      the local adjacency information (e.g., local interface).  It also
      recognizes that this MAC has moved and initiates MAC mobility
      procedures per [RFC7432] by advertising an EVPN MAC/IP
      Advertisement route with only the MAC address filled in along with
      MAC Mobility Extended Community with the sequence number
      incremented by one.

The lead-in had "in addition to MAC-and-IP EVPN routes", but this
step only does the MAC EVPN route.  It looks like the MAC-and-IP route
will be added later, in the third step; would it make sense to switch
this to a numbered list and mention that the additional route will be
added in step 3?

      advertised such route previously).  Furthermore, it searches for
      the corresponding MAC-IP entry and sends an ARP probe for this
      (MAC,IP) pair.  The ARP request message is sent both locally to
      all attached TSes in that subnet as well as it is sent to other
      NVEs participating in that subnet including the target NVE.  Note

I think a comment about the sequencing of this ARP request, similar to
my comment above, may apply.  But here the ARP request is sent both
locally and remotely, whereas previously it was only needed locally.  I
confess I don't understand what the difference is between the cases that
makes it okay for the Section-7.1 scenario to only send the ARP probe
locally.

Also, what happens if we get a response back from the local TSes (in
addition to the actions from the target NVE covered in the next bullet
point)?

      that the PE would need to maintain a correlation between MAC and
      MAC-IP route entries in the MAC-VRF to accomplish this.

I find this "would need to" language very concerning.  If this is
required for correct operation, isn't it a MUST?

  o  All other remote NVE devices upon receiving the MAC/IP
      advertisement route with MAC Mobility extended community compare

Does this only happen when they receive the EVPN MAC/IP Advertisement
route with both MAC and IP address information (vs. the earlier one with
just MAC information)?

  If EVPN-IRB NVEs are configured not to advertise MAC-only routes,
  then upon receiving the first data packet, it learns the MAC address

(editorial) The description for the MAC,MAC+IP case is quite involved and
has many separate steps, but the MAC+IP-only case in this paragraph is
much simpler, and so the description for it does not have a parallel
structure to the MAC,MAC+IP case.  Perhaps splitting into subsections
might help make it clear that, despite the difference in complexity, the
procedures in question are actually quite parallel in the nature of what
role they play.

Section 7.3

  On the source NVE, an age-out timer (for the silent host that has
  moved) is used to trigger an ARP probe.  This age-out timer can be
  either ARP timer or MAC age-out timer and this is an implementation
  choice.  The ARP request gets sent both locally to all the attached
  TSes on that subnet as well as it gets sent to all the remote NVEs
  (including the target NVE) participating in that subnet.  The source
  NVE also withdraw the EVPN MAC/IP Advertisement route with only the
  MAC address (if it has previously advertised such a route).

Does this route withdraw occur at the same time that the ARP probe is
sent, or only if there is not a response?

Also, does this need for a timer-driven full broadcast ARP probe impose
any scaling limits on the number of subnets that can be joined via IRB
for intra-subnet traffic?  (I don't expect there to be fundamentally
different scaling properties than for more typical ARP usage, but it's a
question about whether the joining of subnets will cause the limit to be
reached before other scaling limits would be reached.)

  The target NVE passes the ARP request to its locally attached TSes
  and when it receives the ARP response, it updates its MAC-VRF, IP-
  VRF, and ARP table with the host (MAC, IP) and local adjacency
  information (e.g., local interface).  It also sends an EVPN MAC/IP
  advertisement route with both the MAC and IP address fields filled in
  along with MAC Mobility Extended Community with the sequence number
  incremented by one.

Do we need to mention the possibility of an EVPN MAC/IP Advertisement
route with only the MAC address here, as well?

  All other remote NVE devices upon receiving the MAC/IP Advertisement
  route route with MAC Mobility extended community compare the sequence

[this is the one with both MAC and IP filled in, right?]

Section 8.1

  This extended community is used to carry the PE's MAC address for
  symmetric IRB scenarios and it is sent with RT-2.

It's only needed for specifically the Ethernet NVO tunnel (not MPLS or
IP-only NVO) symmetric IRB scenarios, right?

Section 9

I see that we only give examples/operational models for the symmetric
IRB case and for scenarios with IP subnets behind tenant systems.  Is
there any desire to give a similar example for asymmetric IRB?

Section 9, 9.1

  procedures.  In the following scenarios, without loss of generality,
  it is assumed that a given tenant is represented by a single IP-VPN
  instance.  [...]
  In this scenario, without loss of generality, it is assumed that NVEs
  operate in VLAN-based service interface mode with one Bridge
  Table (BT) per MAC-VRF.  [...]

Just to check my understanding: the "more general" case would have,
e.g., multiple IP-VPN instsances for a single tenant, or ... multiple
BTs per MAC-VRF?  I guess for the former case the IP-VPN instances would
be logically separate (absent some central L3GW), but I'm not really
sure what the latter case would look like.  Though maybe the VLAN-aware
bundling case in the following sentences (with multiple BTs per MAC-VRF)
is exactly the generalization being assumed here?

Section 9.1.1

  o  If the route carries the new Router's MAC Extended Community, and
      if the receiving NVE uses Ethernet NVO tunnel, then the receiving
      NVE imports the IP address into IP-VRF with NVE's MAC address

What does the receiving NVE do if it uses Ethernet NVO tunnels but the
route does not carry the Router's MAC Extended Community?

  o  If the receiving NVE ration MPLS encapsulation, then the receiving
      NVE imports the IP address into IP-VRF with BGP Next Hop address

What does "ration" mean here?

  If the receiving NVE receives a RT-2 with only Label-1 and only a
  single Route Target corresponding to IP-VRF, or if it receives a RT-2
  with only a single Route Target corresponding to MAC-VRF but with
  both Label-1 and Label-2, or if it receives a RT-2 with MAC Address
  Length of zero, then it MUST treat the route as withdraw [RFC7606]
  and log an error message.

Is this changing the error handling for the existing RT-2 advertisement?
How is such a change backwards compatible?

Also, a "MUST log" without rate-limiting risks mandating a DoS channel.

Section 9.1.2

  o  On the egress NVE, if the packet arrives on Ethernet NVO tunnel
      (e.g., it is VxLAN encapsulated), then the NVO tunnel header is

(side note) I can't decide whether it's good or bad that we use VxLAN as
the example here vs. the explicit MPLS/etc. mentions in the previous
point.  On the one hand, it lets us mention all the various different
implementation technologies, but on the other hand we don't have a
coherent thread to tie the steps together.

Section 9.2

  their next hop.  The receiving NVEs perform recursive route
  resolution to resolve the subnet prefix with its associated ingress
  NVE so that they know which NVE to forward the packets to when they
  are destined for that subnet prefix.

nit(?): is the "ingress" part important here?  The "forward the packets
to" operation would seem to be describing a case where that NVE is
acting as egress, at least...

  and MAC-VRF3 across two NVEs.  There are four TSes associated with
  these three MAC-VRFs - i.e., TS1, TS5 are connected to MAC-VRF1 on

[there is no TS5 in the figure]

Section 9.2.1

  o  Label = 0

[draft-ietf-bess-evpn-prefix-advertisement seems to describe this field
as the "MPLS Label".]

  This RT-5 is advertised with one or more Route Targets that have been
  configured as "export route targets" of the IP-VRF from which the
  route is originated.

This is the only place in this document where we use the phrase "export
route target" and no reference is provided; please clarify what is
meant.

  o  It imports the IP prefix into its corresponding IP-VRF that is
      configured with an import RT that is one of the RTs being carried
      by the RT-5 route along with the IP address of the associated TS
      as its next hop.

The phrase "that is one of the RTs being carried by the RT-5 route"
seems like it leaves a significant degree of freedom to the receiving
NVE.  Do we want to make this more precisely specified?

Section 9.2.2

  The following description of the data-plane operation describes just
  the logical functions and the actual implementation may differ.  Lets
  consider data-plane operation when a host on SN1 sitting behind TS1
  wants to send traffic to a host sitting behind SN3 behind TS3.

  o  TS1 send a packet with MAC DA corresponding to the MAC-VRF1 IRB
      interface of NVE1, and VLAN-tag corresponding to MAC-VRF1.

[I guess the step where there's an ethernet frame from SN1 to TS1 is
boring and okay to skip?]

Section 11

I agree with the secdir reviewer's comment that incorporating by
reference the security considerations of the core protocol RFCs would be
worthwhile.

Similarly, a reminder seems appropriate (a la RFC 4365) that this VPN
scheme does not provide the usual quartet of security properties that we
ask about (confidentiality protection, source authentication, integrity
protection, replay protection), and that if such functionality is needed
it must be provided in some other manner.

  Furthermore, the security consideration for layer-3 routing is this
  document follows that of [RFC4365] with the exception for application

The Security Considerations of RFC 4365 notes that RFC 4111 provides a
template "that may be used to evaluate and summarize how a given PPVPN
approach (solution) measures up against the PPVPN Security Framework".
Given that the IP-layer inter-subnet routing introduced by this document
is in some sense a new L3VPN technology, would it be appropriate to fill
out that template as it applies here?  It's unfortunate that RFC 7432
does not itself fill out the template from RFC 4111, as it would be
useful to have that information readily available as well (though I
understand that the L2-only parts of the mechanims described in this
document are essentially unchanged from RFC 7432 and it is only our
responsibility to document otherwise-undocumented critical security
flaws).

I think we should note that the asymmetric IRB scheme leaves tenant MAC
addresses visible in cleartext (absent other cryptographic protections)
over the backbone/underlay, which has potential privacy consequences.

The mechanisms in this document bring additional exposure/usage for
per-NVE IP and MAC addresses, at least some of which may be (locally)
automatically generated in some cases; it might be appropriate to
discuss how the system would fail if there were collisions in these
identifiers between NVEs.

As alluded to in a couple of my previous comments, we may also wish to
discuss any scaling issues (and consequent DoS risks) that may arise due
to the need for NVEs to store additional state in ARP tables and
IP-VRFs.  (If such risks are negligible, then maybe not.)

There probably aren't any additional mobility-related security
considerations other than what RFC 7432 discusses (but mentioning
mobility concerns specifically may still be worthwhile).

Section 14.1

RFC 8214 is listed as a normative reference but not cited anywhere.
Should it be removed, or citations added?

Section 14.2

RFC 7606 seems like it should be normative, since we use it to define
what "treat the route as withdraw" means, which itself is behavior
mandated at a MUST level.

[Ballot discuss]
I'm balloting DISCUSS because the specification in §9.1.1 is not clear, and it is not in sync with draft-ietf-idr-tunnel-encaps.  [Some of the points below are not DISCUSS-worthy, but I'm including them here because they are related to the larger point.]


§9.1.1 talks about using the Encapsulation Extended Community *and* the Router's MAC Extended Community.  However, the requirement for these communities to appear together is not explicit anywhere.  What are the implications for only one of them being present?

The Router's MAC Extended Community "is only required when Ethernet NVO tunnel type is used".  It seems to me that normatively requiring the extended community is important in this case.

What exactly is the "Ethernet NVO tunnel type"?  §1 (Terminology) says that "Examples of this type of tunnels are VXLAN or GENEVE."  A standards track document should be specific about when something is required.  For example, I assume that it would also be required when using NVGRE.  The tunnel types are a finite number, so please be specific.

Where is the GENEVE tunnel type (to be used in the Encapsulation Extended Community) defined?  BTW, the [GENEVE] reference is also missing.

§4 has this text: "the tunnel connecting these IP-VRFs can be either IP-only tunnel (in case of MPLS or GENEVE encapsulation) or Ethernet NVO tunnel (in case of VxLAN encapsulation)."  It confuses me because of the apparent contradiction between GENEVE being an example of an Ethernet NVO tunnel type, but also (?) an IP-only tunnel in this case.

§4.2/draft-ietf-idr-tunnel-encaps mentions possible conflicts created by the Router's MAC Extended Community and how it may be ignored, but this document doesn't mention using the Encapsulation Sub-TLVs (also from draft-ietf-idr-tunnel-encaps) for the same function.  Can the same function be achieved with the Encapsulation Sub-TLVs?

"section 4.5 of [TUNNEL-ENCAP]" is mentioned a couple of times, but there is no §4.5 in draft-ietf-idr-tunnel-encaps, and there's no reference either.  Please remove the specific section number (to avoid becoming out of sync), and instead mention the Encapsulation Extended Community by name.  Add a Normative reference to draft-ietf-idr-tunnel-encaps.

[Ballot comment]
(1) What is the "Tunnel Type Extended Community"?  I'm assuming you mean the Encapsulation Extended Community.

(2) s/MUST treat the route as withdraw/MUST use the treat-as-withdraw approach

(3) I find the use of normative language (MUSTs) in the description of the requirements (§2) unnecessary.

(4) Nits:

s/In case of/In the case of/g

s/forwarding are performed/forwarding is performed

s/back hauled/backhauled/g

s/drawback of centralized/drawback of the centralized

s/relationship among these components/relationship between these components/g

s/can consists/can consist

s/a IP/an IP/g

s/a L3/an L3

s/used in description of/used to describe

s/in prior section/in the prior section

s/Since the packet forwarding is between ingress PE's...and egress PE's ...procedures follows/Because the packet forwarding is between the ingress PE's...and the egress PE's...procedures follow

s/provides different level/provides a different level

s/identifying bridge/identifying the bridge

s/in data plane/in the data plane

s/route route/route

s/with BGP Encapsulation/with the BGP Encapsulation

s/only L2 forwarding (bridging) part/only the L2 forwarding (bridging) part

s/a RT/an RT/g

s/RT-*/EVPN RT-*/g

s/Lets consider/Let's consider/g

s/using EVPN IP Prefix/using the EVPN IP Prefix

s/[EVPN-PREFIX]/[I-D.ietf-bess-evpn-prefix-advertisement]/g

s/Figure below illustrates this scenario/Figure 7 illustrates the scenario

s/as follow/as follows

s/as underlay tunnel/as the underlay tunnel

s/in access-facing/in an access-facing

s/using EVPN IP/using the EVPN IP

s/where, IP lookup/where an IP lookup

s/feedbacks/feedback

s/is this document/in this document

s/for application/for the application

s/MUST not/MUST NOT

"receiving NVE ration MPLS encapsulation"  ??

s/Label-1/Label1/g (for consistency with rfc7432)

s/Label-2/Label2/g

[Ballot comment]
Thank you for the work put into this document.

Please find below a couple of non-blocking COMMENTs (and I would appreciate a reply to each of my COMMENTs).

I hope that this helps to improve the document,

Regards,

-éric

PS: as a side note, I found that this document uses too many acronyms even for short words (e.g., "SN" instead of "Subnet"). There are also very long sentences that, when combined with acronyms, make reading difficult.


== COMMENTS ==

-- Section 2 --
About "to bridge non-IP and intra-subnet traffic and to route inter-subnet IP traffic": suggest to clarify the text when the IP-VRF is IPv6 only, then, (I assume) that IPv4 packets will be bridged and not IP-forwarded (and vice-versa).

-- Section 4.1 --
Suggest to replace "then the IRB interface MAC address MUST be the one used in the initial ARP reply or ND Neighbor Advertisement (NA) for that TS." by "then the IRB interface MAC address MUST be the one used in the initial ARP reply or ND Neighbor Advertisement (NA) or Router Advertisement (RA) for that TS" because routers MAC addresses are also advertised by Router Advertisements.

-- Section 5.1 --
Should also mention NDP when writing "(via an ARP request)" in the first paragraph.

In the same vein, please add "NDP cache" to "Furthermore, it adds this TS's MAC and IP address association to its ARP table".

As I am not an expert in EVPN, I am puzzled by the math about the Length field "either 40 (if IPv4 address is carried) or 52 (if IPv6 address is carried)."

-- Section 5.2 --
This section also only mentions IPv4 ARP table, please add IPv6 NDP cache.

-- Section 6.1 --
Same comments as for section 5.1

-- Section 6.2 --
Same comments as for section 5.2

-- Section 7 --
Good to state "Although the language used in this section is for IPv4 ARP, it equally applies to IPv6 ND."; even if I would have preferred to use by default IPv6 ND ;-)

Please note that in IPv6 there are often at least TWO IPv6 addresses per MAC (one link-local fe80::... and one global); so, "In the following subsections, it is assumed that the MAC and IP addresses of a TS have one-to-one relationship (i.e., there is one IP address per MAC address and vice versa). " is obviously never the case for IPv6. I understand that the rest of the paragraph explains how to handle the case but it could be easier to treat IPv6 in a separate sentence.

-- Section 7.1 --
While about mobility, this section appears to be also applicable to Duplicate Address Detection but is unclear on what to do when the same IP but different MAC (i.e., an actual IP address collision). Or is it covered in other documents?

== NITS ==

-- Section 1 --
"BD and subnet are equivalent terms" while in the rest of the document "IP subnet" is often used. If "subnet IP" and "subnet" are synonyms, then I suggest to keep using one for consistency or at least mention that "IP subnet" and "subnet" are the same concept (or explain the difference if they are not identical).

[Ballot comment]
Bigger stuff:

I'll see Benjamin's DISCUSS (meaning I support it) and raise him the two seemingly normative references to [TUNNEL-ENCAP], which is also undefined.

As someone not from the Routing Area, I found this a little hard to read because it becomes quite dense with acronyms in some places.

Is Section 13 intended to remain in the final published version?  Is it needed?

In the Abstract, please expand all acronyms on first use in the abstract, per the I-D Guidelines.

Lesser stuff:

Thanks for the glossary in Section 1.  I note that, although defined in the glossary here, the terms "DGW", "Ethernet A-D route", "GRE", "GW IP", "IPL", "ML", and "VXLAN" are not used anywhere else in this document.

There are lots of places where a single hyphen is used to break up a sentence, such as to introduce an example.  These should be em dashes, or commas, or maybe semicolons, but not simply hyphens.

A few nits:

Section 2:

* "... all the inter-subnet forwarding are performed ..." -- s/are/is/

* "... connected to the same PE, wanted to communicate ..." -- remove the comma

Section 3:

* "A MAC-VRF can consists of one ..." -- either drop "can", or use "consist"

(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)?  Why
is this the proper type of RFC?  Is this type of RFC indicated in the
title page header?

The document specifies two models of EVPN inter-subnet forwarding
with corresponding procedures and encodings, so it needs to be
a standard track document and is properly indicated so in the
title page header.

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

  EVPN provides an extensible and flexible multi-homing VPN solution
  over an MPLS/IP network for intra-subnet connectivity among Tenant
  Systems and End Devices that can be physical or virtual. However,
  there are scenarios for which there is a need for a dynamic and
  efficient inter-subnet connectivity among these Tenant Systems and
  End Devices while maintaining the multi-homing capabilities of EVPN.
  This document describes an Integrated Routing and Bridging (IRB)
  solution based on EVPN to address such requirements.

Working Group Summary

  The document went through a couple of rounds LCs with
  reviews/comments/revisions, after which onsensus was reached.

Document Quality

  There are implementations from major vendors and deployments by major
  operators. The document has reached WG consensus and passed WG LC.

Personnel

  Document Shepherd is Zhaohui (Jeffrey) Zhang.
  Responsible Area Director is Martin Vigoureux.

(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

Eric Rosen and Document shephered reviewed and provided substantive comments
during the first WG LC in February 2017. The issues have been addressed
and a second WG LC was done in August 2018. Another round of comments from
others have been addressed.


(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?

No.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

No.

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

No. All review comments have been addressed.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

Yes.

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

See https://datatracker.ietf.org/ipr/search/?id=draft-ietf-bess-evpn-inter-subnet-forwarding&submit=draft.
No discussions.

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

Strong interest, extensive invvolvement/review/comments and solid consensus
in the WG.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See https://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

IDnits have been addressed.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

No specific MIB Docotor, meida type and URI type review is done.

(13) Have all references within this document been identified as
either normative or informative?

Yes.

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

draft-ietf-idr-tunnel-encaps has passed WG LC. Waiting for shepherd write-up.
draft-ietf-bess-evpn-prefix-advertisement has been submitted to IESG.


(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

No.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

No.


(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

IANA assignment has already been done.

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

No new registries established in this document.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

Not applicable.

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Functions Operator has completed its review of draft-ietf-bess-evpn-inter-subnet-forwarding-09. If any part of this review is inaccurate, please let us know.

The IANA Functions Operator understands that, upon approval of this document, there is a single action which we must complete.

In the EVPN Extended Community Sub-Types registry on the Border Gateway Protocol (BGP) Extended Communities registry page located at:

https://www.iana.org/assignments/bgp-extended-communities/

the existing registration for:

Sub-Type Value: 0x03
Name: EVPN Router\u2019s MAC Extended Community

will have its reference changed to [ RFC-to-be ].

The IANA Functions Operator understands that this is the only action required to be completed upon approval of this document.

Note:  The actions requested in this document will not be completed until the document has been approved for publication as an RFC. This message is meant only to confirm the list of actions that will be performed.

Thank you,

Sabrina Tanamal
Senior IANA Services Specialist

The following Last Call announcement was sent out (ends 2020-07-03):

From: The IESG
To: IETF-Announce
CC: Zhaohui Zhang , bess-chairs@ietf.org, bess@ietf.org, martin.vigoureux@nokia.com, draft-ietf-bess-evpn-inter-subnet-forwarding@ietf.org, zzhang@juniper.net
Reply-To: last-call@ietf.org
Sender:
Subject: Last Call:  (Integrated Routing and Bridging in EVPN) to Proposed Standard


The IESG has received a request from the BGP Enabled ServiceS WG (bess) to
consider the following document: - 'Integrated Routing and Bridging in EVPN'
  as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2020-07-03. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


  EVPN provides an extensible and flexible multi-homing VPN solution
  over an MPLS/IP network for intra-subnet connectivity among Tenant
  Systems and End Devices that can be physical or virtual.  However,
  there are scenarios for which there is a need for a dynamic and
  efficient inter-subnet connectivity among these Tenant Systems and
  End Devices while maintaining the multi-homing capabilities of EVPN.
  This document describes an Integrated Routing and Bridging (IRB)
  solution based on EVPN to address such requirements.





The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-bess-evpn-inter-subnet-forwarding/


The following IPR Declarations may be related to this I-D:

  https://datatracker.ietf.org/ipr/3352/

(1) What type of RFC is being requested (BCP, Proposed Standard,
Internet Standard, Informational, Experimental, or Historic)?  Why
is this the proper type of RFC?  Is this type of RFC indicated in the
title page header?

The document specifies two models of EVPN inter-subnet forwarding
with corresponding procedures and encodings, so it needs to be
a standard track document and is properly indicated so in the
title page header.

(2) The IESG approval announcement includes a Document Announcement
Write-Up. Please provide such a Document Announcement Write-Up. Recent
examples can be found in the "Action" announcements for approved
documents. The approval announcement contains the following sections:

Technical Summary

  EVPN provides an extensible and flexible multi-homing VPN solution
  over an MPLS/IP network for intra-subnet connectivity among Tenant
  Systems and End Devices that can be physical or virtual. However,
  there are scenarios for which there is a need for a dynamic and
  efficient inter-subnet connectivity among these Tenant Systems and
  End Devices while maintaining the multi-homing capabilities of EVPN.
  This document describes an Integrated Routing and Bridging (IRB)
  solution based on EVPN to address such requirements.

Working Group Summary

  The document went through a couple of rounds LCs with
  reviews/comments/revisions, after which onsensus was reached.

Document Quality

  There are implementations from major vendors and deployments by major
  operators. The document has reached WG consensus and passed WG LC.

Personnel

  Document Shepherd is Zhaohui (Jeffrey) Zhang.
  Responsible Area Director is Alvaro Retana.

(3) Briefly describe the review of this document that was performed by
the Document Shepherd.  If this version of the document is not ready
for publication, please explain why the document is being forwarded to
the IESG.

Eric Rosen and Document shephered reviewed and provided substantive comments
during the first WG LC in February 2017. The issues have been addressed
and a second WG LC was done in August 2018. Another round of comments from
others have been addressed.


(4) Does the document Shepherd have any concerns about the depth or
breadth of the reviews that have been performed?

No.

(5) Do portions of the document need review from a particular or from
broader perspective, e.g., security, operational complexity, AAA, DNS,
DHCP, XML, or internationalization? If so, describe the review that
took place.

No.

(6) Describe any specific concerns or issues that the Document Shepherd
has with this document that the Responsible Area Director and/or the
IESG should be aware of? For example, perhaps he or she is uncomfortable
with certain parts of the document, or has concerns whether there really
is a need for it. In any event, if the WG has discussed those issues and
has indicated that it still wishes to advance the document, detail those
concerns here.

No. All review comments have been addressed.

(7) Has each author confirmed that any and all appropriate IPR
disclosures required for full conformance with the provisions of BCP 78
and BCP 79 have already been filed. If not, explain why.

Yes.

(8) Has an IPR disclosure been filed that references this document?
If so, summarize any WG discussion and conclusion regarding the IPR
disclosures.

See https://datatracker.ietf.org/ipr/search/?id=draft-ietf-bess-evpn-inter-subnet-forwarding&submit=draft.
No discussions.

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

Strong interest, extensive invvolvement/review/comments and solid consensus
in the WG.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See https://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

IDnits have been addressed.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

No specific MIB Docotor, meida type and URI type review is done.

(13) Have all references within this document been identified as
either normative or informative?

Yes.

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

draft-ietf-idr-tunnel-encaps has passed WG LC. Waiting for shepherd write-up.
draft-ietf-bess-evpn-prefix-advertisement has been submitted to IESG.


(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

No.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

No.


(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

IANA assignment has already been done.

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

No new registries established in this document.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

Not applicable.

https://datatracker.ietf.org/ipr/search/?id=draft-ietf-bess-evpn-inter-subnet-forwarding&submit=draft.
No discussions.

(9) How solid is the WG consensus behind this document? Does it
represent the strong concurrence of a few individuals, with others
being silent, or does the WG as a whole understand and agree with it? 

Strong interest, extensive invvolvement/review/comments and solid consensus
in the WG.

(10) Has anyone threatened an appeal or otherwise indicated extreme
discontent? If so, please summarise the areas of conflict in separate
email messages to the Responsible Area Director. (It should be in a
separate email because this questionnaire is publicly available.)

No.

(11) Identify any ID nits the Document Shepherd has found in this
document. (See https://www.ietf.org/tools/idnits/ and the Internet-Drafts
Checklist). Boilerplate checks are not enough; this check needs to be
thorough.

IDnits have been addressed.

(12) Describe how the document meets any required formal review
criteria, such as the MIB Doctor, media type, and URI type reviews.

No specific MIB Docotor, meida type and URI type review is done.

(13) Have all references within this document been identified as
either normative or informative?

Yes.

(14) Are there normative references to documents that are not ready for
advancement or are otherwise in an unclear state? If such normative
references exist, what is the plan for their completion?

draft-ietf-idr-tunnel-encaps has passed WG LC. Waiting for shepherd write-up.
draft-ietf-bess-evpn-prefix-advertisement has been submitted to IESG.


(15) Are there downward normative references references (see RFC 3967)?
If so, list these downward references to support the Area Director in
the Last Call procedure.

No.

(16) Will publication of this document change the status of any
existing RFCs? Are those RFCs listed on the title page header, listed
in the abstract, and discussed in the introduction? If the RFCs are not
listed in the Abstract and Introduction, explain why, and point to the
part of the document where the relationship of this document to the
other RFCs is discussed. If this information is not in the document,
explain why the WG considers it unnecessary.

No.


(17) Describe the Document Shepherd's review of the IANA considerations
section, especially with regard to its consistency with the body of the
document. Confirm that all protocol extensions that the document makes
are associated with the appropriate reservations in IANA registries.
Confirm that any referenced IANA registries have been clearly
identified. Confirm that newly created IANA registries include a
detailed specification of the initial contents for the registry, that
allocations procedures for future registrations are defined, and a
reasonable name for the new registry has been suggested (see RFC 5226).

IANA assignment has already been done.

(18) List any new IANA registries that require Expert Review for future
allocations. Provide any public guidance that the IESG would find
useful in selecting the IANA Experts for these new registries.

No new registries established in this document.

(19) Describe reviews and automated checks performed by the Document
Shepherd to validate sections of the document written in a formal
language, such as XML code, BNF rules, MIB definitions, etc.

Not applicable.

Request for posting confirmation emailed to previous authors: Lucy Yong , Samir Thoria , John Drake , Samer Salam , Jorge Rabadan , bess-chairs@ietf.org, Ali Sajassi

Request for posting confirmation emailed to previous authors: "Jorge Rabadan" , "Lucy Yong" , "Samir Thoria" , "John Drake" , "Samer Salam" , "Ali Sajassi" , bess-chairs@ietf.org

Request for posting confirmation emailed to previous authors: "Yakov Rekhter" , "Samer Salam" , "Linda Dunbar" , "Lucy Yong" , "Samir Thoria" , "John Drake" , "Ali Sajassi" , bess-chairs@ietf.org