Enterprise Multihoming using Provider-Assigned IPv6 Addresses without Network Prefix Translation: Requirements and Solutions
draft-ietf-rtgwg-enterprise-pa-multihoming-12

A few questions and comments:

(1) Section 1. Per “That is, some routers must be capable of some form of Source Address Dependent Routing (SADR), if only as described in [RFC3704]”, I’m not sure what minimal SADR technique is being described in RFC3704.  What section?

(2) Section 3.5.  Per “However, when evaluating scalable implementations of the solution, it would be reasonable to assume that the maximum number of ISPs that a site would connect to is five”, what is the basis of the up to _five_ ISPs?

(3) Section 5. Per “If all of the ISP uplinks are working, the choice of source address by the host may be driven by the desire to load share across ISP uplinks …”, how does an individual host have enough information to make that kind of decision?

(4) Section 5.  Per “An external host initiating communication with a host internal to a PA multihomed site will need to know multiple addresses  for that host in order to communicate with it using different ISPs to the multihomed site.”, what is mean by “in order to communicate with it using different ISPs”?  Why would being multi-homed this matter?  Wouldn’t one IP address be sufficient?
Section 9.  Per “It recommends that a given host verify the original packet header included into ICMPv6 error message was actually sent by the host itself”, is this guidance to the host or to network stack implementers?

(5) Section 9.  Given the current (helpful) text about the threat of a spoofed ICMPv6 message, it would be equally useful to discuss the threat to the other approach in Section 5 (i.e., DHCPv6) – a rogue DHCPv6 server.

A few editorial nits:

** Section 4. s/As as/As/

** Section 5.1.  Typo.  s/such as way/such a way/

** Section 5.2.3 Multiple Typos.  s/signalling/signaling/g 

** Section 5.2.3.  Typo.  s/An site/A site/

** Section 5.6.  Per the use of the term “wall garden”, do you mean “walled garden”?

** Section 5.6 and 5.7.1.  Multiple Typos.  s/envinronments /environments/g

** Section 6.  Typo.  s/mutiple/multiple

** Section 6.  Missing space. /[I-D.ietf-intarea-provisioning-domains]takes/[I-D.ietf-intarea-provisioning-domains] takes/

** Section 9.  Typo.  /mechanim/mechanism/

Thank you all for the work put into this document; this is an important network deployment use case. I second Alvaro's COMMENTs and I sincerely believe that a revised ID could fix a lot of small details in the text in order to improve the quality of the text (see my COMMENTs and NITs)

Also, this is mostly a tutorial / guide. I also wonder why it is in RTGWG rather than V6OPS as part of the document is not about SADR but source address selection. But, this is really a detail.

== COMMENTS ==

-- Abstract --

" a complete solution" while the document is not about a single solution but rather multiple of them.

-- Section 1 --

Unsure whether "one of the goals of IPv6 is to eliminate the need for and the use of NAT or NAPT" is true. Even, if I would hate to use NAT with IPv6, but, this was probably not a design goal for IPv6.

-- Section 1 --

Is there a reason why the issue of stateful device (firewall, ...) requiring to inspect all ingress/egress traffic is not mentioned in the list of issues?

-- Section 3.5 --

While I agree that the scalability of the SADR solution puts some limit on the number of ISP, why does this document select the value 5? More generally, this section could probably be removed.
 
-- Section 4 --

The "way to forward packets" does not read easily. Esp point 2), is it "selected forwarding table" or "selected forwarding table entries" (from point 1). Or should point 1 select a specific source-scoped forwarding table rather than forwarding table entries.

-- Section 5.2.2 --

AFAIK, pfister-6man-sadr-ra is 'dead' since 2015. Is it still worth mentioning here ?

-- Section 5.2.4 --

Why this section have "SHOULD" in uppercase while in the other section it is in lower case ?

-- Section 5.6 --

Using RA for influencing source-address selection is probably not "the most reliable" as RA being multicast can be lost.

-- Section 5.7.1 --

The DNS issue should also probably refer to RFC 7556 (PvD).

-- Section 6 --

Mostly at the end of the document, there is a mention of PvD and of draft-ietf-intarea-provisioning-domains, possibly a little late in the flow.

-- Section 7.3 --

There should be a reference to MPTCP or ICE.

== NITS ==

-- Abstract --

I would suggest to add the word "IPv6" in the abstract as well for clarity.

-- Section 1 --

Sorry but I cannot parse "...without stopping to provide ..." 

-- Section 3 ---

The section title should be changed into "use cases" rather than "requirements" as it already describes part of the solution.

-- Section 3.6 --

s/the aim of this draft/the aim of this document/ also in a couple of places.

-- Section 4 --

Please expand ULA before first use (+ reference)

-- Section 5 --
s/host internal to the multi-homed site/host inside the multi-homed site/

-- Section 5.1 --

Please put all rules explanation in a separate paragraphs (esp rules 1 & 2).

-- Section 5.5.2 --

Please expand GUA before first use.

Thanks for addressing my DISCUSS. I think the disclaimer in Section 6 about default address selection is sufficient to address remaining concerns about the use of the term "host" throughout Section 6. I think Section 6 might be slightly improved if the section headings that talk about "source address selection" instead said "default source address selection."

I agree with the Gen-ART reviewer that a forward reference to Section 8.3 in the last sentence of Section 6.7.1 would be useful.

Thank you for the work on this document!!

I have several comments below -- some of them are substantive and I would like to see them addressed before publication.  

Substantive

(A) It would be nice to add a short terminology section to at least include important terms every reader may not be familiar with: PA, PI, provider-aggregatable.

Other terms that are used but not clearly described, which would benefit from a definition include "SADR domain" and source-prefix-scoped routing/destination/forwarding table...or more-specific-source-prefix-scoped forwarding table.  Note that some of the examples help, but going through them is not the best way to find out the meaning.

(B) §4: "if all routers in a given network support SADR and have a scoped forwarding table, then the unscoped forwarding table can be eliminated which ensures that packets with legitimate source addresses only can leave the network"  This statement is true for traffic existing the network, but not in the general case where the unscoped table has to be used to deliver, for example, packets originated in the Internet to H32 (or any internal host).  If the unscoped forwarding table is eliminated, then how are those packets routed?  Am I missing something?

(C) §5.2.2: "[RFC8028] recommends that a host SHOULD select default routers for each prefix in which it is assigned an address.  It also recommends that hosts SHOULD implement Rule 5.5. of [RFC6724]."  These SHOULDs are not Normative in this document, but come from rfc8028.  I think there should either be a direct quotation or Normative language shouldn't be used.

§5.6/§6 also mention Normative language from rfc8028 without properly quoting it.

(D) The documents talks in several places about SADR support and how it is not necessary for all routers in the enterprise to support it.  There are some mentions of this as examples are described...  However, there is no clear guidance about the considerations for deploying a "SADR domain" in the network, and should be covered in the Deployment Considerations section.

(E) I think that rfc4443, rfc4861, rfc4862, rfc6724 and rfc7078 should all be Normative references.

(F) This point is for the WG Chairs/AD.  I don't think changes to this document are needed, but will leave that up to the Chairs/AD.

§4 says:

   Note that the mechanism described here for converting source-prefix-
   scoped destination prefix routing advertisements into forwarding
   state is somewhat different from that proposed in
   [I-D.ietf-rtgwg-dst-src-routing].  The method described in the
   current document is functionally equivalent, but it is intended to be
   easier to understand for enterprise network operators.

This text makes me wonder about the relationship to I-D.ietf-rtgwg-dst-src-routing, which is currently marked as on the Standards Track (at least on the document itself), and makes no reference to this document (but it does point to rfc8043).  Should the two be more closely related?  This document "attempts to define a complete solution" for the multihoming problem, which is one of the use cases in I-D.ietf-rtgwg-dst-src-routing -- it seems like the answer could be Yes.

The question might be more appropriate in the context of I-D.ietf-rtgwg-dst-src-routing, but I'm asking it now because of the explicit mention (above), and discussion in the WG around the compatibility of the two documents (for example in [1]).   This makes me think that it is an important point to consider before publication of this document.

[1] https://mailarchive.ietf.org/arch/msg/rtgwg/n2K1ZDD_Fco1CO7Oy4ZQ6MYJebg 


Editorial/Nits:

(1) I think a title with "Solutions" (not "Solution") at the end would better reflect the contents.

(2) §1: "Multihoming with provider-assigned addresses is typically less expensive for the enterprise relative to using provider-independent addresses."  I assume you mean "less expensive" from the point of view of acquiring the addresses, right?  However, operationally it may be more expensive because of the need to manage more moving parts, either NATs, or the mechanisms described in this document.  It might be good to clarify.

(3) s/Section 7 discussed other solutions/Section 7 discusses other solutions

(4) s/router(SER)/router (SER)

(5) §3.2: "using GRE for example"  A reference would be nice.

(6) §1: "...some routers must be capable of some form of Source Address Dependent Routing (SADR), if only as described in [RFC3704]."  rfc3704 doesn't talk about SADR, at least not with that name.  Maybe pointing to §4.3 could help.

(7) §4: "...then add the entry to the more source-prefix-scoped forwarding table."   The more what??

(8) s/As as an example/As an example

(9) s/while for `2001:db8:0:b101::31/while for 2001:db8:0:b101::31

(10) s/H01/H101

(11) The first reference to rfc8415 is in §5.2.1.  It would be nice to make it earlier, maybe when DHCPv6 is first mentioned.

(12) "DHCPv6 support is not a mandatory requirement for IPv6 hosts, so this method might not work for all devices." A reference to rfc8504 might be nice.

(13) Given that I-D.pfister-6man-sadr-ra was last updated in 2015, and that it "might need tweaking", I think this document shouldn't even mention it.

(14) s/this is traffic is not following/this traffic is not following

(15) s/An site administrator/A site administrator

(16) The first reference to rfc4443 is in §5.2.3.  It would be nice to make it earlier, maybe when ICMPv6 is first mentioned.

(17) s/reach the a host on the Internet/reach a host on the Internet

(18) [style nit/personal preference] Much of the text is written in first person ("In this document we assume that...").  I find the use of the third person ("In this document it is assumed that..." or "This document assumes...") more appropriate in IETF documents.  Maybe just a manner of taste...

I mostly only have editorial comments, but please note the potential
additional security considerations for ICMPv6 "use this source
address" messages, and the question about leaving a SADR domain being
equivalent to leaving the site.

Abstract

   This document attempts to define a complete solution to this problem.
   It covers the behavior of routers to forward traffic taking into
   account source address, and it covers the behavior of host to select
   appropriate source addresses.  [...]

nit: singular/plural mismatch routers/host

Section 1

       The return packet will be routed over the Internet to ISP-A, but
   it will not be delivered to the multihomed site because its link with
   ISP-A has failed.  [...]

nit: I think formally the subject that "it" refers to in "its link" is
the packet, not the site, so we'd want to disambiguate here.

   Note that the same may be true with a provider that does not
   implement BCP 38, if his upstream provider does, or has no
   corresponding route.  The issue is not that the immediate provider
   implements ingress filtering; it is that someone upstream does, or
   lacks a route.

I'm sure this is just my lack of background, but I didn't see much
introduction of what a "corresponding route" means.

                                   That is, some routers must be capable
   of some form of Source Address Dependent Routing (SADR), if only as
   described in [RFC3704].  [...]

I do not see reference to either "source address dependent routing" or
"SADR" in RFC 3704.

Section 3.2

   In Figure 2, we modify the topology slightly by inserting R7, so that
   SERa and SERb are no longer directly connected.  With this topology,
   it is not enough to just enable SADR routing on SERa and SERb to
   support PA multi-homing.  There are two solutions to ways to enable
   PA multihoming in this topology.

nit: "solutions to ways" seems redundant

Section 4

   3.  Augment each less specific source-prefix-scoped forwarding table
       with all more specific source-prefix-scoped forwarding tables
       entries based on the following rule.  If the destination prefix
       of the less specific source-prefix-scoped forwarding entry
       exactly matches the destination prefix of an existing more
       specific source-prefix-scoped forwarding entry (including
       destination prefix length), then do not add the less specific
       source-prefix-scoped forwarding entry.  [...]

I think this is just editorial, but we start by saying ~"augment
less-specific routes" and thenwe say ~"do not add the less-specific
routes", which doesn't match up -- we can't add X to the baseline when X
is the baseline, and would have to remove X and replace it with the
more-specific thing.

   The forward tables produced by this process are used in the following
   way to forward packets.

nit: "forwarding tables"

   Any traffic that needs to exit the site will eventually hit a SADR-
   capable router.  Once that traffic enters the SADR-capable domain,
   then it will not leave that domain until it exits the site.  [...]

Er, what enforces/provides this property?  (I assume it's not a new
requirement being introduced here...)

   An interesting side-effect of deploying SADR is if all routers in a
   given network support SADR and have a scoped forwarding table, then
   the unscoped forwarding table can be eliminated which ensures that
   packets with legitimate source addresses only can leave the network

nit: s/packets with legitimate source addresses only/only packets with
legitimate source addresses/

                It would prevent accidental leaks of ULA/reserved/link-
   local sources to the Internet as well as ensures that no spoofing is
   possible from the SADR-enabled network.

nit: s/ensures/ensuring/

Section 5

   If all of the ISP uplinks are working, the choice of source address
   by the host may be driven by the desire to load share across ISP
   uplinks, or it may be driven by the desire to take advantage of
   certain properties of a particular uplink or ISP.  If any of the ISP
   uplinks is not working, then the choice of source address by the host
   can determine if packets get dropped.

nit: maybe s/determine if packets get dropped/cause packets to be
dropped/ ?  It seems unlikely that a host would specifically choose a
source address in order to provide the "will be dropped" behavior, since
it could just not send the packet in the first place instead.

               For a session originated from an external host to an
   internal host, the choice of source address used by the internal host
   is simple.  The internal host has no choice but to use the
   destination address in the received packet as the source address of
   the transmitted packet.

(side note) I guess there may be cases where the internal host has a
prearranged agreement with the external host to triangle-route packets,
but (quibbles about "no choice" aside) that doesn't seem pedagogically
relevant to mention here.

Section 5.2

   Again we return to the topology in Figure 3.  Suppose that the site
   administrator wants to implement a policy by which all hosts need to
   use ISP-A to reach H01 at D=2001:db8:0:1234::101.  [...]

nit: I think this wants s/H01/H101/

Section 5.2.3

   We can also use this source-prefix-scoped route originated by SERa to
   communicate the desired routing policy to SERb1.  We can define an
   EXCLUSIVE flag to be advertised together with the IGP route for
   (S=2001:db8:0:a000::/52, D=2001:db8:0:1234::/64).  [...]

Just to check my understanding, is this "we can define" a statement of
future possibilities (viz.
https://tools.ietf.org/html/draft-pioxfolks-6man-pio-exclusive-bit-02)
or something being undertaken in this current document?

   However using ICMPv6 for signalling source address information back
   to hosts introduces new challenges.  [...]

New security risks, too!

                                In addition, the same source prefix
   SHOULD be used for other destinations in the same /64 as the original
   destination address.  The source prefix SHOULD have a specific
   lifetime.  Expiration of the lifetime SHOULD trigger the source
   address selection algorithm again.

nit: I assume this lifetime is for the cached mapping of src/dst
prefixes, and not for using the source prefix at all.

Section 5.2.4

                                As all those options have been
   standardized by IETF and are supported by various operating systems,
   no changes are required on hosts.  [...]

nit: this is a comma splice.

             The policy distribution can be automated by defining an
   EXCLUSIVE flag for the source-prefix-scoped route which can be set on
   the SER that originates the route.  [...]

nit: "can" is  present tense and implies the capability already exists
today; my understanding from the rest of the document is that this
statement refers to potential future work.

Section 5.3.3

                                Potential issue with using ICMPv6 for
   signalling source address issues back to hosts is that uplink to an
   ISP-B failure immediately invalidates source addresses from
   2001:db8:0:b000::/52 for all hosts which triggers a large number of
   ICMPv6 being sent back to hosts - the same scalability/rate limiting
   issues discussed in Section 5.2.3 would apply.

nit: the grammar here is not great.  Also, is the invalidation "for all
hosts" just for use with external destinations?

Section 5.5.2

           In the absence of (S=ULA_Prefix; D=::/0) first-hop routers
   will send dedicated RAs from a unique link-local source LLA_ULA with
   PIO from ULA address space, RIO for the ULA prefix and Router
   Lifetime set to zero.  [...]

(This is still scoped to the "no external connectivity" case, right?)

   particularly useful if all ISPs assign prefixes via DHCP-PD.  In the
   absence of ULAs uplinks failure hosts would lost all their GUAs upon
   prefix lifetime expiration which again makes intra-site communication
   impossible.

nit: I think this is supposed to be "In the absence of ULAs, on uplink
failure hosts would lose [...]"

Section 5.6

[I stopped noting most grammar nits here]

   1.  no new (non-standard) functionality needs to be implemented on
       hosts (except for [RFC4191] support);

RFC 4191 is from 2005; does it really still count as "new"? ;)

   To fully benefit from the RA-based solution, first-hop routers need
   to implement SADR and be able to send dedicated RAs per scoped

It's not just the first-hop routers, though -- won't all the first-hops
need to be part of the same connected SADR domain?

Section 5.7.1

            [RFC8106] defines IPv6 Router Advertisement options to allow
   IPv6 routers to advertise a list of DNS recursive server addresses
   and a DNS Search List to IPv6 hosts.  Using RDNSS together with
   'scoped' RAs as described above would allow a first-hop router (R3 in
   the Figure 3) to send DNS server addresses and search lists provided
   by each ISP (or the corporate DNS servers addresses if the enterprise
   is running its own DNS servers).

I only skimmed RFC 8106, but it seems like this suffers from the same
issue described above for linking PIO and RIO information (that inspired
draft-pfister-6man-sadr-ra) -- we aren't guaranteed an information link
between (source) address to use and DNS recursive to use.  I do see a
note in 8106 that requires this linkage when link-local addresses are
used as DNS recursives, but not in the general case.  While one might
counter that this doesn't matter, since the DNS is a globally consistent
database, in practice that proves to not be the case, with "walled
gardens" being available only within a given ISP, etc., so it does seem
like we could at least mention the potential for issues.  And in fact we
do have such discussion a couple paragraphs later, so maybe all we want
is a hint that there's more to come.

   It should be noted that [RFC8106] explicitly prohibits using DNS
   information if the RA router Lifetime expired: "An RDNSS address or a
   DNSSL domain name MUST be used only as long as both the RA router
   Lifetime (advertised by a Router Advertisement message) and the
   corresponding option Lifetime have not expired.".  Therefore hosts
   might ignore RDNSS information provided in ULA-scoped RAs as those
   RAs would have router lifetime set to 0.  However the updated version
   of RFC6106 ([RFC8106]) has that requirement removed.

It seems that the first reference here needs to be the old one, 6106,
not 8106 as presently indicated.

Section 9

   Section 5.2.3 discusses a mechanism for controlling source address
   selection on hosts using ICMPv6 messages.  It describes how an
   attacker could exploit this mechansim by sending spoofed ICMPv6
   messages.  It recommends that a given host verify the original packet
   header included into ICMPv6 error message was actually sent by the
   host itself.

Section 5.2.3 also talks about a potential extension to ICMPv6 that
would indicate what source address to use, in addition to noting that
the selected source address does not work.  Such an extension would also
have some new security considerations, in that it would provide an
attacker some measure of control over where the resulting traffic ended
up, as (e.g.) might be useful in steering a DDoS.

Much thanks for a very comprehensive document!

Similar to Alvaro's (F),  I find a couple of sentences confusing.

I think it would be very helpful to clarify the scope of this document (in
this document), especially as Alvaro notes, the same working group
is progressing a PS document with another solution.

Examples:

- Abstract: I find the sentence "attempts..complete" solution a bit in conflict.
"Attempts" - either it does or doesn't. "complete" is questionable as it is focused
on a set of use cases. Suggest:

This document attempts to define a complete solution to this problem
/s/
This document examines currently available mechanisms for providing a solution
to this problem for a broad range of enterprise topologies.

- Section 4:
"The method described in the current document is functionally equivalent, but it is
intended to be easier to understand for enterprise network operators."
I don't find the justification "easier to understand for enterprise network operators" to be
convincing. Especially if there is already a PS document being progressed in the same working
group. Hopefully the PS document will also be easy to understand for both operators and vendors.
Suggest a better qualifier, even simply:
but it is intended to be easier to understand for enterprise network operators
/s/
but it is based on application of existing mechanisms for the described scenarios

Regarding Section 7.3 and Section 1 paragraph:

   It may be desirable for an enterprise site to connect to multiple
   ISPs using provider-assigned (PA) addresses, instead of PI addresses.
   Multihoming with provider-assigned addresses is typically less
   expensive for the enterprise relative to using provider-independent
   addresses.  PA multihoming is also a practice that should be
   facilitated and encouraged because it does not add to the size of the
   Internet routing table, whereas PI multihoming does.  Note that PA is
   also used to mean "provider-aggregatable".  In this document we
   assume that provider-assigned addresses are always provider-
   aggregatable.

A possible addition here either in the above paragraph or at least in Section 7.3 that deploying enterprise PA based multi-homing solution actually benefits the usage of multi-path protocols as this ensures that the MP capable transport protocol get a well defined handle to something that likely lead to path diversity. So from my perspective, a working well enough PA based multi-homing solution benefits the deployment of multi-path protocols which in its turn makes the PA based multi-homing work even better than NATed or PI based ones.

Thanks for addressing my discuss as well as the TSV-ART review (thanks Michael for the review!) and adding section 6.7!