Multi-tenant Data Center Use Case for IPsec Load Balancing
draft-bottorff-ipsecme-mtdcuc-ipsec-lb-00

Document Type Active Internet-Draft (individual)
Author Paul Bottorff 
Last updated 2021-07-12
Stream (None)
Intended RFC status (None)
Formats pdf htmlized bibtex
Stream Stream state (No stream defined)
Consensus Boilerplate Unknown
RFC Editor Note (None)
IESG IESG state I-D Exists
Telechat date
Responsible AD (None)
Send notices to (None)
Network Working Group                                       P. Bottorff
Internet-Draft                           Hewlett Packard Enterprise Co.
Intended status: Informational                             July 5, 2021
Expires: December 29, 2021

        Multi-tenant Data Center Use Case for IPsec Load Balancing
                 draft-bottorff-ipsecme-mtdcuc-ipsec-lb-00

Abstract

   IPsec is of increasing importance within data centers to secure
   tunnels used to carry multi-tenant traffic encapsulated using the
   Network Virtualization over L3 (NVO3) protocols. Encrypting NVO3
   tunnels provides defense against bad actors within the physical
   underlay network from monitoring or injecting overlay traffic from
   outside the NVO3 infrastructure. When securing data center tunnels
   using IPsec it becomes crucial to retain entropy within the outer
   IPsec packet headers to facilitate load balancing over the highly
   meshed networks used in these environments. While entropy is
   necessary to support load distribution algorithms it is also
   important that the entropy codes used retain integrity of flows to
   prevent performance deterioration resulting from packet re-ordering.
   Here, we describe a use case for load balancing IPsec traffic within
   multi-tenant data centers.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79. Internet-Drafts are working
   documents of the Internet Engineering Task Force (IETF). Note that
   other groups may also distribute working documents as Internet-
   Drafts.  The list of current Internet-Drafts is at
   https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on December 22, 2021.

Copyright Notice

   Copyright (c) 2021 IETF Trust and the persons identified as the
   document authors. All rights reserved.

Bottorff, et al.      Expires December 29, 2021                [Page 1]
Internet-Draft    Multi-tenant DC IPsec LB Use Case           July 2021

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document. Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1. Introduction...................................................2
   2. Generic Data Center Network Architecture.......................4
   3. Network Virtualization Over L3 (NVO3) Architecture.............5
   4. Load Balancing Secure NVO3 tunnels.............................6
   5. Security Considerations........................................7
   6. IANA Considerations............................................7
   7. Conclusions....................................................7
   8. Normative References...........................................8
   9. Informative References.........................................9
   10. Acknowledgments..............................................10

1. Introduction

   Load balancing is essential within data centers to achieve high
   utilization of the meshed networks common in these environments.
   Typically, the load on the network is scattered over the mesh network
   using a hash of the outer packet headers (i.e. a 5-tuple hash). When
   a tunneling protocol is used over a data center mesh network packets
   are addressed from tunnel end point to tunnel end point (e.g.
   servers) which does not provide the entropy required to spread the
   traffic over the data center mesh network. To provide load balancing
   support, tunnel protocols used in the data center need to provide
   entropy codes within their outer packet headers to support load
   balancing.

   While spreading traffic over a data center mesh network mis-ordering
   of packet flows needs to be avoided to prevent slowing operations
   caused by packet order recovery. To retain flow alignment within
   tunneling protocols entropy codes need to be based on the flow of the
   encapsulated packets.

   Multi-tenant data center using Network Virtualization over L3 (NVO3)
   [RFC7365, RFC8014, RFC8394] create virtual networks interconnecting
   virtual servers within an overlay on top of the physical underlay

Bottorff, et. al.      Expires January 2, 2022                 [Page 2]
Internet-Draft    Multi-tenant DC IPsec LB Use Case           July 2021

   network.  In these NVO3 networks virtual network packets are
   multiplexed into tunnels which extend over the physical underlay
   network. The encapsulations used to in the NVO3 tunnels (i.e.  VxLAN,
   GENEVE, GUE) have an outer UDP header followed by an outer NVO3
   header. The NVO3 header carries a key called the Virtual Network
   Identifier (VNI) which identifies the virtual network within the
   virtual overlay network. Each of the virtual overlay networks is a
   separate subnetwork which can have its own IP and L2 virtual address
   space.

   Since a single NVO3 tunnel is used between communicating servers, any
   server to server connection has the same IP source, destination, and
   UDP destination port. To retain entropy for load balancing the NVO3
   protocols use the UDP source port to hold a hash of the encapsulated
   inner packet. This outer UDP source port provides the entropy
   necessary for spreading traffic over the mesh based on the
   encapsulated flow. Other fields such as the IPv6 flow label could
   also be used, however these are not universally supported in data
   center switching infrastructures, while the use of UDP source port is
   broadly available in data center switches, routers, and middle boxes.

   The NVO3 protocols isolate tenant virtual networks based on the VNI
   identifiers carried in the tunnel headers. Since any bad actor
   connected to the data center underlay network could spoof an
   encapsulation transporting a virtual network and any device in the
   middle of the communication can monitor the tenant networks, NVO3
   networks must operate in a secure perimeter. With the rise of more
   aggressive bad actors it is desirable to provide secure connections
   for NOV3 tunnels to eliminate the threat of a server or switch within
   the data center underlay monitoring or interfering with the operation
   of virtual networks throughout the data center.

   Encryption of [RFC4301, RFC4303, RFC7321] the NVO3 tunnels can
   provide protection against devices outside the virtual overlay from
   monitoring, spoofing or interfering with the virtual networks. This
   can be done using IPsec to encrypt the tunnels carrying virtual
   networks between servers. Since the tunnels can be encrypted using
   smart network interfaces this method can be very efficient, retaining
   the high performance required within data centers.

   If we apply IPsec directly to the NVO3 tunnels the IP source and
   destination as well as the protocol type and SPI will be the same for
   each server to server communication, therefore we will lose the
   entropy needed to support the data center mesh network. Internet
   Draft  "Encapsulating IPsec ESP in UDP for Load-balancing" [IPSEC-
   LB], which proposes using the source port of IPsec transport mode ESP
   in UDP encapsulated packets for entropy, provides the solution needed

Bottorff, et. al.      Expires January 2, 2022                 [Page 3]
Internet-Draft    Multi-tenant DC IPsec LB Use Case           July 2021

   to support the use of IPsec to secure the tunnels used for NVO3
   traffic in data centers. By using transport mode ESP in UDP
   encapsulation of NVO3 tunnels entropy can be provided by using the
   UDP source port just as was done originally for the NVO3 UDP
   encapsulations.

2. Generic Data Center Network Architecture

   Figure 1 depicts a typical Clos mesh network [RFC7938] used in a data
   center. Each server is typically redundantly connected to a set of
   switches located at the Top of the Rack (ToR) switch (also called a
   leaf switch). Each of these switches is, in turn, connected to a set
   of switches for the row of racks commonly called the End of Row (EoR)
   switch (also called a spine switch). Typically these redundant links
   are managed either by L2 link aggregation protocols [IEEE-AX, IEEE-Q]
   or by L3 equal cost multi-path protocols [RFC7938]. The number of
   links interconnecting these layers varies depending on the bandwidth
   and resiliency requirements. The figure shows a two level hierarchy,
   however it is common for data centers to have a three level hierarchy
   where a similar Clos mesh is used to interconnect rows of servers.

                        +--------+   +--------+
                        |   EoR  |   |   EoR  |
                        | switch |   | switch |
                        +--------+   +--------+
                        /  / |  \      / | \  \
                     /    /  +---\----/-----\-----------------+
                  /      /         \/    |   \    \           |
               /        /         /  \   |    \     \         |
             / +-------/--------/------\-+     \      \       |
          /    |      /       /          \      \       \     |
       +--------+   +--------+             +--------+   +--------+
       |   ToR  |   |   ToR  |             |   ToR  |   |   ToR  |
       | switch |   | switch |             | switch |   | switch |
       +--------+   +--------+             +--------+   +--------+
          /   \        /  \                   /   \        /  \
         /      \    /     \                 /      \    /     \
        /         \/        \               /         \/        \
       /         /  \        \             /         /  \        \
      /        /      \       \           /        /      \       \
     /       /          \      \         /       /          \      \
   '-----------'       '-----------'   '-----------'       '-----------'
   :   Server  :       :   Server  :   :   Server  :       :   Server  :
   :           :       :           :   :           :       :           :
   '-----------'       '-----------'   '-----------'       '-----------'

   Figure 1: Typical Data Center Mesh Network

Bottorff, et. al.      Expires January 2, 2022                 [Page 4]
Internet-Draft    Multi-tenant DC IPsec LB Use Case           July 2021

   To distribute packets over the network paths typically a hash
   function is used to reduce the fields within the outer packet headers
   into a group of flows. The group is then allocated to a network link
   or path. In the simplest and most common implementation the
   distributions are done based on a hash. In more sophisticated
   implementation additional load data and timing information may be
   used to move flow groups based on load estimates.

3. Network Virtualization Over L3 (NVO3) Architecture

   In an NVO3 multi-tenant data center the physical interconnect
   depicted in figure 1 is used as the underlay physical IP network
   where IP addresses are assigned to servers.

         +--------+                                    +--------+
         | Tenant +--+                            +----| Tenant |
         | System |  |                           (')   | System |
         +--------+  |    .................     (   )  +--------+
                     |  +---+           +---+    (_)
                     +--|NVE|---+   +---|NVE|-----+
                        +---+   |   |   +---+
                        / .    +-----+      .
                       /  . +--| NVA |--+   .
                      /   . |  +-----+   \  .
                     |    . |             \ .
                     |    . |   Overlay   +--+--++--------+
         +--------+  |    . |   Network   | NVE || Tenant |
         | Tenant +--+    . |             |     || System |
         | System |       .  \ +---+      +--+--++--------+
         +--------+       .....|NVE|.........
                               +---+
                                 |
                                 |
                       =====================
                         |               |
                     +--------+      +--------+
                     | Tenant |      | Tenant |
                     | System |      | System |
                     +--------+      +--------+

   Figure 2: NVO3 Architecture Reference Diagram

   The NVO3 protocols are used on top of the physical underlay network
   to create virtual networks which overlay the physical underlay
   network. The virtual networks are carried over the underlay
   encapsulated in an NVO3 encapsulation protocol such as GENEVE
   [RFC8926], VxLAN [RFC7348], or GUE. These encapsulations indicate the

Bottorff, et. al.      Expires January 2, 2022                 [Page 5]
Internet-Draft    Multi-tenant DC IPsec LB Use Case           July 2021

   virtual network by encoding a Virtual Network Identifier (VNI) within
   the encapsulation header.

   Figure 2 is a copy of the NVO3 architecture [RFC8014] reference
   diagram. In figure 2 the Network Virtual Edge (NVE) entities provide
   the tunnel terminations for the encapsulation protocols. The NVEs can
   be located within the server's hypervisor, within smart NICs on the
   servers, or within switches of the physical network [RFC8394].

   The tenant systems (TS) are virtualized servers. These may be virtual
   machines, containers, or physical servers that are connected over the
   virtual networks and multiplexed into NOV3 tunnel by the NVEs.

   The network virtualization authority (NVA) manages the creation and
   configuration of virtual networks by configuring the NVEs.

4. Load Balancing Secure NVO3 tunnels

   The NVO3 protocols isolate tenant virtual networks based on the
   Virtual Network Identifiers carried in the tunnel header. Since any
   bad actor inside the data center could spoof an encapsulation
   transporting a virtual network and any device in the middle of the
   communication can monitor the tenant networks, these networks are
   only secure when operating within a secure perimeter. With the rise
   of more aggressive bad actors it is desirable to provide secure
   connections for NVO3 tunnels to eliminate the threat of a server or
   switch within the data center underlay monitoring or interfering with
   the operation of overlay virtual networks.

   Encryption of [RFC4301, RFC4303, RFC7321] the NVO3 tunnels provides
   protection against devices outside the virtual overlay from
   monitoring, spoofing or interfering with the virtual networks. This
   can be done using IPsec to encrypt the tunnels carrying virtual
   networks between servers. Since the tunnels can be encrypted using
   smart network interfaces this method can be very efficient, retaining
   the high performance required within data centers. However since the
   resulting tunnel headers don't provide enough entropy to support load
   balancing over the data center mesh networks the resulting network
   bandwidth could be greatly reduced.

   To solve the load balancing problem produced by securing the NVO3
   tunnels using IPsec the method proposed in Internet Draft
   "Encapsulating IPsec ESP in UDP for Load-balancing" [IPSEC-LB] can be
   used. Applying I-D.draft-xu-ipsecme-esp-in-udp-lb the NVO3 tunnels
   are encapsulated as IPsec transparent mode ESP in UDP packets. Given
   the UDP header on the outside of the IPsec tunnel the source port can
   be used for entropy. By copying the source port from the original

Bottorff, et. al.      Expires January 2, 2022                 [Page 6]
Internet-Draft    Multi-tenant DC IPsec LB Use Case           July 2021

   NVO3 encapsulation into the IPsec ESP in UDP header it is possible to
   retain the flow identity of the encrypted virtual network. Since the
   NVO3 encapsulation source port contains an entropy code based on the
   encapsulated overlay packet the resulting packet will provide the
   entropy necessary to support load balancing in the data center mesh
   network.

5. Security Considerations

   Here we use IPsec to secure NVO3 tunnels between data center NVEs to
   prevent attacks from servers or switches located within the data
   center physical underlay, however outside of the overlay networks or
   the NVE tunnel terminations and NVA managers. To fully secure a
   multi-tenant network additional security methods need to be used to
   prevent attackers from infiltrating the overlay infrastructure
   including the Tenant Systems, NVEs and NVA.

6. IANA Considerations

   The Internet Draft "Encapsulating IPsec ESP in UDP for Load-
   balancing" [IPSEC-LB] proposes getting a new UDP destination port
   assignment for use with load balanced IPsec. The use of a new port
   would prevent existing implementations of IKE from operating with a
   load balanced transparent mode ESP in UDP stream. It does not appear
   this is necessary. Instead, the existing ESP in UDP port 4500 could
   be used, provided both ends of the connection are configured to
   exchanging ESP in UDP with an entropy code in the UDP source port. If
   the existing ESP in UDP port 4500 is used, then there are no IANA
   considerations since no new code points are necessary.

7. Conclusions

   IPsec may be used to secure the underlay of a NVO3 multi-tenant data
   center by encrypting the NVO3 tunnels. To make IPsec a viable
   solution the IPsec tunnels need to provide load balancing.

   By applying the proposal in Internet Draft "Encapsulating IPsec ESP
   in UDP for Load-balancing" [IPSEC-LB], entropy can be added to the
   IPsec packet header using the UDP source port of the ESP in UDP IPsec
   packets. In particular, the source port of the original NVO3 tunnel
   header can be copied to the new IPsec ESP in UDP source port
   providing the necessary entropy while retaining the flow identity of
   the encapsulated overlay packet.

Bottorff, et. al.      Expires January 2, 2022                 [Page 7]
Internet-Draft    Multi-tenant DC IPsec LB Use Case           July 2021

8. Normative References

   [IPSEC-LB] Xu, X., Hegde, S., Pismenny, B., Zhang, D., and Xia, L.,
             "Encapsulating IPsec ESP in UDP for Load-balancing",
             December 2020, <https://datatracker.ietf.org/doc/draft-xu-
             ipsecme-esp-in-udp-lb/>.

   [IEEE-AX] "IEEE Standard for Local and Metropolitan Area Networks--
             Link Aggregation," in IEEE Std 802.1AX-2020 (Revision of
             IEEE Std 802.1AX-2014), vol., no., pp.1-333, 29 May 2020,
             doi: 10.1109/IEEESTD.2020.9105034.

   [IEEE-Q]  "IEEE Standard for Local and Metropolitan Area Network--
             Bridges and Bridged Networks," in IEEE Std 802.1Q-2018
             (Revision of IEEE Std 802.1Q-2014), vol., no., pp.1-1993, 6
             July 2018, doi: 10.1109/IEEESTD.2018.8403927.

   [RFC768]  Postel, J., "User Datagram Protocol", STD 6, RFC 768, DOI
             10.17487/RFC0768, August 1980, <https://www.rfc-
             editor.org/info/rfc768/>.

   [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts -
             Communication Layers", STD 3, RFC 112, DOI
             10.17487/RFC1122, October 1989, <https://www.rfc-
             editor.org/info/rfc112/>.

   [RFC1191] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191,
             DOI 10.17487/RFC1191, November 1990, <https://www.rfc-
             editor.org/info/rfc1191/>.

   [RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, DOI
             10.17487/RFC2003, October 1996, <https://www.rfc-
             editor.org/info/rfc2003/>.

   [RFC3948] Huttunen, A., Swander, B., Volpe, V., DiBurro, L., and M.
             Stenberg, "UDP Encapsulation of IPsec ESP Packets", RFC
             3948, DOI 10.17487/RFC3948, January 2005, <https://www.rfc-
             editor.org/info/rfc3948/>.

   [RFC4301] Kent,S., Seo, K., "Security Architecture for the Internet
             Protocol", RFC 4301, December 2005,
             <https://datatracker.ietf.org/doc/rfc4301/>

   [RFC4303] Kent, S., "IP Encapsulating Security Payload", RFC 4303,
             December 2005, <https://datatracker.ietf.org/doc/rfc4303/>

Bottorff, et. al.      Expires January 2, 2022                 [Page 8]
Internet-Draft    Multi-tenant DC IPsec LB Use Case           July 2021

   [RFC4821] Mathis, M. and J. Heffner, "Packetization Layer Path MTU
             Discovery", RFC 4821, DOI 10.17487/RFC4821, March 2007,
             <https://www.rfc-editor.org/info/rfc4821/>.

   [RFC7296] Kaufman, C., Hoffman, P., Nir, Y., Eronen, P., and T.
             Kivinen, "Internet Key Exchange Protocol Version 2
             (IKEv2)", STD 79, RFC 7296, DOI 10.17487/RFC7296, October
             2014, <https://www.rfc-editor.org/info/rfc7296/>.

   [RFC7321] McGrew, D., Hoffman, P., "Cryptographic Algorithm
             Implementation Requirements and Usage Guidance for
             Encapsulating Security Payload (ESP) and Authentication
             Header (AH)", RFC 7321, August 2014,
             https://datatracker.ietf.org/doc/rfc7321/

   [RFC7348] Mahalingam, M., et. al., "Virtual eXtensible Local Area
             Network (VXLAN): A Framework for Overlaying Virtual Layer 2
             Networks over Layer 3 Networks", RFC 7348, August 2014,
             <https://datatracker.ietf.org/doc/rfc7348/>

   [RFC7365] Lasserre, M., et al, "Framework for Data Center (DC)
             Network Virtualization", October 2014,
             <https://datatracker.ietf.org/doc/rfc7365/>

   [RFC7938] Lapukhov, P., Premji, A., Mitchell, J., Ed., "Use of BGP
             for Routing in Large-Scale Data Centers", August 2016,
             <https://datatracker.ietf.org/doc/rfc7938/>.

   [RFC8014] Black, D., et al, "An Architecture for Data-Center Network
             Virtualization over Layer 3 (NVO3)", December 2016,
             <https://datatracker.ietf.org/doc/rfc8014>

   [RFC8394] Li, Y., Eastlake, D., Kreeger, L. Narten, T., Black, D.,
             "Split Network Virtualization Edge (Split-NVE) Control-
             Plane Requirements", RFC 8394, May 2018,
             <https://datatracker.ietf.org/doc/rfc8394/>

   [RFC8926] Gross, J., Ganga, I., Sridhar, T., "Geneve: Generic Network
             Virtualization Encapsulation", November 2020,
             <https://datatracker.ietf.org/doc/rfc8926/>

9. Informative References

   [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label for
             Equal Cost Multipath Routing and Link Aggregation in
             Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011,
             <https://www.rfc-editor.org/info/rfc6438>.

Bottorff, et. al.      Expires January 2, 2022                 [Page 9]
Internet-Draft    Multi-tenant DC IPsec LB Use Case           July 2021

   [RFC8200] Deering, S., Hiden, R., "Internet Protocol, Version 6
             (IPv6) Specification", STD 76, RFC 8200, July 2017,
             <https://datatracker.ietf.org/doc/rfc8200/>

   [RFC8201] McCann, J., Deering, S., Mogul, J., Hiden, R., Ed., "Path
             MTU Discovery for IP version 6", STD 87, RFC 8201, July
             2017, <https://datatracker.ietf.org/doc/rfc8201/>

10. Acknowledgments

   This document was prepared using 2-Word-v2.0.template.dot.

Bottorff, et. al.      Expires January 2, 2022                [Page 10]
Internet-Draft    Multi-tenant DC IPsec LB Use Case           July 2021

Authors' Addresses

   Paul Bottorff
   Aruba a Hewlett Packard Enterprise Co
   8000 Foothill Blvd.
   Roseville, CA 95747

   Email: paul.bottorff@hpe.com

Bottorff, et. al.      Expires January 2, 2022                [Page 11]