Use of BGP for Routing in Large-Scale Data Centers
RFC 7938
Document | Type | RFC - Informational (August 2016; Errata) | |
---|---|---|---|
Authors | Petr Lapukhov , Ariff Premji , Jon Mitchell | ||
Last updated | 2018-08-25 | ||
Replaces | draft-lapukhov-bgp-routing-large-dc | ||
Stream | IETF | ||
Formats | plain text html pdf htmlized bibtex | ||
Reviews | |||
Stream | WG state | Submitted to IESG for Publication | |
Document shepherd | Jeff Tantsura | ||
Shepherd write-up | Show (last changed 2016-02-28) | ||
IESG | IESG state | RFC 7938 (Informational) | |
Action Holders |
(None)
|
||
Consensus Boilerplate | Yes | ||
Telechat date | |||
Responsible AD | Alia Atlas | ||
Send notices to | (None) | ||
IANA | IANA review state | IANA OK - No Actions Needed | |
IANA action state | No IANA Actions |
Internet Engineering Task Force (IETF) P. Lapukhov Request for Comments: 7938 Facebook Category: Informational A. Premji ISSN: 2070-1721 Arista Networks J. Mitchell, Ed. August 2016 Use of BGP for Routing in Large-Scale Data Centers Abstract Some network operators build and operate data centers that support over one hundred thousand servers. In this document, such data centers are referred to as "large-scale" to differentiate them from smaller infrastructures. Environments of this scale have a unique set of network requirements with an emphasis on operational simplicity and network stability. This document summarizes operational experience in designing and operating large-scale data centers using BGP as the only routing protocol. The intent is to report on a proven and stable routing design that could be leveraged by others in the industry. Status of This Memo This document is not an Internet Standards Track specification; it is published for informational purposes. This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are a candidate for any level of Internet Standard; see Section 2 of RFC 7841. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at http://www.rfc-editor.org/info/rfc7938. Lapukhov, et al. Informational [Page 1] RFC 7938 BGP Routing in Data Centers August 2016 Copyright Notice Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Network Design Requirements . . . . . . . . . . . . . . . . . 4 2.1. Bandwidth and Traffic Patterns . . . . . . . . . . . . . 4 2.2. CAPEX Minimization . . . . . . . . . . . . . . . . . . . 4 2.3. OPEX Minimization . . . . . . . . . . . . . . . . . . . . 5 2.4. Traffic Engineering . . . . . . . . . . . . . . . . . . . 5 2.5. Summarized Requirements . . . . . . . . . . . . . . . . . 6 3. Data Center Topologies Overview . . . . . . . . . . . . . . . 6 3.1. Traditional DC Topology . . . . . . . . . . . . . . . . . 6 3.2. Clos Network Topology . . . . . . . . . . . . . . . . . . 7 3.2.1. Overview . . . . . . . . . . . . . . . . . . . . . . 7 3.2.2. Clos Topology Properties . . . . . . . . . . . . . . 8 3.2.3. Scaling the Clos Topology . . . . . . . . . . . . . . 9 3.2.4. Managing the Size of Clos Topology Tiers . . . . . . 10 4. Data Center Routing Overview . . . . . . . . . . . . . . . . 11 4.1. L2-Only Designs . . . . . . . . . . . . . . . . . . . . . 11 4.2. Hybrid L2/L3 Designs . . . . . . . . . . . . . . . . . . 12 4.3. L3-Only Designs . . . . . . . . . . . . . . . . . . . . . 12 5. Routing Protocol Design . . . . . . . . . . . . . . . . . . . 13 5.1. Choosing EBGP as the Routing Protocol . . . . . . . . . . 13 5.2. EBGP Configuration for Clos Topology . . . . . . . . . . 15 5.2.1. EBGP Configuration Guidelines and Example ASN Scheme 15 5.2.2. Private Use ASNs . . . . . . . . . . . . . . . . . . 16 5.2.3. Prefix Advertisement . . . . . . . . . . . . . . . . 17 5.2.4. External Connectivity . . . . . . . . . . . . . . . . 18 5.2.5. Route Summarization at the Edge . . . . . . . . . . . 19 6. ECMP Considerations . . . . . . . . . . . . . . . . . . . . . 20 6.1. Basic ECMP . . . . . . . . . . . . . . . . . . . . . . . 20 6.2. BGP ECMP over Multiple ASNs . . . . . . . . . . . . . . . 21 6.3. Weighted ECMP . . . . . . . . . . . . . . . . . . . . . . 21 6.4. Consistent Hashing . . . . . . . . . . . . . . . . . . . 22 Lapukhov, et al. Informational [Page 2]Show full document text