FCoE over TRILL
draft-mme-trill-fcoe-04
The information below is for an old version of the document.
| Document | Type |
This is an older version of an Internet-Draft that was ultimately published as RFC 6847.
|
|
|---|---|---|---|
| Authors | David T. Melman , Tal Mizrahi , Donald E. Eastlake 3rd | ||
| Last updated | 2012-10-03 | ||
| RFC stream | Independent Submission | ||
| Formats | |||
| IETF conflict review | conflict-review-mme-trill-fcoe, conflict-review-mme-trill-fcoe, conflict-review-mme-trill-fcoe, conflict-review-mme-trill-fcoe, conflict-review-mme-trill-fcoe, conflict-review-mme-trill-fcoe | ||
| Additional resources | |||
| Stream | ISE state | (None) | |
| Consensus boilerplate | Unknown | ||
| Document shepherd | (None) | ||
| IESG | IESG state | Became RFC 6847 (Informational) | |
| Telechat date | (None) | ||
| Responsible AD | Ralph Droms | ||
| IESG note | ** No value found for 'doc.notedoc.note' ** | ||
| Send notices to | davidme@marvell.com, talmi@marvell.com, d3e3e3@gmail.com, draft-mme-trill-fcoe@tools.ietf.org, rfc-ise@rfc-editor.org | ||
| RFC Editor | RFC Editor state | ISR-AUTH | |
| Details |
draft-mme-trill-fcoe-04
Network Working Group David Melman
Internet Draft Tal Mizrahi
Intended status: Informational Marvell
Expires: April 2013 Donald Eastlake
Huawei
October 3, 2012
FCoE over TRILL
draft-mme-trill-fcoe-04.txt
Abstract
Fibre Channel over Ethernet (FCoE) and TRILL are two emerging
standards in the data center environment. While these two protocols
are seemingly unrelated, they have a very similar behavior in the
forwarding plane, as both perform hop-by-hop forwarding over
Ethernet, modifying the packet's MAC addresses at each hop. This
document describes an architecture for the integrated deployment of
these two protocols.
Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on April 3, 2013.
Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved.
Melman, et al. Expires April 3, 2013 [Page 1]
Internet-Draft FCoE over TRILL October 2012
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction ................................................. 2
2. Abbreviations ................................................ 3
3. FCoE over TRILL .............................................. 4
3.1. FCoE over a TRILL Cloud ................................. 4
3.2. FCoE over RBridge ....................................... 6
3.2.1. FCRB ............................................... 6
3.2.2. Topology ........................................... 8
3.2.3. The FCRB Flow ..................................... 10
3.2.3.1. Example - ENode to ENode ..................... 10
3.2.3.2. Example - ENode to Native FC Node ............ 11
3.2.3.3. Example - ENode to ENode with non-FCRB EoR ... 11
3.2.3.4. Example - FCoE Control Traffic through an FCRB 12
4. Security Considerations ..................................... 13
5. IANA Considerations ......................................... 13
6. Acknowledgments ............................................. 14
7. References .................................................. 14
7.1. Normative References ................................... 14
7.2. Informative References ................................. 14
1. Introduction
Data center networks are rapidly evolving towards a consolidated
approach, where Ethernet is used as the common infrastructure for all
types of traffic. Storage traffic was traditionally dominated by the
Fibre Channel (FC) protocol suite. At the intersection between these
two technologies a new technology was born, Fibre Channel over
Ethernet (FCoE), where native Fibre Channel (FC) packets are
encapsulated with an FCoE encapsulation over an Ethernet header. FCoE
is specified in [FC-BB-5] (A future version of FCoE is under
development and is expected to be specified in a document to be
referred to as FC-BB-6; however, this is a work in progress and
beyond the scope of this document.)
Melman, et al. Expires April 3, 2013 [Page 2]
Internet-Draft FCoE over TRILL October 2012
Traffic between two FCoE end nodes (ENodes) is forwarded through one
or more FCoE Forwarders (FCF). An FCF takes a forwarding decision
based on the Fibre Channel destination ID (D_ID), and enforces
security policies between ENodes, also known as zoning. Once an FCF
takes a forwarding decision, it modifies the source and destination
MAC addresses of the packet, to reflect the path to the next hop FCF
or ENode. An FCoE virtual link is an Ethernet link between an ENode
and an FCF, or between two FCFs. An FCoE virtual link may traverse
one or more Layer 2 bridges. FCFs use a routing protocol called
Fabric Shortest Path First (FSPF) to find the optimal path to each
destination. An FCF typically has one or more native Fibre Channel
interfaces, allowing it to communicate with native Fibre Channel
devices, e.g., storage arrays.
TRILL [RFCTRILL] is a protocol for transparent least cost routing,
where RBridges forward traffic to their detination based on a least
cost route, using a TRILL encapsulation header. RBridges route TRILL-
encapsulated packets based on the Egress RBridge Nickname in the
TRILL header. An RBridge routes a TRILL-encapsulated packet after
modifying its MAC addresses to reflect the path to the next-hop
RBridge, and decrementing a Hop Count field.
TRILL and FCoE bear a strong resemblance in their forwarding planes.
Both protocols take a routing decision based on protocol addresses
above Layer 2, and modify the Ethernet MAC addresses on a per-hop
basis. Each of the protocols uses its own routing protocol rather
than using any type of bridging protocol such as spanning tree
protocol [802.1Q] or the Shortest Path Bridging protocol [802.1aq].
FCoE and TRILL are both targeted at the data center environment, and
their concurrent deployment is self-evident. This document describes
an architecture for the integrated deployment of these two protocols.
2. Abbreviations
DCB Data Center Bridging
ENode FCoE Node such as server or storage array
EoR End of Row
FC Fibre Channel
FCF Fibre Channel Forwarder
FCoE Fibre Channel over Ethernet
Melman, et al. Expires April 3, 2013 [Page 3]
Internet-Draft FCoE over TRILL October 2012
FCRB Fibre Channel forwarder over RBridge
FIP FCoE Initialization Protocol
FSPF Fabric Shortest Path First
LAN Local Area Network
RBridge Routing Bridge
SAN Storage Area Network
ToR Top of Rack
TRILL Transparent Interconnection of Lots of Links
WAN Wide Area Network
3. FCoE over TRILL
3.1. FCoE over a TRILL Cloud
The simplest approach for running FCoE traffic over a TRILL network
is presented in Figure 1. The figure illustrates a TRILL-enabled
network, where FCoE traffic is transparently forwarded over the TRILL
cloud. The figure illustrates two ENodes, a Server and an FCoE
Storage Array, an FCF, and a native Fibre Channel SAN connected to
the FCF.
FCoE traffic between the two ENodes is sent from the first ENode over
the TRILL cloud to the FCF, and then back through the TRILL cloud to
the second ENode.
Melman, et al. Expires April 3, 2013 [Page 4]
Internet-Draft FCoE over TRILL October 2012
+---+
| |_________
| | \ ___ _
+---+ \/ \_/ \__ _ __
FCoE Storage _/ \ / \_/ \_
Array / TRILL / +---+ \_ \
(ENode A) \_ Cloud /________| |____/ SAN _/
/ \ | | \__ _/
\__/\_ ___/ +---+ \_/
+---+ / \_/ FCF
| |________/
| |
+---+
Server
(ENode B)
Figure 1 The "Separate Cloud" Approach
The configuration in Figure 1 separates the TRILL cloud(s) and the
FCoE cloud(s). The TRILL cloud routes FCoE traffic as standard
Ethernet traffic, and appears to the ENodes and FCF as an Ethernet
LAN. FCoE traffic routed over the TRILL cloud includes FCoE data
frames, as well as FCoE control traffic, including FCoE
Initialization Protocol (FIP) frames. To eliminate frame loss due to
queue overflow, the switches in any TRILL Cloud used with FCoE would
likely implement and use the relevant DCB protocols [TRILLDCB].
The main drawback of the Separate Cloud approach is that RBridges and
FCFs are separate nodes in the network, resulting in more cabeling
and boxes, and communication between Enodes usually requires two
TRILL cloud traversals with twice as many hops. As mentioned above,
data center networking is converging towards a consolidated and cost
effective approach, where the same infrastructure and equipment is
used for both data and storage traffic, and where high efficiency and
minimal number of hops are important factors when designing the
network topology.
The Separate Cloud approach is not commonly deployed due to its
practical difficulties. It is presented as a background and
motivation for the approach discussed in the next section.
Melman, et al. Expires April 3, 2013 [Page 5]
Internet-Draft FCoE over TRILL October 2012
3.2. FCoE over RBridge
3.2.1. FCRB
Rather than the Separate Cloud approach discussed in the previous
subsection, an alternate approach is presented, where each switch
incorporates both an FCF entity and an RBridge entity. This
consolidated entity is referred to as FCoE-forwarder-over-RBridge
(FCRB).
Figure 2 illustrates an FCRB, and its main building blocks. An FCRB
can be functionally viewed as two independent entities:
o An FCoE Forwarder (FCF) entity.
o An RBridge entity.
The FCF entity is connected to one of the ports of the RBridge, and
appears to the RBridge as a native Ethernet host. A detailed
description of the interaction between the layers is presented in
Section 3.2.3.
Melman, et al. Expires April 3, 2013 [Page 6]
Internet-Draft FCoE over TRILL October 2012
+-------------------+
|FCRB |
| +-----------+ | Native FC
| | FCF |------ Interface
| +-----+-----+ |
| | |
| +-----+-----+ |
| | RBridge | |
| +-+-+---+-+-+ |
| | | | | |
+-----|-|---|-|-----+
FCoE/ / | | |
+---+ Ethernet / / | | FCoE / Ethernet
| |___________________/ / | | over TRILL ___ _
| | / | | / \_/ \__
+---+ / | \_____________ _/ \
FCoE Storage / \_______________/ TRILL /
Array / \_ Cloud /
(ENode A) / / \
/ \__/\_ ___/
+---+ / \_/
| |______________/
| |
+---+
Server
(ENode B)
Figure 2 FCRB Entity in the Network
The FCRB entity maintains layer independence between the TRILL and
FCoE protocols, while enabling both protocols on the same network.
It is noted that FCoE traffic is always forwarded through an FCF, and
cannot be forwarded directly between two ENodes. Thus, FCoE traffic
between ENodes A and B in the topology in Figure 1 is forwarded
through the path
ENode A-->TRILL cloud-->FCF-->TRILL cloud-->ENode B
As opposed to the topology in Figure 1, the FCF in Figure 2 is
adjacent to ENodes A and B. In Figure 2 the FCRB is connected to
ENodes A and B, and functions as the edge RBridge that connects these
two nodes to the TRILL cloud, as well as the FCF that forwards
Melman, et al. Expires April 3, 2013 [Page 7]
Internet-Draft FCoE over TRILL October 2012
traffic between these two nodes. Thus, traffic between A and B in the
topology in Figure 2 is forwarded through the path
ENode A-->FCRB-->ENode B
Hence, the usage of FCRB entities allows TRILL and FCoE to use common
infrastructure and equipment, as opposed to the Separate Cloud
topology presented in Figure 1.
3.2.2. Topology
The network configuration illustrated in Figure 3 shows a typical
topology of a data center network. Servers are hierarchically
connected through Top-of-Rack (ToR) switches, also known as access
switches, and each set of racks is aggregated through an End-of-Row
(EoR) switch. The EoR switches are aggregated to the Core switches,
which may be connected to other clouds, such as an external WAN or a
native FC SAN.
Melman, et al. Expires April 3, 2013 [Page 8]
Internet-Draft FCoE over TRILL October 2012
_ __ _ __
/ \_/ \_ / \_/ \_
\_ \ \_ \ ....
/ SAN _/ / WAN _/
\__ _/ \__ _/
\_/ \_/
| |
| |
| |
+------+ +------+
Core | | | |
FCoE over | | | |
RBridge | | | |
(FCRB) +------+ +------+
| \___ ___/ |
| \ / |
| \/ |
EoR +----+_______/\_______+----+
FCoE over | | | |
RBridge | | | |
(FCRB) +----+ +----+
/ \ / \
/ \ / \
ToR +---+ +---+ +---+ +---+
FCoE over | | | | | | | |
RBridge | | | | | | | |
(FCRB) +---+ +---+ +---+ +---+
/ \ / \ / \ / \
/ \ / \ / \ / \
+-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+
Servers/ | | | | | | | | | | | | | | | |
ENodes +-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+
A B C D E F G H
Figure 3 FCoE over RBridge Topology
Note that in the example in Figure 3 all the ToR, EoR and core
switches are FCRB entities, but it is also possible for some of the
network nodes to be pure RBridges, creating a topology where FCRBs
are interconnected through TRILL clouds.
Melman, et al. Expires April 3, 2013 [Page 9]
Internet-Draft FCoE over TRILL October 2012
3.2.3. The FCRB Flow
3.2.3.1. Example - ENode to ENode
FCoE traffic sent between two ENodes, A and B in Figure 3, is
transmitted through the ToR FCRB, since A and B are connected to the
same ToR. Traffic between A and C must be forwarded through the EoR
FCRB.
+--------+ +--------+ +--------+ +--------+ +--------+
| FCoE |.....| FCF |.....| FCF |.....| FCF |.....| FCoE |
| ENode | +--------+ +--------+ +--------+ | ENode |
| | |RBridge |.....|RBridge |.....|RBridge | | |
+--------+ +--------+ +--------+ +--------+ +--------+
|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet|
+--------+ +--------+ +--------+ +--------+ +--------+
Server ToR 1 EoR ToR 2 FCoE Storage
ENode A FCRB FCRB FCRB Array
ENode C
Figure 4 Traffic between two ENodes - Example
Figure 4 illustrates the traffic between ENodes A and C that are not
connected to the same ToR.
o FCoE traffic from A is sent to the ToR over the Ethernet
interface. The destination MAC address is the address of the FCF
entity at the ToR.
o ToR 1:
o The packet is forwarded to the FCF entity at the ToR. Thus,
forwarding between A and the FCF at the ToR is analogous to
forwarding between two Ethernet hosts.
o The FCF entity at the ToR takes a forwarding decision based
on the FC addresses. This decision is based on the FSPF
routing protocol at the FCF layer. Thus, the next hop at the
FCF layer is either the EoR FCF entity, or the FCF entity in
ToR 2. In this subsection it is assumed that the next hop is
the EoR FCF. The case where the next hop is the FCF in ToR 2
is described in Section 3.2.3.3.
o The FCF then updates the destination MAC address of the
packet to the address of the EoR FCF.
Melman, et al. Expires April 3, 2013 [Page 10]
Internet-Draft FCoE over TRILL October 2012
o The packet is forwarded to the RBridge entity, where it is
encapsulated in a TRILL header, and sent to the RBridge at
the EoR over a single hop of the TRILL network.
o The RBridge entity in the EoR FCRB, acting as the egress RBridge,
decapsulates the TRILL header and forwards the FCoE packet to the
FCF entity. From this point the forwarding process is similar to
the one described above for the ToR.
o A similar forwarding process takes place at the next hop ToR FCRB,
where the FCRB finally forwards the FCoE packet to the target
ENode C.
3.2.3.2. Example - ENode to Native FC Node
+--------+ +--------+ +--------+ +---------+ +--------+
| FCoE |.....| FCF |.....| FCF |.....| FCF |.....| FC |
| ENode | +--------+ +--------+ +----+----+ |protocol|
| | |RBridge |.....|RBridge |.....| RB | | | stack |
+--------+ +--------+ +--------+ +----+ FC | | |
|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Eth | |<===>| |
+--------+ +--------+ +--------+ +----+----+ +--------+
Server ToR EoR Core Native FC
ENode FCRB FCRB FCRB Storage Array
Figure 5 Example Traffic between ENode & Native FC Storage Array
Figure 5 illustrates a second example, where traffic is sent between
an ENode and an FC Storage Array, following the network topology in
Figure 3.
o FCoE traffic from the ENode is sent to the ToR over the Ethernet
interface. The forwarding process through the ToR FCRB and through
the EoR is similar to the corresponding steps in Section 3.2.3.1.
o When the packet reaches the core FCRB, the egress RBridge entity
decapsulates the TRILL header and forwards the FCoE packet to the
FCF entity. The packet is then forwarded as a native FC packet
through the FC interface to the native FC node.
3.2.3.3. Example - ENode to ENode with non-FCRB EoR
The example illustrated in Figure 6 is similar to the one shown in
Figure 4, except that the EoR is an RBridge rather than an FCRB.
Melman, et al. Expires April 3, 2013 [Page 11]
Internet-Draft FCoE over TRILL October 2012
+--------+ +--------+ +--------+ +--------+
| FCoE |.....| FCF |....................| FCF |.....| FCoE |
| ENode | +--------+ +--------+ +--------+ | ENode |
| | |RBridge |.....|RBridge |.....|RBridge | | |
+--------+ +--------+ +--------+ +--------+ +--------+
|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet|
+--------+ +--------+ +--------+ +--------+ +--------+
Server ToR 1 EoR ToR 2 FCoE Storage
ENode A FCRB FCRB FCRB Array
ENode C
Figure 6 Traffic between two ENodes - Example
An FCoE packet sent from A to C is forwarded as follows:
o The packet is sent to the FCF in ToR 1, as in the previous
example.
o The FCF in ToR 1 takes a forwarding decision based on the FC
addresses, and forwards the packet to the next hop FCF, which
resides in ToR 2. This forwarding decision is taken at the FCF
layer, and is based on the FSPF routing protocol.
o The packet is then forwarded to the RBridge entity in ToR 1, where
it is encapsulated in a TRILL encapsulation, and forwarded to the
RBridge at ToR 2. The packet is routed over the TRILL cloud
through the RBridge at the EoR. The path through the TRILL cloud
is determined by TRILL's IS-IS routing protocol.
o Once the packet reaches ToR 2, it is forwarded in a similar manner
to the description in Section 3.2.3.1.
This example demonstrates that it is possible to have a hybrid
network, where some of the nodes are FCRBs, and some of the nodes are
RBridges. A network configuration where some of the nodes are FCFs
and others are not FCoE aware is sometimes referred to in the FCoE
jargon as sparse mode. In dense mode, on the other hand, all nodes in
the network are FCFs. Thus, while the example in Figure 4 illustrates
a dense mode topology where all bridges are FCRBs, the example in
Figure 6 shows a sparse mode, where traffic between FCRBs can be
routed through a TRILL cloud with several RBridge hops.
3.2.3.4. Example - FCoE Control Traffic through an FCRB
The previous subsections focused on the data plane, i.e., storage
data exchanges transported over an FCoE encapsulation. FCoE also
Melman, et al. Expires April 3, 2013 [Page 12]
Internet-Draft FCoE over TRILL October 2012
requires control and management traffic that is used for initializing
sessions (FIP), distributing routing information (FSPF), and fabric
administration and management.
The FCoE Initialization Protocol (FIP) uses Ethernet frames with a
dedicated Ethertype, allowing the FCF to distinguish it from other
traffic. FIP uses both unicast and multicast traffic. The following
example describes the forwarding scheme of a multicast FIP packet
sent through the network depicted in Figure 4:
o ENode A generates a multicast frame to a multicast MAC address
representing all the FCFs (All-FCF-MAC).
o The packet is forwarded to the ToR FCRB node. The RBridge entity
forwards a copy of the packet to its FCF entity, and also sends
the packet through the TRILL cloud as a multicast TRILL
encapsulated packet.
o Each of the FCRBs in turn receives the packet, forwards a copy to
its FCF entity, as well as forwarding the packet through the TRILL
network, allowing all the FCFs to receive the packet.
While FIP packets have a dedicated Ethertype and frame format, other
types of FCoE control and management frames use the same FCoE
encapsulation as FCoE data traffic. Thus, the forwarding scheme for
such control traffic is similar to the examples described in the
previous subsections, with the exception that these frames can be
sent between ENodes, between FCFs, or between ENodes and FCFs.
4. Security Considerations
For general TRILL Security Considerations see [RFCTRILL].
For general FCoE Security Consideration see Annex D of [FC-BB-5].
There are no additional security implications imposed by this
document.
5. IANA Considerations
There are no IANA actions required by this document.
RFC Editor: please delete this section before publication.
Melman, et al. Expires April 3, 2013 [Page 13]
Internet-Draft FCoE over TRILL October 2012
6. Acknowledgments
The authors gratefully acknowledge Ayandeh Siamack and David Black
for their helpful comments. The authors also thank the T11 committee
for reviewing the document, and in particular Pat Thaler and Joe
White for their useful inputs.
This document was prepared using 2-Word-v2.0.template.dot.
7. References
7.1. Normative References
[RFCTRILL] Perlman, R., Eastlake, D., Dutt, D., Gai, S.,
Ghanwani, A., "Routing Bridges (RBridges): Base
Protocol Specification", RFC 6325, July 2011.
[FC-BB-5] ANSI INCITS 462: Information Technology - Fibre
Channel - Backbone - 5 (FC-BB-5).
7.2. Informative References
[802.1Q] "IEEE Standard for Local and metropolitan area
networks - Virtual Bridged Local Area Networks", IEEE
Std 802.1Q-2011, May 2011.
[802.1aq] "IEEE Standard for Local and metropolitan area
networks - Media Access Control (MAC) Bridges and
Virtual Bridged Local Area Networks - Shortest Path
Bridging", IEEE Std 802.1aq-2012, June 2012.
[TRILLDCB] Eastlake, D., Wadekar, M., Ghanwani, A., Agarwal, P.,
Mizrahi, T., "TRILL: Support of IEEE 802.1Qbb,
802.1Qaz, and Congestion Notification", draft-
eastlake-trill-rbridge-dcb, work in progress, 2012.
Authors' Addresses
David Melman
Marvell
6 Hamada St.
Yokneam, 20692 Israel
Email: davidme@marvell.com
Melman, et al. Expires April 3, 2013 [Page 14]
Internet-Draft FCoE over TRILL October 2012
Tal Mizrahi
Marvell
6 Hamada St.
Yokneam, 20692 Israel
Email: talmi@marvell.com
Donald Eastlake 3rd
Huawei USA R&D
155 Beaver Street
Milford, MA 01757 USA
Phone: +1-508-333-2270
EMail: d3e3e3@gmail.com
Melman, et al. Expires April 3, 2013 [Page 15]