Applicability of Remote Direct Memory Access Protocol (RDMA) and Direct Data Placement (DDP)
draft-ietf-rddp-applicability-08

Bestler & Coene         Expires December 23, 2006              [Page 14]
Internet-Draft           RDMA/DDP Applicability                June 2006

6.  LLP Comparisons

   Normally the choice of underlying IP transport is irrelevant to the
   ULP.  RDMAP and DDP provides the same services over either.  There
   may be performance impacts of the choice, however.  It is the
   responsibility of the ULP to determine which IP transport is best
   suited to its needs.

   SCTP provides for preservation of message boundaries.  Each DDP
   segment will be delivered within a single SCTP packet.  The
   equivalent services are only available with TCP through the use of
   the MPA (Marker PDU Alignment) adaptation layer.

6.1.  Multistreaming Implications

   SCTP also provides multi-streaming.  When the same pair of hosts have
   need for multiple DDP streams this can be a major advantage.  A
   single SCTP association carries multiple DDP streams, consolidating
   connection setup, congestion control and acknowledgements.

   Completions are controlled by the DDP Source Sequence Number (DDP-
   SSN) on a per stream basis.  Therefore combining multiple DDP Streams
   into a single SCTP association cannot result in a dropped packet
   carrying data for one stream delaying completions on others.

6.2.  Out of Order Reception Implications

   The use of unordered Data Chunks with SCTP guarantees that the DDP
   layer will be able to perform placements when IP datagrams are
   received out of order.

   Placement of out-of-order DDP Segments carried over MPA/TCP is not
   guaranteed, but certainly allowed.  The ability of the MPA receiver
   to process out-of-order DDP Segments may be impaired when alignment
   of TCP segments and MPA FPDUs is lost.  Using SCTP, each DDP Segment
   is encoded in a single Data Chunk and never spread over multiple IP
   datagrams.

6.3.  Header and Marker Overhead

   MPA and TCP headers together are smaller than the headers used by
   SCTP and its adaptation layer.  However, this advantage can be
   reduced by the insertion of MPA markers.  The different in ULP
   payload per IP Datagram is not likely to be a signifigant factor.

6.4.  Middlebox Support

   Even with the MPA adaptation layer, DDP traffic carried over MPA/TCP

Bestler & Coene         Expires December 23, 2006              [Page 15]
Internet-Draft           RDMA/DDP Applicability                June 2006

   will appear to all network middleboxes as a normal TCP connection.
   In many environments there may be a requirement to use only TCP
   connections to satisfy existing network elements and/or to facilitate
   monitoring and control of connections.  While SCTP is certainly just
   as monitorable and controllable as TCP, there is no guarantee that
   the network management infrastructure has the required support for
   both.

6.5.  Processing Overhead

   A DDP stream delivered via MPA/TCP will require more processing
   effort that one delivered over SCTP.  However this extra work may be
   justified for many deployments where full SCTP support is unavailable
   in the endpoints of the network, or where middleboxes impair the
   usability of SCTP.

6.6.  Data Integrity Implications

   Both the SCTP and MPA/TCP adaptation provide end-to-end CRC32c
   protection against data accidental corruption, or its equivalent.

   A ULP that requires a greater degree of protection may add it own.
   However, DDP and RDMAP headers will only be guaranteed to have the
   equivalent of end-to-end CRC32c protection.  A ULP that requires data
   integrity checking more thorough than an end-to-end CRC32c should
   first invalidate all STags that reference a buffer before applying
   their own integrity check.

   CRC32c only provides protection against random corruption.  To
   protect against unauthorized alteration or forging of data packets
   security methods must be applied.  The security draft [RDMA-Security]
   [6] specifies usage of RFC2406 [1] for both adaptation layers.

6.6.1.  MPA/TCP Specifics

   It is mandatory for MPA/TCP implementations to implement CRC32c, but
   it is NOT mandatory to use the CRC32c during an RDMA connection.  The
   activating or deactivating of the CRC in MPA/TCP is an administrative
   configuration operation at the local and remote end.  The
   administration of the CRC(ON/OFF) is invisible to the ULP.

   Applications SHOULD trust that this administrative option will only
   be used when the end-to-end protection is at least as effective as a
   transport layer CRC32c.  Applications SHOULD NOT apply additional
   protection as a guard against this administrative option being turned
   on inadvertently.

   Administrators MUST NOT enable CRC32c suppression unless the end-to-

Bestler & Coene         Expires December 23, 2006              [Page 16]
Internet-Draft           RDMA/DDP Applicability                June 2006

   end protection is truly equivalent.

   If the CRC is active/used for one direction/end , then the use of the
   CRC is mandatory in both directions/ends.

   If both ends have been configured NOT to use the CRC, then this is
   allowed as long as an equivalent protection(comparable or better
   than/to CRC) from undetected errors on the connection is provided.

6.6.2.  SCTP Specifics

   SCTP provides CRC32c protection automatically.  The adaptation to
   SCTP provides for no option to suppress SCTP CRC32c protection.

6.7.  Non-IP Transports

   DDP is defined to operate over ubiquitous IP transports such as SCTP
   and TCP.  This enabled a new DDP-enabled node to be added anywhere to
   an IP network.  No DDP-specific support from middle-boxes is
   required.

   There are non-IP transport fabric offering RDMA capabilities.
   Because these capabilities are integrated with the transport protocol
   they have some technical advantages when compared to RDMA over IP.
   For example fencing of RDMA operations can be based upon transport
   level acks.  Because DDP is cleanly layered over an IP transport, any
   explicit RDMA layer ack must be separate from the transport layer
   ack.

   There may be deployments where the benefits of RDMA/transport
   integration outweigh the benefits of being on an IP network.

6.7.1.  No RDMA Layer Ack

   DDP does not provide for its own acknowledgements.  The only form of
   ack provided at the RDMAP layer is an RDMA Read Response.  DDP and
   RDMAP rely almost entirely upon other layers for flow control and
   pacing.  The LLP is relied upon to guarantee delivery and avoid
   network congestion, and ULP level acking is relied upon for ULP
   pacing and to avoid ULP buffer overruns.

   Previous RDMA protocols, such as InfiniBand, have been able to use
   their integration with the transport layer to provide stronger
   ordering guarantees.  It is important that application designers that
   require such guarantees to provide them through ULP interaction.

   Specifically:

Bestler & Coene         Expires December 23, 2006              [Page 17]
Internet-Draft           RDMA/DDP Applicability                June 2006

      There is no ability for a local interface to "fence" outbound
      messages to guarantee that prior tagged messages have been placed
      prior to sending a tagged message.  The only guarantees available
      from the other side would be an RDMA Read Response (coming from
      the RDMAP layer) or a response from the ULP layer.  Remember that
      the normal ordering rules only guarantee when the Data Sink ULP
      will be notified of untagged messages, it does not control when
      data is placed into receive buffers.

      Re-use of tagged buffers must be done with extreme care.  The fact
      that an untagged message indicates that all prior tagged messages
      have been placed does not guarantee that no later tagged message
      have.  The best strategy is to only change the state of any given
      advertised buffers with with untagged messages.

      As covered elsewhere in this document, flow control of untagged
      messages MUST be provided by the ULP itself.

6.8.  Other IP Transports

   Both TCP and SCTP provide DDP with reliable transport with TCP
   friendly rate control.  As currently DDP is defined to work over
   reliable transports and implicitly relies upon some form of rate
   control.

   DDP is fully compatible with a non-reliable protocol.  Out-of-order
   placement is obviously not dependent on whether the other DDP
   Segments ever actually arrive.

   However, RDMAP requires the LLP to provide reliable service.  An
   alternate completion handling protocol would be required if DDP were
   to be deployed over an unreliable IP transport.

   As noted in the prior section on tagged buffers as ULP credits,
   neither RDMAP or DDP provide any flow control for tagged messages.
   If no transport layer flow control is provided, an RDMAP/DDP
   application would be only limited by the link layer rate, almost
   inevitably resulting in severe network congestion.

   RDMAP encourages applications to be ignorant of the underlying
   transport PMTU.  The ULP is only notified when all messages ending in
   a single untagged message have completed.  The ULP is not aware of
   the granularity or ordering of the underlying message.  This approach
   assumes that the ULP is only interested in the complete set of
   messages, and has no use for a subset of them.

Bestler & Coene         Expires December 23, 2006              [Page 18]
Internet-Draft           RDMA/DDP Applicability                June 2006

6.9.  LLP Independent Session Establishment

   For an RDMAP/DDP application, the transport services provided by a
   pair of SCTP Streams and by a TCP connection both provide the same
   service (reliable delivery of DDP Segments between two connected
   RDMAP/DDP endpoints).

6.9.1.  RDMA-only Session Establishment

   It is also possible to allow for transport neutral establishment of
   RDMAP/DDP sessions between endpoints.  Combined, these two features
   would allow most applications to be unconcerned as to which LLP was
   actually in use.

   Specifically, the procedures for DDP Stream Session establishment
   discussed in section 3 of the SCTP mapping, and section 13.3 of the
   MPA/TCP mapping, both allow for the exchange of ULP specific data
   ("Private Data") before enabling the exchange of DDP Segments.  This
   delay can allow for proper selection and/or configuration of the
   endpoints based upon the exchanged data.  For example, each DDP
   Stream Session associated with a single client session might be
   assigned to the same DDP Protection Domain.

   To be transport neutral, the applications should exchange Private
   Data as part of session establishment messages to determine how the
   RDMA endpoints are to be configured.  One side must be the Initiator,
   and the other the Responder.

   With SCTP, a pair of SCTP streams can be used for successive sessions
   while the SCTP association remains open.  With MPA/TCP each
   connection can be used for at most one session.  However, the same
   source/destination pair of ports can be re-used sequentially subject
   to normal TCP rules.

   Both SCTP and MPA limit the private data size to a maximum of 512
   bytes.

   MPA/TCP requires the end of the TCP connection that initiated the
   conversion to MPA mode to send the first DDP Segment.  SCTP does not
   have this requirement.  ULPs which wish to be transport neutral
   should require the initiating end to send the first message.  A zero-
   length RDMA Write can be used for this purpose if the ULP logic
   itself does naturally support this restriction.

6.9.2.  RDMA-Conditional Session Establishment

   It is sometimes desirable for the active side of a session to connect
   with the passive side before knowing whether the passive side

Bestler & Coene         Expires December 23, 2006              [Page 19]
Internet-Draft           RDMA/DDP Applicability                June 2006

   supports RDMA.

   This style of session establishment can be supported with either TCP
   or SCTP, but not as transparently as for RDMA-only sessions.  Pre-
   existing non-RDMA servers are also far more likely to be using TCP
   than SCTP.

   With TCP. a normal TCP connection is established.  It is then used by
   the ULP to determine whether or not to convert to MPA mode and use
   RDMA.  This will typically be integral with other session
   establishment negotiations.

   With SCTP, the establishment of an association tests whether RDMA is
   supported.  If not supported, the application simply requests the
   association without the RDMA adaptation indication.

   One key difference is that with SCTP the determination as to whether
   the peer can support RDMA is made before the transport layer
   association/connection is established while with TCP the established
   connection itself is used to determine whether RDMA is supported.

Bestler & Coene         Expires December 23, 2006              [Page 20]
Internet-Draft           RDMA/DDP Applicability                June 2006

7.  Local Interface Implications

   Full utilization of DDP and RDMAP capabilities requires a local
   interface that explicitly requests these services.  Protocols such as
   Sockets Direct Protocol (SDP) can allow applications to keep their
   traditional byte-stream or message-stream interface and still enjoy
   many of the benefits of the optimized wire level protocols.

Bestler & Coene         Expires December 23, 2006              [Page 21]
Internet-Draft           RDMA/DDP Applicability                June 2006

8.  IANA Considerations

   There are no IANA considerations in this document.

Bestler & Coene         Expires December 23, 2006              [Page 22]
Internet-Draft           RDMA/DDP Applicability                June 2006

9.  Security considerations

   RDMA security considerations are discussed in [RDMA-SEC] [6].  This
   document will only deal with the more usage oriented aspects, and
   where there are implications in the choice of underlying transport.

9.1.  Connection/Association Setup

   Both the SCTP and TCP adaptations allow for existing procedures to be
   followed for the establishment of the SCTP association or TCP
   connection.  Use of DDP does not impair the use of any security
   measures to filter, validate and/or log the remote end of an
   association/connection.

9.2.  Tagged Buffer Exposure

   DDP only exposes ULP memory to the extent explicitly allowed by ULP
   actions.  These include posting of receive operations and enabling of
   Steering Tags.

   Neither RDMAP or DDP place requirements on how ULP's advertise
   buffers.  A ULP may use a single Steering Tag for multiple buffer
   advertisements.  However, the ULP should be aware that enforcement on
   STag usage is likely limited to the overall range that is enabled.
   If the remote peer writes into the 'wrong' advertised buffer, neither
   the DDP or RDMAP layer will be aware of this.  Nor is there any
   report to the ULP on how the remote peer specifically used tagged
   buffers.

   Unless the ULP peers have an adequate basis for mutual trust, the
   receiving ULP might be well advised to use a distinct STag for each
   interaction, and to invalidate it after each use or to require its
   peer to use the RDMAP option to invalidate the STag with its
   responding untagged message.

9.3.  Impact of Encrypted Transports

   While DDP is cleanly layered over the LLP, its maximum benefit may be
   limited when the LLP Stream is secured with a streaming cypher, such
   as Transport Layer Security (TLS) RFC4346 [9].  If the LLP must
   decrypt in order, it cannot provide out-of-order DDP Segments to the
   DDP layer for placement purposes.  IPsec RFC2406 [1]. tunnel mode
   encrypts entire IP Datagrams.  IPsec transport mode encrypts TCP
   Segments or SCTP packets, as does use of DTLS RFC4347 [10] over UDP
   beneath TCP or SCTP.  Neither IPsec nor this use of DTLS preclude
   providing out-of-order DDP Segments to the DDP layer for placement.

   Note that end-to-end use of cryptographic integrity protection may

Bestler & Coene         Expires December 23, 2006              [Page 23]
Internet-Draft           RDMA/DDP Applicability                June 2006

   allow suppression of MPA CRC generation and checking under certain
   circumstances.  This is one example where the LLP may be judged to
   have "or equivalent" protection to an end-to-end CRC32c.

Bestler & Coene         Expires December 23, 2006              [Page 24]
Internet-Draft           RDMA/DDP Applicability                June 2006

10.  References

10.1.  Normative references

   [1]  Kent, S. and R. Atkinson, "IP Encapsulating Security Payload
        (ESP)", RFC 2406, November 1998.

   [2]  Recio, R., "An RDMA Protocol Specification",
        draft-ietf-rddp-rdmap-05 (work in progress), July 2005.

   [3]  Shah, H., "Direct Data Placement over Reliable Transports",
        draft-ietf-rddp-ddp-05 (work in progress), July 2005.

   [4]  Stewart, R., "Stream Control Transmission Protocol (SCTP) Remote
        Direct Memory Access  (RDMA) Direct Data Placement (DDP)
        Adaptation", draft-ietf-rddp-sctp-03 (work in progress),
        June 2006.

   [5]  Culley, P., "Marker PDU Aligned Framing for TCP Specification",
        draft-ietf-rddp-mpa-04 (work in progress), May 2006.

   [6]  Pinkerton, J., "DDP/RDMAP Security", draft-ietf-rddp-security-09
        (work in progress), May 2006.

10.2.  Informative References

   [7]   Ko, M., "iSCSI Extensions for RDMA Specification",
         draft-ietf-ips-iser-05 (work in progress), October 2005.

   [8]   Callaghan, B. and T. Talpey, "NFS Direct Data Placement",
         draft-ietf-nfsv4-nfsdirect-02 (work in progress), October 2005.

   [9]   Dierks, T. and E. Rescorla, "The Transport Layer Security (TLS)
         Protocol Version 1.1", RFC 4346, April 2006.

   [10]  Rescorla, E. and N. Modadugu, "Datagram Transport Layer
         Security", RFC 4347, April 2006.

Bestler & Coene         Expires December 23, 2006              [Page 25]
Internet-Draft           RDMA/DDP Applicability                June 2006

Authors' Addresses

   Caitlin Bestler
   Broadcom Corporation
   16215 Alton Parkway
   P.O. Box 57013
   Irvine, CA  92619-7013
   USA

   Phone: 949-926-6383
   Email: caitlinb@broadcom.com

   Lode Coene
   Siemens
   Atealaan 26
   Herentals,   2200
   Belgium

   Phone: +32-14-252081
   Email: lode.coene@siemens.com

Bestler & Coene         Expires December 23, 2006              [Page 26]
Internet-Draft           RDMA/DDP Applicability                June 2006

Intellectual Property Statement

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at
   http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at
   ietf-ipr@ietf.org.

Disclaimer of Validity

   This document and the information contained herein are provided on an
   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
   ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
   INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
   INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Copyright Statement

   Copyright (C) The Internet Society (2006).  This document is subject
   to the rights, licenses and restrictions contained in BCP 78, and
   except as set forth therein, the authors retain all their rights.

Acknowledgment

   Funding for the RFC Editor function is currently provided by the
   Internet Society.

Bestler & Coene         Expires December 23, 2006              [Page 27]