Remote Direct Memory Access - Connection Manager (RDMA-CM) Private Data for RPC-over-RDMA Version 1
RFC 8797

Document Type RFC - Proposed Standard (June 2020; No errata)
Updates RFC 8166
Author Chuck Lever 
Last updated 2020-06-25
Stream Internent Engineering Task Force (IETF)
Formats plain text html xml pdf htmlized (tools) htmlized bibtex
Stream WG state Submitted to IESG for Publication
Document shepherd Brian Pawlowski
Shepherd write-up Show (last changed 2019-10-30)
IESG IESG state RFC 8797 (Proposed Standard)
Action Holders
Consensus Boilerplate Yes
Telechat date
Responsible AD Magnus Westerlund
Send notices to Spencer Shepler <>, Thomas Haynes <>, Brian Pawlowski <>
IANA IANA review state Version Changed - Review Needed
IANA action state RFC-Ed-Ack

Internet Engineering Task Force (IETF)                          C. Lever
Request for Comments: 8797                                        Oracle
Updates: 8166                                                  June 2020
Category: Standards Track                                               
ISSN: 2070-1721

Remote Direct Memory Access - Connection Manager (RDMA-CM) Private Data
                      for RPC-over-RDMA Version 1


   This document specifies the format of Remote Direct Memory Access -
   Connection Manager (RDMA-CM) Private Data exchanged between RPC-over-
   RDMA version 1 peers as part of establishing a connection.  The
   addition of the Private Data payload specified in this document is an
   optional extension that does not alter the RPC-over-RDMA version 1
   protocol.  This document updates RFC 8166.

Status of This Memo

   This is an Internet Standards Track document.

   This document is a product of the Internet Engineering Task Force
   (IETF).  It represents the consensus of the IETF community.  It has
   received public review and has been approved for publication by the
   Internet Engineering Steering Group (IESG).  Further information on
   Internet Standards is available in Section 2 of RFC 7841.

   Information about the current status of this document, any errata,
   and how to provide feedback on it may be obtained at

Copyright Notice

   Copyright (c) 2020 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   ( in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction
   2.  Requirements Language
   3.  Advertised Transport Properties
     3.1.  Inline Threshold Size
     3.2.  Remote Invalidation
   4.  Private Data Message Format
     4.1.  Using the R Field
     4.2.  Send and Receive Size Values
   5.  Interoperability Considerations
     5.1.  Interoperability with RPC-over-RDMA Version 1
     5.2.  Interoperability amongst RDMA Transports
   6.  Updating the Message Format
   7.  Security Considerations
   8.  IANA Considerations
     8.1.  Guidance for Designated Experts
   9.  References
     9.1.  Normative References
     9.2.  Informative References
   Author's Address

1.  Introduction

   The RPC-over-RDMA version 1 transport protocol [RFC8166] enables
   payload data transfer using Remote Direct Memory Access (RDMA) for
   upper-layer protocols based on Remote Procedure Calls (RPCs)
   [RFC5531].  The terms "Remote Direct Memory Access" (RDMA) and
   "Direct Data Placement" (DDP) are introduced in [RFC5040].

   The two most immediate shortcomings of RPC-over-RDMA version 1 are as

   1.  Setting up an RDMA data transfer (via RDMA Read or Write) can be
       costly.  The small default size of messages transmitted using
       RDMA Send forces the use of RDMA Read or Write operations even
       for relatively small messages and data payloads.

       The original specification of RPC-over-RDMA version 1 provided an
       out-of-band protocol for passing inline threshold values between
       connected peers [RFC5666].  However, [RFC8166] eliminated support
       for this protocol, making it unavailable for this purpose.

   2.  Unlike most other contemporary RDMA-enabled storage protocols,
       there is no facility in RPC-over-RDMA version 1 that enables the
       use of remote invalidation [RFC5042].

   Each RPC-over-RDMA version 1 Transport Header follows the External
   Data Representation (XDR) definition [RFC4506] specified in
   [RFC8166].  However, RPC-over-RDMA version 1 has no means of
   extending this definition in such a way that interoperability with
   existing implementations is preserved.  As a result, an out-of-band
   mechanism is needed to help relieve these constraints for existing
   RPC-over-RDMA version 1 implementations.

   This document specifies a simple, non-XDR-based message format
   designed to be passed between RPC-over-RDMA version 1 peers at the
   time each RDMA transport connection is first established.  The
   mechanism assumes that the underlying RDMA transport has a Private
   Data field that is passed between peers at connection time, such as
   is present in the Marker PDU Aligned Framing (MPA) protocol
   (described in Section 7.1 of [RFC5044] and extended in [RFC6581]) or
   the InfiniBand Connection Manager [IBA].

   To enable current RPC-over-RDMA version 1 implementations to
Show full document text