A Remote Direct Memory Access Protocol Specification
RFC 5040

Network Working Group                                           R. Recio
Request for Comments: 5040                                    B. Metzler
Category: Standards Track                                IBM Corporation
                                                               P. Culley
                                                              J. Hilland
                                                 Hewlett-Packard Company
                                                               D. Garcia
                                                            October 2007

          A Remote Direct Memory Access Protocol Specification

Status of This Memo

   This document specifies an Internet standards track protocol for the
   Internet community, and requests discussion and suggestions for
   improvements.  Please refer to the current edition of the "Internet
   Official Protocol Standards" (STD 1) for the standardization state
   and status of this protocol.  Distribution of this memo is unlimited.

Abstract

   This document defines a Remote Direct Memory Access Protocol (RDMAP)
   that operates over the Direct Data Placement Protocol (DDP protocol).
   RDMAP provides read and write services directly to applications and
   enables data to be transferred directly into Upper Layer Protocol
   (ULP) Buffers without intermediate data copies.  It also enables a
   kernel bypass implementation.

Recio, et al.               Standards Track                     [Page 1]
RFC 5040              RDMA Protocol Specification           October 2007

Table of Contents

   1. Introduction ....................................................4
      1.1. Architectural Goals ........................................4
      1.2. Protocol Overview ..........................................5
      1.3. RDMAP Layering .............................................7
   2. Glossary ........................................................8
      2.1. General ....................................................8
      2.2. LLP .......................................................10
      2.3. Direct Data Placement (DDP) ...............................11
      2.4. Remote Direct Memory Access (RDMA) ........................13
   3. ULP and Transport Attributes ...................................15
      3.1. Transport Requirements and Assumptions ....................15
      3.2. RDMAP Interactions with the ULP ...........................16
   4. Header Format ..................................................19
      4.1. RDMAP Control and Invalidate STag Field ...................20
      4.2. RDMA Message Definitions ..................................23
      4.3. RDMA Write Header .........................................24
      4.4. RDMA Read Request Header ..................................24
      4.5. RDMA Read Response Header .................................26
      4.6. Send Header and Send with Solicited Event Header ..........26
      4.7. Send with Invalidate Header and Send with SE and
           Invalidate Header .........................................26
      4.8. Terminate Header ..........................................26
   5. Data Transfer ..................................................32
      5.1. RDMA Write Message ........................................32
      5.2. RDMA Read Operation .......................................33
           5.2.1. RDMA Read Request Message ..........................33
           5.2.2. RDMA Read Response Message .........................35
      5.3. Send Message Type .........................................36
      5.4. Terminate Message .........................................37
      5.5. Ordering and Completions ..................................38
   6. RDMAP Stream Management ........................................41
      6.1. Stream Initialization .....................................41
      6.2. Stream Teardown ...........................................42
           6.2.1. RDMAP Abortive Termination .........................43
   7. RDMAP Error Management .........................................43
      7.1. RDMAP Error Surfacing .....................................44
      7.2. Errors Detected at the Remote Peer on Incoming
           RDMA Messages .............................................45
   8. Security Considerations ........................................46
      8.1. Summary of RDMAP-Specific Security Requirements ...........46
           8.1.1. RDMAP (RNIC) Requirements ..........................47
           8.1.2. Privileged Resource Manager Requirements ...........48
      8.2. Security Services for RDMAP ...............................49
           8.2.1. Available Security Services ........................49
           8.2.2. Requirements for IPsec Services for RDMAP ..........50
   9. IANA Considerations ............................................51