STORM A. Kanevsky, Ed.
Internet-Draft VMware
Updates: 5043, 5044 (if approved) C. Bestler, Ed.
Intended status: Standards Track Nexenta Systems
Expires: December 15, 2011 R. Sharp
Intel
S. Wise
Open Grid Computing
June 13, 2011
Enhanced RDMA Connection Establishment
draft-ietf-storm-mpa-peer-connect-05
Abstract
This document updates RFC5043 and RFC5044 by extending MPA
negotiation for RDMA connection establishment. The first enhancement
extends RFC5043, enabling peer-to-peer connection establishment over
MPA/TCP. The second enhancement extends both RFC5043 and RFC5044, by
providing an option for standardized exchange of RDMA-layer
connection configuration.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on December 15, 2011.
Copyright Notice
Copyright (c) 2011 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
Kanevsky, et al. Expires December 15, 2011 [Page 1]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Summary of changes from RFC 5044 . . . . . . . . . . . . . 4
1.2. Summary of changes from RFC 5043 . . . . . . . . . . . . . 4
2. Requirements Language . . . . . . . . . . . . . . . . . . . . 4
3. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 4
4. Motivations . . . . . . . . . . . . . . . . . . . . . . . . . 5
4.1. Enabling MPA Mode . . . . . . . . . . . . . . . . . . . . 6
4.2. Lack of Explicit RTR in MPA Request/Reply Exchange . . . . 6
4.3. Limitations on ULP Workaround . . . . . . . . . . . . . . 7
4.3.1. Transport Neutral APIs . . . . . . . . . . . . . . . . 8
4.3.2. Work/Completion Queue Accounting . . . . . . . . . . . 8
4.3.3. Host-based Implementation of MPA Fencing . . . . . . . 9
4.4. Standardized RDMA Parameter Negotiation . . . . . . . . . 9
5. Enhanced MPA Connection Establishment . . . . . . . . . . . . 10
6. Enhanced MPA Request/Reply Frames . . . . . . . . . . . . . . 11
7. Enhanced SCTP Session Control Chunks . . . . . . . . . . . . . 12
8. MPA Error Reporting . . . . . . . . . . . . . . . . . . . . . 14
9. Enhanced RDMA Connection Establishment Data . . . . . . . . . 15
10. Interoperability . . . . . . . . . . . . . . . . . . . . . . . 17
11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 18
12. Security Considerations . . . . . . . . . . . . . . . . . . . 18
13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 18
14. References . . . . . . . . . . . . . . . . . . . . . . . . . . 18
14.1. Normative References . . . . . . . . . . . . . . . . . . . 18
14.2. Informative References . . . . . . . . . . . . . . . . . . 19
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 20
Kanevsky, et al. Expires December 15, 2011 [Page 2]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
1. Introduction
When used over Transmission Control Protocol TCP, the current Remote
Direct Data Placement (RDDP) suite of protocols relies on Marker
protocol Data Unit PDU Alignment (MPA) [RFC5044] protocol for both
connection establishment and for markers for TCP layering.
A typical model for establishing an RDMA connection has the following
steps:
o The passive side Upper Layer Protocol (ULP) listens for connection
requests.
o The active side ULP submits a connection request using an RDMA
endpoint, the desired destination and the parameters to be used
for the connection. Those parameters include both RDMA layer
characteristics, such as the RDMA Read credits to be allowed and
application specific data.
o The passive side ULP receives the Connection Request, which
includes the identity of the active side and the requested
connection characteristics. The passive side ULP uses this
information to decide whether to accept the connection, and if it
is to be accepted, how to create and/or configure the local RDMA
endpoint.
o If accepting, the passive side ULP submits its acceptance of the
Connection Request. This local accept operation includes the RDMA
endpoint to be used and the connection characteristics (both the
RDMA configuration and any application specific private data to be
returned).
o The active side receives confirmation that the connection has been
accepted, what the configured connection characteristics are, and
any application supplied private data.
Currently, MPA only supports a client-server model for connection
establishment, forcing peer-to-peer applications to interact as
though they had a client/server relationship. In addition
negotiation of some of Remote Direct Memory Access Protocol (RDMAP)
[RFC5040] specific parameters are left to ULP negotiation. Providing
an optional ULP-independent format for exchanging these parameters
would be of benefit to transport neutral Remote Direct Memory Access
(RDMA) applications.
Kanevsky, et al. Expires December 15, 2011 [Page 3]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
1.1. Summary of changes from RFC 5044
This draft enhances [RFC5044] MPA connection setup protocol. First,
it adds exchange and negotiation of the maximum number of RDMA Read
Incoming (IRD) and Outgoing (ORD) messages. Second, it adds a Ready
to Receive (RTR) frame from Initiator to Responder as the last
message of connection establishment and adds negotiation of an RTR
message frame type into MPA request/response frames.
1.2. Summary of changes from RFC 5043
This draft enhances [RFC5043] by adding new Enhanced Session Control
Chunks that extends the currently defined Chunks with the addition of
IRD and ORD negotiation.
2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
3. Definitions
FULPDU: Framed Upper Layer Protocol PDU. See FPDU of [RFC5044].
CRC: Cyclic Redundancy Check.
Completion Queue (CQ): A consumer accessible queue where the RDMA
device reports completions of Work Requests. A Consumer is able
to reap completions from a CQ without requiring per transaction
support from the kernel or other privileged entity. See [RDMAC].
Completion Queue Entry (CQE): Transport and device specific
representation of a Work Completion. A Completion Queue holds
CQEs. See [RDMAC].
Inbound RDMA Read Queue Depth (IRD): The maximum number of incoming
outstanding RDMA Read Request Messages an RDMA connection can
handle. See [RDMAC].
IRD: See Inbound RDMA Read Queue Depth.
MPA Fencing: MPA Responder Connection Establishment logic that
ensures that no ULP messages will be transferred until Initiator
first message has been received.
Kanevsky, et al. Expires December 15, 2011 [Page 4]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
ORD: See Outbound RDMA Read Queue Depth.
Outbound RDMA Read Queue Depth (ORD): The maximum number of
outstanding RDMA Read Requests that can be issued for the RDMA
connection. This should be less than or equal to the peer's IRD.
See [RDMAC].
Queue Pair (QP): The traditional name for a local Endpoint in a
[VIA] derived local interface. A Queue Pair is the set of Work
Queues associated exclusively with a single Endpoint. The Send
Queue (SQ), Receive Queue (RQ) and Inbound RDMA Read Queue (IRQ)
are considered to be part of the Queue Pair. The potentially
shared Completion Queue (CQ) and Shared Receive Queue (SRQ) are
not. See [RDMAC].
Ready to Receive (RTR): RTR is the last connection establishment
message sent from Initiator to Responder indicating that Initiator
is ready to receive messages and that connection establishment is
completed. See [IBTA].
Shared Receive Queue(SRQ): A shared pool of Receive Work Requests
posted by the Consumer that can be allocated by multiple RDMA
endpoints (Queue Pair). See [RDMAC].
Tagged (DDP) Message: - A DDP Message that targets a Tagged Buffer
that is explicitly Advertised to the Remote Peer through exchange
of an STag (memory handle), offset in the memory region identified
by STag, and length [RFC5040].
Work Queue: An element of a [VIA] derived local interface that
allows user-space applications to submit Work Requests directly to
network hardware. Specific Work Queues include the Send Queue
(SQ) for transmit requests, Receive Queue (RQ) for receive
requests specific to a single Endpoint and Shared Receive Queues
(SRQs) for receive requests that can be allocated by one or more
Endpoints. See [RDMAC].
Work Queue Element (WQE): Transport and device specific
representation of a Work Request. See [RDMAC].
Work Request: An elementary object used by Consumers to enqueue a
requested operation (WQEs) onto a Work Queue. See [RDMAC].
4. Motivations
The goal of this draft is twofold. One is to extend support from the
current client-server model for RDMA connection setup to a peer-to-
Kanevsky, et al. Expires December 15, 2011 [Page 5]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
peer model. The second is to add negotiation of RDMA Read queue size
for both sides of an RDMA connection.
4.1. Enabling MPA Mode
MPA defines encoding of DDP Segments in FULPDUs (Framed Upper Layer
Protocol PDUs). Generation of FULPDUs requires the ability to
periodically insert MPA Markers and to generate the MPA CRC-32c for
each frame. Reception may require parsing/removing the markers after
using them to identify MPA Frame boundaries, and validation of the
MPA-CRC32c.
A major design objective for MPA was to ensure that the resulting TCP
stream would be a fully compliant TCP stream for any and all TCP-
aware middle-boxes. The challenge is that while only some TCP
payload streams are a valid stream of MPA FULPDUs, any sequence of
bytes is a valid TCP payload stream. The determination that a given
stream is in a specific MPA mode cannot be made at the MPA or TCP
layer. Therefore enabling of MPA mode is handled by the Upper Level
Protocol (ULP).
The MPA protocol can be viewed as having two parts.
o a specification of generation and reception of MPA FULPDUs. This
is unchanged by enhanced RDMA connection establishment.
o a pre-MPA exchange of messages to enable a specific MPA mode for
the TCP connection. enhanced RDMA connection establishment extends
this protocol with two new features.
In typical implementations, generation and reception of MPA FULPDUs
is handled by hardware. The exchange of the MPA Request and Reply
frames is then handled by host software. As will be explained, this
implementation split impedes applications which are not compatible
with the client-server assumptions in the current MPA Request/Reply
exchange.
4.2. Lack of Explicit RTR in MPA Request/Reply Exchange
The exchange of MPA Request and Reply messages to place a TCP
connection in MPA mode is specified in [RFC5044]. This protocol
provides many benefits to the design of MPA FULPDU hardware:
o The ULP is responsible for specifying the exact MPA Mode (Markers
enabled or disabled, CRC-32c enabled or suppressed) and the point
in the TCP streams (inbound and outbound) where MPA frames will
begin.
Kanevsky, et al. Expires December 15, 2011 [Page 6]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
o Before the first MPA frame is transmitted, all pre-MPA mode TCP
payload will have been acknowledged by the peer. Therefore it is
never necessary to generate a retransmission that mixes pre-MPA
and MPA payload.
o Before MPA reception is enabled, all incoming pre-MPA mode TCP
payload will have been acknowledged. Therefore the host will
never receive a TCP segment that mixes pre-MPA and MPA payload.
The limitation of the current MPA Request/Reply exchange is that it
does not define a Ready to Receive (RTR) message that the active side
would send, so that the passive side can know that the last non-MPA
payload (the MPA Reply) had been received.
Instead, the role of an RTR message is piggy-backed on the first MPA
FULPDU sent by the active side. This is actually a valuable
optimization for all applications that fit the classic client/server
model. The client only initiates the connection when it has a
request to send to the server, and the server has nothing to send
until it has received and processed the client request.
Even applications where the server sends some configuration data
immediately can easily send the same information as application
private data in the MPA Reply. So the currently defined exchange
works for almost all applications.
Many peer-to-peer applications, especially those involving cluster
calculations (frequently using MPI [UsingMPI], or [RDS]), have no
natural client or server roles ([PPMPI], [OpenMP]). Typically one
member of the cluster is arbitrarily selected to initiate the
connection when the distributed task is launched, while the other
accepts it. At startup time, however, there is no way to predict
which node will have the first message to actually send.
Establishing the connections immediately, however, is valuable
because it reduces latency once results are ready to transmit and it
validates connectivity throughout the cluster.
The lack of an explicit RTR message in the MPA Request/Reply exchange
forces all applications to have a first message from the connection
Initiator, whether this matches the application communication model
or not.
4.3. Limitations on ULP Workaround
The requirement that the RDMA connection Initiator sends the first
message does not appear to be onerous on first examination. The
natural question is why the application layer would not simply
generate a "nop" message when there was no other message to submit.
Kanevsky, et al. Expires December 15, 2011 [Page 7]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
There are three factors that make this workaround unsuitable for many
peer-to-peer applications.
o Transport Neutral APIs.
o Work/Completion Queue Accounting.
o Host-based implementation of MPA Fencing.
4.3.1. Transport Neutral APIs
Many of these applications access RDMA services using a transport
neutral API such as [DAPL] or [OFA]. Only RDDP over TCP [RFC5044]
has a first message requirement. Other RDMA transports, including
RDDP over SCTP (see [RFC5043]) and InfiniBand (see [IBTA]), do not.
Application or middleware communications can be expressed as
transport neutral RDMA operations, allowing lower software layers to
translate to transport and device specifics. Having a distinct extra
message that is required only for one transport undermines the
application's goal of being transport neutral.
4.3.2. Work/Completion Queue Accounting
RDMA local APIs conventionally use work queues to submit requests
(work queue elements or WQEs) and to asynchronously receive
completions (in completion queues or CQs).
Each work request can generate a completion queue entry (CQE).
Completions for successful transmit work requests are frequently
suppressed, but the completion queue capacity must account for the
possibility that each will complete in error. A completion queue can
receive completions from multiple work queues.
Completion Queues are defined so as to allow hardware RDMA
implementations to generate CQEs directly to a user-space mapped
buffer. This enables a user-space RDMA consumer to reap completions
without requiring kernel intervention.
A hardware RDMA implementation cannot reasonably wait for an
available slot in the completion queue. The queue must be sized such
that an overflow will not occur. When an overflow does occur it is
considered catastrophic and will typically require tearing down all
RDMA connections using that CQ.
This style of interface is very efficient, but places a burden on the
application to properly size each Completion Queue to match the Work
Queues that feed it.
Kanevsky, et al. Expires December 15, 2011 [Page 8]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
While the format of both WQEs and CQEs is transport and device
dependent, a transport neutral API can deal with WQEs and CQEs as
abstract transport and device neutral objects. Therefore the number
of WQEs and CQEs required for an application can be transport and
device neutral.
The capacity of the work queues and completion queues can be
calculated in an abstract transport/device neutral fashion. If a nop
approach was used, it would require lower layers to know the usage
model, and would disrupt the calculations by inserting a dummy "nop"
Work Request and filtering out the matching completion. The lower
layer does not know the usage model on which the queue sizes are
built, nor does it know how frequently an insertion will be required.
4.3.3. Host-based Implementation of MPA Fencing
Many hardware implementations of RDDP using MPA/TCP do not handle the
MPA Request/Reply exchange in hardware, rather they are handled by
the host processor in software. With such designs it is common for
the MPA Fencing to be implemented in the user-space device-specific
library (commonly referred to as a 'User Verbs' library or module).
When the generation and reception of MPA FULPDUs is already dedicated
to hardware, a Work Completion can only be generated by an untagged
message since arrival of a message for tagged buffer does not
necessarily generates a completion and is done without any
interaction with ULP [RFC5040].
4.4. Standardized RDMA Parameter Negotiation
Most RDMA applications are developed using a transport neutral API to
access RDMA services based on a "queue pair" paradigm as originally
defined by the Virtual Interface Architecture [VIA], refined by the
Direct Access Programming Library [DAPL] and most commonly deployed
with the OpenFabrics API [OFA].
These transport neutral APIs seek to provide a common set of RDMA
services whether the underlying transport is, for example, RDDP over
MPA, RDDP over SCTP or InfiniBand.
The common model for establishing an RDMA connection has the
following steps:
o The passive side ULP listens for connection requests.
o The active side ULP submits a connection request using an RDMA
endpoint ("queue pair"), the desired destination and the
parameters to be used for the connection. Those parameters
Kanevsky, et al. Expires December 15, 2011 [Page 9]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
include both RDMA layer characteristics, such as the RDMA Read
credits to be allowed and application specific data (typically
referred to as "private data").
o The passive side ULP receives the Connection Request, which
includes the identity of the active side and the requested
connection characteristics. The passive side ULP uses this
information to decide whether to accept the connection, and if it
is to be accepted, how to create and/or configure the RDMA
endpoint.
o If accepting, the passive side ULP submits its acceptance of the
Connection Request. This local accept operation includes the RDMA
endpoint to be used and the connection characteristics (both the
RDMA configuration and any application specific private data to be
returned).
o The active side receives confirmation that the connection has been
accepted, what the configured connection characteristics are, and
any application supplied private data.
As currently defined, DDP connection establishment requires the ULP
to encode the RDMA configuration in the application specific private
data. This results undesirable duplication of logic to cover both
InfiniBand and RDDP, and to specify the extraction of the RDMA
characteristics from the ULP for each specific Upper Layer Protocol.
Both RDDP and InfiniBand support an initial private data exchange,
therefore a standard definition of the RDMA characteristics within
the private data section would enable common connection establishment
APIs to format the RDMA characteristics based on the same API
information used when establishing either protocol to form the
connection. The application would then only have to indicate that it
was using this standard format to enable common connection
establishment procedures to apply common code to properly parse these
fields and configure the RDMA endpoints accordingly.
5. Enhanced MPA Connection Establishment
Below we provide an overview of Enhanced Connection Setup. The goal
is to allow standard negotiation of ORD/IRD setting on both sides of
the RDMA connection and/or to negotiate the initial data transfer
operation by Initiator when the existing 'client sends first' rule
does not match application requirements.
The RDMA connection Initiator sends an MPA Request, as specified in
[RFC5044]; the new format defined here allows for:
Kanevsky, et al. Expires December 15, 2011 [Page 10]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
o Standardized negotiation of ORD and IRD.
o Negotiation of an RTR message.
The RDMA connection Responder processes the MPA Request and generates
an MPA Reply, as specified in [RFC5044]; the new format completes the
negotiation.
The local interface SHOULD require the ULP to explicitly configure on
a per-application or per-connection basis when an explicit RTR
message will be required. Piggy-backing the RTR on a Client's first
message is a valuable optimization for most connections.
The RDMA connection Initiator MUST NOT allow any later FULPDUs to be
transmitted before the RTR message. One method to achieve that is to
delay notifying the ULP that the RDMA connection has been established
until after any required RTR Message has been transmitted.
All MPA exchanges are performed via TCP prior to RDMA establishment,
and are therefore signaled via TCP and not via RDMA completion.
6. Enhanced MPA Request/Reply Frames
Enhanced RDMA connection establishment uses an alternate format for
MPA Requests and Replies, as follows:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 | |
+ Key (16 bytes containing "MPA ID Req Frame") +
4 | (4D 50 41 20 49 44 20 52 65 71 20 46 72 61 6D 65) |
+ Or (16 bytes containing "MPA ID Rep Frame") +
8 | (4D 50 41 20 49 44 20 52 65 70 20 46 72 61 6D 65) |
+ +
12 | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
16 |M|C|R|S| Res | Rev | PD_Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ ~
~ Private Data ~
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Kanevsky, et al. Expires December 15, 2011 [Page 11]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
Key: Unchanged from [RFC5044].
M: Unchanged from [RFC5044].
C: Unchanged from [RFC5044].
R: Unchanged from [RFC5044].
S: One if the Private Data begins with the enhanced RDMA connection
establishment data. Zero otherwise.
Res: One bit smaller than in [RFC5044], otherwise unchanged. In
[RFC5044] this field plus one more bit S used for enhancement is
reserved for future use and since it MUST be set to zero when
sending, and not checked on reception. Hence, it is backward
compatible to the original MPA frame format. With S bit set to
zero when no extra private data used for enhanced RDMA connection
establishment the MPA request/reply frame is identical to
unenhanced protocol. Unenhenced Responder/Initiator MUST NOT
check this field, hence no impact on existing implementations.
Rev: This field contains the revision of MPA. To use any enhanced
connection establishment feature this MUST be set to two or
higher, If no enhanced connection establishment features are
desired it MAY be set to one. A host accepting MPA connections
MUST continue to accept MPA Requests with version one even if it
supports version two.
PD_Length: Unchanged from [RFC5044]. This is the total length of
the Private Data field, including the enhanced RDMA connection
establishment data if present.
Private Data: Unchanged from [RFC5044]. However, if the 'S' flag is
set, Private Data begins with enhanced RDMA connection
establishment data.
7. Enhanced SCTP Session Control Chunks
The type of the SCTP Session Control Chunk is defined by a Function
Code (see [RFC4960]). [RFC5043] already defines codes for 'DDP
Stream Session Initiate' and 'DDP Stream Session Accept', which are
equivalent to a MPA Request Frame and an accepting MPA Reply Frame.
Enhanced RDMA connection establishment requires three additional
Function codes. All DDP Stream Session Functional Codes are listed
below:
Kanevsky, et al. Expires December 15, 2011 [Page 12]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
DDP Stream Session Initiate: 0x001
DDP Stream Session Accept: 0x002
DDP Stream Session Reject: 0x003
DDP Stream Session Terminate: 0x004
Enhanced DDP Stream Session Initiate: 0x005
Enhanced DDP Stream Session Accept: 0x006
Enhanced DDP Stream Session Reject: 0x007
The Enhanced Reject function code MUST be used to indicate rejection
of enhanced DDP stream session for a configuration that would have
been accepted for unenhanced DDP Stream Session negotiation.
The Enhanced DDP stream session establishment follows the same rules
as the standard DDP stream session establishment as defined in
[RFC5043]. ULP-supplied Private Data MUST be included for Enhanced
DDP Stream Session Initiate, Enhanced DDP Stream Session Accept, and
Enhanced DDP Stream Session Reject messages.
Private Data length MUST NOT exceed 512 bytes in any message,
including enhanced RDMA connection establishment data.
Private Data MUST NOT be included in the DDP Stream Session Terminate
message.
Received Extended DDP Stream Session Control messages SHOULD be
reported to the ULP. If reported, any supplied Private Data MUST be
available for the ULP to examine. An example, when received Extended
DDP Stream Session Control message is not reported to ULP if none of
the requested RTR message types are supported by receiver. In this
case, Provider CAN generate reject reply message indicating which RTR
message types it supports.
The enhanced DDP stream management MUST use the DDP stream session
termination function code to terminate a stream established using
enhanced DDP stream session function codes.
[RFC5043] already supports either side sending the first DDP Message
since the Payload Protocol Identifier (PPID) already distinguishes
between Session Establishment and DDP Segments. The enhanced RDMA
Connection Establishment provides to the ULP a transport independent
way to support peer-to-peer model.
Kanevsky, et al. Expires December 15, 2011 [Page 13]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
The following additional Legal Sequences of DDP Stream Session
messages are defined:
o Enhanced Active/Passive Session Accepted: as with section 6.2 of
[RFC5043], but with the extended opcodes as defined in this
document.
o Enhanced Active/Passive Session Rejected: as with section 6.3 of
[RFC5043], but with the extended opcodes as defined in this
document.
o Enhanced Active/Passive Session Non-ULP Rejected: as with section
6.4 of [RFC5043], but with the extended opcodes as defined in this
document.
8. MPA Error Reporting
The RDMA connection establishment protocol is layered upon [RFC5040]
and [RFC5041]. Any enhanced RDMA connection establishment error
generates an MPA termination message to a peer. [RFC5040] defines a
triplet of protocol layers, error types and error codes for error
specification. MPA negotiation for RDMA connection establishment
uses the following layer and error type for MPA error reporting:
Layer: 0010b - LLP/MPA
Error Type: 0x3 - Lower Layer Protocol (LLP)
While [RFC5044] defines four error codes, [RFC5043] does not define
any. Enhanced RDMA connection establishment extends [RFC5044] error
codes by adding two new error codes. Thus, enhanced RDMA connection
establishment is backward compatible with both [RFC5043] and
[RFC5044].
The following error codes are defined for enhanced RDMA connection
establishment negotiation:
Error Code Description
--------------------------------------------------------
0x5 Local Catastrophic
0x6 Insufficient IRD resources
0x7 No matching RTR option
Kanevsky, et al. Expires December 15, 2011 [Page 14]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
9. Enhanced RDMA Connection Establishment Data
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 |A|B| IRD |C|D| ORD |
4 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
IRD: In request: Maximum IRD Initiator can support for the requested
connection. In reply: Responder local endpoint IRD setting.
ORD: In request: Initiator ORD setting for the connection that
corresponds to Initiator requested IRD at the peer. In reply:
Responder local endpoint ORD setting.
A: Control Flag that indicates whether peer-to-peer or client-server
model is requested. The TRUE value represents peer-to-peer model
with B, C, and D flags representing types of Ready to Receive
(RTR) message supported by Initiator and accepted by Responder,
respectively. The FALSE value represents client-server model and
values of flags B, C, D MUST be set to zero on transmit and
ignored on receive by both Initiator and Responder.
B: Control Flag for using a zero length FULPDU as the RTR message.
C: Control Flag for using a zero length RDMA Write as the RTR
message.
D: Control Flag for using a zero length RDMA Read as the RTR message.
For ORD/IRD negotiation both Responder and Initiator MUST pass the
remote side provided IRD and ORD to ULP. Responder SHOULD set its
IRD at least to Initiator ORD, unless it does not have sufficient
resources for it. Responder SHOULD NOT set its IRD higher than
Initiator ORD. For example, Responder MAY set IRD to one if
Initiator ORD is zero and Responder supports zero-size RDMA Read RTR
message. Responder MUST set its ORD at most to the Initiator IRD.
Responder CAN reject connection request if Initiator IRD is not
sufficient for ULP required ORD and specify in MPA Reject frame
Responder required ORD.
An all ones value (0x3FFF) indicates that automatic negotiation of
the IRD and ORD is not desired, and that the ULP will be responsible
for doing this configuration.
Upon receiving MPA Accept frame from Responder, Initiator MUST set
Kanevsky, et al. Expires December 15, 2011 [Page 15]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
its IRD at least to Responder ORD. Upon receiving MPA Accept frame
from Responder, Initiator MUST set its ORD at most to Responder IRD.
If Initiator does not have sufficient resources to satisfy Responder
ORD it MUST terminate the connection and report to local ULP
Responder's ORD and IRD with an indicator of insufficient resources
to satisfy Responder ORD, and MUST send termination message to
Responder indicating insufficient resources. Thus, TERM message MUST
contain Layer 2, Error Type 3, Error Code 6. Initiator MUST pass the
remote side provided IRD and ORD to ULP for both MPA Accept and
Reject. Initiator ULP CAN terminate the established connection if
Responder IRD is not sufficient.
In the following rules, TRUE value for each Control Flag field
corresponds to one, and FALSE corresponds to zero.
In the MPA Request, Initiator MUST set Control Flag A to TRUE if
peer-to-peer model is requested, zero otherwise.
In the MPA Request, Initiator MUST set each Control Flags B, C and D
to TRUE for each of the options it supports, if Control Flag A is set
to TRUE. If Control Flag A is not set, then Initiator MUST set
Control Flags B, C, and D to zero.
In the MPA Reply, the Control Flag A MUST match the Control Flag
value of MPA Request.
If Control Flag A is not set in MPA Reply, then Responder MUST ignore
values of remaining Control Flags.
In the MPA Reply, if Control Flag A is set then Responder SHOULD set
Control Flags B, C and D for the options that Responder will accept
as an RTR message. Responder MAY include only RTR options it CAN
support that match Initiator Control Flags specified RTR options.
MPA Reply SHOULD include all options that Responder CAN support
without requiring a connection specific enabling action. For
example, if Responder only supports one choice of RTR message, then
Responder SHOULD set Control Flag for it only, irrespective what
Initiator options specified. Options which require connection
specific enabling actions SHOULD NOT be set unless the corresponding
flag was set in the MPA Request. Responder MAY choose to limit the
number of RTR modes that it enables. For example, it may clear the C
flag to disallow zero size RDMA read as RTR message if it does not
have sufficient resources to support it, or if negotiated IRD/ORD do
not allow this option.
When Control Flags B, C, and D are not set, but A is set in the MPA
Reply indicates that an RDMA Send message will be required. As this
option will require Initiator ULP to be involved it MUST NOT be used
Kanevsky, et al. Expires December 15, 2011 [Page 16]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
unless necessary. It SHOULD only be used if ULP requires peer-to-
peer model support and Responder does not support any RTR options
presented by Initiator.
Upon receiving MPA accept frame with Control Flag A set, Initiator
MUST generate any of the negotiated RTR messages. If Initiator CAN
NOT generate any of the Responder supported RTR messages, then it
MUST terminate the connection and report to local ULP inability to
support peer-to-peer connection model to satisfy Responder RTR
message options, and MUST send termination message to Responder
indicating no matching RTR option. Thus, TERM message MUST contain
Layer 2, Error Type 3, Error Code 7. ULP CAN negotiate ULP level RTR
message when local and remote Providers do not have an RTR match.
The RTR message type and ORD/IRD negotiation follows the following
order:
Initiator -->: Sets Control Flags to indicate Initiator connection
model, and for peer-to-peer one the Initiator-supported forms of RTR
and Initiator initial IRD, and ORD setting on local Endpoint of the
connection
Responder <--: Sets Control Flags to indicate Responder-supported
forms of RTR or peer-to-peer connection model and the ORD/IRD depth
Responder will support
Initiator -->: After Initiator modifies its ORD/IRD to match
Responder's ones s stated above, and if peer-to-peer model is
requested and at least one form of RTR is supported by both Initiator
and Responder, then the first message sent by Initiator MUST be an
RTR using a form supported by both the Initiator and Responder.
Initiator or Responder MUST generate the TERM message that contains
Layer 2, Error Type 3, Error Code 5 when it encounters any error
locally for which the special Error Code is not defined in section
Section 8 before resetting the connection.
10. Interoperability
Initiator MUST NOT use the enhanced RDMA connection establishment
formats or function codes when no enhanced functionality is desired.
Responder MUST continue to accept the unenhanced connection requests.
There are three Initiator/Responder cases that involve enhanced MPA:
both Initiator and Responder, only Responder, and only Initiator.
The enhanced MPA frame is defined by field 'S' set to 1.
Kanevsky, et al. Expires December 15, 2011 [Page 17]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
Enhanced MPA Initiator and Responder: If Responder receives an
enhanced MPA message, it MUST respond with an enhanced MPA
message.
Enhanced MPA Responder only: If Responder receives an unenhanced MPA
message ('S' is set to 0), it MUST respond with an unenhanced MPA
message.
Enhanced MPA Initiator only: If Responder receives an enhanced MPA
message and it does not support enhanced RDMA connection
establishment, it MUST close the TCP connection and exit MPA.
From standard RDMA connection establishment point of view enhanced
MPA frame is improperly formatted as stated in [RFC5044]. Thus,
both Initiator and Responder report TCP connection termination to
an application locally. In this case Initiator MAY attempt to
establish RDMA connection using unenhanced MPA protocol as defined
in [RFC5044] if this protocol is compatible with the application,
and let ULP deal with ORD and IRD, and peer-to-peer negotiations.
11. IANA Considerations
This document has no IANA considerations.
12. Security Considerations
The security considerations from RFC 5044 apply and the changes in
this document do not introduce new security considerations.
13. Acknowledgements
The authors wish to thank Sean Hefty, Dave Minturn, Tom Talpey and
David Black for their valuable contributions and reviews of this
document.
14. References
14.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC4960] Stewart, R., "Stream Control Transmission Protocol",
RFC 4960, September 2007.
Kanevsky, et al. Expires December 15, 2011 [Page 18]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
[RFC5040] Recio, R., Metzler, B., Culley, P., Hilland, J., and D.
Garcia, "A Remote Direct Memory Access Protocol
Specification", RFC 5040, October 2007.
[RFC5041] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
Data Placement over Reliable Transports", RFC 5041,
October 2007.
[RFC5043] Bestler, C. and R. Stewart, "Stream Control Transmission
Protocol (SCTP) Direct Data Placement (DDP) Adaptation",
RFC 5043, October 2007.
[RFC5044] Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
Carrier, "Marker PDU Aligned Framing for TCP
Specification", RFC 5044, October 2007.
14.2. Informative References
[DAPL] "Direct Access Programming Library",
<http://www.datcollaborative.org>.
[IBTA] "InfiniBand Architecture Specification Release 1.2.1", <ht
tp://www.infinibandta.org/content/
pages.php?pg=technology_overview>.
[OFA] "OFA verbs & APIs", <http://www.openfabrics.org/>.
[OpenMP] McGraw-Hill, "Parallel Programming in C with MPI and
OpenMP", 2003.
[PPMPI] Morgan Kaufmann Publishers Inc., "Parallel Programming
with MPI", 2008.
[RDMAC] "RDMA Protocol Verbs Specification (Version 1.0)", <http:/
/www.rdmaconsortium.org/home/
draft-hilland-iwarp-verbs-v1.0-RDMAC.pdf>.
[RDS] Open Fabrics Association, "Reliable Datagram Socket",
2008, <http://www.openfabrics.org/archives/
spring2008sonoma/Tuesday/sonoma_2008_0408%20Oracle.ppt>.
[UsingMPI]
MIT Press, "Using MPI-2: Advanced Features of the Message
Passing Interface", 1999.
[VIA] Compaq, Intel, Microsoft, "Virtual Interface Architecture
Specification", 1997, <http://pllab.cs.nthu.edu.tw/cs5403/
Readings/EJB/san_10.pdf>.
Kanevsky, et al. Expires December 15, 2011 [Page 19]
Internet-Draft Enhanced RDMA Connection Establishment June 2011
Authors' Addresses
Arkady Kanevsky (editor)
VMware
5 Cambridge Center
Cambridge, MA 02142
USA
Phone: +1-617-528-7721
Email: arkady.kanevsky@gmail.com
Caitlin Bestler (editor)
Nexenta Systems
555 E El Camino Real #104
Sunnyvale, CA 94087
USA
Phone: +1-949-528-3085
Email: Caitlin.Bestler@nexenta.com
Robert Sharp
Intel
LAD High Performance Message Passing, Mailstop: AN1-WTR1
1501 South Mopac, Suite 400
Austin, TX 78746
USA
Phone: +1-512-493-3242
Email: robert.o.sharp@intel.com
Steve Wise
Open Grid Computing
4030 Braker Lane STE 130
Austin, TX 78759
USA
Phone: +1-512-343-9196 x101
Email: swise@opengridcomputing.com
Kanevsky, et al. Expires December 15, 2011 [Page 20]