STORM A. Kanevsky, Ed.
Internet-Draft EMC
Updates: 5043, 5044 C. Bestler, Ed.
(if approved) Consultant
Intended status: Standards Track D. Minturn
Expires: August 24, 2010 Intel
S. Wise
Open Grid Computing
February 20, 2010
Enhanced RDMA Connection Establishment
draft-ietf-storm-mpa-peer-connect-01
Abstract
Extensions to MPA are specified for RDMA Connection establishment.
The first extension extends RFC5043, enabling peer-to-peer connection
establishment over MPA/TCP. The second extension extends both
RFC5043 and RFC5044, by providing an option for standardized exchange
of RDMA-layer connection configuration.
Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on August 24, 2010.
Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the
Kanevsky, et al. Expires August 24, 2010 [Page 1]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Summary of changes from RFC 5044 . . . . . . . . . . . . . 3
1.2. Summary of changes from RFC 5043 . . . . . . . . . . . . . 3
2. Requirements Language . . . . . . . . . . . . . . . . . . . . 3
3. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 3
4. Motivations . . . . . . . . . . . . . . . . . . . . . . . . . 4
4.1. Enabling MPA Mode . . . . . . . . . . . . . . . . . . . . 5
4.2. Lack of Explicit RTR in MPA Request/Reply Exchange . . . . 5
4.3. Limitations on ULP Workaround . . . . . . . . . . . . . . 6
4.3.1. Transport Neutral APIs . . . . . . . . . . . . . . . . 7
4.3.2. Work/Completion Queue Accounting . . . . . . . . . . . 7
4.3.3. Host-based Implementation of MPA Fencing . . . . . . . 8
4.4. Standardized RDMA Parameter Negotiation . . . . . . . . . 8
5. MPA Connection Setup . . . . . . . . . . . . . . . . . . . . . 9
6. Enhanced MPA Request/Reply Frames . . . . . . . . . . . . . . 10
7. Enhanced SCTP Session Control Chunks . . . . . . . . . . . . . 11
8. Enhanced RDMA Connection Establishment Data . . . . . . . . . 12
9. Interoperability . . . . . . . . . . . . . . . . . . . . . . . 13
10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13
11. Security Considerations . . . . . . . . . . . . . . . . . . . 13
12. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 13
13. References . . . . . . . . . . . . . . . . . . . . . . . . . . 13
13.1. Normative References . . . . . . . . . . . . . . . . . . . 13
13.2. Informative References . . . . . . . . . . . . . . . . . . 14
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 14
Kanevsky, et al. Expires August 24, 2010 [Page 2]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
1. Introduction
When used over TCP, the current RDDP suite of protocols relies on MPA
RFC 5044 [RFC5044] protocol for both connection establishment and for
markers for TCP layering. Currently MPA only supports a client-
server model for connection establishment, forcing peer-to-peer
applications to interact as though they had a client/server
relationship. In addition negotiation of some of RDMAP RFC 5040
[RFC5040] specific parameters are left to ULP negotiation. Providing
an optional ULP-independent format for exchanging these parameters
would be of benefit to transport neutral RDMA applications.
1.1. Summary of changes from RFC 5044
This draft extends RFC 5044 [RFC5044] MPA connection setup protocol.
First, it add exchange and negotiation of maximum number of RDMA Read
Incoming (IRD) and Outgoing messages (ORD). Second, it adds one more
Ready to Receive (RTR) frame from requestor to responder as the last
message of connection establishment and adds negotiation of RTR frame
message type into MPA request/response frames.
1.2. Summary of changes from RFC 5043
This draft extends RFC 5043 [RFC5043] by adding new Enhanced Session
Control Chunks that extend the currently defined Chunks with the
addition of IRD and ORD negotiation.
2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
3. Definitions
FULPDU: Framed Upper Layer Protocol PDU. See [RFC5044].
Completion Queue (CQ): A consumer accessible queue where the RDMA
device reports completions of Work Requests. A Consumer is able
to reap completions from a CQ without requiring per transaction
support from the kernel or other privileged entity. See [RDMAC]
Completion Queue Entry (CQE): Transport and device specific
representation of a Work Completion. A Completion Queue holds
CQEs. See [RDMAC]
Kanevsky, et al. Expires August 24, 2010 [Page 3]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
Inbound RDMA Read Queue Depth (IRD): The maximum number of incoming
outstanding RDMA Read Request Messages an RDMA connection can
handle. See [RDMAC]
IRD: See Inbound RDMA Read Queue Depth.
ORD: See Outbound RDMA Read Queue Depth.
Outbound RDMA Read Queue Depth (ORD): The maximum number of
outstanding RDMA Read Requests that can be issued for the RDMA
connection. This should be less than or equal to the peer's IRD.
See [RDMAC]
Queue Pair (QP): The traditional name for a local Endpoint in a
[VIA] derived local interface. A Queue Pair is the set of Work
Queues associated exclusively with a single Endpoint. The Send
Queue (SQ), Receive Queue (RQ) and Inbound RDMA Read Queue (IRQ)
are considered to be part of the Queue Pair. The potentially
shared Completion Queue (CQ) and Shared Receive Queue (SRQ) are
not. See [RDMAC].
Shared Receive Queue(SRQ): A shared pool of Receive Work Requests
posted by the Consumer that can be allocated by multiple RDMA
endpoints (Queue Pair). See [DAPL].
Work Queue: An element of a [VIA] derived local interface that
allows user-space applications to submit Work Requests directly to
network hardware. Specific Work Queues include the Send Queue
(SQ) for transmit requests, Receive Queue (RQ) for receive
requests specific to a single Endpoint and Shared Receive Queues
(SRQs) for receive requests that can be allocated by one or more
Endpoints. See [RDMAC].
Work Queue Element (WQE): Transport and device specific
representation of a Work Request. See [RDMAC].
Work Request: An elementary object used by Consumers to enqueue a
requested operation (WQEs) onto a Work Queue. See [RDMAC].
4. Motivations
The goal of this draft is twofold. One is to extend support from the
current client-server model for RDMA connection setup to a peer-to-
peer model. The second is to add negotiation of RDMA Read queue size
for both sides of an RDMA connection.
Kanevsky, et al. Expires August 24, 2010 [Page 4]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
4.1. Enabling MPA Mode
MPA defines encoding of DDP Segments in FULPDUs (Framed Upper Layer
Protocol PDUs). Generation of FULPDUs requires the ability to
periodically insert MPA Markers and to generate the MPA CRC-32c for
each frame. Reception may require parsing/removing the markers after
using them to identify MPA Frame boundaries, and validation of the
MPA-CRC32c.
A major design objective for MPA was to ensure that the resulting TCP
stream would be a fully compliant TCP stream for any and all TCP-
aware middle-boxes. The challenge is that while only some TCP
payload streams are a valid stream of MPA FULPDUs, any sequence of
bytes is a valid TCP payload stream. The determination that a given
stream is in a specific MPA mode cannot be made at the MPA or TCP
layer. Therefore enabling of MPA mode is handled by the ULP.
The MPA protocol can be viewed as having two parts.
o a specification of generation and reception of MPA FULPDUs. This
is unchanged by Enhanced RDMA Connection Establishment.
o a pre-MPA exchange of messages to enable a specific MPA mode for
the TCP connection. Enhanced RDMA Connection Establishment
extends this protocol with two new features.
In typical implementations, generation and reception of MPA FULPDUs
is handled by hardware. The exchange of the MPA Request and Reply
frames is then handled by host software. As will be explained, this
implementation split prevents applications from working around the
client-server assumptions in the current MPA Request/Reply exchange.
4.2. Lack of Explicit RTR in MPA Request/Reply Exchange
The exchange of MPA Request and Reply messages to place a TCP
connection in MPA mode is specified in RFC 5044 [RFC5044]. This
protocol provides many benefits to the design of MPA FULPDU hardware:
o The ULP is responsible for specifying the exact MPA Mode (Markers
enabled or disabled, CRC-32c enabled or suppressed) and the point
in the TCP streams (inbound and outbound) where MPA frames will
begin.
o Before the first MPA frame is transmitted, all pre-MPA mode TCP
payload will have been acknowledged by the peer. Therefore it is
never necessary to generate a retransmission that mixes pre-MPA
and MPA payload.
Kanevsky, et al. Expires August 24, 2010 [Page 5]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
o Before MPA reception is enabled, all incoming pre-MPA mode TCP
payload will have been acknowledged. Therefore the host will
never receive a TCP segment that mixes pre-MPA and MPA payload.
The limitation of the current MPA Request/Reply exchange is that it
does not define a Ready to Receive (RTR) message that the active side
would send, so that the passive side can know that the last non-MPA
payload (the MPA Reply) had been received.
Instead, the role of an RTR message is piggy-backed on the first MPA
FULPDU sent by the active side. This is actually a valuable
optimization for all applications that fit the classic client/server
model. The client only initiates the connection when it has a
request to send to the server, and the server has nothing to send
until it has received and processed the client request.
Even applications where the server sends some configuration data
immediately can easily send the same information as application
private data in the MPA Reply. So the currently defined exchange
works for almost all applications.
Many peer-to-peer applications, especially those involving cluster
calculations (frequently using MPI [UsingMPI], or [RDS]), have no
natural client or server roles ([PPMPI], [OpenMP]). Typically one
member of the cluster is arbitrarily selected to initiate the
connection when the distributed task is launched, while the other
accepts it. At startup time, however, there is no way to predict
which node will have the first message to actually send.
Establishing the connections immediately, however, is valuable
because it reduces latency once results are ready to transmit and it
validates connectivity throughout the cluster.
The lack of an explicit RTR message in the MPA Request/Reply exchange
forces all applications to have a first message from the connection
initiator, whether this matches the application communication model
or not.
4.3. Limitations on ULP Workaround
The requirement that the RDMA connection initiator sends the first
message does not appear to be that onerous on first examination. The
natural question is why the application layer would not simply
generate a "nop" message when there was no other message to submit.
There are three factors that make this workaround unsuitable for many
peer-to-peer applications.
Kanevsky, et al. Expires August 24, 2010 [Page 6]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
o Transport Neutral APIs.
o Work/Completion Queue Accounting.
o Host-based implementation of MPA Fencing.
4.3.1. Transport Neutral APIs
Many of these applications access RDMA services using a transport
neutral API just as [DAPL] or [OFA]. Only MPA has a first message
requirement. Other RDMA transports, including SCTP and InfiniBand,
do not.
Application or middleware communications can be expressed as
transport neutral RDMA operations, allowing lower software layers to
translate to transport and device specifics. Having a distinct extra
message that is required only for one transport undermines the
application's goal of being transport neutral.
4.3.2. Work/Completion Queue Accounting
RDMA local APIs conventionally use work queues to submit requests
(work queue elements or WQEs) and to asynchronously receive
completions (in completion queues or CQs).
Each work request can generate a completion queue entries (CQE).
Completions for successful transmit work requests are frequently
suppressed, but the completion queue capacity must account for the
possibility that each will complete in error. A completion queue can
receive completions from multiple work queues.
Completion Queues are defined so as to allow hardware RDMA
implementations to generate CQEs directly to a user-space mapped
buffer. This enables a user-space RDMA consumer to reap completions
without requiring kernel intervention.
A hardware RDMA implementation cannot reasonably wait for an
available slot in the completion queue. The queue must be sized such
that an overflow will not occur. When an overflow does occur it is
considered catastrophic and will typically require tearing down all
RDMA connections using that CQ.
This style of interface is very efficient, but places a burden on the
application to properly size each Completion Queue to match the Work
Queues that feed it.
While the format of both WQEs and CQEs is transport and device
dependent, a transport neutral API can deal with WQEs and CQEs as
Kanevsky, et al. Expires August 24, 2010 [Page 7]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
abstract transport and device neutral objects. Therefore the number
of WQEs and CQEs required for an application can be transport and
device neutral.
The capacity of the work queues and completion queues can be
calculated in an abstract transport/device neutral fashion. Lower
layers of the protocol stack cannot disrupt these calculations by
inserting a dummy "nop" Work Request and filtering out the matching
completion. The lower layer does not know the usage model on which
the queue sizes are built, nor does it know how frequently an
insertion will be required.
4.3.3. Host-based Implementation of MPA Fencing
Many hardware implementations of iWARP using MPA/TCP do not handle
the MPA Request/Reply exchange in hardware, rather they are handled
by the host processor in software. With such designs it is common
for the MPA Fencing to be implemented in the user-space device-
specific library (commonly referred to as a 'User Verbs' library or
module).
When the generation and reception of MPA FULPDUs is already dedicated
to hardware, a Work Completion can only be generated by an untagged
message.
4.4. Standardized RDMA Parameter Negotiation
Most RDMA applications are developed using a transport neutral API to
access RDMA services based on a "queue pair" paradigm as originally
defined by the Virtual Interface Architecture [VIA], refined by the
Direct Access Programming Library [DAPL] and most commonly deployed
with the OpenFabrics API [OFA]
These transport neutral APIs seek to provide a common set of RDMA
services whether the underlying transport is, for example, iWARP over
MPA, iWARP over SCTP or InfiniBand.
The common model for establishing an RDMA connection has the
following steps:
o The passive side ULP listens for connection requests.
o The active side ULP submits a connection request using an RDMA
endpoint ("queue pair"), the desired destination and the
parameters to be used for the connection. Those parameters
include both RDMA layer characteristics, such as the RDMA Read
credits to be allowed and application specific data (typically
referred to as "private data").
Kanevsky, et al. Expires August 24, 2010 [Page 8]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
o The passive side ULP receives the Connection Request, which
includes the identity of the active side and the requested
connection characteristics. The passive side ULP uses this
information to decide whether to accept the connection, and if it
is to be accepted, how to create and/or configure the RDMA
endpoint.
o If accepting, the passive side ULP submits its acceptance of the
Connection Request. This local accept operation includes the RDMA
endpoint to be used and the connection characteristics (both the
RDMA configuration and any application specific private data to be
returned).
o The active side receives confirmation that the connection has been
accepted, what the configured connection characteristics are, and
any application supplied private data.
As currently defined, DDP connection establishment requires the ULP
to encode the RDMA configuration in the application specific private
data. This results undesirable duplication of logic to cover both
InfiniBand and iWARP, and to specify the extraction of the RDMA
characteristics from the ULP for each specific Upper Layer Protocol.
A standard definition of the RDMA characteristics within the private
data section would enable common connection establishment APIs to
format the RDMA characteristics based on the same API information
used when establishing an InfiniBand connection. The application
would then only have to indicate that it was using this standard
format to enable common connection establishment procedures to apply
common code to properly parse these fields and configure the RDMA
endpoints accordingly.
5. MPA Connection Setup
Below we provide overview of Enhanced Connection Setup. The goal is
to allow standard negotiation of ORD/IRD setting on both sides of the
RDMA connection and/or to negotiate the initial data transfer
operation by an initiator when the existing 'client sends first' rule
does not match application requirements.
The RDMA connection initiator sends an MPA Request, as specified in
[RFC5044]; the new format defined here allows for:
o Standardized negotiation of ORD and IRD.
o Negotiation of an RTR message.
Kanevsky, et al. Expires August 24, 2010 [Page 9]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
The RDMA connection responder processes the MPA Request and generates
an MPA Reply, as specified in [RFC5044]; the new format completes the
negotiation.
The local interface SHOULD require the ULP to explicitly configure on
a per-application or per-connection basis when an explicit RTR
message will be required. Piggy-backing the RTR on a Client's first
message is a valuable optimization for most connections.
The RDMA connection initiator MUST NOT inform its ULP that the RDMA
connection has been established until after any required RTR Message
has been transmitted.
6. Enhanced MPA Request/Reply Frames
Enhanced RDMA Connection Establishment uses an alternate format for
MPA Requests and Replies, as follows:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 | |
+ Key (16 bytes containing "MPA ID Req Frame") +
4 | (4D 50 41 20 49 44 20 52 65 71 20 46 72 61 6D 65) |
+ Or (16 bytes containing "MPA ID Rep Frame") +
8 | (4D 50 41 20 49 44 20 52 65 70 20 46 72 61 6D 65) |
+ +
12 | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
16 |M|C|R|S| Res | Rev | PD_Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ ~
~ Private Data ~
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
MPA Request/Reply Frame
Key: Unchanged from [RFC5044].
M: Unchanged from [RFC5044].
Kanevsky, et al. Expires August 24, 2010 [Page 10]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
C: Unchanged from [RFC5044].
R: Unchanged from [RFC5044].
S: One if the Private Data begins with the Enhanced RDMA Connection
Establishment Data. Zero otherwise.
Res: One bit smaller than in [RFC5044], otherwise unchanged.
Rev: This field contains the revision of MPA. To use any Enhanced
Connection Establishment feature this MUST be set to two, If no
Enhanced Connection Establishment features are desired it MAY be
set to one. A host accepting MPA connections SHOULD continue to
accept MPA Requests with version one even if it supports version
two.
PD Length: Unchanged from [RFC5044]. This is the total length of
the Private Data field, including the Enhanced RDMA Connection
Establishment Data if present.
Private Data: Unchanged from [RFC5044]. However, if the 'S' flag is
set, Private Data begins with Enhanced RDMA Connection
Establishment Data.
7. Enhanced SCTP Session Control Chunks
The type of the SCTP Session Control Chunk is defined by a Function
Code. [RFC5043] already defines codes for 'DDP Stream Session
Initiate' and 'DDP Stream Session Accept', which are equivalent to a
MPA Request Frame and an accepting MPA Reply Frame.
Enhanced RDMA Connection Establishment requires three additional
Function codes, as follows:
Enhanced DDP Stream Session Initiate: 0x05
Enhanced DDP Stream Session Accept: 0x06
Enhanced DDP Stream Session Reject: 0x07
The Enhanced Reject function code SHOULD be used to indicate a
configuration that would have been accepted.
It should be noted that [RFC5043] already supports either side
sending the first DDP Message. The Payload Protocol Identifier
(PPID) already distinguishes between Session Establishment and DDP
Segments.
Kanevsky, et al. Expires August 24, 2010 [Page 11]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
8. Enhanced RDMA Connection Establishment Data
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 |A|B| IRD |C|D| ORD |
4 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Enhanced RDMA Connection Establishment Data
IRD: In request: the Initiator requested responder IRD for the
connection. In reply: the depth the Responder will support. An
all ones value (0x3FFF) indicates that automatic negotiation of
the IRD is not desired, and that the ULP will be responsible for
doing this configuration.
ORD: In request: the Initiator initial ORD setting for the
connection. In reply: the depth the Responder will support. An
all ones value (0x3FFF) indicates that automatic negotiation of
the IRD is not desired, and that the ULP will be responsible for
doing this configuration.
A: Control Flag for using a zero length ULPDU as the RTR message.
B: Control Flag for using a zero length RDMA Write as the RTR
message.
C: Control Flag for using a zero length RDMA Read as the RTR message.
D: Reserved. Must be sent as zero and ignored when received.
In the MPA Request, the Control Flag MUST be set unless the Initiator
cannot support this option, or can only do so in collaboration with
the ULP.
In the MPA Reply, the Control Flag is set for the set of options that
the passive side will accept as an RTR message. This response MUST
include all options that the responder will support without requiring
a connection specific enabling action. For example, if the responder
will always unblock an MPA connection when it receives a zero length
MPA Write, it should so indicate without regard to what was in the
MPA Request. Options which require connection specific enabling
actions SHOULD NOT be set unless the corresponding flag was set in
the MPA Request. The respondent MAY choose to limit the number of
modes that it enables.
Kanevsky, et al. Expires August 24, 2010 [Page 12]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
If there is no Standard RDMAP Configuration Data in the MPA Reply
Frame, and the Enhanced Connection Establishment version number is
used, it is the equivalent of setting 'A', 'B' and 'C'.
Setting no Control Flags in the MPA Reply indicates that an RDMA Send
message will be required. As that this option will require the
initiator ULP to be involved it SHOULD NOT be used unless necessary.
9. Interoperability
An initiator SHOULD NOT use the Enhanced DDP Connection Establishment
formats or function codes when no enhanced functionality is desired.
A responder SHOULD continue to accept the unenhanced connection
requests.
10. IANA Considerations
This document has no IANA considerations.
11. Security Considerations
The security considerations from RFC 5044 apply and the changes in
this document do not introduce new security considerations.
12. Acknowledgements
The authors wish to thank Sean Hefty, Tom Talpey and David Black for
their valuable contributions and review of this document.
13. References
13.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC5043] Bestler, C. and R. Stewart, "Stream Control Transmission
Protocol (SCTP) Direct Data Placement (DDP) Adaptation",
RFC 5043, October 2007.
[RFC5044] Culley, P., Elzur, U., Recio, R., Bailey, S., and J.
Carrier, "Marker PDU Aligned Framing for TCP
Kanevsky, et al. Expires August 24, 2010 [Page 13]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
Specification", RFC 5044, October 2007.
13.2. Informative References
[DAPL] DAT Collaborative, "Direct Access Programming Library",
<http://www.datcollaborative.org/>.
[OFA] Open Fabrics Alliance, "Wiki",
<https://wiki.openfabrics.org/tiki-index.php>.
[OpenMP] Quinn, M., "Parallel Programming in C with MPI and
OpenMP", 2003.
[PPMPI] Pacheco, P., "Parallel Programming with MPI", 2008.
[RDMAC] Hilland, J., Culley, P., Pinkerton, J., and R. Recio,
"RDMA Protocol Verbs Specification (Version 1.0)", <http:/
/www.rdmaconsortium.org/home/
draft-hilland-iwarp-verbs-v1.0-RDMA.pdf>.
[RDS] Open Fabrics Association, "Reliable Datagram Socket",
2008, <http://www.openfabrics.org/archives/
spring2008sonoma/Tuesday/sonoma_2008_0408%20Oracle.ppt>.
[RFC5040] Recio, R., Metzler, B., Culley, P., Hilland, J., and D.
Garcia, "A Remote Direct Memory Access Protocol
Specification", RFC 5040, October 2007.
[UsingMPI]
Gropp, W., Lusk, E., and R. Thakur, "Using MPI-2: Advanced
Features of the Message Passing Interface", 1999.
[VIA] Compaq, Intel, Microsoft, "Virtual Interface Architecture
Specification", 1997, <http://pllab.cs.nthu.edu.tw/cs5403/
Readings/EJB/san_10.pdf>.
Authors' Addresses
Arkady Kanevsky (editor)
EMC
11 Cambridge Technology Center
Cambridge, MA 02142
USA
Phone: +1-617-300-7131
Email: arkady.kanevsky@gmail.com
Kanevsky, et al. Expires August 24, 2010 [Page 14]
Internet-Draft Enhanced RDMA Connection Establishment February 2010
Caitlin Bestler (editor)
Consultant
555 E El Camino Real #104
Sunnyvale, CA 94087
USA
Phone: +1-949-528-3085
Email: cait@asomi.com
Dave Minturn
Intel
5200 NE Elam Young Pkwy, JF3-410
Hillsboro, OR 97123
USA
Phone: +1-503-712-4106
Email: dave.b.minturn@intel.com
Steve Wise
Open Grid Computing
4030 Braker Lane STE 130
Austin, TX 78759
USA
Phone: +1-512-343-9196 x101
Email: swise@opengridcomputing.com
Kanevsky, et al. Expires August 24, 2010 [Page 15]
