Internet-Draft Stephen Bailey (Sandburst)
Expires: March 2004 Tom Talpey (NetApp)
The Architecture of Direct Data Placement (DDP)
and Remote Direct Memory Access (RDMA)
on Internet Protocols
draft-ietf-rddp-arch-03
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as "work in
progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Copyright Notice
Copyright (C) The Internet Society (2003). All Rights Reserved.
Abstract
This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
run on Internet Protocol-suite transports. This architecture does
not necessarily reflect the proper way to implement such protocols,
but is, rather, a descriptive tool for defining and understanding
the protocols. DDP allows the efficient placement of data into
buffers designated by Upper Layer Protocols (e.g. RDMA). RDMA
provides the semantics to enable Remote Direct Memory Access
between peers in a way consistent with application requirements.
Bailey & Talpey Expires March 2004 [Page 1]
Internet-Draft DDP & RDMA Architecture September 2003
Table Of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . 2
2. Architecture . . . . . . . . . . . . . . . . . . . . . . 3
2.1. Direct Data Placement (DDP) Protocol Architecture . . . 3
2.1.1. Transport Operations . . . . . . . . . . . . . . . . . . 5
2.1.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . 6
2.1.3. Transport Characteristics in DDP . . . . . . . . . . . . 10
2.2. Remote Direct Memory Access Protocol Architecture . . . 11
2.2.1. RDMA Operations . . . . . . . . . . . . . . . . . . . . 12
2.2.2. Transport Characteristics in RDMA . . . . . . . . . . . 15
3. Security Considerations . . . . . . . . . . . . . . . . 16
4. IANA Considerations . . . . . . . . . . . . . . . . . . 16
5. Acknowledgements . . . . . . . . . . . . . . . . . . . . 16
Informative References . . . . . . . . . . . . . . . . . 16
Authors' Addresses . . . . . . . . . . . . . . . . . . . 17
Full Copyright Statement . . . . . . . . . . . . . . . . 18
1. Introduction
This document defines an abstract architecture for Direct Data
Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
run on Internet Protocol-suite transports. This architecture does
not necessarily reflect the proper way to implement such protocols,
but is, rather, a descriptive tool for defining and understanding
the protocols.
The first part of the document describes the architecture of DDP
protocols, including what assumptions are made about the transports
on which DDP is built. The second part describes the architecture
of RDMA protocols layered on top of DDP.
Before introducing the protocols, three definitions will be useful
to guide discussion:
o Placement - writing to a data buffer.
o Delivery - informing the Upper Layer Protocol (ULP) (e.g.
RDMA) that a particular message is available for use.
Delivery therefore may be viewed as the "control" signal
associated with a unit of data. Note that the order of
delivery is defined more strictly than it is for placement.
o Completion - informing the ULP or application that a
particular RDMA operation has finished. A completion, for
instance, may require the delivery of several messages, or it
may also reflect that some local processing has finished.
Bailey & Talpey Expires March 2004 [Page 2]
Internet-Draft DDP & RDMA Architecture September 2003
The goal of the DDP protocol is to allow the efficient placement of
data into buffers designated by Upper Layer Protocols (e.g. RDMA).
This is described in detail in [ROM]. Efficiency may be
characterized by the minimization of the number of transfers of the
data over the receiver's system buses.
The goal of the RDMA protocol is to provide the semantics to enable
Remote Direct Memory Access between peers in a way consistent with
application requirements. The RDMA protocol provides facilities
immediately useful to existing and future networking, storage, and
other application protocols. [DAFS, FCVI, IB, MYR, SDP, SRVNET,
VI]
The DDP and RDMA protocols work together to achieve their
respective goals. DDP provides facilities to safely steer payloads
to specific buffers at the Data Sink. RDMA provides facilities to
a ULP for identifying these buffers, controlling the transfer of
data between ULP peers, and signalling completion to the ULP. ULPs
that do not require the features of RDMA may be layered directly on
top of DDP.
The DDP and RDMA protocols are transport independent. The
following figure shows the relationship between RDMA, DDP, Upper
Layer Protocols and Transport.
+---------------------------------------------------+
| ULP |
+---------+------------+----------------------------+
| | | RDMA |
| | +----------------------------+
| | DDP |
| +-----------------------------------------+
| Transport |
+---------------------------------------------------+
2. Architecture
The Architecture section is presented in two parts: Direct Data
Placement Protocol architecture and Remote Direct Memory Access
Protocol architecture.
2.1. Direct Data Placement (DDP) Protocol Architecture
The central idea of general-purpose DDP is that a data sender will
supplement the data it sends with placement information that allows
the receiver's network interface to place the data directly at its
final destination without any copying. DDP can be used to steer
received data to its final destination, without requiring layer-
Bailey & Talpey Expires March 2004 [Page 3]
Internet-Draft DDP & RDMA Architecture September 2003
specific behavior for each different layer. Data sent with such
DDP information is said to be `tagged'.
The central component of the DDP architecture is the `buffer',
which is an object with beginning and ending addresses, and a
method (set()) to set the value of an octet at an address. In many
cases, a buffer corresponds directly to a portion of host user
memory. However, DDP does not depend on this---a buffer could be a
disk file, or anything else that can be viewed as an addressable
collection of octets. Abstractly, a buffer provides the interface:
typedef struct {
const address_t start;
const address_t end;
void set(address_t a, data_t v);
} ddp_buffer_t;
address_t
a reference to local memory
data_t
an octet data value.
The protocol layering and in-line data flow of DDP is:
Client Protocol
(e.g. ULP or RDMA)
| ^
untagged messages | | untagged message delivery
tagged messages | | tagged message delivery
v |
DDP+---> data placement
^
| transport messages
v
Transport
(e.g. SCTP, DCP, framed TCP)
^
| IP datagrams
v
. . .
In addition to in-line data flow, the client protocol registers
buffers with DDP, and DDP performs buffer update (set()) operations
Bailey & Talpey Expires March 2004 [Page 4]
Internet-Draft DDP & RDMA Architecture September 2003
as a result of receiving tagged messages.
DDP messages may be split into multiple, smaller DDP messages, each
in a separate transport message. However, if the transport is
unreliable or unordered, messages split across transport messages
may or may not provide useful behavior, in the same way as
splitting arbitrary upper layer messages across unreliable or
unordered transport messages may or may not provide useful
behavior. In other words, the same considerations apply to
building client protocols on different types of transports with or
without the use of DDP.
A DDP message split across transport messages looks like:
DDP message: Transport messages:
stag=s, offset=o, message 1:
notify=y, id=i |type=ddp |
message= |stag=s |
|aabbccddee|-------. |offset=o |
~ ... ~----. \ |notify=n |
|vvwwxxyyzz|-. \ \ |id=? |
| \ `--->|aabbccddee|
| \ ~ ... ~
| +----->|iijjkkllmm|
| |
+ | message 2:
\ | |type=ddp |
\ | |stag=s |
\ + |offset=o+n|
\ \ |notify=y |
\ \ |id=i |
\ `-->|nnooppqqrr|
\ ~ ... ~
`---->|vvwwxxyyzz|
Although this picture suggests that DDP information is carried in-
line with the message payload, components of the DDP information
may also be in transport-specific fields, or derived from
transport-specific control information if the transport permits.
2.1.1. Transport Operations
For the purposes of this architecture, the transport provides:
Bailey & Talpey Expires March 2004 [Page 5]
Internet-Draft DDP & RDMA Architecture September 2003
void xpt_send(socket_t s, message_t m);
message_t xpt_recv(socket_t s);
msize_t xpt_max_msize(socket_t s);
socket_t
a transport address, including IP addresses, ports and other
transport-specific identifiers.
message_t
a string of octets.
msize_t (scalar)
a message size.
xpt_send(socket_t s, message_t m)
send a transport message.
xpt_recv(socket_t s)
receive a transport message.
xpt_max_msize(socket_t s)
get the current maximum transport message size. Corresponds,
roughly, to the current path Maximum Transfer Unit (PMTU),
adjusted by underlying protocol overheads.
Real implementations of xpt_send() and xpt_recv() typically return
error indications, but that is not relevant to this architecture.
2.1.2. DDP Operations
The DDP layer provides:
void ddp_send(socket_t s, message_t m);
void ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d,
ddp_notify_t n);
void ddp_post_recv(socket_t s, bdesc_t b);
ddp_ind_t ddp_recv(socket_t s);
bdesc_t ddp_register(socket_t s, ddp_buffer_t b);
void ddp_deregister(bhand_t bh);
msizes_t ddp_max_msizes(socket_t s);
Bailey & Talpey Expires March 2004 [Page 6]
Internet-Draft DDP & RDMA Architecture September 2003
ddp_addr_t
the buffer address portion of a tagged message:
typedef struct {
stag_t stag;
address_t offset;
} ddp_addr_t;
stag_t (scalar)
a Steering Tag. A stag_t identifies the destination buffer
for tagged messages. stag_ts are generated when the buffer is
registered, communicated to the sender by some client protocol
convention and inserted in DDP messages. stag_t values in
this DDP architecture are assumed to be completely opaque to
the client protocol, and implementation-dependent. However,
particular implementations, such as DDP on a multicast
transport (see below), may provide the buffer holder some
control in selecting stag_ts.
ddp_notify_t
the notification portion of a DDP message, used to signal that
the message represents the final fragment of a multi-segmented
DDP message:
typedef struct {
boolean_t notify;
ddp_msg_id_t i;
} ddp_notify_t;
ddp_msg_id_t (scalar)
a DDP message identifier. msg_id_ts are chosen by the DDP
message receiver (buffer holder), communicated to the sender
by some client protocol convention and inserted in DDP
messages. Whether a message reception indication is requested
for a DDP message is a matter of client protocol convention.
Unlike stag_ts, the structure of msg_id_ts is opaque to DDP,
and therefore, completely in the hands of the client protocol.
bdesc_t
a description of a registered buffer:
Bailey & Talpey Expires March 2004 [Page 7]
Internet-Draft DDP & RDMA Architecture September 2003
typedef struct {
bhand_t bh;
ddp_addr_t a;
} bdesc_t;
`a.offset' is the starting offset of the registered buffer,
which may have no relationship to the `start' or `end'
addresses of that buffer. However, particular
implementations, such as DDP on a multicast transport (see
below), may allow some client protocol control over the
starting offset.
bhand_t
an opaque buffer handle used to deregister a buffer.
recv_message_t
a description of a completed untagged receive buffer:
typedef struct {
bdesc_t b;
length l;
} recv_message_t;
ddp_ind_t
an untagged message, a tagged message reception indication, or
a tagged message reception error:
typedef union {
recv_message_t m;
ddp_msg_id_t i;
ddp_err_t e;
} ddp_ind_t;
ddp_err_t
indicates an error while receiving a tagged message, typically
`offset' out of bounds, or `stag' is not registered to the
socket.
msizes_t
Bailey & Talpey Expires March 2004 [Page 8]
Internet-Draft DDP & RDMA Architecture September 2003
The maximum untagged and tagged messages that fit in a single
transport message:
typedef struct {
msize_t max_untagged;
msize_t max_tagged;
} msizes_t;
ddp_send(socket_t s, message_t m)
send an untagged message.
ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n)
send a tagged message to remote buffer address d.
ddp_post_recv(socket_t s, bdesc_t b)
post a registered buffer to accept a single received untagged
message. Each buffer is returned to the caller in a
ddp_recv() untagged message reception indication, in the order
in which it was posted. The same buffer may be enabled on
multiple sockets, receipt of an untagged message into the
buffer from any of these sockets unposts the buffer from all
sockets.
ddp_recv(socket_t s)
get the next received untagged message, tagged message
reception indication, or tagged message error.
ddp_register(socket_t s, ddp_buffer_t b)
register a buffer for DDP on a socket. The same buffer may be
registered multiple times on the same or different sockets.
The same buffer registered on different sockets may result in
a common registration. Different buffers may also refer to
portions of the same underlying addressable object (buffer
aliasing).
ddp_deregister(bhand_t bh)
remove a registration from a buffer.
ddp_max_msizes(socket_t s)
Bailey & Talpey Expires March 2004 [Page 9]
Internet-Draft DDP & RDMA Architecture September 2003
get the current maximum untagged and tagged message sizes that
will fit in a single transport message.
2.1.3. Transport Characteristics In DDP
Certain characteristics of the transport on which DDP is mapped
determine the nature of the service provided to client protocols.
Specifically, transports are:
o reliable or unreliable,
o ordered or unordered,
o single source or multisource,
o single destination or multidestination (multicast or anycast).
Some transports support several combinations of these
characteristics. For example, SCTP [SCTP] is reliable, single
source, single destination (point-to-point) and supports both
ordered and unordered modes.
DDP messages carried by transport are framed for processing by the
receiver, and may be further protected for integrity or privacy in
accordance with the transport capabilities. DDP does not provide
such functions.
In general, transport characteristics equally affect transport and
DDP message delivery. However, there are several issues specific
to DDP messages.
A key component of DDP is how the following operations on the
receiving side are ordered among themselves, and how they relate to
corresponding operations on the sending side:
o set()s,
o untagged message reception indications, and
o tagged message reception indications.
These relationships depend upon the characteristics of the
underlying transport in a way which is defined by the DDP protocol.
For example, if the transport is unreliable and unordered, the DDP
protocol might specify that the client protocol is subject to the
consequences of transport messages being lost or duplicated, rather
than requiring different characteristics be presented to the client
protocol.
Bailey & Talpey Expires March 2004 [Page 10]
Internet-Draft DDP & RDMA Architecture September 2003
Multidestination data delivery is the other transport
characteristic which may require specific consideration in a DDP
protocol. As mentioned above, the basic DDP model assumes that
buffer address values returned by ddp_register() are opaque to the
client protocol, and can be implementation dependent. The most
natural way to map DDP to a multidestination transport is to
require all receivers produce the same buffer address when
registering a multidestination destination buffer. Restriction of
the DDP model to accommodate multiple destinations involves
engineering tradeoffs comparable to those of providing non-DDP
multidestination transport capability.
The same buffer may be enabled by ddp_post_recv() on multiple
sockets. In this case the ddp_recv() untagged message reception
indication may be provided on a different socket from that on which
the buffer was posted. Such indications are not ordered among
multiple DDP sockets.
When multiple sockets reference an untagged message reception
buffer, local interfaces are responsible for managing the
mechanisms of allocating posted buffers to received untagged
messages, the handling of received untagged messages when no buffer
is available, and of resource management among multiple sockets.
Where underprovisioning of buffers on multiple sockets is allowed,
mechanisms should be provided to manage buffer consumption on a
per-socket or group of related sockets basis.
2.2. Remote Direct Memory Access (RDMA) Protocol Architecture
Remote Direct Memory Access (RDMA) extends the capabilities of DDP
with the ability to read from buffers registered to a socket (RDMA
Read). This allows a client protocol to perform arbitrary,
bidirectional data movement without involving the remote client.
When RDMA is implemented in hardware, arbitrary data movement can
be performed without involving the remote host CPU at all.
In addition, RDMA protocols usually specify a transport-independent
untagged message service (Send) with characteristics which are both
very efficient to implement in hardware, and convenient for client
protocols.
The RDMA architecture is patterned after the traditional model for
device programming, where the client requests an operation using
Send-like actions (programmed I/O), the server performs the
necessary data transfers for the operation (DMA reads and writes),
and notifies the client of completion. The programmed I/O+DMA
model efficiently supports a high degree of concurrency and
flexibility for both the client and server, even when operations
Bailey & Talpey Expires March 2004 [Page 11]
Internet-Draft DDP & RDMA Architecture September 2003
have a wide range of intrinsic latencies.
RDMA is layered as a client protocol on top of DDP:
Client Protocol
| ^
Sends | | Send reception indications
RDMA Read Requests | | RDMA Read Completion indications
RDMA Writes | | RDMA Write Completion indications
v |
RDMA
| ^
untagged messages | | untagged message delivery
tagged messages | | tagged message delivery
v |
DDP+---> data placement
^
| transport messages
v
. . .
In addition to in-line data flow, read (get()) and update (set())
operations are performed on buffers registered with RDMA as a
result of RDMA Read Requests and RDMA Writes, respectively.
An RDMA `buffer' extends a DDP buffer with a get() operation that
retrieves the value of the octet at address `a':
typedef struct {
const address_t start;
const address_t end;
void set(address_t a, data_t v);
data_t get(address_t a);
} rdma_buffer_t;
2.2.1. RDMA Operations
The RDMA layer provides:
Bailey & Talpey Expires March 2004 [Page 12]
Internet-Draft DDP & RDMA Architecture September 2003
void rdma_send(socket_t s, message_t m);
void rdma_write(socket_t s, message_t m, ddp_addr_t d,
rdma_notify_t n);
void rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d);
void rdma_post_recv(socket_t s, bdesc_t b);
rdma_ind_t rdma_recv(socket_t s);
bdesc_t rdma_register(socket_t s, rdma_buffer_t b,
bmode_t mode);
void rdma_deregister(bhand_t bh);
msizes_t rdma_max_msizes(socket_t s);
Although, for clarity, these data transfer interfaces are
synchronous, rdma_read() and possibly rdma_send() (in the presence
of Send flow control), can require an arbitrary amount of time to
complete. To express the full concurrency and interleaving of RDMA
data transfer, these interfaces should also be reentrant. For
example, a client protocol may perform an rdma_send(), while an
rdma_read() operation is in progress.
rdma_notify_t
RDMA Write notification information, used to signal that the
message represents the final fragment of a multi-segmented
RDMA message:
typedef struct {
boolean_t notify;
rdma_write_id_t i;
} rdma_notify_t;
identical in function to ddp_notify_t, except that the type
rdma_write_id_t may not be equivalent to ddp_msg_id_t.
rdma_write_id_t (scalar)
an RDMA Write identifier.
rdma_ind_t
a Send message, or an RDMA error:
typedef union {
recv_message_t m;
rdma_err_t e;
} rdma_ind_t;
Bailey & Talpey Expires March 2004 [Page 13]
Internet-Draft DDP & RDMA Architecture September 2003
rdma_err_t
an RDMA protocol error indication. RDMA errors include buffer
addressing errors corresponding to ddp_err_ts, and buffer
protection violations (e.g. RDMA Writing a buffer only
registered for reading).
bmode_t
buffer registration mode (permissions). Any combination of
permitting RDMA Read (BMODE_READ) and RDMA Write (BMODE_WRITE)
operations.
rdma_send(socket_t s, message_t m)
send a message, delivering it to the next untagged RDMA buffer
at the remote peer.
rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n)
RDMA Write to remote buffer address d.
rdma_read(socket_t s, ddp_addr_t s, length l, ddp_addr_t d)
RDMA Read l octets from remote buffer address s to local
buffer address d.
rdma_post_recv(socket_t s, bdesc_t b)
post a registered buffer to accept a single Send message, to
be filled and returned in-order to a subsequent caller of
rdma_recv(). As with DDP, buffers may be enabled on multiple
sockets, in which case ordering guarantees are relaxed. Also
as with DDP, local interfaces must manage the mechanisms of
allocation and management of buffers posted to multiple
sockets.
rdma_recv(socket_t s);
get the next received Send message, RDMA Write completion
identifier, or RDMA error.
rdma_register(socket_t s, rdma_buffer_t b, bmode_t mode)
register a buffer for RDMA on a socket (for read access, write
access or both). As with DDP, the same buffer may be
registered multiple times on the same or different sockets,
and different buffers may refer to portions of the same
Bailey & Talpey Expires March 2004 [Page 14]
Internet-Draft DDP & RDMA Architecture September 2003
underlying addressable object.
rdma_deregister(bhand_t bh)
remove a registration from a buffer.
rdma_max_msizes(socket_t s)
get the current maximum Send (max_untagged) and RDMA Read or
Write (max_tagged) operations that will fit in a single
transport message. The values returned by rdma_max_msizes()
are closely related to the values returned by
ddp_max_msizes(), but may not be equal.
2.2.2. Transport Characteristics In RDMA
As with DDP, RDMA can be used on transports with a variety of
different characteristics that manifest themselves directly in the
service provided by RDMA.
Like DDP, an RDMA protocol must specify how:
o set()s,
o get()s,
o Send messages, and
o RDMA Read completions
are ordered among themselves and how they relate to corresponding
operations on the remote peer(s). These relationships are likely
to be a function of the underlying transport characteristics.
There are some additional characteristics of RDMA which may
translate poorly to unreliable or multipoint transports due to
attendant complexities in managing endpoint state:
o Send flow control
o RDMA Read
These difficulties can be overcome by placing restrictions on the
service provided by RDMA. However, many RDMA clients, especially
those that separate data transfer and application logic concerns,
are likely to depend upon capabilities only provided by RDMA on a
point-to-point, reliable transport.
Bailey & Talpey Expires March 2004 [Page 15]
Internet-Draft DDP & RDMA Architecture September 2003
3. Security Considerations
System integrity must be maintained in any RDMA solution.
Mechanisms must be specified to prevent RDMA or DDP operations from
impairing system integrity. For example, the threat caused by
potential buffer overflow needs full examination, and prevention
mechanisms must be spelled out.
Because a Steering Tag exports access to a memory region, one
critical aspect of security is the scope of this access. It must
be possible to individually control specific attributes of the
access provided by a Steering Tag, including remote read access,
remote write access, and others that might be identified. DDP and
RDMA specifications must provide both implementation requirements
relevant to this issue, and guidelines to assist implementors in
making the appropriate design decisions.
Resource issues leading to denial-of-service attacks, overwrites
and other concurrent operations, the ordering of completions as
required by the RDMA protocol, and the granularity of transfer are
all within the required scope of any security analysis of RDMA and
DDP.
4. IANA Considerations
IANA considerations are not addressed in by this document. Any
IANA considerations resulting from the use of DDP or RDMA must be
addressed in the relevant standards.
5. Acknowledgements
The authors wish to acknowledge the valuable contributions of
Caitlin Bestler, David Black, Jeff Mogul and Allyn Romanow.
6. Informative References
[DAFS]
DAFS Collaborative, "Direct Access File System Specification
v1.0", September 2001, available from
http://www.dafscollaborative.org
[FCVI]
ANSI Technical Committee T11, "Fibre Channel Standard Virtual
Interface Architecture Mapping", ANSI/NCITS 357-2001, March
2001, available from http://www.t11.org/t11/stat.nsf/fcproj
[IB] InfiniBand Trade Association, "InfiniBand Architecture
Specification Volumes 1 and 2", Release 1.1, November 2002,
Bailey & Talpey Expires March 2004 [Page 16]
Internet-Draft DDP & RDMA Architecture September 2003
available from http://www.infinibandta.org/specs
[MYR]
VMEbus International Trade Association, "Myrinet on VME
Protocol Specification", ANSI/VITA 26-1998, August 1998,
available from http://www.myri.com/open-specs
[ROM]
A. Romanow, J. Mogul, T. Talpey and S. Bailey, "RDMA over IP
Problem Statement", draft-ietf-rddp-problem-statement-02, Work
in Progress, June 2003
RFC Editor note: Replace problem statement draft-ietf- name, status and
date with appropriate reference when assigned.
[SCTP]
R. Stewart et al., "Stream Transmission Control Protocol", RFC
2960, Standards Track
[SDP]
InfiniBand Trade Association, "Sockets Direct Protocol v1.0",
Annex A of InfiniBand Architecture Specification Volume 1,
Release 1.1, November 2002, available from
http://www.infinibandta.org/specs
[SRVNET]
R. Horst, "TNet: A reliable system area network", IEEE Micro,
pp. 37-45, February 1995
[VI] Compaq Computer Corp., Intel Corporation and Microsoft
Corporation, "Virtual Interface Architecture Specification
Version 1.0", December 1997, available from
http://www.vidf.org/info/04standards.html
Authors' Addresses
Stephen Bailey
Sandburst Corporation
600 Federal Street
Andover, MA 01810 USA
USA
Phone: +1 978 689 1614
Email: steph@sandburst.com
Bailey & Talpey Expires March 2004 [Page 17]
Internet-Draft DDP & RDMA Architecture September 2003
Tom Talpey
Network Appliance
375 Totten Pond Road
Waltham, MA 02451 USA
Phone: +1 781 768 5329
Email: thomas.talpey@netapp.com
Full Copyright Statement
Copyright (C) The Internet Society (2003). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain
it or assist in its implementation may be prepared, copied,
published and distributed, in whole or in part, without restriction
of any kind, provided that the above copyright notice and this
paragraph are included on all such copies and derivative works.
However, this document itself may not be modified in any way, such
as by removing the copyright notice or references to the Internet
Society or other Internet organizations, except as needed for the
purpose of developing Internet standards in which case the
procedures for copyrights defined in the Internet Standards process
must be followed, or as required to translate it into languages
other than English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on
an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET
ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Bailey & Talpey Expires March 2004 [Page 18]
