Remote Direct Memory Access (RDMA) over IP Problem Statement
RFC 4297
Network Working Group A. Romanow
Request for Comments: 4297 Cisco
Category: Informational J. Mogul
HP
T. Talpey
NetApp
S. Bailey
Sandburst
December 2005
Remote Direct Memory Access (RDMA) over IP Problem Statement
Status of This Memo
This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Copyright Notice
Copyright (C) The Internet Society (2005).
Abstract
Overhead due to the movement of user data in the end-system network
I/O processing path at high speeds is significant, and has limited
the use of Internet protocols in interconnection networks, and the
Internet itself -- especially where high bandwidth, low latency,
and/or low overhead are required by the hosted application.
This document examines this overhead, and addresses an architectural,
IP-based "copy avoidance" solution for its elimination, by enabling
Remote Direct Memory Access (RDMA).
Romanow, et al. Informational [Page 1]
RFC 4297 RDMA over IP Problem Statement December 2005
Table of Contents
1. Introduction ....................................................2
2. The High Cost of Data Movement Operations in Network I/O ........4
2.1. Copy avoidance improves processing overhead. ...............5
3. Memory bandwidth is the root cause of the problem. ..............6
4. High copy overhead is problematic for many key Internet
applications. ...................................................8
5. Copy Avoidance Techniques ......................................10
5.1. A Conceptual Framework: DDP and RDMA ......................11
6. Conclusions ....................................................12
7. Security Considerations ........................................12
8. Terminology ....................................................14
9. Acknowledgements ...............................................14
10. Informative References ........................................15
1. Introduction
This document considers the problem of high host processing overhead
associated with the movement of user data to and from the network
interface under high speed conditions. This problem is often
referred to as the "I/O bottleneck" [CT90]. More specifically, the
source of high overhead that is of interest here is data movement
operations, i.e., copying. The throughput of a system may therefore
be limited by the overhead of this copying. This issue is not to be
confused with TCP offload, which is not addressed here. High speed
refers to conditions where the network link speed is high, relative
to the bandwidths of the host CPU and memory. With today's computer
systems, one Gigabit per second (Gbits/s) and over is considered high
speed.
High costs associated with copying are an issue primarily for large
scale systems. Although smaller systems such as rack-mounted PCs and
small workstations would benefit from a reduction in copying
overhead, the benefit to smaller machines will be primarily in the
next few years as they scale the amount of bandwidth they handle.
Today, it is large system machines with high bandwidth feeds, usually
multiprocessors and clusters, that are adversely affected by copying
overhead. Examples of such machines include all varieties of
servers: database servers, storage servers, application servers for
transaction processing, for e-commerce, and web serving, content
distribution, video distribution, backups, data mining and decision
support, and scientific computing.
Note that such servers almost exclusively service many concurrent
sessions (transport connections), which, in aggregate, are
responsible for > 1 Gbits/s of communication. Nonetheless, the cost
Romanow, et al. Informational [Page 2]
RFC 4297 RDMA over IP Problem Statement December 2005
of copying overhead for a particular load is the same whether from
few or many sessions.
The I/O bottleneck, and the role of data movement operations, have
been widely studied in research and industry over the last
approximately 14 years, and we draw freely on these results.
Historically, the I/O bottleneck has received attention whenever new
networking technology has substantially increased line rates: 100
Show full document text