Remote Direct Memory Access (RDMA) over IP Problem Statement
RFC 4297

Network Working Group                                         A. Romanow
Request for Comments: 4297                                         Cisco
Category: Informational                                         J. Mogul
                                                                      HP
                                                               T. Talpey
                                                                  NetApp
                                                               S. Bailey
                                                               Sandburst
                                                           December 2005

      Remote Direct Memory Access (RDMA) over IP Problem Statement

Status of This Memo

   This memo provides information for the Internet community.  It does
   not specify an Internet standard of any kind.  Distribution of this
   memo is unlimited.

Copyright Notice

   Copyright (C) The Internet Society (2005).

Abstract

   Overhead due to the movement of user data in the end-system network
   I/O processing path at high speeds is significant, and has limited
   the use of Internet protocols in interconnection networks, and the
   Internet itself -- especially where high bandwidth, low latency,
   and/or low overhead are required by the hosted application.

   This document examines this overhead, and addresses an architectural,
   IP-based "copy avoidance" solution for its elimination, by enabling
   Remote Direct Memory Access (RDMA).

Romanow, et al.              Informational                      [Page 1]
RFC 4297             RDMA over IP Problem Statement        December 2005

Table of Contents

   1. Introduction ....................................................2
   2. The High Cost of Data Movement Operations in Network I/O ........4
      2.1. Copy avoidance improves processing overhead. ...............5
   3. Memory bandwidth is the root cause of the problem. ..............6
   4. High copy overhead is problematic for many key Internet
      applications. ...................................................8
   5. Copy Avoidance Techniques ......................................10
      5.1. A Conceptual Framework: DDP and RDMA ......................11
   6. Conclusions ....................................................12
   7. Security Considerations ........................................12
   8. Terminology ....................................................14
   9. Acknowledgements ...............................................14
   10. Informative References ........................................15

1.  Introduction

   This document considers the problem of high host processing overhead
   associated with the movement of user data to and from the network
   interface under high speed conditions.  This problem is often
   referred to as the "I/O bottleneck" [CT90].  More specifically, the
   source of high overhead that is of interest here is data movement
   operations, i.e., copying.  The throughput of a system may therefore
   be limited by the overhead of this copying.  This issue is not to be
   confused with TCP offload, which is not addressed here.  High speed
   refers to conditions where the network link speed is high, relative
   to the bandwidths of the host CPU and memory.  With today's computer
   systems, one Gigabit per second (Gbits/s) and over is considered high
   speed.

   High costs associated with copying are an issue primarily for large
   scale systems.  Although smaller systems such as rack-mounted PCs and
   small workstations would benefit from a reduction in copying
   overhead, the benefit to smaller machines will be primarily in the
   next few years as they scale the amount of bandwidth they handle.
   Today, it is large system machines with high bandwidth feeds, usually
   multiprocessors and clusters, that are adversely affected by copying
   overhead.  Examples of such machines include all varieties of
   servers: database servers, storage servers, application servers for
   transaction processing, for e-commerce, and web serving, content
   distribution, video distribution, backups, data mining and decision
   support, and scientific computing.

   Note that such servers almost exclusively service many concurrent
   sessions (transport connections), which, in aggregate, are
   responsible for > 1 Gbits/s of communication.  Nonetheless, the cost

Romanow, et al.              Informational                      [Page 2]
RFC 4297             RDMA over IP Problem Statement        December 2005

   of copying overhead for a particular load is the same whether from
   few or many sessions.

   The I/O bottleneck, and the role of data movement operations, have