Internet Engineering Task Force Greg Minshall INTERNET-DRAFT Siara Systems draft-minshall-nagle-01.txt June 17, 1999 A Proposed Modification to Nagle's Algorithm Status of This Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This draft proposes a modification to Nagle's algorithm (as specified in RFC896) to allow TCP, under certain conditions, to send a small sized packet immediately after one or more maximum segment sized packet. Abstract The Nagle algorithm is one of the primary mechanisms which protects the internet from poorly designed and/or poorly implemented applications. However, for a certain class of applications (notably, request-response protocols) the Nagle algorithm interacts poorly with delayed acknowledgements to give these applications poorer performance. This draft is NOT suggesting that these applications should disable the Nagle algorithm. This draft suggests a fairly small and simple modification to the Nagle algorithm which preserves the Nagle algorithm as a means of protecting the internet while at the same time giving better performance to a wider class of applications. Introduction to the Nagle algorithm The Nagle algorithm [RFC896] protects the internet from applications (most notably Telnet [RFC854], at the time the algorithm was developed) which tend to dribble small amounts of data to TCP. Without the Nagle algorithm, TCP would transmit a packet, with a small amount of data, in response to each of the application's writes to TCP. With the Nagle algorithm, a first small packet will be transmitted, then subsequent writes from the application will be buffered at the sending TCP until either i) enough application data has accumulated to enable TCP to transmit a maximum sized packet, or ii) the initial small packet is acknowledged by the receiving TCP. This limits the number of small packets to one per round trip time. While the current Nagle algorithm does a very good job of protecting the internet from such applications, there are other applications, such as request-response protocols (with HTTP [RFC2068] being a topical example) in which the current Nagle algorithm interacts with TCP's ``delayed ACK'' policy [RFC1122] to produce non-optimal results. Delayed ACKs A receiving TCP tries to avoid acknowledging every received data packet in the hope of ``piggy-backing'' the acknowledgement on a data packet flowing in the reverse direction or combining the acknowledgement with a window update flowing in the reverse direction. This process, known as ``delayed ACKing'' [RFC1122], typically causes an ACK to be generated for every other received (full-sized) data packet. In the case of an ``isolated'' TCP packet (i.e., where a second TCP packet is not going to arrive anytime soon), the delayed ACK policy causes an acknowledgement for the data in the isolated packet to be delayed up to 200 milliseconds of the receipt of the isolated packet (the actual maximum time the acknowledgement can be delayed is 500ms [RFC1122], but most systems implement a maximum of 200ms, and we shall assume that number in this document). The way delayed ACKs are implemented in some systems causes the delayed ACK to be generated anytime between 0ms and 200ms; in this case, the average amount of time before the delayed ACK is generated is 100ms. The interaction of delayed ACKs and Nagle If a TCP has more application data to transmit than will fit in one packet, but less than two full-sized packets' worth of data, it will transmit the first packet. As a result of Nagle, it will not transmit the second packet until the first packet has been acknowledged. On the other hand, the receiving TCP will delay acknowledging the first packet until either i) a second packet arrives (which, in this case, won't arrive), or ii) approximately 100ms (and a maximum of 200ms) has elapsed. When the sending TCP receives the delayed ACK, it can then transmit its second packet. In a request-response protocol, this second packet will complete either a request or a response, which then enables a succeeding response or request. Note two (related) bad results of the interaction of delayed ACKs and the Nagle algorithm in this case: the request-response time may be increased by up to 400ms (if both the request and the response are delayed); and, consequently, the number of transactions per second is substantially reduced. A proposed modification to the Nagle algorithm In the following discussion we make use of the following variables defined in the TCP RFC [RFC793] and in the host requirements RFC [RFC1122]: ``snd.nxt'' is a TCP variable which names the next byte of data to be transmitted; ``snd.una'' is a TCP variable which names the next byte of data to be acknowledged (if snd.nxt equals snd.una, then all previous packets have been acknowledged); Eff.snd.MSS is the largest TCP payload (user data) that can be transmitted in one packet. The current Nagle algorithm does not require any other state to be kept by TCP on a system. The proposed modification to the Nagle algorithm does, unfortunately, require one new state variable to be kept by TCP: ``snd.sml'' is a TCP variable which names the last byte of data in the most recently transmitted small packet. The current Nagle algorithm can be described as follows: "If a TCP has less than a full-sized packet to transmit, and if any previous packet has not yet been acknowledged, do not transmit a packet." and in pseudo-code: if ((packet.size < Eff.snd.MSS) && (snd.nxt > snd.una)) { do not send the packet; } The proposed Nagle algorithm modifies this as follows: "If a TCP has less than a full-sized packet to transmit, and if any previously transmitted less than full-sized packet has not yet been acknowledged, do not transmit a packet." and in pseudo-code: if (packet.size < Eff.snd.MSS) { if (snd.sml > snd.una)) { do not send the packet; } else { snd.sml = snd.nxt+packet.size; send the packet; } } In other words, when running Nagle, only look at the recent transmission (and acknowledgement) of small packets (rather than all packets, as in the current Nagle). (In writing the above, I am aware that TCP acknowledges bytes, not packets. However, expressing the algorithm in terms of packets seems to make the explanation a bit clearer.) Implementing Nagle at Send The above description of the current Nagle algorithm and of the proposed modification assumes that the Nagle algorithm is being implemented just as TCP is about to hand a packet to IP to be transmitted, i.e., the algorithm is looking at the sizes of the packets it transmits. In reality, many TCPs essentially implement Nagle at the interface where applications present data to TCP to be transmitted (i.e., in the call to ``SEND'', as defined in section 3.8 of the TCP specification [RFC793]). The motivation for this is to not penalize applications that provide data to TCP in large chunks (ideally a multiple of Eff.snd.MSS). This allows a single application send to be broken into zero or more full-sized packets, possibly followed by one small packet, without forcing any delay on the trailing small packet. For example, one implementation with which the author is familiar first captures the boolean ``snd.nxt > snd.una'' in a temporary variable (``busy''): busy = (snd.nxt > snd.una); then goes into a loop transmitting packets out of the data which has been presented to TCP by the application; the loop contains the following code to implement the current Nagle algorithm: if ((packet.size < Eff.snd.MSS) && busy) { do not send the packet; } Since ``busy'' is a constant in the loop transmitting packets, a trailing small packet will be transmitted (after zero or more large packets transmitted by the same call to send) if the connection had no outstanding data at the time the application presented data to TCP for transmission (assuming the TCP window allows this). To implement the modified Nagle algorithm in such a system, we replace snd.sml with two variables: ``snd.sml.add'' is a TCP variable which names the last byte presented to TCP by the application with a ``small'' send (i.e., the application called SEND with fewer than Eff.snd.MSS bytes of data); and ``snd.sml.snt'' is a TCP variable which names the highest value of snd.sml.add which has, in fact, been transmitted. The send routine contains the following code: if (byte.count < Eff.snd.MSS) { snd.sml.add = snd.una + snd.bytes.queued; } (where ``snd.bytes.queued'' is the number of bytes queued for transmission, and has already been updated with ``byte.count'', the number of bytes being presented to TCP in this call to SEND). The loop that transmits packets contains the following code: if (packet.size < Eff.snd.MSS) { if (snd.sm.snt > snd.una) { do not send the packet; } else { if ((snd.nxt + packet.size) <= snd.sm.add) { snd.sm.snt = snd.sm.add; } send the packet; } } (In most implementations, the most deeply nested ``if'' statement above is unnecessary, as a small-sized packet will contain all the data available to be transmitted, and so will include, or be beyond, snd.sm.add. In this case, the modified Nagle algorithm adds one test, one addition, and one assignment in the send routine, and one assignment in the output routine.) A Failure Mode If an application sends a large amount of data, followed by a small amount of data, followed by a large amount of data, the current Nagle algorithm would perform better than the proposed modification. The current Nagle algorithm would send at most one small packet (possibly the last packet), delaying the middle (small) amount of data which would allow the application to send the following large amount of data; the modified Nagle algorithm would send as many as two small packets (the middle packet, plus possibly a last packet). A separate, but desirable, system facility In addition to the Nagle algorithm (or the modification proposed by this draft), it would be desirable for a system providing TCP service to applications to allow the application to set TCP into a mode in which the TCP would only transmit small packets at the explicit direction of the application. For example, a system based on BSD might implement a socket option (using setsockopt(2)) SO_EXPLICITPUSH, as well as a flag to sendto(2) (possibly overloading the semantics of an existing flag, such as MSG_EOF). In this scenario, an application would set a socket into SO_EXPLICITPUSH mode, then enter a mode of writing data to the socket and, at the last write, using send(2) with the MSG_EOF flag. The underlying TCP would recognize the MSG_EOF flag as an indicator to transmit the (possibly) small packet. Like the proposed modification to the Nagle algorithm, this is fairly simple to implement. If a system were to implement this interface, it would be important to NOT disable Nagle when using this interface. In other words, when using this interface, the default mode for TCP would be to NOT transmit a small packet (even in the presence of MSG_EOF) if a previously transmitted small packet was as yet unacknowledged. Note, also, that implementing this interface does not eliminate the desirability of using the modification of the Nagle as the default for applications. More sophisticated networking applications might well use the new interface, but naive applications will often be adequately served by the modified Nagle algorithm. Application scenarios that will not be helped by this modification The proposed modification helps applications which do not need to transmit more than one small packet in a single round-trip time. This characterizes one way file transfer applications (such as FTP [RFC959]) and request/response protocols (such as NNTP [RFC977] and HTTP [RFC2068] without pipelining). However, applications that need to transmit more than one small packet in a single round-trip time are not served by this modification. An example of such an application is HTTP [RFC2068] using ``pipelining'', in which multiple requests (responses) are transmitted asynchronously. Applications needing to transmit more than one small packet in a single round-trip time will need other mechanisms to satisfy their requirements. (One possible such mechanism would be to use more than one TCP connection.) If an application developer is considering disabling the Nagle algorithm, they should be very careful to ensure that their application will generally provide data to TCP in chunks larger than two full-sized segments (> 2*Eff.snd.MSS), and they should verify after their development that this is, in fact, true. With Nagle disabled, many writes of small blocks of data can add significant load to the network, reducing the network's performance. Acknowledgements Jim Gettys, Henrik Frystyk Nielsen, Jeff Mogul, and Yasushi Saito, as well as a message forwarded to the end2end-interest list by Sean Doran, have motivated my current interest in the Nagle algorithm. John Heidemann's work related to the Nagle algorithm has informed some of the thinking in this draft; discussions with John have also been helpful. Members of the End-to-End Research Group (under the direction of Bob Braden) patiently listened to my discussion of the current state of the Nagle algorithm and to the modifications proposed in this document. Members of the TCP implementors mailing list <tcp-impl@lerc.nasa.gov> have been very helpful in refining this proposal. In particular, Rick Jones, Neal Cardwell, Vernon Schryver, Bernie Volz, Sam Manthorpe, Art Shelest, David Borman, Kacheong Poon, Jon Snader, Eric Hall, Joe Touch, and Alan Cox. Security Considerations The Nagle algorithm does not have major security consequences. Implementation of this algorithm should not negatively impact the performance of the internet. The negative impact of implementation of this algorithm should be significantly less than disabling the Nagle algorithm. Appendix -- Sample application code The following code is provided to give application developers a model for buffering. We assume a BSD-style sockets API. #include <stdio.h> #include <stdlib.h> #include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <netinet/tcp.h> #define SNDBUF_MULT 3 /* * 2 * TCP_MAXSEG -> SO_SNDBUF */ /* * Given a connected socket (s), configure the socket * with good buffer size defaults, and return the * the size the application should use for issuing * writes to the socket. * * Returns size to use for application buffering, or * zero (0) on error. */ int getbufsize(int s) { unsigned long bufsize, parm; int buflen; buflen = sizeof bufsize; if (getsockopt(s, IPPROTO_TCP, TCP_MAXSEG, &bufsize, &buflen) == -1) { perror("getsockopt(...TCP_MAXSEG...)"); return 0; } /* Set socket transmit buffer */ parm = 2*SNDBUF_MULT*bufsize; if (setsockopt(s, SOL_SOCKET, SO_SNDBUF, &parm, sizeof parm) == -1) { perror("setsockopt(SO_SNDBUF)"); return 0; } /* Now, set socket low water threshhold */ parm = 2*bufsize; if (setsockopt(s, SOL_SOCKET, SO_SNDLOWAT, &parm, sizeof parm) == -1) { perror("setsockopt(...SO_SNDLOWAT...)"); return 0; } return 2*bufsize; } int main(int argc, char *argv[]) { char *buffer = 0; int buflen; int sock; /* * ... allocate a socket (sock) and get it connected * via either connect(2) or listen(2)/accept(2). */ buflen = getbufsize(sock); if (buflen == 0) { fprintf(stderr, "aborting\n"); exit(1); } buffer = malloc(buflen); if (buffer == 0) { fprintf(stderr, "no room for buffer of size %d\n", buflen); exit(1); } /* * ... loop generating ``buflen'' data in buffer * and using send(2) to hand it to TCP. * When there is no more data to send, call * send(2) one last time with <= ``buflen'' * bytes. */ return 0; } References [RFC793] Postel, J. (ed), "Transmission Control Protocol", Sep-1981. [RFC854] Postel, J., J. Reynolds, "Telnet Protocol Specification", May-1983. [RFC959] Postel, J., J. Reynolds, "File Transfer Protocol (FTP)", Oct-1985. [RFC977] Kantor, B., P. Lapsley, "Network News Transfer Protocol", Feb-1986. [RFC896] Nagle, J., "Congestion control in IP/TCP internetworks", Jan-06-1984. [RFC1122] Braden, R. T., "Requirements for Internet hosts - communication layers", Oct-01-1989. [RFC2068] Fielding, R., J. Gettys, J. Mogul, H. Frystyk, T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1". Author's Address Greg Minshall Siara Systems 300 Ferguson Drive, 2nd floor Mountain View, CA 94043 USA <minshall@siara.com>