Internet Engineering Task Force                      K. Subramaniam, Ed.
Internet-Draft                                                  D. Loher
Intended status: Informational                                 Microsoft
Expires: April 30, 2015                                 October 27, 2014

                     Router Buffer Sizes In The WAN


   This draft identifies the set of data that needs to be collected, and
   analyzed to quantify router buffer sizes used in routers in the Wide
   Area Network (WAN).  The scope of this draft is limited to WAN links
   that have link latencies of 40 to 150 milliseconds.

   Reducing router buffer sizes has many advantages, the most important
   being cost.  However, there is not much data available today to
   effectively calculate this.  This draft details use cases for the
   study, and lists data that needs to be taken into consideration to be
   able to quantify the size of router buffers.  The details of the
   individual measurement metrics are beyond the scope of this document.
   Neither does the draft identify methods to gather the data.  What it
   identifies is a need to be able to collect, and report this empirical
   data in a readable fashion thus providing the ability to study and
   compare data in a more standardized method.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on April 30, 2015.

Subramaniam & Loher      Expires April 30, 2015                 [Page 1]

Internet-Draft       Router Buffer Sizes in the WAN         October 2014

Copyright Notice

   Copyright (c) 2014 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   ( in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

   This document may not be modified, and derivative works of it may not
   be created, except to format it for publication as an RFC or to
   translate it into languages other than English.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   4
   3.  Use Case  . . . . . . . . . . . . . . . . . . . . . . . . . .   4
     3.1.  Discards with small buffer sizes  . . . . . . . . . . . .   4
     3.2.  Discards with large buffer sizes  . . . . . . . . . . . .   4
   4.  List of required data for study of router buffer sizes  . . .   4
     4.1.  Number of concurrent flows, N . . . . . . . . . . . . . .   5
     4.2.  Length of a flow, L . . . . . . . . . . . . . . . . . . .   6
     4.3.  Packet Discards, D  . . . . . . . . . . . . . . . . . . .   6
     4.4.  Reason for Packet Discards, R . . . . . . . . . . . . . .   6
     4.5.  Resolution of time interval, T  . . . . . . . . . . . . .   6
     4.6.  5 Tuple Flow Identity, I  . . . . . . . . . . . . . . . .   7
   5.  Conclusion  . . . . . . . . . . . . . . . . . . . . . . . . .   7
   6.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   7
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   7
   8.  Security Considerations . . . . . . . . . . . . . . . . . . .   7
   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   7
     9.1.  Normative References  . . . . . . . . . . . . . . . . . .   8
     9.2.  Informative References  . . . . . . . . . . . . . . . . .   8
   Appendix A.  Additional Stuff . . . . . . . . . . . . . . . . . .   9
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   9

1.  Introduction

   "How much buffering do core links need?" is a question that has been
   under study for a while.  The question boils down to quantify buffer

Subramaniam & Loher      Expires April 30, 2015                 [Page 2]

Internet-Draft       Router Buffer Sizes in the WAN         October 2014

   sizes and yet achieve 100% utilization on links with maximum
   throughput at a feasible cost.

   Buffer design could substantially increase costs.  While over-
   buffering seems intuitive it can complicate the design of high speed
   routers, lead to higher power consumption, more board space, and
   lower density.  It can actually increase end-to-end delay in the
   presence of congestion.  This can make congestion more persistent.
   Additionally, there is always a tradeoff between buffer sizes and the
   capacity of a router.

   On the other hand, under-buffering while doing away from the above
   cons of over-buffering could lead us away from our primary goal of
   100 percent link utilization.  This could happen in a scenario using
   a simple Additive Increase Multiplicative Decrease (AIMD) for TCP
   flows when the sender has packets to send but the window size
   advertised is less and as a result the receiver consumes far less
   that it could.

   The rule of thumb for router buffers has been defined as [Villamizar]
   : B = 2RTT*C.  Where B, was the buffer size, RTT the Round Trip Time,
   and C the capacity of the bottleneck link.  [RFC3429]  also talks
   about the buffer size being at least one TCP window size.

   However later studies [Appenzeller], show that the rule of thumb
   works either for a single flow or a perfectly synchronized large
   number of flows.  Further they postulate that the buffer size is
   actually (2RTT * C)/sqrt(n), where n is the number of flows.  This
   indicates a significant reduction in the buffer chip promoting lower

   As seen, there have been proponents for large buffers and small.
   However, most of these studies are based on theoretical models and
   simulations.  Today, there is no model or protocol to mine big data
   from a providers network to be able to answer this question
   efficiently.  The nature of WAN traffic can be uncertain and varying.
   Furthermore the traffic could vastly vary between individual ISPs.
   This document implored the need for a model of mining empirical big
   data in a providers network to be able to build a network that drives
   down the $/GB and at the same time maximizing link utilization.

   This document outlines use cases for the study of router buffer sizes
   in the WAN and identifies the data that needs to be collected and
   analyzed.  It could be further extended to the edge and datacenters,
   but it is outside the scope of this draft.

Subramaniam & Loher      Expires April 30, 2015                 [Page 3]

Internet-Draft       Router Buffer Sizes in the WAN         October 2014

2.  Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   document are to be interpreted as described in RFC 2119 [RFC2119].

3.  Use Case

   From an operator's perspective it is imperative to monitor discards
   and link utilization over WAN links to be able to study the router
   buffer sizes.  But these alone will be unable to provide an operator
   with enough information as to why the discards happened.  The two use
   cases outlined here argue that more data needs to be collected,
   reported, and analyzed.

3.1.  Discards with small buffer sizes

   Trans-pacific and trans-atlantic links of latencies in the range of
   150 ms and 90 ms respectively, with low link utilization of 30
   percent, and small buffers have seen dropped packets.  The most
   intuitive method has been to increase the buffer sizes for these
   links on noticing packet discards.  While this might alleviate the
   issue temporarily, unless the right problem has been identified this
   could readily lead to buffer bloat which has many issues on its own.

3.2.  Discards with large buffer sizes

   Operators have also observed dropped packets on WAN links within
   North America with as large buffers as 125 MB per port with link
   utilizations of 60%. If this happens even if the router has not been
   specifically configured to drop certain type of packets, or there are
   no routing misconfigurations, then clearly the issue here is not the
   size of the router buffer.

4.  List of required data for study of router buffer sizes

   This section talks about the absolute minimum requirements of the
   type of data that needs to be collected to be able to effectively
   quantify router buffer size.

Subramaniam & Loher      Expires April 30, 2015                 [Page 4]

Internet-Draft       Router Buffer Sizes in the WAN         October 2014

   |   |           Data          |               Details               |
   | 1 |   Number of concurrent  |        For aggregate traffic        |
   |   |         flows, N        |                                     |
   | 2 |  Length of the flow, L  |  [Flow start time - flow end time]  |
   | 3 |    Packet Discards, P   |            Per Interface            |
   | 4 |    Reason for Packet    |   Buffer overflow, configuration,   |
   |   |       Discards, R       |                 etc.                |
   | 5 |    Resolution of Time   |  [Flow start time - flow end time]  |
   |   |       Interval, T       |                                     |
   | 6 |  5 tuple flow identity, |   Src IP, Dest IP, Src port, Dest   |
   |   |            I            |           Port, Protocol.           |

          Table 1: List of required data for Router Buffer Sizes

   A service provider needs to take into consideration several
   attributes to determine the right buffer size for its WAN routers.
   This section enlists the details as to why the five above have been
   identified as the minimum essential data needed to aid the study of
   router buffer sizes.

4.1.  Number of concurrent flows, N

   Studies [Feldmann]  and [Stevens] show that 95% of flows in the
   internet today are attributed to TCP [Postel] flows.  The nature of
   these flows can vary significantly not only with various time
   periods, but also between providers.  Flows that spend most of their
   time in slow-start require significantly less buffering than flows
   that live mostly in congestion avoidance.  Due to this it is
   important to identify the type of concurrent flows that can live on a
   WAN link.

   Short (non-persistent) flows are those that live for less than one
   RTT, and large (persistent) flows are those whose lifetime is larger
   that one RTT with congestion overhead.  Internet measurements [Avra]
   show that while a smaller number of large flows contribute to maximum
   packet transfer, short flows dominate most TCP sessions and large
   flows are known to have a larger effect on buffer sizes.  These
   combination flows could in turn have an effect on Round Trip Time
   (RTT), loss probability and flow lengths.  The ability to detect
   large flows is necessary because while the flows can be constant in
   steady state, the aggregate traffic can keep changing due to various
   arrival and departure rates.  There needs to be a way for the number
   of concurrent flows to be collected and analyzed with the granularity
   of the lifetime of short flows, as low as one millisecond.

Subramaniam & Loher      Expires April 30, 2015                 [Page 5]

Internet-Draft       Router Buffer Sizes in the WAN         October 2014

4.2.  Length of a flow, L

   Length of a flow can be defined as its duration: [flow stop time -
   flow start time], or the number of packets/bytes sent in this time
   duration.  Identifying the length of flow in a provider's network
   will give information of the mix of short and large flows that are
   present in the WAN.  This will lead to modeling implications in TCP
   flow control.

4.3.  Packet Discards, D

   Number of packet discards per interface is probably the most
   important metric.  Of this the number of outward (WAN) facing
   interface discards would be more intuitive to the study of buffer
   sizes.  Interface discards can be referred to in [RFC2893]

4.4.  Reason for Packet Discards, R

   There can be several reasons for packet discards especially when it
   is observed on less utilized links.  Some of them could be due to
   routing misconfigurations, or designed to drop certain packets due to
   configurations.  Clearly stating a reason as insufficient buffer will
   help narrow down the data required.  This is especially true in the
   case of smart buffer allocations when some ports run out of buffers
   but not others.  We could observe that a port has been allocated
   only, say, 30 percent of the available total buffer space but is
   experiencing the highest utilization and as a result of that is
   seeing packet drops pointing to the fact that dynamic buffers' smart
   allocations scheme is not adaptive and predictive to the nature of
   the WAN traffic.

4.5.  Resolution of time interval, T

   The time interval should be granular such that it captures not only
   the number of concurrent flows in steady state but also the aggregate
   traffic over the lifetime of a short flow.  It should also be able to
   correlate the discards per interface to the number of concurrent

   Today via IPFIX we can calculate the number of concurrent flows.  Via
   Sflow counters or flows, we can calculate the discards.  Using
   counters requires upto two times the granularity set for any changes
   to be visible due to Nyquist rate.  Reducing the counter export
   interval would increase the responsiveness, but at the cost of
   increased overhead and reduced scalability.  On the other hand,
   packet sampling automatically allocates monitoring resources to busy
   links, providing a highly scaleable way to quickly detect traffic

Subramaniam & Loher      Expires April 30, 2015                 [Page 6]

Internet-Draft       Router Buffer Sizes in the WAN         October 2014

   flows wherever they occur in the network.  Responsiveness is
   important for a more stable control.

4.6.  5 Tuple Flow Identity, I

   5 tuple flows have a source IP, destination IP, source port,
   destination port, and protocol to identify endpoints for
   unidirectional flows.  Having this functionality gives the network
   operator a way to identify the offending flows, legitimate elephant
   flows, and high priority flows which may happen at certain periods
   during the day.  Being able to separate traffic using the 5 tuple,
   further increases the strength of the sample set of empirical data
   available for the study of router buffer sizes.

5.  Conclusion

   We see that there are numerous issues at different layers that have
   an effect (directly or indirectly) on the sizing of router buffers.
   We also notice that there is no study that takes empirical data into
   consideration.  Ideally, what would be required is an all knowing
   oracle that sees the traffic flow on an end-to-end network across all
   layers.  Due to a lack of the resource, the first step to the study
   of router buffer sizes is to effectively mine the big data repository
   of a provider for the data identified in this draft.

6.  Acknowledgements

7.  IANA Considerations

   This memo includes no request to IANA.

   All drafts are required to have an IANA considerations section (see
   the update of RFC 2434 [I-D.narten-iana-considerations-rfc2434bis]
   for a guide).  If the draft does not require IANA to do anything, the
   section contains an explicit statement that this is the case (as
   above).  If there are no requirements for IANA, the section will be
   removed during conversion into an RFC by the RFC Editor.

8.  Security Considerations

   This document does not introduce new security issues.

9.  References

Subramaniam & Loher      Expires April 30, 2015                 [Page 7]

Internet-Draft       Router Buffer Sizes in the WAN         October 2014

9.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

9.2.  Informative References

              G. Appenzeller, I. Klesassy, and N. McKeown, "Some
              Internet Architectural Guidelines and Philosophy", 2004,
              <SIGCOMM '04 Proceedings of the 2004 conference on
              Applications, technologies, architectures, and protocols
              for computer communications>.

   [Avra]     Konstantin Avrachenkov, INRIA Sophia Antipolis,
              "Differentiation Between Short and Long TCP Flows:
              Predictability of the Response Time", 2004, <IEEE

              A. Feldmann, J. Rexford, and R. Caceres, "Efficient
              policies for carrying Web traffic over flow-switched
              networks", Dec. 1998, <IEEE/ACM Trans. Networking, vol. 6,
              pp. 673-685>.

              Narten, T. and H. Alvestrand, "Guidelines for Writing an
              IANA Considerations Section in RFCs", draft-narten-iana-
              considerations-rfc2434bis-09 (work in progress), March

   [Postel]   J. Postel, "Transmission Control Protocol", Sep. 1981,
              <RFC 793>.

   [RFC2893]  K. McCloghrie, F. Kastenholz, "The Interfaces Group MIB",
              Jun. 2000, <RFC 2893>.

   [RFC3429]  R. Bush and D. Meyer, "Some Internet Architectural
              Guidelines and Philosophy", Dec. 2002, <RFC 3429>.

   [Stevens]  W. R. Stevens, "Transmission Control Protocol", 1994,
              <TCP/IP Illustrated. Reading, MA: Addison-Wesley, vol. 1>.

              C. Villamizar and C. Song, "High performance tcp in
              ansnet", 1994, <ACM Computer Communications Review,

Subramaniam & Loher      Expires April 30, 2015                 [Page 8]

Internet-Draft       Router Buffer Sizes in the WAN         October 2014

Appendix A.  Additional Stuff

   This becomes an Appendix.

Authors' Addresses

   Kamala Subramaniam (editor)
   Mountain View, CA  94043

   Phone: +1 919 345 8778

   Darren Loher
   Redmond, WA  98052


Subramaniam & Loher      Expires April 30, 2015                 [Page 9]