Internet Engineering Task Force                                A. Biswas
Internet-Draft                                              NetApp, Inc.
Intended status: Standards Track                            May 25, 2010
Expires: November 26, 2010

   Support for Stronger Error Detection Codes in TCP for Jumbo Frames


   There is a class of data serving protocols and applications that
   cannot tolerate undetected data corruption on the wire.  Data
   corruption could occur at the source in software, in the network
   interface card, out on the link, on intermediate routers or at the
   destination network interface card or node.  The Ethernet CRC and the
   16-bit checksum in the TCP/UDP headers are used to detect data
   errors.  Most applications rely on these checksums to detect data
   corruptions and do not use any checksums or CRC checks at their
   level.  Research has shown that the TCP/UDP checksums are catching a
   significant number of errors, however, the research suggests that one
   packet in 10 billion will have an error that goes undetected for
   Ethernet MTU frames (MTU of 1500).  Under certain situations, "bad"
   hosts can introduce undetected errors at a much higher frequency and
   order.  With the use of Jumbo frames on the rise, and therefore more
   data bits on the wire that could be corrupted, the current 16-bit
   TCP/UDP checksum, or the Ethernet 32-bit CRC are simply not
   sufficient for detecting errors.  This document specifies a proposal
   to use stronger checksum algorithms for TCP Jumbo Frames for IPv4 and
   IPv6 networks.  The Castagnoli CRC 32C algorithm used in iSCSI and
   SCTP is proposed as the error detection code of choice.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on November 26, 2010.

Biswas                  Expires November 26, 2010               [Page 1]

Internet-Draft        Stronger TCP Error Detection              May 2010

Copyright Notice

   Copyright (c) 2010 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   ( in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
     1.1.  Conventions  . . . . . . . . . . . . . . . . . . . . . . .  4
   2.  Calculating the CRC-32C value  . . . . . . . . . . . . . . . .  4
   3.  Negotiating the use of CRC 32C . . . . . . . . . . . . . . . .  6
   4.  IPv6 Considerations  . . . . . . . . . . . . . . . . . . . . .  8
   5.  Conclusions and Acknowledgements . . . . . . . . . . . . . . .  8
   6.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . .  9
   7.  Security Considerations  . . . . . . . . . . . . . . . . . . .  9
   8.  References . . . . . . . . . . . . . . . . . . . . . . . . . .  9
     8.1.  Normative References . . . . . . . . . . . . . . . . . . .  9
     8.2.  Informative References . . . . . . . . . . . . . . . . . .  9
   Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 10

Biswas                  Expires November 26, 2010               [Page 2]

Internet-Draft        Stronger TCP Error Detection              May 2010

1.  Introduction

   There is a class of data serving applications that host business and
   financial data.  Detecting and recovering from data corruption is
   paramount to the success of this class of applications.  Data
   corruption can occur while data is transiting from the source to a
   desired destination.  Data can get corrupted right at the source due
   to software errors, within the network interface card, out on the
   wire or link, in intermediate routers and at the destination network
   interface or node.  Link errors are detected using the Ethernet 32-
   bit CRC.  Node or router errors are detected using the 16-bit
   checksum in the transport headers of TCP and UDP.  Most applications
   do not have built-in error detection capability and typically rely on
   the checksums in the underlying networking layers.  Stone et al.
   [Stone] have recommended applications employ their own checksums to
   detect errors that go undetected by lower levels.  They have made
   this recommendation for the standard Ethernet MTU.  They have done so
   considering situations where a "bad" host can introduce undetected
   errors at a much higher frequency and order.  It must also be said
   that the physical layer already does encodings with bit error
   rates(BER) of 10^-12 ti 10^-14 and therefore the current checksum
   algorithms may be sufficient.  However, stronger checksumming
   accounts for the cases where noisy hardware, bad cables can introduce
   noise at a much higher frequency and order.  It is also to be noted
   that increasing speed of the physical medium (to 40G and 100G) can
   also lead to higher BER.

   Another dynamic, very much in the rise is the use and deployment of
   Jumbo Frames.  Jumbo Frames reduce per packet overheads significantly
   and are a cheap way of improving the performance of bulk data
   applications.  Combining the use of Jumbo frames with noisy physical
   medium increases the risk of undetected bit errors as there simply
   are more bits that can get corrupted.  This is rather concerning as
   business and financial data typically are transported over the
   network using file access based protocols like NFS, CIFS, HTTP over

   The strength of the Ethernet CRC checksum and the 16-bit Transport
   checksum has been found to reduce for data segments that are larger
   than the standard Ethernet MTU.  Koopman et. al.  [Koopman] have
   explored a number of CRC polynomials as well as the polynomial used
   in the Ethernet CRC calculation.  They have measured the
   effectiveness of these CRC polynomials for different data word
   lengths, where a data word is a bit stream from 64 bits to 128 Kbits.
   These data word lengths cover lengths equivalent to Ethernet MTUs and
   Jumbo frames and also frame lengths larger than Jumbo frames.  They
   found that the Castagnoli polynomial x^32 + x^28 + x^27 + x^26 + x^25
   + x^23 + x^22 + x^20 + x^19 + x^18 + x^14 + x^13 + x^11 + x^10 + x^9

Biswas                  Expires November 26, 2010               [Page 3]

Internet-Draft        Stronger TCP Error Detection              May 2010

   + x^8 + x^6 + x^0 represented as the 32-bit code 0x8F6E37A0 bests
   other CRC polynomials for Jumbo frames and larger segments.  This
   polynomial has been adopted by the iSCSI and SCTP standards.  It is
   to be noted that this polynomial is represented as the 32-bit code
   0x11EDC6F41 in SCTP in accordance to the convention adopted for bit-
   ordering at the transport-level, i.e., bit-ordering for mapping SCTP
   messages to polynomials is that bytes are taken most significant
   first, but within each bytes, bits are taken least-significant first.

   Given the ubiquity of TCP, it is the layer where we can introduce
   stronger error detection capability without duplicating the effort in
   higher layers.  TCP options provide an easy path to introduce
   stronger checksum without hindering interoperability.  TCP options
   allow a TCP stack supporting a TCP option to interoperate seamlessly
   with a TCP stack that does not support the new TCP option (RFC 1122
   [RFC1122] requires the interoperability in Section

   This document proposes that the use of the Castagnoli polynomial,
   also known as the CRC 32C as the "checksum" of choice for TCP
   protocol.  Other summation based checksum algorithms like Fletcher
   and Adler's algorithm were evaluated in RFC 3385 [RFC3385] and found
   to behave substanially worse than CRCs and hence are not considered
   in this proposal.

   By standardizing a stronger checksum at the TCP level, we can quickly
   drive the offloading of this checksum to NIC hardware, just as the
   16-bit TCP checksum is offloaded by most NIC vendors today.
   Offloading computation to hardware allows us to get rid of the in-
   software computation overheads of stronger checksum algorithms.

   Another positive effect of implementing strong TCP checksumming is
   that this will drive the rapid adoption of 9K Jumbo frames and make
   it considerably easier to consider even larger Jumbo Frames.

1.1.  Conventions

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   document are to be interpreted as described in RFC 2119 [RFC2119].

2.  Calculating the CRC-32C value

   The 16-bit TCP checksum does a checksum of the TCP header and
   payload.  It also includes the pseudo header values of Source
   Address, Destination Address, Protocol and TCP Length.  The addition
   of the bytes of a pseudo header into a summation based checksum
   algorithm is simpler than the inclusion of the bytes of a pseudo

Biswas                  Expires November 26, 2010               [Page 4]

Internet-Draft        Stronger TCP Error Detection              May 2010

   header into a CRC computation.  This is because a CRC computation
   assumes a contiguous bit stream when translating the bit stream to a
   polynomial for doing the polynomial division.  The psuedo header was
   added to the TCP checksum computation in order to detect errors
   introduced in one of the IP header fields that could possibly cause
   the packet to be sent to an incorrect destination.  These fields also
   get included in the IP header checksum.  The intent was to include
   them in two separate checksums for better data integrity.  One can
   question the need for including the pseudo-header fields twice.  The
   pseudo-header currently get included thrice if one considers the fact
   that the Ethernet CRC is computed over the entire Ethernet frame and
   Ethernet is ubiquitous today.  So for the purposes of this draft, all
   the fields used in the current TCP checksum except the pseudo-header
   must be included in the CRC-32c calculation.  If this draft's
   proposal is accepted for standardization, IETF may elect to add back
   the pseudo-header into the CRC-32C calculation or add only a smaller
   subset of the fields.  But it is to be noted that in this proposal we
   do have room to consider changes like this without disrupting current

   It may also be questionable whether one needs to compute the 16-bit
   TCP checksum if the new TCP checksum option is present.  To avoid a
   chicken and egg problem, this document proposes that the 16-bit
   checksum field be zeroed out and included in the CRC 32C checksum as
   part of the TCP header bit stream.  The standardization process may
   choose a different approach and decide to do both the 16-bit TCP
   checksum and the CRC 32C checksum, in which case, a method will need
   to be defined as to the order of checksumming and the fields used in
   each of the checksum computations.

   This document also recommends the use of the CRC-32C when the
   negotiated Maximum Segment Size (MSS) value is equal or greater than
   8948 bytes (excluding frame and TCPIP header bytes), the most common
   Jumbo Frame size, but does not explicitly recommend the use of CRC-
   32C for standard Ethernet MTU frames.

   The CRC-32C MAY be used even for regular Ethernet MTU frames also if
   the application so desires for stricter data integrity checking,
   since CRC-32C can detect more independent bit errors than Ethernet
   CRC for Ethernet MTU sized packets.  The use of CRC-32C can be made
   settable by the application, by providing a socket option to the
   application.  The provision for an application to enable/disable the
   use of the new checksum option is left as an API detail of the
   particular TCP/IP socket layer implementation.

   The following section describes two possible approaches to
   negotiating the proposed 32-bit TCP checksum.  The common thread in
   the two approaches is the use of TCP options to negotiate the use of

Biswas                  Expires November 26, 2010               [Page 5]

Internet-Draft        Stronger TCP Error Detection              May 2010

   this checksum during the connection setup phase.  Once the connection
   is setup, all subsequent packets sent during the connection transfer
   phase MUST carry the stronger checksum except as described below.

   It is also possible that Path MTU discovery causes a connection to
   reduce the negotiated MSS value post connection establishment.  So,
   during connection establishment, an MSS equal or greater than 9K
   might have been negotiated along with stronger TCP checksumming, and
   then later the MSS reduced to be equal to the discovered path MTU.
   If the reduced MSS value is equal or less than an Ethernet MSS
   (typically 1460 without other TCP options), then the TCP end point
   that reduced its MTU may choose to NOT send the TCP checksum option
   in subsequent data packets.  The peer must then rely on the 16-bit
   TCP checksum for end to end data integrity which is okay since the
   Ethernet CRC has comparable data integrity checking capability for
   Ethernet sized packets.

   Now, let us discuss the method for computing the CRC 32c value:

   The CRC computation uses polynomial division.  The TCP header and
   payload is mapped to a polynomial and the CRC is calculated by
   dividing the bit stream with the CRC 32C polynomial.  Stone et. al.
   in Appendix B of RFC 4960 [RFC4960] describe a convention for mapping
   the bytes of the bit stream into the polynomial.  The same MUST be
   adopted for TCP transport too.

3.  Negotiating the use of CRC 32C

   There are two possible approaches to negotiating the proposed CRC 32C
   checksum during the TCP connection setup phase.

   o  A new TCP option

   o  Using the TCP Alternate Checksum Data Option

   The first approach introduces a new TCP option to be negotiated by
   TCP endpoints during the connection setup phase.  It will be of the
   same format as other defined TCP options and will have Type, Length
   and Value fields.  A new type will be requested from IANA.  The
   length field will be the sum total length of the new TCP checksum
   option which is 6 bytes.  The value field will hold the 32-bit CRC
   32C checksum.

   If either one of the peers does not add this option to its TCP
   options list in its SYN segment, the CRC-32C checksum must not be
   used by the other peer.  Most TCP implementations are written to
   process the TCP options they recognize and ignore unknown options on

Biswas                  Expires November 26, 2010               [Page 6]

Internet-Draft        Stronger TCP Error Detection              May 2010

   SYN segments so an endpoint that supports the new TCP option can
   interoperate with an endpoint that does not support the proposed TCP

   Since we have seen that the 16-bit TCP checksum is insufficient for
   detecting multiple independent errors for Jumbo frames, this proposal
   says that a peer supporting this option MUST send the new TCP
   checksum option if its link MTU is equal or greather than 9K.
   However, if the remote peer does not recognize the new option, the
   initiating peer MUST NOT use this TCP extension for the connection
   transfer phase.  If the remote peer recognizes the option and also
   has a Maximum Segment Size equal to the peer's advertised MSS or a
   minimum MSS of 9K, it MUST respond with the TCP checksum option.
   Every subsequent packet from both peers must include this option in
   the TCP header.  The extra overhead for adding this option is minimal
   for Jumbo frame sized segments and the higher data integrity pays for

   Note that all TCP control packets sent after succesfully negotiating
   this TCP option may carry this TCP option also, although this draft
   does not mandate it.

   TCP CRC Checksum Option.

   | Kind = X | Length = 6 | Value = 4 bytes of CRC 32C |


                                 Figure 1

   The second approach utilizes a pair of existing TCP options called
   the "TCP Alternate Checksum Options" specified in RFC 1146 [RFC1146].
   The current checksum types specified by that option are TCP checksum,
   8-bit Fletcher's algorithm and 16-bit Fletcher's algorithm.  A new
   checksum type can be added to this list for CRC-32C checksums.  The
   negotiation rules for selecting the checksum type would follow the
   rules described in RFC1146.  That is, if both SYN segments carry the
   Alternate Checksum Request option, and both specify the same
   algorithm, that algorithm must be used for the remainder of the
   connection.  Otherwise, the standard TCP checksum must be used for
   the entire connection.

   Once the CRC 32C checksum algorithm is negotiated, the TCP Alternate
   Checksum Data Option is sent whose data will equal 4 bytes for the
   CRC-32C checksum.

Biswas                  Expires November 26, 2010               [Page 7]

Internet-Draft        Stronger TCP Error Detection              May 2010

   TCP Alternate Checksum Request Option
   | Kind = 14 | Length = 3 | Value = CRC-32C |

   Here the value for CRC32C would need to be defined, and may possibly
   be the next undefined value '3', following the definitions for 8-bit
   and 16-bit fletcher's algorithms.

   TCP Alternate Checksum Data Option
   | Kind = 15 | Length = 6 | Value = CRC-32C computed value |

   The TCP Alternate Checksum Data Option must be sent only during the
   connection transfer and tear down phase.  Again, the 16-bit TCP
   checksum field must be zeroed out before computing the 32-bit CRC 32C

   One or more padding bytes may be used when sending any of the above
   options to align to a 4 or 8 byte boundary for faster parsing on both
   32-bit and 64-bit machines.

   At this stage of draft development, the author is evaluating and
   seeking inputs for both approaches.

4.  IPv6 Considerations

   The TCP extension for CRC 32C can be applied equally to IPv4 and
   IPv6.  The pseudo header for IPv6 includes 128 bit source and
   destination addresses.  This pseudo header, the TCP header and
   payload MUST be included in the CRC 32C checksum of a TCP/IPv6
   segment as there is no IPv6 header checksum.

5.  Conclusions and Acknowledgements

   This document proposes the use of stronger error detection codes for
   TCP connections sending Jumbo Frames.  It does not provide a solution
   for UDP based applications.  I would also like to thank Tom Kessler
   ( for his review comments.  He specifically
   pointed out his concerns about the safety of TCP checksum + Ethernet
   CRC at 40G and 100G speeds with even 9K jumbo frames.  He also
   provided information on the Intel instruction set that can be used to
   speed up CRC-32c computation.  Special thanks to Janet Takami
   ( for her comments as well as for pointing out
   that there is no IPv6 header checksum and so the pseudo header must

Biswas                  Expires November 26, 2010               [Page 8]

Internet-Draft        Stronger TCP Error Detection              May 2010

   be included in the CRC 32c checksum.

6.  IANA Considerations

   This memo includes a request to IANA for a new Type Number for the
   new TCP Checksum Option if we do not go with the TCP Alternate
   Checksum Option.  If we go with the TCP Alternate Checksum option,
   then a new checksum type will need to be defined for CRC 32C,
   probably after the defined values for Fletcher's 8-bit and 16-bit
   algorithm types.

7.  Security Considerations

   The CRC 32C codes can detect unintentional changes to data such as
   those caused by noise.  If an attacker changes the data, it can also
   change the error-detection code to match the changed data.  Hence,
   these codes are not intended for security purposes.

8.  References

8.1.  Normative References

   [RFC1122]  IETF, "Requirements for Internet Hosts -- Communication
              Layers", October 1989.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

8.2.  Informative References

   [Koopman]  Koopman, P., "32-Bit Cyclic Redundancy Codes for Internet
              Applications", 2002.

   [Stone]    Stone, J., Partridge, C., "When the CRC and TCP Checksum

   [RFC1146]  Zweig, J., Partridge, C., "TCP Alternate Checksum Options"
              March 1990.

   [RFC3385]  Sheinwald, D., et. al. "Internet Protocol Small Computer
              System Interface (iSCSI) Cyclic Redundance Check (CRC)/
              Checksum Considerations", September 2002.

   [RFC4960]  Stewart, R., "Stream Control Transmission Protocol",
              September 2007.

Biswas                  Expires November 26, 2010               [Page 9]

Internet-Draft        Stronger TCP Error Detection              May 2010

Author's Address

   Anumita Biswas
   NetApp, Inc.
   495, E. Java Dr
   Sunnyvale, CA  95054

   Phone: +14088223204

Biswas                  Expires November 26, 2010              [Page 10]