IPv4 Reassembly Errors at High Data Rates
draft-heffner-frag-harmful-05

[Ballot discuss]
I asked Eric Rescorla to review this and he sent the following which does get at exactly my concerns.

The -04 revision of this document doesn't really address most of
my issues. In a number of cases the authors provided some comments
in e-mail but they don't seem to have made it into the document.

1. - This was deal with.

2. If we assume an initial packet size of 10K (10^4 bytes)
then the 10 TB (10^13) bytes represents (10^9 packets).
Stone et al. report error rates of of between 10^-3 and 10^-5.
Accordingly, even ignoring fragmentatio issues we would
expect to see btwn 10^4 and 10^6 errored packets. The authors
report 10^7 errored packets. OTOH, if we assume a more plausible
1500 byte MTU and that the intervening links are smaller, then
the number of errored packets due to fragmentation is commensurate
with the number of pure data errors. I would think the bottom line
here is that you can't trust the TCP/UDP checksum if you can't
accept a base packet error rate of 10^-5 or so packets.

This was to some extent addressed in e-mail but I don't see
anything in the draft. I was hoping the authors would add
some discussion of Stone's results, but that doesn't seem
to have happened.


3. The effective data rate here is quite high (100 Mb/s) for the
Internet environment. What would the situation look like at a more
practical real-world data rate of (e.g., 10 Mb/s). Certainly, we
should confine this to WAN contexts since in a LAN you can directly
observe your MTU and Kent et al. should have convinced people not to
send data which has to be fragmented at their outgoing interface.

Again, this was sort-of addressed in e-mail but I don't see anything in the
draft addressing it.

4. Regarding the flow control/false congestion issue, how much worse
is this setting going to be than one would expect from Kent et al.'s
work? If you only have burst losses, wouldn't you expect that a
TCP-friendly protocol would back off to the point where the
the fragment reassembly buffers start to time out and the cycles time
out.

Again, this was sort-of addressed in e-mail but I don't see anything in the
draft addressing it.


Finally, I'm trying to figure out what the take-home message of this
document is. How would implementors behave differently after reading
this document than they would have before?

After exchanging e-mail with the authors I'm even more confused about
this. Here's the relevant portion:

  Further, the problem definitely does exist on LANs.  One of the
  earliest reported cases of this issue was on NFS servers using
  early 100 Mbps NICs.  NFS transmits one block per datagram, and if
  you configure it with a normal block size of 2k or 4k, it
  fragments.  (You end up getting better performance with the larger
  block size than with 1k blocks.)  The high rate fragmentation led
  to occasional file corruption, in a situation where otherwise the
  bit error rate was very low.

  Part of our motivation in writing this is that it seemed Kent, et
  al. were not always fully persuasive in deterring the use of
  fragmentation.  There have been a number of UDP bulk transport
  tools developed to work around some TCP issues.  For example, see
  , .  Some of these tools use fragmentation
  to reduce syscall CPU overhead, since it increases their overall
  performance, regardless of the Kent/Mogul issues.

Isn't the take-home message here that fragmentation is sometimes
good, provided that you use a strong checksum to detect and compensate
for occasional corruption?

[Ballot comment]
I agree with Dave Kessens that the title and abstract need to be changed to make it clear that this is really only talking about IP Fragmentation problems for extremely high bandwidth communication (which may indeed be quite important for supercomputer centers, but are not applicable for normal Internet users). However, I don't see any need to enter a "discuss" because Dave already has one for the same issue and I agree with Dave's proposed solution of changing the title and abstract to make it clear what the scope of this document actually is.

[Ballot discuss]
I asked Eric Rescorla to review this and he sent the following which does get at exactly my concerns.

$Id: draft-heffner-frag-harmful-03-rev.txt,v 1.2 2006/12/14 05:39:35 ekr Exp $

This document argues that as a result of the IP fragmentation field
being only 16 bytes, it is possible to confuse fragments from multiple
packets and as a consequence produce pathological results. This is
potentially an issue in two cases:

- Bulk data transfer via UDP
- Where TCP implementations don't set DF (rare) or where gateways
  don't respect it.

The pathological results come in two flavors:

- Corrupt packets being caught by the TCP/UDP checksum and
  being interpreted as congestion
- Corrupt data passing the too-short TCP/UDP checksum and
  being passed to the application.

After reviewing the experimental data in S 4 I'm not sure.
it supports the author's conclusions.

1. It seems to be incompletely described. In particular, the authors
don't describe the pre-fragmentation datagram size, which is important
for interpreting the results.  After all, if the datagram size is MTU
then presumably they wouldn't get any fragmentation at all whereas if
it's the same as the file size then presumably effective throughput
would be near zero. This may be fixed in QUANTA but it needs to be
described here.

2. If we assume an initial packet size of 10K (10^4 bytes)
then the 10 TB (10^13) bytes represents (10^9 packets).
Stone et al. report error rates of of between 10^-3 and 10^-5.
Accordingly, even ignoring fragmentatio issues we would
expect to see btwn 10^4 and 10^6 errored packets. The authors
report 10^7 errored packets. OTOH, if we assume a more plausible
1500 byte MTU and that the intervening links are smaller, then
the number of errored packets due to fragmentation is commensurate
with the number of pure data errors. I would think the bottom line
here is that you can't trust the TCP/UDP checksum if you can't
accept a base packet error rate of 10^-5 or so packets.

3. The effective data rate here is quite high (100 Mb/s) for the
Internet environment. What would the situation look like at a more
practical real-world data rate of (e.g., 10 Mb/s). Certainly, we
should confine this to WAN contexts since in a LAN you can directly
observe your MTU and Kent et al. should have convinced people not to
send data which has to be fragmented at their outgoing interface.

4. Regarding the flow control/false congestion issue, how much worse
is this setting going to be than one would expect from Kent et al.'s
work? If you only have burst losses, wouldn't you expect that a
TCP-friendly protocol would back off to the point where the
the fragment reassembly buffers start to time out and the cycles time
out.


Finally, I'm trying to figure out what the take-home message of this
document is. How would implementors behave differently after reading
this document than they would have before?

[Ballot discuss]
How real is this problem really for most Internet users ?

The document says:

  For example, a host sending 1500
  byte packets with a 30 second maximum packet lifetime could send at
  only about 26 Mbits/s before exceeding 65535 packets per packet
  lifetime.  Or, filling a 1 Gbit/s interface with 1500 byte packets
  requires sending 65536 packets in less than 1 second, an unreasonably
  short maximum packet lifetime, being less than the round-trip time on
  some paths.  This requirement is widely ignored.

Basically, how many people are able to blast 1 Gbit/s out of a single
interface but have such bad connectivity that they regurlarly have
roundtrip times of 1 second or more. Or for the other example, a 30
second maximum lifetime seems rather high for most Internet communications
that don't leave earth.

This is not to say that fragmentation is not extremely harmful, for many other reasons that are not mentioned in this draft.

Basically, this draft really only supports a title and abstract that says
something like 'ipv4 fragmentation problems with
1500 byte packets and datarates higher than around 1Gbs'.

This problem is really easy to fix: just change the title,
abstract and introduction a little bit that makes it clear that the scope
is quite limited (until most people get to enjoy 1gbs on every desktop).

[Ballot comment]
5.  Implications
...
  IPv6 is less vulnerable to this type of problem, since its fragment
  header contains a 32-bit identification field [RFC2460].  Mis-
  association will only be a problem at packet rates 65536 times higher
  than for IPv4.

Should note that IPv6 fragmentation only occurs e2e and there is no
DF bit; hence errors caused by non-respect of the DF bit cannot occur.

From Gen-ART reviewer Robert Sparks:
"Some comments from a personal preference point-of-view:

Consider changing the title to something describing the results  directly - make this more likely to find when someone in the future uses the rfc-index to find issues  with reassembly.

Tuning the abstract to reflect the results rather than the  consequenses of the results might also help draw eyes to the document, but I'm not sure how many people  filter/choose documents based on the abstract text. On the other hand, a lot of the people you  probably want to reach have been
de-sensitized to fragmentation/congestion klaxons - your message  might get out faster without them. "

[Ballot comment]
5.  Implications
...
  IPv6 is less vulnerable to this type of problem, since its fragment
  header contains a 32-bit identification field [RFC2460].  Mis-
  association will only be a problem at packet rates 65536 times higher
  than for IPv4.

Should note that IPv6 fragmentation only occurs e2e and there is no
DF bit; hence errors caused by non-respect of the DF bit cannot occur.