Internet-Draft                                               Matt Mathis
                                                            John Heffner
                                                             Kevin Lahey
                                                            14 Feb, 2004

                           Path MTU Discovery

Status of this Memo

   This document is an Internet-Draft and is in full conformance with
   all provisions of Section 10 of RFC2026.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups. Note that other
   groups may also distribute working documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at

   The list of Internet-Draft Shadow Directories can be accessed at


   This document describes a robust new method for Path MTU Discovery
   that relies on TCP or other Packetization Layer to probe an Internet
   path with progressively larger packets.  This method is described as
   an extension to RFC 1191 and RFC 1981, which specify ICMP based Path
   MTU Discovery for IP versions 4 and 6.  This document does not define
   a protocol, but rather a method to use features of existing protocols
   to discover the path MTU.

   The general strategy of the new algorithm is to start with a small
   MTU and probe upward, testing successively larger MTUs by probing

Mathis, et al                                                   [Page 1]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   with single packets.  If the probe is successfully delivered, then
   the MTU is raised.  If the probe is lost, it is treated as an MTU
   limitation and not as a congestion signal.

Table of Contents

   1. Introduction  . . . . . . . . . . . . . . . . . . . . . . . 3
   2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . 3
   3. Overview  . . . . . . . . . . . . . . . . . . . . . . . . . 6
   3.1. General Method  . . . . . . . . . . . . . . . . . . . . . 8
   3.2. Generating Probes . . . . . . . . . . . . . . . . . . . . 9
   3.3. Normal sequence of events to raise the MTU  . . . . . . . 10
   3.4. Processing MTU Indications  . . . . . . . . . . . . . . . 11
   3.4.1. Processing Packet Too Big Messages  . . . . . . . . . . 11
   3.4.2. Packetization Layer retransmits lost packets  . . . . . 11
   3.4.3. Packetization Layer Retransmission Timeout  . . . . . . 13
   3.5. Probing Intervals . . . . . . . . . . . . . . . . . . . . 14
   3.6. Interoperation with prior algorithms  . . . . . . . . . . 15
   4. Requirements  . . . . . . . . . . . . . . . . . . . . . . . 15
   5. Implementation Issues . . . . . . . . . . . . . . . . . . . 16
   5.1. Layering and Accounting for Header Sizes. . . . . . . . . 17
   5.2. Storing PMTU information  . . . . . . . . . . . . . . . . 18
   5.3. Host fragmentation  . . . . . . . . . . . . . . . . . . . 19
   5.4. Multicast . . . . . . . . . . . . . . . . . . . . . . . . 19
   5.5. Path MTU Search Strategy  . . . . . . . . . . . . . . . . 20
   5.5.1. Search  . . . . . . . . . . . . . . . . . . . . . . . . 20
   5.5.2. Monitor . . . . . . . . . . . . . . . . . . . . . . . . 21
   5.5.3. Suspend . . . . . . . . . . . . . . . . . . . . . . . . 21
   5.6. Implementation issues for specific Packetization Layers . 21
   5.6.1. Probing method using TCP  . . . . . . . . . . . . . . . 21
   5.6.2. Probing method using SCTP . . . . . . . . . . . . . . . 22
   5.6.3.  Issues for tunnels . . . . . . . . . . . . . . . . . . 23
   5.6.4.  Issues for other transport protocols . . . . . . . . . 23
   5.7.  Diagnostic tools . . . . . . . . . . . . . . . . . . . . 23
   5.8.  Management interface . . . . . . . . . . . . . . . . . . 23
   6. Normative references  . . . . . . . . . . . . . . . . . . . 24
   7. Informative references  . . . . . . . . . . . . . . . . . . 24
   8. Security considerations . . . . . . . . . . . . . . . . . . 24
   9. IANA considerations . . . . . . . . . . . . . . . . . . . . 25
   10. Contributors . . . . . . . . . . . . . . . . . . . . . . . 25
   11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . 25
   12. Authors' addresses . . . . . . . . . . . . . . . . . . . . 25
   13. Intellectual Property  . . . . . . . . . . . . . . . . . . 25
   14. Full copyright statement . . . . . . . . . . . . . . . . . 26

Mathis, et al                                                   [Page 2]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

1. Introduction

   This document describes a method for Packetization Layer Path MTU
   Discovery (PLPMTUD) which is an extension to existing Path MTU
   discovery methods.  The proper MTU is determined by starting with
   small packets and probing with successively larger packets.   The
   bulk of the algorithm is implemented above IP, in the transport layer
   (e.g. TCP) or other "Packetization Protocol" that is responsible for
   determining packet boundaries.

   This document draws heavily RFC1191 and RFC1981 for terminology,
   ideas and some of the text.

   The methods described in this document apply both IPv4 and IPv6, and
   to many transport protocols such as TCP.   This document does not
   define a protocol, but rather a method to use features of existing
   protocols to discover the path MTU.  It does not require cooperation
   from the lower layers (except that they are consistent about what
   packet sizes are acceptable) or the far node.  Variants in
   implementations will not cause interoperability problems.

   The methods described in this document are carefully designed to
   maximize robustness in the presence of less than ideal
   implementations of other protocols or Internet components.

   For sake of clarity we uniformly prefer TCP and IPv6 terminology.  In
   the terminology section we also present the analogous IPv4 terms and
   concepts for the IPv6 terminology.  In a few situations we describe
   specific details that are different between IPv4 and IPv6.

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   document are to be interpreted as described in [RFC 2119].

2. Terminology

   IP          - Either IPv4 [IPv4-SPEC] or IPv6 [IPv6-SPEC].

   node        - A device that implements IP.

   router      - A node that forwards IP packets not explicitly
                 addressed to itself.

   host        - Any node that is not a router.

Mathis, et al                                                   [Page 3]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   upper layer - A protocol layer immediately above IP.  Examples are
                 transport protocols such as TCP and UDP, control
                 protocols such as ICMP, routing protocols such as OSPF,
                 and Internet or lower-layer protocols being "tunneled"
                 over (i.e., encapsulated in) IP such as IPX,
                 AppleTalk, IP itself.

   link        - A communication facility or medium over which nodes can
                 communicate at the link layer, i.e., the layer
                 immediately below IP.  Examples are Ethernets (simple
                 or bridged); PPP links; X.25, Frame Relay, or ATM
                 networks; and Internet (or higher) layer "tunnels",
                 such as tunnels over IPv4 or IPv6.   In some earlier
                 documents the term "lower layer" was used for this

   interface   - A node's attachment to a link.

   address     - An IP-layer identifier for an interface or a set of

   packet      - An IP header plus payload.

   MTU         - Maximum Transmission Unit, the size in bytes of the
                 largest IP packet, including the IP header and payload,
                 that can be transmitted on a link or path.   Note that
                 this could more properly be called the IP MTU, to be
                 consistent with how other standards organizations use
                 the acronym MTU.

   link MTU    - The Maximum Transmission Unit, i.e., maximum IP packet
                 size in octets, that can be conveyed in one piece over
                 a link.    Beware that this definition differers from
                 the definition used by other standards organizations.

                 For IETF documents, link MTU is uniformly defined as
                 the IP MTU over the link.  This includes the IP header,
                 but excludes link layer headers and other framing which
                 is not part of IP or the IP payload.

                 Other standards organizations generally define link MTU
                 to include the link layer headers.

   path        - The set of links traversed by a packet between a source
                 node and a destination node

   path MTU    - The minimum link MTU of all the links in a path between
                 a source node and a destination node.

Mathis, et al                                                   [Page 4]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   PMTU        - Path MTU

   classical PMTU discovery,
               - Process described in RFC 1191 and RFC 1981, in which
                 nodes rely on ICMP messages to learn the MTU of a path.

   PL, packetization layer
               - The layer of the network stack which segments data into

   PLPMTUD     - Packetization Layer Path MTU Discovers, the method
                 described in this document, which is an extension to
                 classical PMTU discovery.

   Packet Too Big message
               - An ICMP message reporting that an IP packet is too
                 large to forward.  This is the IPv6 term that
                 corresponds to the IPv4 "ICMP Can't fragment" message.

   flow        - A context in which MTU discovery is applied.  This is
                 naturally an instance of the packetization protocol, e.g.
                 one side of a TCP connection.

   MPS         - The maximum IP payload size available over a specific
                 path.  This is typically the path MTU minus the IP header
                 As an example, this is the maximum TCP packet size,
                 including TCP payload and headers but not including
                 IP headers.   This has also been called the "L3 MTU".

   MSS         - The TCP Maximum Segment Size, the maximum payload
                 size available to the TCP layer.   This is typically the
                 path MPS minus the size of the TCP headers.

   probe packet- A packet which is being used to test for a larger MTU.

   probe size  - The size of a packet being used to probe for a larger MTU.

   successful probe
               - The probe packet was delivered through the network.

   inconclusive probe
               - The probe packet was not delivered, but there were other lost
                 packets close enough to the probe where can not presume that
                 the probe was lost due to MTU. By implication the probe
                 might have been lost due to something other than MTU (such
                 congestion), so the results are inconclusive.   Inconclusive
                 probes are generally repeated at the same probe size, after
                 a suitable delay.

Mathis, et al                                                   [Page 5]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   failed probe
               - The probe packet was not delivered and there were no other
                 lost packets close to the probe.   This is taken as an
                 indication that the probe was larger than the path MTU,
                 and future probes should generally be for at smaller sizes.

   errored probe
               - There were losses or timeouts during the verification
                 phase which suggest a potentially disruptive failure
                 or network condition.   These are generally retried
                 only after substantially longer intervals.
                 @@@ not used

   probe gap   - The expected missing payload data that will need to be
                 retransmitted if the probe is not delivered.

   probe phase - The interval (time or protocol events)
                 between when a probe is sent, and when
                 it is determined that the the probe succeeded, failed or was

   verification phase - An additional interval during which the new path MTU
                 is considered provisional.   Packet losses or timeouts are
                 treated as an indication that there may be a problem with
                 the provisional MTU.

   Transition phase - The interval between the probe phase and the
                 verification phase, during which packets using the new MTU
                 propagate to the far node and the acknowledgment propagates

   full stop timeout - a timeout where the none of the packets transmitted
                 after some specific event at the sender (e.g. entering the
                 probe or verification phase) is acknowledged by the receiver.
                 This is taken as an indication that the MTU change caused
                 some failure in the network.

   search strategy - the heuristics used to choose successive probe sizes
                 to converge to the proper path MTU, as described in
                 section 5.5.

3. Overview

   This document describes a method for TCP or other packetization

Mathis, et al                                                   [Page 6]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   protocols to dynamically discover the MTU of a path without relying
   on explicit signals from the network.  These procedures are
   applicable to TCP and other transport- or application-level
   packetization protocols in which the receiver always reports to the
   sender complete information about which packets were lost in the

   The general strategy of the new procedure is for the packetization
   layer to find the proper MTU by probing with progressively larger
   packets, without disrupting its normal protocol operation.  If a
   probe packet is successfully delivered, then the path MTU is
   provisionally raised.  If there are no additional losses during the
   subsequent verification phase, then the path MTU is confirmed
   (verified) to be at least as large as the provisional MTU.  PLPMTUD
   can then probe again with an even larger MTU, according to MTU search
   strategy described in section 5.5.

   The verification phase is used to detect situations where raising the
   MTU greatly raises the packet loss rate.  For example this might
   happen if some link is striped across multiple physical channels and
   the stripes have inconsistent MTUs.

   A conservative implementation of PLPMTUD would use a full round trip
   time for the verification phase.  In this case each time PLPMTUD
   raises the MTU it takes three full round trip times to do so.  It
   takes one round trip for the probe phase, during which the probe
   propagates to the far node and an acknowledgment is returned.   The
   second round trip is the transitional phase, during which data
   packets using the provisional MTU propagate to the far node and are
   acknowledged.  During he third and final round trip time, it is
   verified that raising the MTU does not cause excessive loss.

   The isolated loss of a probe packet (with or without a Packet Too Big
   message) is treated as an indication of an MTU limit, and not as a
   congestion indicator.  In this case alone, the packetization protocol
   is permitted to retransmit the probe gap without adjusting the
   congestion window.

   If there is a timeout or additional lost packets during any of the
   three phases, the loss is treated as a congestion indication as well
   as a indication of some sort of failure of the PLPMTUD process.   The
   congestion indication is treated like any other congestion
   indication: window or rate adjustments are mandatory per the relevant
   congestions control standards [Congestion].   Probing can resume with
   some new probe size after a delay which is determined by the nature
   of the indicated failure.

   The most likely (and least serious) PLPMTUD failure is the link

Mathis, et al                                                   [Page 7]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   experiencing legitimate congestion related losses at about the same
   time as the probe.   In this case, it is appropriate to retry the
   probe (with the same probe size) as soon as the packetization layer
   has fully adapted to the congestion and recovered from the losses.

   In other cases, additional losses or timeouts indicate problems with
   the link or packetization layer, and that probes may be disruptive.
   In these situations it is desirable to use progressively longer
   delays depending on the severity of the failure and if it is

   PLPMTUD can optionally process Packet Too Big messages to select the
   provisional MTU for faster convergence in exchange for a slight
   decrease in robustness.  Processing malicious or erroneous Packet Too
   Big messages can cause PLPMTUD to arrive at the incorrect MTU for a
   path, which is likely to reduce protocol performance.  The document
   describes three options for processing Packet Too Big messages:
   completely ignore them, only accept them in response to probes or
   accept all Packet Too Big messages (fully implementing classic PMTUD
   within PLPMTUD).  Theses are further described in section 3.8.

   Relatively few details of this procedure affect interoperability with
   other standards or Internet protocols.  These details are specified
   in RFC2119 standards language in the requirements section.  The vast
   majority of the implementation details are recommendations based on
   experiences with earlier versions of path MTU discovery.  These are
   motivated by a desire to maximize robustness of PLPMTUD in the
   presence of less than ideal implementations as they exist in the

3.1. General Method

   Most of the difficulty in implementing PLPMTUD arises because it
   needs to be implemented in several different places within a single
   node.  In general each packetization protocol needs to have it's own
   implementation of PLPMTUD.  Furthermore, the natural mechanism to
   share path MTU information between concurrent or subsequent
   connections over the same path is a path information cache in the IP
   layer.  The various packetization protocols need to have the means to
   access and update the shared cache in the IP layer.

   Rather than prescribing implementation details this memo describes
   PLPMTUD in terms of its primary subsystems, without fully describing
   how they are integrated into a complete implementation.  The
   subsystems are: generating probes, processing probe responses, the
   search strategy and, the supporting infrastructure (including the

Mathis, et al                                                   [Page 8]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   path cache).   The first two are introduced in this section and are
   subject to the requirements specified in the following section.   The
   probe strategy and issues related to the support infrastructure and
   cache are covered in the implementation section.

3.2. Generating Probes

   A new candidate MTU is tested by sending one "probe packet", which is
   larger than the current MTU.  In this section we present a couple of
   possible ways to alter packetization layers to generate probe
   packets.   The different techniques incur different overheads in
   three areas: difficulty in generating the probe packet (in terms of
   packetization layer implementation complexity and computational
   overhead) possible additional network capacity consumed by the probes
   and the overhead of recovering from failed probes (both network and
   protocol overhead).

   For example a protocol such as SCTP might be extended to allow
   padding with dummy data inside the SCTP packets.  This greatly
   simplify the implementation because the probing can be performed
   without participation from the application and if the probe fails,
   the missing data (the "probe gap") is assured to fit within the
   current MTU when it is retransmitted.  However, the padding does
   consume network capacity without carrying any useful payload.

   This technique does not work for TCP, because there is not a separate
   length field or other mechanism to differentiate between padding and
   real payload data.  With TCP the natural approach is to send
   additional payload data in an over-sized segment.   There are several
   variants which have different tradeoffs.

   In one method, after a TCP probe segment has been sent the subsequent
   segment(s) may be sent as though the probe segment was not over-
   sized.  Thus if the probe segment is lost, it will leave a gap in the
   sequence space that is exactly the correct size to be filled by one
   segment at the current MTU.   Since this method generates overlapping
   data, it will cause duplicate acknowledgments if the probe is
   successfully delivered.  The sender must be capable of ignoring these
   expected duplicate acknowledgments in a manner which will not cause
   unnecessary retransmission or congestion window reduction.

   In the second method, after a TCP probe segment has been sent,
   subsequent TCP segments are sent in a non-overlapping manner.  If the
   probe segment is lost, it will leave a gap which will require
   retransmission of multiple segments to fill.  This method has lower
   overhead for successful probes, but it requires more complexity in
   the retransmit logic to correctly retransmit the missing data with
   multiple segments that fit into the old MTU, while properly

Mathis, et al                                                   [Page 9]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   suppressing the congestion adjustments for this one situation and no

   Under all conditions it is important that the packetization layer
   sends additional data (packets) after the probe, such that Fast
   Retransmit or equivalent algorithms in the packetization layer will
   trigger the retransmission of the probe if it is lost in the network.

3.3. Normal sequence of events to raise the MTU

   If the probe size is smaller than the path MTU and there are no other
   losses, the normal sequence of events will be:

 Step 1) The probe is sent, followed by more packets at the current MTU.
   By definition PLPMTUD enters the probe phase.   The probe propagates
   through the network and the far node acknowledges it (or possibly
   latter data, if ACKs are cumulative and delayed ACK is in effect).

 Step 2) The ACK for the probe reaches the data sender.   By definition,
   this ends the probe phase.

 Step 3) The packetization layer provisionally raises the MTU to the
   probe size.  PLPMTUD enters the transitional phase when it starts
   sending data using the provisional MTU.

   Note that implementations that use packet counts for congestion
   accounting (e.g. keep cwnd in units of packets) must rescale their
   congestion accounting such that raising the MTU does not raise the
   total congestion window or data rate.

   If the implementation packetizes the data at the application API, it
   may transmit already queued data at the current MTU before raising
   the MTU.  In this case this data is not part of either the probing or
   transition phases, because all of the packets in flight fit within
   the current MTU.

 Step 4) Once the first packet of the transitional phase is
   acknowledged, PLPMTUD enters the verification phase to determine if
   raising the MTU causes packet loss.   In principle the verification
   phase can be of arbitrary duration, however at this time we are
   recommending one full window of data (i.e one full round trip time).

 Step 5) Once there has been sufficient data delivered and acknowledged
   in the provisional MTU is considered verified and the path MTU is
   updated.   PLPMTUD can then probe for an even larger MTU, as
   described in the searching strategy in section 5.5.

Mathis, et al                                                  [Page 10]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

3.4. Processing MTU Indications

   Other events described below are treated as exceptions and alter or
   cancel some of the steps above.

3.4.1. Processing Packet Too Big Messages

   Classical PMTU discovery specifies the generation of Packet Too Big
   Messages if an over-sized packet (e.g. a probe) encounters a link
   that has a smaller MTU.  Since these messages can not be
   authenticated they introduce a number of well documented denial of
   service attacks against classical PMTUD [DOS].

   In PLPMTUD these messages are not required for correct operation, so
   in principle they can be summarily ignored at the expense of slower
   convergence to the proper MTU.   However we believe that a slightly
   better compromise is to process Packet too big messages in two
   specific contexts: in conjunction with a PLPMTUD probe or a
   retransmission timeout in the packetization layer (indication a re-
   route to a link with a smaller MTU).

   Every Packet Too Big Message should be subjected to the following

 o If globally forbidden then discard the message.

 o If forbidden by the application then discard the message.

 o If this path has been tagged "bogus ICMP messages" then discard the

 o If the reported MTU fails consistency checks then set "bogus ICMP
   messages" flag for this path and discards the message.  These
   consistency checks include: unrecognized or unparseable enclosed
   header, reported MTU is larger than the size indicated by the
   enclosed header or larger than the current MTU, provisional MTU or
   probe size as appropriate.

 o If the Packet Too Big Message is acceptable under all of these
   checks, save the "ICMP MTU", pending another packetization layer
   protocol event.

3.4.2. Packetization Layer retransmits lost packets

Mathis, et al                                                  [Page 11]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   Each packetization protocol has it's own mechanism to detect lost
   packets and request the retransmission of missing data.  The primary
   signals used by the packetization layer are these protocol specific
   loss indications.   In all cases the packetization layer is
   responsible for retransmitting the lost data and notifying PLPMTUD
   that there was a loss.

 o If the probe itself was lost, and there were no other losses during
   the probe phase (The RTT between when the probe was sent and the loss
   detected) than it is taken as an indication that the path MTU is
   smaller than the probe size.  In this situation alone the
   Packetization Layer is permitted to retransmit the missing data (the
   "probe gap") without adjusting its congestion window or data
   transmission rate.

   If an accepted Packet Too Big Message was received after the probe
   was sent, and it passes the additional checks that the ICMP MTU is
   greater than the current MTU, then set the provisional MTU to the
   ICMP MTU and proceed from step 3 in section 3.3 above.

   If there was not a accepted Packet Too Big Message, then the
   indicated event is a "probe failure", which can be retried with a
   smaller probe size after a suitable delay for a probe_failure_event.
   See section 3.7 for more complete descriptions of failure events.

 o If there are losses during the probe phase and the probe was not
   lost, then the probe was successful.  However, since additional loses
   have the potential to spoil the verification phase, it is important
   that PLPMTUD not progress into the transition phase (step 3 above)
   until after the Packetization Layer has fully recovered from the
   losses and completed the congestion window (or rate) adjustment.

 o If there are losses during the probe phase and the probe was also
   lost the outcome depends on the presence an ICMP MTU set by an
   acceptable Packet Too Big Message.

   If there was an accepted Packet Too Big Message received since the
   probe was sent, and it passes the additional checks that the ICMP MTU
   is greater than the current MTU, then set the provisional MTU to the
   ICMP MTU, and once the Packetization Layer has fully recovered from
   the losses and completed the congestion window (or rate) adjustment
   then proceed to step 3 in section 3.3 above.

   If there was not an accepted Packet Too big Message, then the probe
   is inconclusive because the lost probe might have been caused by
   congestion.   The probe can be retried  after a suitable delay for a

Mathis, et al                                                  [Page 12]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

 o Losses during the transition phase do not receive special treatment.

 o Losses during the verification phase are taken as a indication that
   the path may have a non-uniform MTU or some other problems such that
   raising the MTU substantially raises the loss rate.  If so, this is
   potentially a very serious problem, so the provisional MTU is
   considered to have failed the verification phase and the path MTU is
   set back to the previously verified MTU (the previously the current

   Packet loss during the verification phase might also be due to
   coincidental congestion on the path, unrelated to the probe, so it
   would seem to be desirable that PLPMTUD re-probes the path.  The risk
   is that this effectively raises the tolerated loss threshold because
   even though raising the MTU causes additional loss, there is a
   statistical chance that repeated attempts to verify a the new MTU may
   yield as false pass.    The compromise is to re-probe once with the
   same probe size (after delay inconclusive_probe_event), and if this
   also fails, then the probe may not be retried until after a suitable
   delay for a verification_fail_event, which exponentially increases on
   each successive failure.

   Losses during the verification phase may indicate that a Packet Too
   Big Message reported the incorrect ICMP MTU, so if the provisional
   MTU was updated from the ICMP MTU (which was from an earlier Packet
   Too Big Message), set the "bogus ICMP message" flag for this path.
   This will prevent PLPMTUD from processing further "Packet Too Big
   Messages" for this path.   If the provisional MTU was correct, the
   re-probe above will correctly use it.   If it was not correct, then
   by definition the path reported at least one incorrect  "Packet Too
   Big Message", and should not process any additional messages.

3.4.3. Packetization Layer Retransmission Timeout

   If there is a retransmission timeout during the probe or verification
   phase it may be an indication of a serious problem with the path or
   the Packetization Layer.  We first define the notion of a "full stop
   timeout" to be a timeout where the none of the packets transmitted
   after some specific event at the sender (e.g. entering the probe or
   verification phase) is acknowledged by the receiver.    If a
   retransmission timeout is not full stop it is processed above as
   loss, except using longer delays before re-probing.
   (probe_timeout_event, verification_timeout_event)

   If there is a full stop timeout following a probe then it is taken as
   an indication that probing may be disruptive to either the network or
   the far node (e.g. it triggered a bug halt due to a buffer overrun,

Mathis, et al                                                  [Page 13]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   etc).   The probe should not be retried until after a long delay, for
   probe_stop_event.  Not that this makes it particularly important that
   probes are only sent when the sender can send sufficient additional
   data to assure the correct operation of Fast Retransmit and similar
   algorithms in the Packetization Layer.

   If there is a full stop timeout when the path MTU is raised to the
   provisional MTU and the provisional MTU was updated from the ICMP
   MTU, then it is assumed that the MTU reported in the Packet Too Big
   Message was incorrect.  Set the "bogus ICMP message" flag for this
   path and re-probe with a smaller probe size after a suitable delay
   for an ICMP_fail_event.

   If there is a full stop timeout when the path MTU is raised to the
   provisional MTU and the provisional MTU is the same as the probe size
   (because the probe packet was not lost), then something truly
   unexpected happened.   It is possible that the timeout is unrelated
   to the probe, so in this situation re-probe with a smaller probe size
   after a suitable delay for an verification_stop_event.

3.5. Probing Intervals

   Section 3.4 above describes a number of probe failure events.   In
   all cases the basic response is the same: to wait some time interval
   (dependent on the specific event and possibly the history) and then
   to probe again.  For events that are "inconclusive", is is generally
   appropriate to re-probe with the same probe size.   For events that
   are identified as "failed probes" is is generally appropriate to re-
   probe with a smaller probe size.   The search strategy described in
   section 5.5 is used to select probe sizes.

   Many of the intervals below are specified in terms of elapsed round
   trips relative to the current congestion window.   This is because
   TCP and other Packetization Layer protocols tend to exhibit periodic
   loses which cause periodic variations of the congestion window and
   possibly the data rate.  It is preferable that the PLPMTUD probes are
   scheduled near the low point of these cycles.

   In order from least to most serious:

   inconclusive_probe_event - Packet loss near the lost probe marked the
   result ambiguous.   Since the loss of non-probe causes a window (or
   data rate) reduction, it is desirable to schedule the re-probe (of
   the same probe size) a few round trips after the end of the recovery.

   ICMP_fail_event - Since this is detected by a timeout, it is first
   desirable for the packetization protocol to come back into

Mathis, et al                                                  [Page 14]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   equilibrium with the network (for TCP, this generally means recover
   it's self clock by completing slowstart up to one half of the old
   congestion window) before probing again with a smaller probe segment.

   @@@ TODO finish

   probe_failure_event -

   verification_fail_event -

   verification_timeout_event -

   probe_timeout_event -

   verification_stop_event -

3.6. Interoperation with prior algorithms

   To cache or not.  To ICMP or not -

   Three choices for processing packet too big: ignore all, accept all
   or only on probes.

   Three choices for starting size: cached, small, or interface

4. Requirements

   All Internet nodes SHOULD implement PLPMTUD in order to discover and
   take advantage of the largest MTU supported along the Internet path.

   Links MUST not deliver packets that are larger than their MTU.  Links
   that have parametric limitations (e.g. MTU bounds due to limited
   clock stability) MUST include explicit mechanisms to consistently
   reject packets that might otherwise be nondeterministically

   The requirements below only apply to those implementations that
   include PLPMTUD.

   If the IP protocol is IPv4 the DF bit must be set.

   A packetization protocol MUST use a loss reporting mechanism
   mechanism (e.g. SACK) which avoids spurious retransmission of any
   other data when a probe packet is lost.

Mathis, et al                                                  [Page 15]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   A Packetization Layer SHOULD NOT send a probe packet unless the flow
   is expected to have at least the 3 round trips worth of data needed
   to successfully complete the probing and verification process.

   A Packetization Layer MUST NOT send a probe unless it has sufficient
   data available to send such that a lost packet will trigger Fast
   Retransmit or similar algorithm.

   Failed and inconclusive probes MUST NOT be sent more frequently than
   the normal congestion interval for the current average window size.
   @@@@ too TCP specific

   During the probe, the normal congestion control machinery MUST remain
   in effect except when only the probe gap is detected as lost.  In
   this case the normal multiplicative congestion window reduction is
   suppressed.  If any other lost data is detected, all normal
   congestion control MUST take place.

   If the probe is successful, the current MPS is updated to the
   candidate MPS.  If window and other congestion state variables are
   kept in units of packets, they MUST be rescaled to preserve the
   current window size in bytes.   @@ move

5. Implementation Issues

   This section discusses a number of issues related to the
   implementation of Path MTU Discovery.  This is not a specification,
   but rather a set of notes provided as an aid for implementers.

   The issues include:

   - What layer or layers implement Path MTU Discovery?

   - Accounting for headers

   - How is the PMTU information cached?

   - How are ICMP messages processed

   - How to implement PMTUD with TCP?

   - What should other transport and higher layers do?

Mathis, et al                                                  [Page 16]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   - What should tunnels above IP do?

5.1. Layering and Accounting for Header Sizes.

   Packetization Layer Path MTU Discovery is most easily implemented by
   splitting its functions between layers.  The IP layer is in the best
   place to keep shared state, collect the ICMP messages, track IP
   headers sizes and manage MTU information from the link layer
   interfaces.  However the procedures that PLPMTUD uses for probing,
   verifications and scanning for the path MTU are very tightly coupled
   to the data recovery and congestion control state machines in the
   Packetization Layer.   The most difficult part of implementing
   PLPMTUD is properly splitting the implementation between the layers.

   Note that this layering is constant with the advice in the current
   PMTUD specifications [RFC1191, RFC1981].  Today, many implementations
   of classical PMTU Discovery are already split along these same

   Early implementation of PLPMTUD revealed that it is critically
   important to have a good clean mechanism for accounting header sizes
   at all layers.  This is because the Packetization Layer does its
   calculations in its own natural data unit, which are almost always a
   reflection of the service that the Packetization Layer provides to
   the application or other upper layers.  For example, TCP naturally
   performs all of its calculations in terms of sequence numbers and
   segment sizes, and the probe gap is the segment that was carried by
   the probe packet.  However, the probe size, ICMP MTU, etc are
   measures of full packets, which not only include the TCP data and
   fixed IP and TCP headers, but may also include IP extension headers
   or options, TCP options and even IPsec AH or ESP headers.

   PLPMTUD requires frequent bidirectional translation between these two
   domains: the Packetization Layer's natural data unit and full IP
   packet sizes.  While there are a number of possible ways to
   accurately implement this duality of size measures, our experience
   has been that it is best if the boundary between the IP layer and the
   Packetization layer communicate in terms of the IP Maximum Payload
   Size or MPS.  The MPS is the only size measure that is common to both
   the IP and Packetization Layers, because it exactly matches the
   boundary between the layers.  The IP Layer is responsible for adding
   or deducting it's own headers when translating between MTU and MPS.
   Likewise the Packetization Layer is responsible for adding or
   deducting its own headers when calculations in it's own natural data

   This document does not take a stance on the placement of IPsec, which

Mathis, et al                                                  [Page 17]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   logically sits between IP and the Packetization Layer.  IPsec can be
   treated either as part of IP or as part of the Packetization Layer,
   as long as the accounting is consistent within any given
   implementation.  If IPsec is treated as part of the IP layer, then
   each security association to a remote node may need to be treated as
   a separate flow if they have different length security headers.  If
   IPsec is treated as part of the packetization layer, the IPsec header
   size has to be included in the Packetization Layer's header size

5.2. Storing PMTU information

   This memo uses the concept of a "flow" to define the scope in which
   path MTU information is used.  Each flow locally stores its maximum
   payload size (MPS), which is used for packetizing data.  Flows may
   communicate with the IP layer to store or access cached PMTU values,
   providing a means by which similar flows may share information.  To
   do so, the flow must convert between these two values by adding or
   subtracting the size of the IP header plus any additional
   intermediate headers.  The IP layer also stores PMTU information from
   the ICMP layer when it receives Packet Too Big messages.

   Ideally, a PMTU value should be associated with a specific path
   traversed by packets exchanged between the source and destination
   nodes.  However, in most cases a node will not have enough
   information to completely and accurately identify such a path.
   Rather, a node must associate a PMTU value with some local
   representation of a path.  It is left to the implementation to select
   the local representation of a path.

   An implementation could use the destination address as the local
   representation of a path.  The PMTU value associated with a
   destination would be the minimum PMTU learned across the set of all
   paths in use to that destination.  The set of paths in use to a
   particular destination is expected to be small, in many cases
   consisting of a single path.  This approach will result in the use of
   optimally sized packets on a per-destination basis.  This approach
   integrates nicely with the conceptual model of a host as described in
   [ND]: a PMTU value could be stored with the corresponding entry in
   the destination cache.   However, NAT and other forms of middle boxes
   may exhibit differing MTUs at as single IP address.

   If IPv6 flows [IPv6-SPEC] are in use, an implementation could use the
   IPv6 flow id as the local representation of a path.  Packets sent to
   a particular destination but belonging to different flows may use
   different paths, with the choice of path depending on the flow id.
   This approach will result in the use of optimally sized packets on a
   per-flow basis, providing finer granularity than PMTU values

Mathis, et al                                                  [Page 18]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   maintained on a per-destination basis.

   For source routed packets (i.e. packets containing an IPv6 Routing
   header, or IPv4 LSRR or SSRR options), the source route may further
   qualify the local representation of a path.    An implementation
   could use source route information in the local representation of a

   If IPsec is in use, the security association can also be used to
   represent paths.

5.3. Host fragmentation

   Packetization layers are encouraged to avoid sending messages that
   will require fragmentation (for the case against fragmentation, see
   [FRAG]).  However this is not always possible.  Some packetization
   layers, such as a UDP application outside the kernel, may be unable
   to change the size of messages it sends.  This may result in packet
   sizes that exceeds the Path MTU.

   IPv4 permitted such applications to send packets without DF set.
   These packets would be fragmented in the IP layer in the host or
   fragmented by the network.  This approach is no longer recommended.

   We recommend that IPv4 implementation use a new strategy to mimic
   IPv6 functionality.   When the application sends packets that are too
   large for the path they should be fragmented in the host IP layer.
   However, the DF bit should be set on the fragments, so they will not
   be fragmented again in the network.   Note that in principle the IP
   fragmentation layer is an example of a Packetization Layers, it could
   implement full PLPMTUD in the fragmentation process.

   At lease one major operating system already uses this strategy.

5.4. Multicast

   In the case of a multicast destination address, copies of a packet
   may traverse many different paths to reach many different nodes.  The
   local representation of the "path" to a multicast destination must in
   fact represent a potentially large set of paths.

   Minimally, an implementation could maintain a single MPS value to be
   used for all packets originated from the node.  This MPS value would
   be the minimum MPS learned across the set of all paths in use by the
   node.  This approach is likely to result in the use of smaller
   packets than is necessary for many paths.

   Alternatively, if the application using multicast gets complete

Mathis, et al                                                  [Page 19]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   delivery reports (unlikely because this requirement  has poor scaling
   properties), PLPMTUD could be implemented in multicast protocols.

5.5. Path MTU Search Strategy

   The probe strategy described here is a recommended baseline
   algorithm.  It is not presented in formal standards language because
   the probe strategy can include heuristics to help select an optimal
   MSS for a given path.  As a consequence there is opportunity for
   future improvements to this algorithms.

   The probing strategy has three major states: Search, Monitor and
   Suspend.  In the Search state, it sequentially searches for the
   largest MSS that the path can support.  Once the appropriate MPS has
   been discovered, the probing algorithm enters the Monitor state where
   it probes infrequently to detect if the path MPS has become larger.

   If the MPS probing persistently fails it may be desirable to suspend
   MPS probing and heuristically select one of the common default MSSs:
   576, 1240, or 1460 Bytes.

5.5.1. Search

   @@@ this entire section still needs to be rewritten @@@

   The recommended search strategy is a multi-phase scan: First, a
   coarse scan for the approximate MTU using factor of 2 steps starting
   at 1024 Bytes until a probe fails, followed by successively finer
   scans between the largest previously successful and unsuccessful
   probes.  The TCP should use its best knowledge of the lower@@ layer
   header sizes to appropriately determine the MPS from the MTUs listed
   in the table below.

          Table 1: Recommended MTU scanning sequence
          (Coarse scan down column 1, fine scan across each row)
          512, [Use only after repeated timeouts]
          1024,  1492, 1500, 2002
          4096, 4352
          8192, 9000
          16384, 17914
          ((Additional values needed))

   During the scan it is recommended that the MPS not be raised if cwnd

Mathis, et al                                                  [Page 20]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   is too small as determined by a heuristic.  The recommended heuristic
   is that the MPS is only raised when the cwnd is larger than 20
   segments.  @@@ This may be too high.

5.5.2. Monitor

   Once the scan has found an appropriate MPS, the probe strategy enters
   the Monitor state, where it re-probes the most recent failed MTU,
   once every MONITOR_INTERVAL seconds.  If the probe fails, it remains
   in the Monitor state.  If it succeeds, it enters the scanning state.

   If the network becomes too congested during either the Search or the
   Monitor states, it is recommended that the MPS be reduced to a
   smaller size as determined by a heuristic.  The recommended heuristic
   is to reduce the MSS if ssthresh is reduced to 5 segments or smaller.
   The recommended reduction is to the next smaller coarse step in Table

   When there are repeated timeouts (MAX_TIMO or more retransmissions,
   without any received ACKs), it is presumed that the connection was
   re-routed onto a link with a smaller MSS, and that ICMP messages are
   not being delivered.  The MSS probing algorithms is reset by pulling
   back the MSS to 1024 Bytes, rescaling the congestion control
   variables and reentering the Search state.

5.5.3. Suspend

   If there is a timeout, and cwnd prior to the timeout was smaller than
   6 packets, then the probe strategy can enter the Suspend state and
   set the MSS to 512 or 1240 Bytes.  This has the effect of reducing
   the minimum data rate that TCP can stably manage.

5.6. Implementation issues for specific Packetization Layers

   Different protocols introduce specific problems.

5.6.1. Probing method using TCP

   TCP has no mechanism that could be used to distinguish between real
   application data and some other form of padding that might be used to
   fill out probe packets.  Therefor, TCP must generate probes by
   sending oversized segments that are carrying real application date.
   As previously mentioned there are two approaches that TCP might use
   to minimize the overheads associated with the probing process.

Mathis, et al                                                  [Page 21]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   A TCP implementation of PLPMTUD can elect to send subsequent segments
   as though the probe segment was not oversized.  This has the
   advantage that TCP only need to retransmit a segment at the current
   MTU to recover from failed probes.  However the duplicate data in the
   probe does consume network resources and will cause duplicate
   acknowledgments.   It is important that these extra duplicate
   acknowledgments not trigger Fast Retransmit.  This can be guaranteed
   by limiting the largest probe segment size to twice the current
   segment size (causing at most 1 duplicate acknowledgment) three times
   the current segment size (causing at most 2 duplicate

   The other approach is to send non-overlapping segments following the
   probe.  Although this is cleaner from a protocol architecture
   standpoint it clashes with many of the optimizations used improve the
   efficiency of data motion withing many operating systems.  In
   particular many implementations divide the data into segments and
   precompute checksums as the data is copied out of user space.  In
   these implementation it can be very expensive to adjust segment
   boundaries after the data is already queued.

   If TCP is using SACK or any other variable length headers, the
   headers on the probe and verification packets should be padded to the
   maximum possible length.  Otherwise, large headers may cause delivery
   problems for future segments.

   Note that the header size and overhead calculations described in
   section @@@ apply here.  TCP's natural data accounting units are
   sequence space and Maximum Segment Size.  However the the PLPMTUD
   process is described in terms of total packet size, which is larger
   than the MSS by all fixed and optional headers.

   At the point when TCP is ready to start verification, it is permitted
   to not re-packetize already queued data.  This postpones the
   verification process by the time required to send the queued data.

   If the verification phase experiences any segment losses, TCP is
   required to pull back to the prior MSS.   Since failing the
   verification phase should be an infrequent error condition it  is
   less important  that this be  as efficient as probing.

5.6.2. Probing method using SCTP

   In the SCTP protocol packetization is the responsibility of the
   application or protocol above SCTP.   By implication SCTP can not
   easily generate probes sending additional application data.

Mathis, et al                                                  [Page 22]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   For SCTP the recommended method for generating probes is to pad
   messages by @@@@@@ or by sending a message that consists entirely of
   padding and no application data.

   The verification phase ......

5.6.3.  Issues for tunnels

   @@@ to be written

5.6.4.  Issues for other transport protocols

   Some transport protocols (such as ISO TP4 [ISOTP]) are not allowed to
   repacketize when doing a retransmission.  That is, once an attempt is
   made to transmit a segment of a certain size, the transport cannot
   split the contents of the segment into smaller segments for
   retransmission.  In such a case, the original segment can be
   fragmented by the IP layer during retransmission.  Subsequent
   segments, when transmitted for the first time, should be no larger
   than allowed by the Path MTU.

5.7.  Diagnostic tools

   All implementations MUST include a mechanism to implement diagnostic
   tools that do not rely on the operating systems implementation of
   path MTU discovery.   This requires an mechanism where an application
   can send oversized packets that are not subjected to the operating
   systems notion of the current path MTU, up to the physical MTU limit
   as supported by the network interface, as well as a mechanism to
   collect any Packet Too Big Messages.

5.8.  Management interface

   It is suggested that an implementation provide a way for a system
   utility program to:

   - Specify that Path MTU Discovery not be done on a given path.

   - Change the PMTU value associated with a given path.

   - Global controls on ICMP processing

   - Per connection or per application controls on ICMP processing

   The former can be accomplished by associating a flag with the path;
   when a packet is sent on a path with this flag set, the IP layer does
   not send packets larger than the IPv6 minimum link MTU.

Mathis, et al                                                  [Page 23]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   These features might be used to work around an anomalous situation,
   or by a routing protocol implementation that is able to obtain Path
   MTU values.

   The implementation should also provide a way to change the timeout
   period for aging stale PMTU information.

6. Normative references

 [RFC1191]  Path MTU discovery. J.C. Mogul, S.E. Deering. Nov-01-1990.
            (Format: TXT=47936 bytes) (Obsoletes RFC1063) (Status: DRAFT

 [RFC1981]  Path MTU Discovery for IP version 6. J. McCann, S. Deering,
            J. Mogul. August 1996. (Status: PROPOSED STANDARD)

 [RFC2119]  Key words for use in RFCs to Indicate Requirement Levels. S.
            Bradner.  March 1997. (Status: BEST CURRENT PRACTICE)

7. Informative references

 [RFC1063]  IP MTU discovery options. J.C. Mogul, C.A. Kent, C. Par-
            tridge, K. McCloghrie. Jul-01-1988. (Obsoleted by RFC1191)

 [RFC1435]  IESG Advice from Experience with Path MTU Discovery. S.
            Knowles. March 1993. (Format: TXT=2708 bytes) (Status:

 [RFC1626]  Default IP MTU for use over ATM AAL5. R. Atkinson. May 1994.
            (Status: PROPOSED STANDARD)

 [RFC1791]  TCP And UDP Over IPX Networks With Fixed Path MTU. T. Sung.
            April 1995. (Status: EXPERIMENTAL)

 [RFC2923]  TCP Problems with Path MTU Discovery. K. Lahey. September
            2000. (Status: INFORMATIONAL)

8. Security considerations

   Since the MTU reported in the ICMP messages is constrained to be

Mathis, et al                                                  [Page 24]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   between the old MTU and the candidate MTU, this algorithm is more
   difficult to attack through fraudulent ICMP messages.

   Furthermore, since this algorithm can function properly without ICMP
   messages that part of the algorithm can be disabled for additional
   robustness in hostile environments.

9. IANA considerations

10. Contributors

11. Acknowledgements

   Matt Mathis and John Heffner are supported by a grant from Cisco Sys-
   tems, Inc.

12. Authors' addresses

   Please send comments and suggestions to

   Matt Mathis and John Heffner
   Pittsburgh Supercomputing Center
   4400 Fifth Ave.
   Pittsburgh, PA 15213

   Kevin Lahey

13. Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   intellectual property or other rights that might be claimed to  per-
   tain to the implementation or use of the technology described in this
   document or the extent to which any license under such rights might
   or might not be available; neither does it represent that it has made
   any effort to identify any such rights.  Information on the IETF's
   procedures with respect to rights in standards-track and standards-
   related documentation can be found in BCP-11.  Copies of claims of
   rights made available for publication and any assurances of licenses

Mathis, et al                                                  [Page 25]

Internet-Draft Expires Sept 2004                            14 Feb, 2004

   to be made available, or the result of an attempt made to obtain a
   general license or permission for the use of such proprietary rights
   by implementers or users of this specification can be obtained from
   the IETF Secretariat.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights which may cover technology that may be required to practice
   this standard.  Please address the information to the IETF Executive

14. Full copyright statement

   Copyright (C) The Internet Society 14 Feb, 2004. All Rights Reserved.

   This document and translations of it may be copied and furnished to
   others, and derivative works that comment on or otherwise explain it
   or assist in its implementation may be prepared, copied, published
   and distributed, in whole or in part, without restriction of any
   kind, provided that the above copyright notice and this paragraph are
   included on all such copies and derivative works.  However, this doc-
   ument itself may not be modified in any way, such as by removing the
   copyright notice or references to the Internet Society or other
   Internet organizations, except as needed for the  purpose of develop-
   ing Internet standards in which case the procedures for copyrights
   defined in the Internet Standards process must be followed, or as
   required to translate it into languages other than English.

   The limited permissions granted above are perpetual and will not be
   revoked by the Internet Society or its successors or assigns.

Mathis, et al                                                  [Page 26]