Internet Area WG                                               J. Touch
Internet Draft                                                  USC/ISI
Intended status: Informational                              M. Townsley
Expires: September 2010                                           Cisco
                                                          March 5, 2010

                   Tunnels in the Internet Architecture

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   This document may contain material from IETF Documents or IETF
   Contributions published or made publicly available before November
   10, 2008. The person(s) controlling the copyright in some of this
   material may not have granted the IETF Trust the right to allow
   modifications of such material outside the IETF Standards Process.
   Without obtaining an adequate license from the person(s) controlling
   the copyright in such materials, this document may not be modified
   outside the IETF Standards Process, and derivative works of it may
   not be created outside the IETF Standards Process, except to format
   it for publication as an RFC or to translate it into languages other
   than English.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at

   The list of Internet-Draft Shadow Directories can be accessed at

   This Internet-Draft will expire on September 5, 2010.

Touch, Townsley       Expires September 1, 2010                [Page 1]

Internet-Draft         Tunnels in the Internet               March 2010

Copyright Notice

   Copyright (c) 2010 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   ( in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document. Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.


   This document discusses the role of tunnels in the Internet
   architecture. It explains their relationship to existing protocol
   layers, and the challenges in supporting tunneling.

Table of Contents

   1. Introduction...................................................3
   2. Conventions used in this document..............................4
   3. Known Issues...................................................4
      3.1. MTU discovery.............................................5
      3.2. Fragmentation.............................................6
         3.2.1. Outer Fragmentation..................................6
         3.2.2. Inner Fragmentation..................................7
         3.2.3. Fragmentation efficiency.............................8
         3.2.4. Packing (ala GigE bursting).........................10
         3.2.5. IP ID exhaustion....................................11
      3.3. Signaling................................................12
   4. Current Tunnel Standards......................................13
      4.1. IP in IP.................................................13
         4.1.1. MTU discovery.......................................13
         4.1.2. Fragmentation.......................................14
         4.1.3. Signaling...........................................14
      4.2. IPsec....................................................14
         4.2.1. MTU discovery.......................................15
         4.2.2. Fragmentation.......................................15
         4.2.3. Signaling...........................................15
   5. Issues........................................................15
      5.1. Tunnel model.............................................15
      5.2. Parties participating....................................16

Touch, Townsley       Expires September 1, 2010                [Page 2]

Internet-Draft         Tunnels in the Internet               March 2010

   6. Potential Ways Forward........................................17
   7. Notes for future updates......................................18
   8. Security Considerations.......................................19
   9. IANA Considerations...........................................19
   10. References...................................................20
      10.1. Normative References....................................20
      10.2. Informative References..................................20
   11. Acknowledgments..............................................22

1. Introduction

   The Internet is loosely based on the ISO seven layer stack, in which
   data units traverse the stack by being wrapped inside data units one
   layer down (Figure 1). A tunnel is a mechanism for transmitting data
   units between endpoints by wrapping them inside data units other
   layers, e.g., IP in IP, or IP in UDP (Figure 2).

                    +  Eth | IP | TCP |     Data     |

                  Figure 1 TCP inside IP inside Ethernet

              +  Eth | IP'| UDP | IP | TCP |     Data     |

                   Figure 2 IP in UDP in IP in Ethernet

   Tunnels help decouple topology from that provided by the physical
   network components. For example, they were critical in the
   development of multicast, where not all routers were capable of
   processing multicast packets. Multicast routers were interconnected
   by tunnels where not directly connected. Similar techniques have been
   used to support other protocols, such as IPv6.

   Use of tunnels is common in the Internet. The word "tunnel" occurs in
   over 100 RFCs, and is supported within numerous protocols, including:

   o  IPsec - hides the original traffic destination [RFC4301]

   o  L2TP - Tunnels PPP over IP, used largely in DSL/FTTH access
      networks to extend a subscriber's connection from an access line
      provider to an ISP [RFC3931]

   o  Mobile IP - forwards traffic to the home agent [RFC2003]

Touch, Townsley       Expires September 1, 2010                [Page 3]

Internet-Draft         Tunnels in the Internet               March 2010

   o  L2VPNs - provides a link topology different from that provided by
      physical links [RFC4664]

   o  L3VPNs - provides a network topology different from that provided
      by ISPs [RFC4176]

   o  SEAL - a generic mechanism for IP in IP tunneling designed to
      overcome the limitations of RFC2003 [RFC5320]

   o  LISP - reduces routing table load within an enclave of routers

   o  TRILL - enables L3 routing in an enclave of bridges

   o  MPLS - ? {need description/ref}

   o  PWE3 - ? {need description/ref}

   The variety of tunnel mechanisms begs the question of the roles of
   tunnels in the Internet architecture, and the potential need for
   coordination of these mechanisms. In particular, the ways in which
   MTU mismatch, error signals (e.g., ICMP), and is handled may benefit
   from a coordinated approach.

   It is useful to note that, regardless of the layer in which
   encapsulation occurs, tunnels emulate a link. As links, they are
   subject to link issues, e.g., MTU discovery, signaling, and the
   potential utility of native support for broadcast and multicast
   [RFC3819]. They have advantages over native links, being potentially
   easier to reconfigure and control.

2. Conventions used in this document

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   document are to be interpreted as described in RFC-2119 [RFC2119].

3. Known Issues

   Most of the known issues with tunnels arise from the complications of
   encapsulation, or from the introduction of artificial endpoints along
   a data path. Encapsulation exacerbates MTU issues, often because a
   data unit will traverse at least one layer of a protocol stack more
   than once (e.g., as in Figure 2), which requires space for additional
   headers. This space complicates MTU discovery, and often results in

Touch, Townsley       Expires September 1, 2010                [Page 4]

Internet-Draft         Tunnels in the Internet               March 2010

   Tunnel encapsulation and decapsulation nodes act as network
   endpoints. They may source and sink much higher bandwidth streams
   from single IP addresses, and thus can be affected by many of the
   issues of other high bandwidth edge devices, such as fragmentation
   efficiency and IP ID exhaustion (in IPv4). These endpoints also
   introduce complexity in end-to-end and path signaling, in the
   translation between signals inside a tunnel and signals outside on
   the end-to-end path.

3.1. MTU discovery

   MTU discovery is a known challenge in the current Internet, and
   tunnels can complicate its proper operation. Encapsulation increases
   the size of a packet during tunnel transit that can exceed the MTU of
   the links of the tunnel path. This is especially true for recursive
   tunnels, i.e., tunnels that reuse layers of the protocol stack (e.g.,
   IPv4 over IPv4). These issues are discussed in detail in [RFC4459];
   the following provides a brief overview of the issues. Note that the
   impact of tunnels on MTU discovery may be mitigated somewhat by the
   ubiquity of workarounds already needed in the Internet, e.g., the
   deduction of a 'tunnel tax' for all MTUs (i.e., maxing out the MTU at
   1200-1400 bytes, rather than 1500).

   Conventional path MTU discovery (PMTUD) relies on explicit negative
   feedback from routers along the path (ICMP "message to big" signals)
   [RFC1191]. This technique is susceptible to the "black hole"
   phenomenon, in which the ICMP messages never return to the source
   [RFC2923]. In the typical Internet case, lost ICMPs are often the
   result of filtering, e.g., for policy reasons.

   A more recent alternative is packetization-layer path MTU discovery
   (PLPMTUD) [RFC4821]. This variant relies on feedback from the
   endpoint, indicating either the success or failure of probe packets.
   It is not susceptible to "black holing", but requires explicit
   participation by the receiver.

   Either of these techniques (PMTUD, PLPMTUD) can be applied to
   tunnels. The encapsulator must react to "message to big" signals in
   either case, by either adjusting its fragmentation, relaying a
   corresponding signal to the packet origin outside the tunnel, or
   both. Fragmentation adjustment is easy to incorporate, but can result
   in inefficient transmission of packets over the tunnel (e.g., where
   every source packet is fragmented). Relaying the signal to the source
   can be much more efficient, but it can be difficult to determine what
   signal to forward. E.g., in PMTUD, routers along the tunnel may not
   return a sufficiently long prefix to determine the decapsulated
   packet origin.

Touch, Townsley       Expires September 1, 2010                [Page 5]

Internet-Draft         Tunnels in the Internet               March 2010

   Tunnels thus may need to participate in MTU discovery, either
   forwarding or recomputing ICMPs received inside the tunnel path. The
   tunnel may incorporate its own MTU discovery between ingress and
   egress, e.g., as proposed in SEAL [RFC5320].

3.2. Fragmentation

   There are two places where fragmentation can occur in a tunnel,
   called Outer Fragmentation and Inner Fragmentation.

3.2.1. Outer Fragmentation

   The simplest case is Outer Fragmentation, as shown in Figure 3. The
   bottom of the figure shows the network toplogy, where packets start
   at the source, enter the tunnel at the encapsulator, exit the tunnel
   at the decapsulator, and arrive finally at the destination. The
   packet traffic is shown above the topology, where the end-to-end
   packets are shown at the top. The packets are composed of an inner
   header (iH) and inner data (iD); the term "inner") is relative to the
   tunnel, as will become apparent. When the packet (iH,iD) arrives at
   the encapsulator, it is placed inside the tunnel packet structure,
   here shown as adding just an outer header, oH, in step (a).

   When the encapsulated packet exceeds the MTU of the tunnel, the
   packet needs to be fragmented. In this case we fragment the packet at
   the outer header, with the fragments shown as (b1) and (b2). Note
   that the outer header indicates fragmentation (as ' and "),the inner
   header occurs only in the first fragment, and the inner data is
   broken across the two packets. These fragments are reassembled at the
   encapsulator in step (c), and the resulting packet is decapsulated
   and sent on to the destination.

Touch, Townsley       Expires September 1, 2010                [Page 6]

Internet-Draft         Tunnels in the Internet               March 2010

    +----+----+                                              +----+----+
    | iH | iD |------+ -  -  -  -  -  -  -  -  -  -  +------>| iH | iD |
    +----+----+      |                               |       +----+----+
                     v                               |
              +----+----+----+               +----+----+----+
          (a) | oH | iH | iD |               | oH | iH | iD | (c)
              +----+----+----+               +----+----+----+
                     |                               ^
                     |       +----+----+-----+       |
                (b1) +----- >| oH'| iH | iD1 |-------+
                     |       +----+----+-----+       |
                     |                               |
                     |       +----+-----+            |
                (b2) +----- >| oH"| iD2 |------------+

   +-----+         +---+                           +---+         +-----+
   |     |        /     \ ======================= /     \        |     |
   | Src |=======|  Enc  |=======================|  Dec  |=======| Dst |
   |     |        \     / ======================= \     /        |     |
   +-----+         +---+                           +---+         +-----+

                Figure 3 Fragmentation of the outer packet

   Outer fragmentation isolates Source and Destination from tunnel
   encapsulation duties. This can be considered a benefit in clean,
   layered network design, but also may result in complex decapsulator
   design, especially where tunnels aggregate large amounts of traffic,
   such as IP ID overload (see Sec. 3.2.5). Outer fragmentation is valid
   for any tunnel encapsulation protocol that supports fragmentation
   (e.g., IPv4 or IPv6), where the tunnel endpoints act as the host
   endpoints of that protocol.

   Along the tunnel, the inner header is contained only in the first
   fragment, which can interfere with mechanisms that 'peek' into lower
   layer headers, e.g., as for ICMP, as discussed in Sec. 3.3.

3.2.2. Inner Fragmentation

   Inner Fragmentation distributes the impact of tunneling across both
   the decapsulator and destination, and is shown in Figure 4. Again,
   the network topology is shown at the bottom of the figure, and the
   original packets show at the top. Packets arrive at the encapsulator,
   and are fragmented there based on the inner header into (a1) and
   (a2). The fragments arrive at the decapsulator, which removes the
   outer header and forwards the resulting fragments on to the

Touch, Townsley       Expires September 1, 2010                [Page 7]

Internet-Draft         Tunnels in the Internet               March 2010

   destination. The destination is then responsible for reassembling the
   fragments into the original packet.

   +----+----+                                               +----+----+
   | iH | iD |-------+-  -  -  -  -  -  -  -  -  -  -  -  - >| iH | iD |
   +----+----+       |                                       +----+----+
                     v                                            ^
                +----+-----+                    +----+-----+      |
           (a1) | iH'| iD1 |                    | iH'| iD1 |------+
                +----+-----+                    +----+-----+      |
                +----+---                       +----+-----+      |
           (a2) | iH"| iD2 |                    | iH"| iD2 |------+
                +----+-----+                    +----+-----+
                     |                               ^
                     |       +----+----+-----        |
                (b1) +----- >| oH | iH'| iD1 |-------+
                     |       +----+----+-----+       |
                     |                               |
                     |       +----+----+-----+       |
                (b2) +----- >| oH | iH"| iD2 |-------+

   +-----+         +---+                           +---+         +-----+
   |     |        /     \ ======================= /     \        |     |
   | Src |=======|  Enc  |=======================|  Dec  |=======| Dst |
   |     |        \     / ======================= \     /        |     |
   +-----+         +---+                           +---+         +-----+

                Figure 4 Fragmentation of the inner packet

   As noted, inner fragmentation distributes the effort of tunneling
   across the decapsulator and destinations; this can be especially
   important when the tunnel aggregates large amounts of traffic. Note
   that this mechanism is thus valid only when the original source
   packets can be fragmented on-path, e.g., as in IPv4.

   Along the tunnel, the inner headers are copied into each fragment,
   and so are available to mechanisms that 'peek' into headers (e.g.,
   ICMP, as discussed in Sec. 3.3). Because fragmentation happens on the
   inner header, the impact of IP ID is reduced.

3.2.3. Fragmentation efficiency

   There are different ways to fragment a packet. Consider a network
   with an MTU as shown in Figure 5, where packets are encapsulated over
   the same network layer as they arrive on (e.g., IP in IP). If a

Touch, Townsley       Expires September 1, 2010                [Page 8]

Internet-Draft         Tunnels in the Internet               March 2010

   packet as large as the MTU arrives, it must be fragmented to
   accommodate the additional header.

                 X===========================X (MTU)
                 | iH | DDDDDDDDDDDDDDDDDDDD |
                   |  X===========================X (MTU)
                   |  +---+----+------------------+
               (a) +->| H'| iH | DDDDDDDDDDDDDDDD |
                   |  +---+----+------------------+
                   |      |
                   |      |  X===========================X (MTU)
                   |      |  +----+---+----+-------------+
                   | (a1) +->| nH'| H | iH | DDDDDDDDDDD |
                   |      |  +----+---+----+-------------+
                   |      |
                   |      |  +----+-------+
                   | (a2) +->| nH"| DDDDD |
                   |         +----+-------+
                   |  +---+------+
               (b) +->| H"| DDDD |
                          |  +----+---+------+
                     (b1) +->| nH'| H"| DDDD |

                   Figure 5 Fragmenting via maximum fit

   Figure 5 shows this process, using Outer Fragmentation as an example
   (the situation is the same for Inner Fragmentation, but the headers
   that are affected differ). The arriving packet is first split into
   (a) and (b), where (a) is of the MTU of the network. However, this
   tunnel then traverses over another tunnel, whose impact the first
   tunnel ingress has not accommodated. The packet (a) arrives at the
   second tunnel ingress, and needs to be encapsulated again, but
   because it is already at the MTU, it needs to be fragmented as well,
   into (a1) and (a2). In this case, packet (b) arrives at the second
   tunnel ingress and is encapsulated into (b1) without fragmentation,
   because it is already below the MTU size.

   In Figure 6, the fragmentation is done evenly, i.e., by splitting the
   original packet into two roughly equal-sized components, (c) and (d).
   Note that (d) contains more packet data, because (c) includes the

Touch, Townsley       Expires September 1, 2010                [Page 9]

Internet-Draft         Tunnels in the Internet               March 2010

   original packet header because this is an example of Outer
   Fragmentation. The packets (c) and (d) arrive at the second tunnel
   encapsulator, and are encapsulated again; this time, neither packet
   exceeds the MTU, and neither requires further fragmentation.

                 X===========================X (MTU)
                 | iH | DDDDDDDDDDDDDDDDDDDD |
                   |  X===========================X (MTU)
                   |  +---+----+----------+
               (c) +->| H'| iH | DDDDDDDD |
                   |  +---+----+----------+
                   |      |
                   |      |  X===========================X (MTU)
                   |      |  +----+---+----+----------+
                   | (c1) +->| nH | H'| iH | DDDDDDDD |
                   |         +----+---+----+----------+
                   |  +---+--------------+
               (d) +->| H"| DDDDDDDDDDDD |
                          |  +----+---+--------------+
                     (d1) +->| nH | H"| DDDDDDDDDDDD |

                        Figure 6 Fragmenting evenly

3.2.4. Packing (ala GigE bursting)

   Encapsulating individual packets to traverse a tunnel can be
   inefficient, especially where headers are large relative to the
   packets being carried. In that case, it can be more efficient to
   encapsulate many small packets in a single, larger tunnel payload.
   This technique, similar to the effect of packet bursting in Gigabit
   Ethernet, reduces the overhead of the encapsulation headers (Figure
   7). It reduces the work of header addition and removal at the tunnel
   endpoints, but increases other work involving the packing and
   unpacking of the component packets carried.

Touch, Townsley       Expires September 1, 2010               [Page 10]

Internet-Draft         Tunnels in the Internet               March 2010

                     | iHa | iDa |
                           |     +-----+-----+
                           |     | iHb | iDb |
                           |     +-----+-----+
                           |           |
                           |           |     +-----+-----+
                           |           |     | iHc | iDc |
                           |           |     +-----+-----+
                           |           |           |
                           v           v           v
                | oH | iHa | iHa | iHb | iDb | iHc | iDc |

                  Figure 7 Packing packets into a tunnel

3.2.5. IP ID exhaustion

   In IPv4, the IP Identification (ID) field is a 16-bit value that is
   unique for every packet for a given source address, destination
   address, and protocol, such that it does not repeat within the
   Maximum Segment Lifetime (MSL) [RFC791][RFC1122]. Although the ID
   field was originally intended for fragmentation and reassembly, it
   can also be used to detect and discard duplicate packets, e.g., at
   congested routers (see Sec. of [RFC1122]). For this reason,
   and even more so that IPv4 packets can be fragmented anywhere along a
   path, all packets between a source and destination of a given
   protocol must have unique ID values over a period of an MSL, which is
   typically interpreted as two minutes (120 seconds).

   The uniqueness of the IP ID is a known problem for high speed
   devices, because it limits the speed of a single protocol between two
   endpoints [RFC4963]. With the maximum IP packet size of 64KB, a 16-
   bit ID field that does not repeat within 120 seconds means that the
   sum of all TCP connections between two endpoints is limited to
   roughly 286 Mbps; for more typical MTUs of 1500 bytes, this drops to
   6.4 Mbps.

   Although this strongly suggests that the uniqueness of the IP ID is
   moot, tunnels exacerbate this condition. A tunnel often aggregates
   traffic from a number of different source and destination addresses,
   of different protocols, and encapsulates them in a header with the
   same ingress and egress addresses, all using a single encapsulation
   protocol. The result is one of the following:

Touch, Townsley       Expires September 1, 2010               [Page 11]

Internet-Draft         Tunnels in the Internet               March 2010

   1. The IP ID rules are enforced, and the tunnel throughput is
      severely limited.

   2. The IP ID rules are enforced, and the tunnel consumes large
      numbers of ingress/egress IP addresses solely to ensure ID

   3. The IP ID rules are ignored.

   The last case is the most obvious solution, because it corresponds to
   how endpoints currently behave. Fortunately, fragmentation is
   somewhat rare in the current Internet at large, but it can be common
   along a tunnel. Fragments that repeat the IP ID risk being
   reassembled incorrectly, especially when fragments are reordered or
   lost. Although such errors may be detected at the transport layer,
   this results in excessive overall packet loss, as well as wasting
   bandwidth between the egress and ultimate packet destination.

3.3. Signaling

   In the current Internet architecture, signals tend to go upstream,
   either from routers along a path or from the destination, back toward
   the source (Figure 8). Such signals are typically contained in ICMP
   messages, but can involve other protocols such as RSVP, transport
   protocol signals (e.g., TCP RSTs), or multicast.

     |                                                              |
     | +---------------------------+                                |
     | |                           |                                |
     v v                           |                                |
   +-----+                         |                             +-----+
   |     |                         |                             |     |
   | Src |=========================R=============================| Dst |
   |     |                                                       |     |
   +-----+                                                       +-----+

                  Figure 8 Signaling paths in an Internet

   Tunnels interfere with these known signaling paths. As shown in
   Figure 9, signals from routers along the tunnel path (R2), as well as
   those from the tunnel egress, need to be relayed by the ingress. This
   relaying may be difficult, because R2 may not return enough
   information to the ingress to support relaying (e.g., when ICMP
   returns only the outermost headers in a "message to big", and the
   source transport port information is lost). Signals from routers

Touch, Townsley       Expires September 1, 2010               [Page 12]

Internet-Draft         Tunnels in the Internet               March 2010

   downstream of the egress (R3 in Figure 9) need to traverse the tunnel
   in reverse.

   In all cases, the tunnel ingress needs to determine how to relay the
   signals from inside the tunnel into signals back to the source. For
   some protocols this is either simple or impossible (such as for
   ICMP), for others, it can even be undefined (e.g., multicast).

      +  -  -  -  -  +-------------------------------+
      |              |                               |
      v              v                               |
   +-----+         +---+                           +---+         +-----+
   |     |        /     \ ======================= /     \        |     |
   | Src |==R1===|  Enc  |==========R2===========|  Dec  |===R3==| Dst |
   |     |        \     / ======================= \     /        |     |
   +-----+         +---+             |             +---+         +-----+
      ^              ^               |
      |              |               |
      +  -  -  -  -  +---------------+

              Figure 9 Signaling paths introduced by a tunnel

4. Current Tunnel Standards

   This section reviews two common Internet tunnel standards. They are
   notable because they both ultimately rely on IP in IP encapsulation,
   although they each handle MTU discovery, fragmentation, and signaling

   [There are other tunnel mechanisms, such as IPv4 in IPv6, which may
   be added to this discussion later.]

4.1. IP in IP

   The simplest tunnel encapsulation mechanism is IP in IP, explained
   here for IPv4 [RFC2003]. This protocol was standardized for use in
   mobile IP, so that packets sent from a source to a Home Agent could
   be forwarded unmodified to the different address of the Mobile Node
   [RFC3344]. It has come to be used much more generally, e.g., to
   support multicast, as well as in overlay network systems

4.1.1. MTU discovery

   When an IPv4 packet arrives at an IP-in-IP ingress, the DF flag from
   the inner packet is copied to the outer header. This enforces DF of
   the packet within the tunnel when requested by the packet source.

Touch, Townsley       Expires September 1, 2010               [Page 13]

Internet-Draft         Tunnels in the Internet               March 2010

   Packets which are too large are dropped at the ingress, and a
   corresponding ICMP "message to big" is returned to the source.
   Internally, IP-in-IP tunneling requires that the tunnel MUST support
   ICMP-based path MTU discovery (i.e., PMTUD). Note that due to common
   filtering of ICMP messages, this requirement is impossible to
   determine and thus to enforce.

4.1.2. Fragmentation

   IP-in-IP tunneling supports Inner Fragmentation. The inner packet MAY
   be fragmented if DF=0, otherwise the packet would have been dropped
   if too big, as noted earlier. The tunnel MUST NOT fragment at the
   outer header if DF=1 is set, i.e., this tunnel protocol assumes the
   network honors the DF bit (note that some tunnels, as well as some
   network devices, do not honor the DF bit). Further, if the DF bit is
   set in the inner header, it MUST be set in the outer; if not, it MAY
   be set in the outer.

4.1.3. Signaling

   IP-in-IP tunnels MAY relay ICMPs from inside the tunnel to the
   source, i.e., at the ingress. They SHOULD relay network and host
   unreachable messages, and MUST relay "message too big" messages;
   these reflect network conditions that the source should be informed
   about. They MUST NOT relay port unreachable messages, because these
   are meaningless for encapsulated packets, and thus reflect internal
   link conditions that the source should not care about at all. They
   MUST NOT relay and SHOULD handle locally messages that affect the
   ingress as if it were a host, e.g., source quench and router errors.

   Most notably, IP-in-IP notes that the tunnel SHOULD keep sufficient
   soft state to assist with relaying. Such state may involve keeping
   copies of recently sent packets, to have sufficient context to relay
   when lacking in the received ICMP message.

4.2. IPsec

   The Internet network security standard, IPsec, incorporates IP-in-IP
   encapsulation as part of its tunnel mode of operation [RFC4301].
   Although IP-in-IP packets can be secured via IPsec transport mode,
   resulting in identical packets [RFC3884], the rules affecting IPsec
   tunnel mode MTU discovery, fragmentation, and signaling mode are
   specified by IPsec, rather than IP-in-IP.

Touch, Townsley       Expires September 1, 2010               [Page 14]

Internet-Draft         Tunnels in the Internet               March 2010

4.2.1. MTU discovery

   Tunnel mode IPsec MTU discovery supports ICMP-based path MTU
   discovery (PMTUD), but only as a SHOULD. If an IPv4 packet arrives
   with DF=1, or an IPv6 packet arrives, and either is too large for the
   tunnel, the ingress SHOULD discard and send an ICMP to the source. If
   IPv4 and DF=0, the ingress SHOULD perform Outer Fragmentation, and
   SHOULD NOT send an ICMP to the source.

4.2.2. Fragmentation

   IPsec performs only Outer Fragmentation; this distinguishes it from
   IP-in-IP, which performs only Inner Fragmentation.

   It requires that implementations of tunnel mode allow the security
   policy to decide how the IPv4 DF bit should propagate from the inner
   to the outer header. It may be copied, cleared, or set, again,
   differing from IP-in-IP which allows only copy or set.

4.2.3. Signaling

   IPsec, like IP-in-IP, relays ICMP "message to big" signals from the
   ingress back to the source. The size indicated is adjusted to take
   into account for the space for both encapsulation and security
   information. Further, it allows that any ICMP message may be blocked,
   on a per-security association basis; this filtering is for security
   reasons, but also can directly result in "black holing".

5. Issues

   As has been shown in only two examples, even similar mechanisms for
   encapsulation can result in very different approaches to tunneling.
   Although these approaches result in different MTU discovery,
   fragmentation, and signaling mechanisms, they result from different
   architectural perspectives on the role of tunnels in the Internet.
   This section discusses these more fundamental perspectives, and their
   impact on the mechanisms.

5.1. Tunnel model

   The Internet architecture is composed of hosts, gateways (i.e.,
   routers), and links [Cl88]. A host is a source or sink of network
   packet traffic, a router redirects packets from one set of links to
   another, and links interconnect hosts and routers. Although
   originally described for the Internet's network layer, this
   architecture, with a bit of renaming (e.g., routers become bridges),
   applies equally well for link layers.

Touch, Townsley       Expires September 1, 2010               [Page 15]

Internet-Draft         Tunnels in the Internet               March 2010

   Tunnels could, in principle, be related to this basic model in one of
   three ways:

   o  Tunnel as a link

   o  Tunnel as a router/bridge

   o  Tunnel as invisible

   Tunnels require distinct ingress and egress addresses, to use during
   encapsulation, and to direct encapsulated traffic from the ingress to
   the egress. As a result, a tunnel is most usefully considered a link
   in the architecture in which they are deployed. As a result, tunnel
   designers should consider and apply link design issues [RFC3819].
   This also implies that operating systems designers should represent
   tunnels as links; this may be conveniently represented as virtual

   [this includes tunnel as point-point vs. tunnel as multipoint]

5.2. Parties participating

   The description of a tunnel focuses on the functions of the ingress
   and egress, but not all functions need be located at one of these two
   points. Recall inner fragmentation, in which fragment reassembly
   occurs at the destination, not the egress - this imposes load on the
   destination as a result of behavior of the ingress.

   Containing all tunnel functions solely inside the tunnel endpoints,
   as with outer fragmentation, is architecturally clean. It also obeys
   the 'clean up your own mess' principle; the impact of encapsulation
   and fragmentation caused by the ingress is then handled by the
   egress, without imposing load on the destination.

   Distributing tunnel functions across both egress and destination, as
   with inner fragmentation, can be more efficient. The impact of the
   limited IPv4 IP ID space is more prominent in the outer header, due
   to aggregation of traffic at the ingress. Using the inner header for
   fragmentation allows use of a larger effective IP ID space because of
   the additional IP source/destination addresses present there.
   Reassembly can be distributed among a large number of destinations
   (where present), and the impact of reassembly can be isolated to only
   affected destinations. Further, fragmenting once at the ingress can
   avoid repeated fragmentation/reassembly steps when packets traverse
   multiple tunnels in succession.

Touch, Townsley       Expires September 1, 2010               [Page 16]

Internet-Draft         Tunnels in the Internet               March 2010

   The primary case in favor of distributed tunnel functions, and thus
   inner encapsulation is that high speed ingress devices can be
   implemented, but that corresponding high speed egresses are difficult
   or costly. Unfortunately, network operators cannot always know in
   advance that high-speed ingresses are being deployed where the
   destination traffic is sufficiently diffuse; deploying such a device
   where the traffic focuses on a single destination puts an undue
   burden on that destination.

6. Potential Ways Forward

   There are a number of issues which may benefit from a coordinated
   review. These include unification of various tunneling standards, and
   revision of tunnel standards to address:

   o  Relation of inner/outer headers (i.e., which fields are copied,
      derived, etc.)

   o  MTU discovery

   o  Fragmentation

   o  Signaling

   This revision may suggest the utility of a single, configurable
   tunnel mechanism that includes various solutions as alternatives,
   rather than developing custom tunnel solutions on-demand. It may also
   suggest the development of new solutions, such as:

   o  The use of PLPMTUD for tunnels

   o  Addressing the IP ID issue and fragmentation

   o  New ICMP signals

   o  Optimization solutions, such as packing

   SEAL addresses a few of these issues, notably the first two
   [RFC5320]. It adds an active signal exchange between ingress and
   egress for intra-tunnel MTU discovery, and an extension to the IP ID
   space to detect collisions.

   Tunnels are further evidence that the current requirements for IPv4
   ID uniqueness may need revision. In particular, it is clear that even
   moderate speed transport connections already violate these
   requirements. We recommend revisiting the requirements as suggested
   in [To10].

Touch, Townsley       Expires September 1, 2010               [Page 17]

Internet-Draft         Tunnels in the Internet               March 2010

   Note that this document does not argue for a single, generic
   tunneling protocol or mechanism. Such a mechanism is no more likely
   to be useful than would a 'one size fits all' transport protocol. It
   does argue, however, for consistency in tunnel design, and
   abstraction and reuse of mechanism where possible.

7. Notes for future updates

   [This area includes notes for future updates which have been reported
   but not yet fully included - it represents a holding area for
   comments, and should not appear in the final document.]

   tunnel as virtualization - Stewart Bryant (SB)

   tunnel as endpoint only, not on-path (not MPLS, e.g.) - JT/coauthor

   gigE packing like PWE3 ATM packing - SB

   PPP chopping and coalescing - MT/coauthor

   end sec 2 "we need large seq num and to frag at the tunnel" / maybe,
   but do we want recommendations? - SB

   security should add addr management and ACLs (?) - SB

   MTU as part of BGP? - SB (Will this even work - JT)

   section 2 it says: "The IPv6 fragment header is present only when a
   packet has been fragmented", but I know of at least one effort in
   MANET that is proposing to include the fragment header even for
   unfragmented IPv6 packets. That would seem to bend the rules set
   forth in RFC2460, but I just thought it might be worth pointing out
   that some people are considering bending them. - Fred Templin

   NATs - i.e., One other thought; where the IP ID problem becomes truly
   pathological is for tunnels that traverse IPv4 NATs. First, the NATs
   could rewrite the ID to something the ingress tunnel endpoint never
   intended. Secondly, multiple ingress tunnel endpoints that traverse
   the same NAT could have IP ID "collisions" from the perspective of
   the outside world.  This may deserve a section unto itself? - FT

   NAT as half-tunnel - JT

   tunnel endpoint as following host rules - JT (as with ECN in CAPWAP,
   per Magnus' email of 10/10/08)

   the need for larger min MTU - FT (see SEAL)

Touch, Townsley       Expires September 1, 2010               [Page 18]

Internet-Draft         Tunnels in the Internet               March 2010

   describe relationship to [Ho08] - JT (as per INTAREA meeting notes,
   don't cover Teredo-specific issues in Ho08, but include generic
   issues here)

8. Security Considerations

   Tunnels may introduce vulnerabilities, or add to the potential for
   receiver overload and thus DOS attacks. These issues are primarily
   related to the fact that a tunnel is a link that traverses a network
   path, and to fragmentation and reassembly. Regarding ICMP signals,
   tunnels have similar security issues to routers, in that they SHOULD
   throttle ICMPs sent to a given source, and SHOULD send ICMPs that
   correspond to events inside the tunnel. Such ICMPs MUST have the
   tunnel ingress IP address as the source IP, because IP addresses
   inside a tunnel path may have no meaning outside the tunnel.

   Tunnels traverse multiple hops of a network path from ingress to
   egress. Traffic along such tunnels may be susceptible to on-path and
   off-path attacks, including fragment injection, reassembly buffer
   overload, and ICMP attacks. Some of these attacks may not be as
   visible to the endpoints of the architecture into which tunnels are
   deployed, and may result in these attacks being more difficult to

   Inner fragmentation can present an undue burden on destinations where
   traffic is not sufficiently diffuse; tunnels SHOULD NOT employ inner
   fragmentation except where such diffusion is confirmed either by the
   tunnel mechanism or network designer. All tunnel fragmentation -
   inner and outer - MUST obey all existing fragmentation requirements,
   i.e., IPv6 tunnels MUST NOT employ inner fragmentation, and IPv4
   tunnels MUST NOT use inner fragmentation where the inner header DF=1.

   Tunnels MUST obey all existing IP requirements, such as the
   uniqueness of the IP ID field, until otherwise exceptioned or
   revoked. Failure to either limit encapsulation traffic, or use
   additional ingress/egress IP addresses, can result in high speed
   traffic fragments being incorrectly reassembled.

9. IANA Considerations

   This document has no IANA considerations.

   The RFC Editor should remove this section prior to publication.

Touch, Townsley       Expires September 1, 2010               [Page 19]

Internet-Draft         Tunnels in the Internet               March 2010

10. References

10.1. Normative References

   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
             Requirement Levels", BCP 14, RFC 2119, March 1997.

10.2. Informative References

   [Cl88]    Clark, D., "The design philosophy of the DARPA internet
             protocols," Proc. Sigcomm 1988, p.106-114, 1988.

   [Er94]    Eriksson, H., "MBone: The Multicast Backbone,"
             Communications of the ACM, Aug. 1994, pp.54-60.

   [Fa10]    Farinacci, D., V. Fuller, D. Meyer, D. Lewis, "Locator/ID
             Separation Protocol (LISP)," (work in progress), draft-
             ietf-lisp-06, Jan. 2010.

   [Ho08]    Hoagland, J., S. Krishnan, D. Thaler, "Security Concerns
             With IP Tunneling," (work in progress), draft-ietf-v6ops-
             tunnel-security-concerns-01, Oct. 2008.

   [Pe10]    Perlman, R., D. Eastlake, D. Dutt, S. Gai, A. Ghanwani,
             "RBridges: Base Protocol Specification," (work in
             progress), trill draft-ietf-trill-rbridge-protocol-15, Jan.

   [RFC791]  Postel, J., "Internet Protocol," RFC 791 / STD 5, September

   [RFC1122] Braden, R., Ed., "Requirements for Internet Hosts -
             Communication Layers," RFC 1122 / STD 3, October 1989.

   [RFC1191] Mogul, J., S. Deering, "Path MTU discovery," RFC 1191,
             November 1990.

   [RFC2003] Perkins, C., "IP Encapsulation within IP," RFC 2003,
             October 1996.

   [RFC2923] Lahey, K., "TCP Problems with Path MTU Discovery," RFC
             2923, September 2000.

   [RFC3344] Perkins, C., Ed., "IP Mobility Support for IPv4," RFC 3344,
             August 2002.

Touch, Townsley       Expires September 1, 2010               [Page 20]

Internet-Draft         Tunnels in the Internet               March 2010

   [RFC3819] Karn, P., Ed., C. Bormann, G. Fairhurst, D. Grossman, R.
             Ludwig, J. Mahdavi, G. Montenegro, J. Touch, L. Wood,
             "Advice for Internet Subnetwork Designers," RFC 3819 / BCP
             89, July 2004.

   [RFC3884] Touch, J., L. Eggert, Y. Wang, "Use of IPsec Transport Mode
             for Dynamic Routing," RFC 3884, September 2004.

   [RFC3931] Lau, J., Ed., M. Townsley, Ed., I. Goyret, Ed., "Layer Two
             Tunneling Protocol - Version 3 (L2TPv3)," RFC 3931, March

   [RFC4176] El Mghazli, Y., Ed., T. Nadeau, M. Boucadair, K. Chan, A.
             Gonguet, "Framework for Layer 3 Virtual Private Networks
             (L3VPN) Operations and Management," RFC 4176, October 2005.

   [RFC4301] Kent, S., and K. Seo, "Security Architecture for the
             Internet Protocol," RFC 4301, December 2005.

   [RFC4459] Savola, P., "MTU and Fragmentation Issues with In-the-
             Network Tunneling," RFC 4459, April 2006.

   [RFC4664] Andersson, L., Ed., E. Rosen, Ed., "Framework for Layer 2
             Virtual Private Networks (L2VPNs)," RFC 4664, September

   [RFC4821] Mathis, M., J. Heffner, "Packetization Layer Path MTU
             Discovery," RFC 4821, March 2007.

   [RFC4963] Heffner, J., M. Mathis, B. Chandler, "IPv4 Reassembly
             Errors at High Data Rates," RFC 4963, July 2007.

   [RFC5320] Templin, F., Ed., "The Subnetwork Encapsulation and
             Adaptation Layer (SEAL)," RFC 5320, Feb. 2010.

   [RFC5556] Touch, J., R. Perlman, "Transparently Interconnecting Lots
             of Links (TRILL): Problem and Applicability Statement," RFC
             5556, May 2009.

   [To01]    Touch, J., "Dynamic Internet Overlay Deployment and
             Management Using the X-Bone," Computer Networks, July 2001,
             pp. 117-135.

   [To10]    Touch, J., "Updated Specification of the IPv4 ID Field,"
             (work in progress), draft-touch-intarea-ipv4-id-update,
             Feb. 2010.

Touch, Townsley       Expires September 1, 2010               [Page 21]

Internet-Draft         Tunnels in the Internet               March 2010

11. Acknowledgments

   This document originated as the result of numerous discussions among
   the authors, Jari Arkko, Stuart Bryant, Lars Eggert, Dino Farinacci,
   Matt Mathis, and Fred Templin, as well as members participating in
   the Internet Area Working Group.

   This document was prepared using

Authors' Addresses

   Joe Touch
   4676 Admiralty Way
   Marina del Rey, CA 90292-6695

   Phone: +1 (310) 448-9151

   W. Mark Townsley
   L'Atlantis, 11, Rue Camille Desmoulins
   Issy Les Moulineaux, ILE DE FRANCE 92782


Touch, Townsley       Expires September 1, 2010               [Page 22]