NVO3 Workgroup                                           S. Boutros, Ed.
Internet-Draft                                                     Ciena
Intended Status: Informational                          D. Eastlake, Ed.
Expires: January 28, 2022                                  July 29, 2021

                   NVO3 Encapsulation Considerations

   As communicated by the WG Chairs, the IETF NVO3 chairs and Routing
   Area director have chartered a design team to take forward the
   encapsulation discussion and see if there is potential to design a
   common encapsulation that addresses the various technical concerns.

   There are implications of different encapsulations in real
   environments consisting of both software and hardware implementations
   and spanning multiple data centers.  For example, OAM functions such
   as path MTU discovery become challenging with multiple encapsulations
   along the data path.

   The design team recommends Geneve with a few modifications as the
   common encapsulation. This document provides more details,
   particularly in Section 7.

Status of This Document

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Distribution of this document is unlimited. Comments should be sent
   to the authors or the IDR Working Group mailing list <nvo3@ietf.org>.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   https://www.ietf.org/1id-abstracts.html. The list of Internet-Draft
   Shadow Directories can be accessed at

S. Boutros & D. Eastlake  Expires January 2022                  [Page 1]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

Copyright Notice

   Copyright (c) 2021 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document. Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

S. Boutros & D. Eastlake  Expires January 2022                  [Page 2]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

Table of Contents

      1. Introduction............................................4
      2. Design Team Goals.......................................4
      3. Terminology.............................................5
      4. Abbreviations and Acronyms..............................5

      5. Issues with Current Encapsulations......................6
      5.1. Geneve................................................6
      5.2. GUE (Generic UDP Encapsulation).......................6
      5.3. VXLAN-GPE.............................................6

      6. Common Encapsulation Considerations.....................7
      6.1. Current Encapsulations................................7
      6.2. Useful Extensions Use Cases...........................7
      6.2.1. Telemetry Extensions................................7
      6.2.2. Security/Integrity Extensions.......................8
      6.2.3 Group Based Policy...................................8
      6.3. Hardware Considerations...............................9
      6.4. Extension Size........................................9
      6.5. Ordering of Extension Headers........................10
      6.6. TLV versus Bit Fields................................10
      6.7. Control Plane Considerations.........................11
      6.8. Split NVE............................................12
      6.9. Larger VNI Considerations............................12

      7. Design Team Recommendations............................14

      8. Acknowledgements.......................................17
      9. Security Considerations................................17
      10. IANA Considerations...................................17

      11. References............................................18
      11.1 Normative References.................................18
      11.2 Informative References...............................18

      Appendix A: Encapsulations Comparison.....................20
      A.1. Overview.............................................20
      A.2. Extensibility........................................20
      A.2.1. Native Extensibility Support.......................20
      A.2.2. Extension Parsing..................................20
      A.2.3. Critical Extensions................................21
      A.2.4. Maximal Header Length..............................21
      A.3. Encapsulation Header.................................21
      A.3.1. Virtual Network Identifier (VNI)...................21
      A.3.2. Next Protocol......................................21
      A.3.3. Other Header Fields................................22
      A.4. Comparison Summary...................................23


S. Boutros & D. Eastlake  Expires January 2022                  [Page 3]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

1. Introduction

   As communicated by the WG Chairs, the NVO3 WG Charter states that it
   may produce requirements for network virtualization data planes based
   on encapsulation of virtual network traffic over an IP-based underlay
   data plane.  Such requirements should consider OAM and security.
   Based on these requirements the WG will select, extend, and/or
   develop one or more data plane encapsulation format(s).

   This has led to WG drafts and an RFC describing three encapsulations
   as follows:

      - [RFC8926] Geneve: Generic Network Virtualization Encapsulation

      - [I-D.ietf-intarea-gue] Generic UDP Encapsulation

      - [I-D.ietf-nvo3-vxlan-gpe] Generic Protocol Extension for VXLAN

   Discussion on the list and in face-to-face meetings has identified a
   number of technical problems with each of these encapsulations.
   Furthermore, there was clear consensus at the 96th IETF meeting in
   Berlin that it is undesirable for the working group to progress more
   than one data plane encapsulation.  Although consensus could not be
   reached on the list, the overall consensus was for a single
   encapsulation [RFC2418], Section 3.3.

   Nonetheless there has been resistance to converging on a single
   encapsulation format.

2. Design Team Goals

   As communicated by the WG Chairs, the design team (DT) should take
   one of the proposed encapsulations and enhance it to address the
   technical concerns.  The simple evolution of deployed networks as
   well as applicability to all locations in the NVO3 architecture are
   goals.  The DT should specifically avoid a design that is burdensome
   on hardware implementations but should allow future extensibility.
   The chosen design should also operate well with ICMP and in ECMP
   environments.  If further extensibility is required, then it should
   be done in such a manner that it does not require the consent of an
   entity outside of the IETF.

S. Boutros & D. Eastlake  Expires January 2022                  [Page 4]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

3. Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

4. Abbreviations and Acronyms

      DT   NVO3 encapsulation Design Team

      EVPN Ethernet VPN [RFC8365]

      GUE  Generic UDP Encapsulation [I-D.ietf-intarea-gue]

      NVO3 Network Virtualization Overlays over Layer 3

      OAM Operations, Administration, and Maintenance

      TLV Type, Length, and Value

      VNI Virtual Network Identifier

      NVE Network Virtualization Edge

      NVA Network Virtualization Authority

      NIC Network interface card

      TCAM Ternary Content-Addressable Memory

      Transit device - Underlay network devices between NVE(s).

S. Boutros & D. Eastlake  Expires January 2022                  [Page 5]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

5. Issues with Current Encapsulations

   The following subsections describe issues with current encapsulations
   as summarized by the WG Chairs:

5.1. Geneve

   - Can't be implemented cost-effectively in all use cases because
   variable length header and order of the TLVs makes is costly (in
   terms of number of gates) to implement in hardware.

   - Header doesn't fit into largest commonly available parse buffer
   (256 bytes in NIC).  Cannot justify doubling buffer size unless it is
   mandatory for hardware to process additional option fields.

5.2. GUE (Generic UDP Encapsulation)

   - There were a significant number of objections to GUE
   [I-D.ietf-intarea-gue] related to the complexity of implementation in
   hardware, similar to those noted for Geneve above.


   - GPE is not day-1 backwards compatible with VXLAN [RFC7348].
   Although the frame format is similar, it uses a different UDP port,
   so would require changes to existing implementations even if the rest
   of the GPE frame is the same.

   - GPE is insufficiently extensible.  Numerous extensions and options
   have been designed for GUE and Geneve.  Note that these have not yet
   been validated by the WG.

   - Security, e.g., of the VNI, has not been addressed by GPE.
   Although a shim header could be used for security and other
   extensions, this has not been defined yet and its implications on
   offloading in NICs are not understood.

S. Boutros & D. Eastlake  Expires January 2022                  [Page 6]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

6. Common Encapsulation Considerations

6.1. Current Encapsulations

   Appendix A includes a detailed comparison between the three proposed
   encapsulations.  The comparison indicates several common properties
   but also three major differences among the encapsulations:

   - Extensibility: Geneve and GUE were defined with built-in
   extensibility, while VXLAN-GPE is not inherently extensible.  Note
   that any of the three encapsulations can be extended using the
   Network Service Header (NSH [RFC8300]).

   - Extension method: Geneve is extensible using Type/Length/Value
   (TLV) fields, while GUE uses a small set of possible extensions, and
   a set of flags that indicate which extensions are present.

   - Length field: Geneve and GUE include a Length field, indicating the
   length of the encapsulation header while VXLAN-GPE does not include
   such a field.

6.2. Useful Extensions Use Cases

   Non vendor specific TLVs MUST follow the standardization process.
   The following use cases for extensions shows that there is a strong
   requirement to support variable length extensions with possible
   different subtypes.

6.2.1. Telemetry Extensions

   In several scenarios it is beneficial to make information about the
   path a packet took through the network or through a network device as
   well as associated telemetry information available to the operator.

   This includes not only tasks like debugging, troubleshooting, and
   network planning and optimization but also policy or service level
   agreement compliance checks.

   Packet scheduling algorithms, especially for balancing traffic across
   equal cost paths or links, often leverage information contained
   within the packet, such as protocol number, IP address, or MAC
   address.  Probe packets would thus either need to be sent between the
   exact same endpoints with the exact same parameters, or probe packets
   would need to be artificially constructed as "fake" packets and

S. Boutros & D. Eastlake  Expires January 2022                  [Page 7]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

   inserted along the path.  Both approaches are often not feasible from
   an operational perspective, be it that access to the end-system is
   not feasible, or that the diversity of parameters and associated
   probe packets to be created is simply too large.  An extension
   providing an in-band telemetry mechanism [I-D.ietf-ippm-ioam-data] is
   an alternative in those cases.

6.2.2. Security/Integrity Extensions

   Since the currently proposed NVO3 encapsulations do not protect their
   headers, a single bit corruption in the VNI field could deliver a
   packet to the wrong tenant.  Extension headers are needed to use any
   sophisticated security.

   The possibility of VNI spoofing with an NVO3 protocol is exacerbated
   by using UDP.  Systems typically have no restrictions on applications
   being able to send to any UDP port so an unprivileged application can
   trivially spoof VXLAN [RFC7348] packets for instance, including using
   arbitrary VNIs.

   One can envision HMAC-like support in an NVO3 extension to
   authenticate the header and the outer IP addresses, thereby
   preventing attackers from injecting packets with spoofed VNIs.

   Another aspect of security is payload security.  Essentially this is
   to make packets that look like IP|UDP|NVO3 Encap|DTLS/IPSEC-ESP
   Extension|payload.  This is desireable since we still have the UDP
   header for ECMP, the NVO3 header is in plain text so it can be read
   by network elements, and different security or other payload
   transforms can be supported on a single UDP port (we don't need a
   separate UDP port for DTLS/IPSEC).

6.2.3 Group Based Policy

   Another use case would be to carry the Group Based Policy (GBP)
   source group information within a NVO3 header extension in a similar
   manner as has been implemented for VXLAN
   [I-D.smith-vxlan-group-policy].  This allows various forms of policy
   such as access control and QoS to be applied between abstract groups
   rather than coupled to specific endpoint addresses.

S. Boutros & D. Eastlake  Expires January 2022                  [Page 8]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

6.3. Hardware Considerations

   Hardware restrictions should be taken into consideration along with
   future hardware enhancements that may provide more flexible metadata
   processing.  However, the set of options that need to and will be
   implemented in hardware will be a subset of what is implemented in
   software, since software NVEs are likely to grow features, and hence
   option support, at a more rapid rate.

   We note that it is hard to predict which options will be implemented
   in which piece of hardware and when.  That depends on whether the
   hardware will be in the form of a NIC providing increasing offload
   capabilities to software NVEs, or a switch chip being used as an NVE
   gateway towards non-NVO3 parts of the network, or even a transit
   device that participates in the NVO3 dataplane, e.g., for OAM

   A result of this is that it doesn't look useful to prescribe some
   order of the option so that the ones that are likely to be
   implemented in hardware come first; we can't decide such an order
   when we define the options, however a control plane can enforce such
   an order for some hardware implementation.

   We do know that hardware needs to initially be able to efficiently
   skip over the NVO3 header to find the inner payload.  That is needed
   both for NICs implementing various TCP offload mechanisms and for
   transit devices and NVEs applying policy/ACLs to the inner payload.

6.4. Extension Size

   Extension header length has a significant impact on hardware and
   software implementations.  A total header length that is too small
   will unnecessarily constrain software flexibility.  A total header
   length that is too large will place a nontrivial cost on hardware
   implementations.  Thus, the DT recommends that there be a minimum and
   maximum total extension header length specified.  The maximum total
   header length is determined by the size of the bit field allocated
   for the total extension header length field.  The risk with this
   approach is that it may be difficult to extend the total header size
   in the future.  The minimum total header length is determined by a
   requirement in the specifications that all implementations must meet.
   The risk with this approach is that all implementations will only
   implement the minimum total header length which would then become the
   de facto maximum total header length.  The recommended minimum total
   header length is 64 bytes.

   The size of an extension header should always be 4 byte aligned.

S. Boutros & D. Eastlake  Expires January 2022                  [Page 9]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

   The maximum length of a single option should be large enough to meet
   the different extension use case requirements, e.g., in-band
   telemetry and future use.

6.5. Ordering of Extension Headers

   To support hardware nodes at the target NVE or at a transit device
   that can process one or a few extension headers in TCAM, a control
   plane in such a deployment can signal a capability to ensure a
   specific extension header will always appear in a specific order, for
   example the first one in the packet.

   The order of the extension headers should be hardware friendly for
   both the sender and the receiver and possibly the transit device

   Transit devices don't participate in control plane communication
   between the end points and are not required to process the extension
   headers; however, if they do, they may need to process only a small
   subset of extension headers that will be consumed by target NVEs.

6.6. TLV versus Bit Fields

   If there is a well-known initial set of options that are likely to be
   implemented in software and in hardware, it can be efficient to use
   the bit fields approach as in GUE.  However, as described in section
   6.3, if options are added over time and different subsets of options
   are likely to be implemented in different pieces of hardware, then it
   would be hard for the IETF to specify which options should get the
   early bit fields.  TLVs are a lot more flexible, which avoids the
   need to determine the relative importance different options.
   However, general TLV of arbitrary order, size, and repetition of the
   same order is difficult to implement in hardware.  A middle ground is
   to use TLVs with restrictions on their size and alignment, observing
   that individual TLVs can have a fixed length, and support via the
   control plane a method such that an NVE will only receive options
   that it needs and implements.  The control plane approach can
   potentially be used to control the order of the TLVs sent to a
   particular NVE.  Note that transit devices are not likely to
   participate in the control plane; hence, to the extent that they need
   to participate in option processing, some other method must be used.
   Transit devices would have issues with future GUE bit fields being
   defined for future options as well.

   A benefit of TLVs from a hardware perspective is that they are self
   describing, i.e., all the information is in the TLV.  In a bit field

S. Boutros & D. Eastlake  Expires January 2022                 [Page 10]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

   approach, the hardware needs to look up the bit to determine the
   length of the data associated with the bit through some separate
   table, which would add hardware complexity.

   There are use cases where multiple modules of software are running on
   an NVE.  This can be modules such as a diagnostic module by one
   vendor that does packet sampling and another module from a different
   vendor that implements a firewall.  Using a TLV format, it is easier
   to have different software modules process different TLVs, which
   could be standard extensions or vendor specific extensions defined by
   the different vendors, without conflicting with each other.  This can
   help with hardware modularity as well.  There are some
   implementations with options that allows different software modules,
   like MAC learning and security, to process different options.

6.7. Control Plane Considerations

   Given that we want to allow considerable flexibility and
   extensibility for, e.g., software NVEs, yet be able to support
   important extensions in less flexible contexts such as hardware NVEs,
   it is useful to consider the control plane.  By control plane in this
   section we mean both protocols, such as EVPN [RFC8365] and others,
   and deployment specific configuration.

   If each NVE can express in the control plane that it only supports
   certain extensions (could be a single extension, or a few), and the
   source NVEs only include supported extensions in the NVO3 packets,
   then the target NVE can both use a simpler parser (e.g., a TCAM might
   be usable to look for a single NVO3 extension) and the depth of the
   inner payload in the NVO3 packet will be minimized.  Furthermore, if
   the target NVE cares about a few extensions and can express in the
   control plane the desired order of those extensions in the NVO3
   packets, then it can provide useful functionality with simplified
   hardware requirements for the target NVE.

   Note that transit devices that are not aware of the NVO3 extensions
   somewhat benefit from such an approach, since the inner payload is
   less deep in the packet if no extraneous extension headers are
   included in the packet.  In general, a transit device is not likely
   to participate in the NVO3 control plane.  (However, configuration
   mechanisms can take into account limitations of the transit devices
   used in particular deployments.)

   Note that with this approach different NVEs could desire different
   extensions or sets of extensions, which means that the source NVE
   needs to be able to place different sets of extensions in different
   NVO3 packets, and perhaps in different order.  It also assumes that
   underlay multicast or replication servers are not used together with

S. Boutros & D. Eastlake  Expires January 2022                 [Page 11]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

   NVO3 extension headers.

   There is a need to consider mandatory extensions versus optional
   extensions.  Mandatory extensions require the receiver to drop the
   packet if the extension is unknown.  A control plane mechanism can
   prevent the need for dropping unknown extensions, since they would
   not be included to target NVEs that do not support them.

   The control planes defined today need to add the ability to describe
   the different encapsulations.  Thus, perhaps EVPN [RFC8365] and any
   other control plane protocol that the IETF defines should have a way
   to indicate the supported NVO3 extensions and their order, for each
   of the encapsulations supported.

   The WG should consider developing a separate draft on guidance for
   option processing and control plane participation.  This should
   provide examples/guidance on range of usage models and deployments
   scenarios for specific options and ordering that are relevant for
   that specific deployment.  This includes end points and middle boxes
   using the options.  So, having the control plane negotiate the
   constraints is the most appropriate and flexible way to address these

6.8. Split NVE

   If the working group sees a need for having the hosts send and
   receive options in a split NVE case [RFC8394], this is possible using
   any of the existing extensible encapsulations (Geneve, GUE, GPE+NSH)
   by defining a way to carry those over other transports.  NSH can
   already be used over different transports.

   If we need to do this with other encapsulations it can be done by
   defining an Ethertype for other encapsulations so that it can be
   carried over Ethernet and 802.1Q.

   If we need to carry other encapsulations over MPLS, it would require
   an EVPN control plane to signal that other encapsulation header +
   options will be present in front of the L2 packet.  The VNI can be
   ignored in the header, and the MPLS label will be the one used to
   identify the EVPN L2 instance.

6.9. Larger VNI Considerations

   We discussed whether we should make the VNI 32-bits or larger.  The
   benefit of a 24-bit VNI would be to avoid unnecessary changes with
   existing proposals and implementations that are almost all, if not

S. Boutros & D. Eastlake  Expires January 2022                 [Page 12]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

   all, using 24-bit VNI.  If we need a larger VNI, an extension can be
   used to support that.

S. Boutros & D. Eastlake  Expires January 2022                 [Page 13]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

7. Design Team Recommendations

   We concluded that Geneve is most suitable as a starting point for a
   proposed standard for network virtualization, for the following

   1.  We studied whether VNI should be in the base header or in an
   extension header and whether it should be a 24-bit or 32-bit field.
   The design team agreed that VNI is critical information for network
   virtualization and MUST be present in all packets.  The design team
   also agreed that a 24-bit VNI matches the existing widely used
   encapsulation formats, i.e., VXLAN [RFC7348] and NVGRE [RFC7637], and
   hence is more suitable to use going forward.

   2.  The Geneve header has the total options length which allows
   skipping over the options for NIC offload operations and will allow
   transit devices to view flow information in the inner payload.

   3.  We considered the option of using NSH [RFC8300] with VXLAN-GPE
   but given that NSH is targeted at service chaining and contains
   service chaining information, it is less suitable for the network
   virtualization use case.  The other downside for VXLAN-GPE was lack
   of a header length in VXLAN-GPE which makes skipping over the headers
   to process inner payload more difficult.  Total Option Length is
   present in Geneve.  It is not possible to skip any options in the
   middle with VXLAN-GPE.  In principle a split between a base header
   and a header with options is interesting (whether that options header
   is NSH or some new header without ties to a service path).  We
   explored whether it would make sense to either use NSH for this, or
   define a new NVO3 options header.  However, we observed that this
   makes it slightly harder to find the inner payload since the length
   field is not in the NVO3 header itself.  Thus, one more field would
   have to be extracted to compute the start of the inner payload.
   Also, if the experience with IPv6 extension headers is a guide, there
   would be a risk that key pieces of hardware might not implement the
   options header, resulting in future calls to deprecate its use.
   Making the options part of the base NVO3 header has less of those
   issues.  Even though the implementation of any particular option can
   not be predicted ahead of time, the option mechanism and ability to
   skip the options is likely to be broadly implemented.

   4.  We compared the TLV vs bit fields style extension and it was
   deemed that parsing both TLV and bit fields is expensive and while
   bit fields may be simpler to parse, it is also more restrictive and
   requires guessing which extensions will be widely implemented so they
   can get early bit assignments, given that half the bits are already
   assigned in GUE, a widely deployed extension may appear in a flag
   extension, and this will require extra processing, to dig the flag
   from the flag extension and then look for the extension itself.  Also
   bit fields are not flexible enough to address the requirements from

S. Boutros & D. Eastlake  Expires January 2022                 [Page 14]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

   OAM, Telemetry, and security extensions, for variable length option
   and different subtypes of the same option.  While TLV are more
   flexible, a control plane can restrict the number of option TLVs as
   well the order and size of the TLVs to make it simpler for a
   dataplane implementation to handle.

   5.  We briefly discussed the multi-vendor NVE case, and the need to
   allow vendors to put their own extensions in the NVE header.  This is
   possible with TLVs.

   6.  We also agreed that the C bit in Geneve is helpful to allow a
   receiver NVE to easily decide whether to process options or not, for
   example a UUID based packet trace, and how an optional extension such
   as that can be ignored by a receiver NVE and thus make it easy for
   NVE to skip over the options.  Thus, the C bit remains as defined in

   7.  There are already some extensions that are being discussed (see
   section 6.2) of varying sizes. By using Geneve options it is possible
   to get in band parameters like switch id, ingress port, egress port,
   internal delay, and queue in telemetry defined extension TLV from
   switches.  It is also possible to add security extension TLVs like
   HMAC and DTLS/IPSEC to authenticate the Geneve packet header and
   secure the Geneve packet payload by software or hardware tunnel
   endpoints.  A Group Based Policy extension TLV can be carried as

   8.  There are already implementations of Geneve options deployed in
   production networks as of this writing.  There are as well new
   hardware supporting Geneve TLV parsing.  In addition, an In-band
   Telemetry [INT] specification is being developed by P4.org that
   illustrates the option of INT meta data carried over Geneve.  OVN/OVS
   have also defined some option TLV(s) for Geneve.

   9.  The DT has addressed the usage models while considering the
   requirements and implementations in general that includes software
   and hardware.

      There seems to be interest to standardize some well-known secure
      option TLVs to secure the header and payload to guarantee
      encapsulation header integrity and tenant data privacy.  The
      design team recommends that the working group consider
      standardizing such option(s).

      We recommend the following enhancements to Geneve to make it more
      suitable to hardware and yet provide the flexibility for software:

      We would propose a text such as, while TLV are more flexible, a
      control plane can restrict the number of option TLVs as well the
      order and size of the TLVs to make it simpler for a data plane

S. Boutros & D. Eastlake  Expires January 2022                 [Page 15]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

      implementation in software or hardware to handle.  For example,
      there may be some critical information such as a secure hash that
      must be processed in a certain order at lowest latency.

      A control plane can negotiate a subset of option TLVs and certain
      TLV ordering, as well as limiting the total number of option TLVs
      present in the packet, for example, to allow for hardware capable
      of processing fewer options.  Hence, the control plane needs to
      have the ability to describe the supported TLVs subset and their

      The Geneve draft should specify that the subset and order of
      option TLVs should be configurable for each remote NVE in the
      absence of a protocol control plane.

      We recommend that Geneve follow fragmentation recommendations in
      overlay services like PWE3 and the L2/L3 VPN recommendations to
      guarantee larger MTU for the tunnel overhead ([RFC3985] Section

      We request that Geneve provide a recommendation for critical bit
      processing - text could specify how critical bits can be used with
      control plane specifying the critical options.

      Given that there is a telemetry option use case for a length of
      256 bytes, we recommend that Geneve increase the Single TLV option
      length to 256.

      We request that Geneve address Requirements for OAM considerations
      for alternate marking and for performance measurements that need a
      2 bit field in the header and clarify the need for the current OAM
      bit in the Geneve Header.

      We recommend that the WG work on security options for Geneve.

S. Boutros & D. Eastlake  Expires January 2022                 [Page 16]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

8. Acknowledgements

   The authors would like to thank Tom Herbert for providing the
   motivation for the Security/Integrity extension, and for his valuable
   comments, T. Sridhar for his valuable comments and feedback, and
   Anoop Ghanwani for his extensive comments.

9. Security Considerations

   This document does not introduce any additional security constraints.

10. IANA Considerations

   This document requires no IANA actions.

S. Boutros & D. Eastlake  Expires January 2022                 [Page 17]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

11. References

11.1 Normative References

   [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
             Requirement Levels", BCP 14, RFC 2119, DOI
             10.17487/RFC2119, March 1997, <https://www.rfc-

   [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119
             Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May
             2017, <https://www.rfc-editor.org/info/rfc8174>.

11.2 Informative References

   [I-D.herbert-gue-extensions] Herbert, T., Yong, L., and F. Templin,
             "Extensions for Generic UDP Encapsulation",
             draft-herbert-gue-extensions-01 (work in progress), October

   [I-D.ietf-intarea-gue] Herbert, T., Yong, L., and O. Zia, "Generic
             UDP Encapsulation", draft-ietf-intarea-gue (work in
             progress), October 2019.

   [I-D.ietf-ippm-ioam-data] F. Brockers, S. Bhandari, T. Mizrahi, "Data
             Fields for In-situ OAM", draft-ietf-ippm-ioam-data (work in
             progress), June 2021.

   [I-D.ietf-nvo3-vxlan-gpe] Maino, F., Kreeger, L., and U. Elzur,
             "Generic Protocol Extension for VXLAN",
             draft-ietf-nvo3-vxlan-gpe (work in progress), March 2021.

   [I-D.smith-vxlan-group-policy] Smith, M. and L. Kreeger, "VXLAN Group
             Policy Option", draft-smith-vxlan-group-policy-05 (work in
             progress), October 2018.

   [INT]     P4.org, "In-band Network Telemetry (INT) Dataplane
             Specification", November 2020,

   [RFC2418] Bradner, S., "IETF Working Group Guidelines and
             Procedures", BCP 25, RFC 2418, DOI 10.17487/RFC2418,
             September 1998, <https://www.rfc-editor.org/info/rfc2418>.

   [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation
             Edge-to-Edge (PWE3) Architecture", RFC 3985, DOI

S. Boutros & D. Eastlake  Expires January 2022                 [Page 18]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

             10.17487/RFC3985, March 2005,

   [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
             L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
             eXtensible Local Area Network (VXLAN): A Framework for
             Overlaying Virtualized Layer 2 Networks over Layer 3
             Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
             <https://www.rfc- editor.org/info/rfc7348>.

   [RFC7637] Garg, P., Ed., and Y. Wang, Ed., "NVGRE: Network
             Virtualization Using Generic Routing Encapsulation", RFC
             7637, DOI 10.17487/RFC7637, September 2015,

   [RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed.,
             "Network Service Header (NSH)", RFC 8300, DOI
             10.17487/RFC8300, January 2018,

   [RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
             Uttaro, J., and W. Henderickx, "A Network Virtualization
             Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365, DOI
             10.17487/RFC8365, March 2018,

   [RFC8394] Li, Y., Eastlake 3rd, D., Kreeger, L., Narten, T., and D.
             Black, "Split Network Virtualization Edge (Split-NVE)
             Control-Plane Requirements", RFC 8394, DOI
             10.17487/RFC8394, May 2018,

   [RFC8926] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed.,
             "Geneve: Generic Network Virtualization Encapsulation", RFC
             8926, DOI 10.17487/RFC8926, November 2020,

S. Boutros & D. Eastlake  Expires January 2022                 [Page 19]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

Appendix A: Encapsulations Comparison

A.1. Overview

   This section presents a comparison of the three NVO3 encapsulation
   proposals, Geneve, GUE, and VXLAN-GPE.  The three encapsulations use
   an outer UDP/IP transport.  Geneve and VXLAN-GPE use an 8-octet
   header, while GUE uses a 4-octet header.  In addition to the base
   header, optional extensions may be included in the encapsulation, as
   discussed in Section A.2 below.

A.2. Extensibility

A.2.1. Native Extensibility Support

   The Geneve and GUE encapsulations both enable optional headers to be
   incorporated at the end of the base encapsulation header.

   VXLAN-GPE does not provide native support for header extensions.
   However, as discussed in [I-D.ietf-nvo3-vxlan-gpe], extensibility can
   be attained to some extent if the Network Service Header (NSH)
   [RFC8300] is used immediately following the VXLAN-GPE header.  NSH
   supports either a fixed-size extension (MD Type 1), or a variable-
   size TLV-based extension (MD Type 2).  It should be noted that NSH-
   over-VXLAN-GPE implies an additional overhead of the 8-octets NSH
   header, in addition to the VXLAN-GPE header.

A.2.2. Extension Parsing

   The Geneve Variable Length Options are defined as Type/Length/Value
   (TLV) extensions.  Similarly, VXLAN-GPE, when using NSH, can include
   NSH TLV-based extensions.  In contrast, GUE defines a small set of
   possible extension fields (proposed in [I-D.herbert-gue-extensions]),
   and a set of flags in the GUE header that indicate for each extension
   type whether it is present or not.

   TLV-based extensions, as defined in Geneve, provide the flexibility
   for a large number of possible extension types.  Similar behavior can
   be supported in NSH-over-VXLAN-GPE when using MD Type 2.  The flag-
   based approach taken in GUE strives to simplify implementations by
   defining a small number of possible extensions used in a fixed order.

S. Boutros & D. Eastlake  Expires January 2022                 [Page 20]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

   The Geneve and GUE headers both include a length field, defining the
   total length of the encapsulation, including the optional extensions.

   The length field simplifies the parsing of transit devices that skip
   the encapsulation header without parsing its extensions.

A.2.3. Critical Extensions

   The Geneve encapsulation header includes the 'C' field, which
   indicates whether the current Geneve header includes critical
   options, that is to say, options which must be parsed by the target
   NVE.  If the endpoint is not able to process a critical option, the
   packet is discarded.

A.2.4. Maximal Header Length

   The maximal header length in Geneve, including options, is 260
   octets.  GUE defines the maximal header to be 128 octets.  VXLAN-GPE
   uses a fixed-length header of 8 octets, unless NSH-over-VXLAN-GPE is
   used, yielding an encapsulation header of up to 264 octets.

A.3. Encapsulation Header

A.3.1. Virtual Network Identifier (VNI)

   The Geneve and VXLAN-GPE headers both include a 24-bit VNI field.
   GUE, on the other hand, enables the use of a 32-bit field called
   VNID; this field is not included in the GUE header, but was defined
   as an optional extension in [I-D.herbert-gue-extensions].

   The VXLAN-GPE header includes the 'I' bit, indicating that the VNI
   field is valid in the current header.  A similar indicator is defined
   as a flag in the GUE header [I-D.herbert-gue-extensions].

A.3.2. Next Protocol

   The three encapsulation headers include a field that specifies the
   type of the next protocol header, which resides after the NVO3
   encapsulation header.  The Geneve header includes a 16-bit field that
   uses the IEEE Ethertype convention.  GUE uses an 8-bit field, which

S. Boutros & D. Eastlake  Expires January 2022                 [Page 21]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

   uses the IANA Internet protocol numbering.  The VXLAN-GPE header
   incorporates an 8-bit Next Protocol field, using a VXLAN-GPE-specific
   registry, defined in [I-D.ietf-nvo3-vxlan-gpe].

   The VXLAN-GPE header also includes the 'P' bit, which explicitly
   indicates whether the Next Protocol field is present in the current

A.3.3. Other Header Fields

   The OAM bit, which is defined in Geneve and in VXLAN-GPE, indicates
   whether the current packet is an OAM packet.  The GUE header includes
   a similar field, but uses different terminology; the GUE 'C-bit'
   specifies whether the current packet is a control packet.  Note that
   the GUE control bit can potentially be used in a large set of
   protocols that are not OAM protocols.  However, the control packet
   examples discussed in [I-D.ietf-intarea-gue] are OAM-related.

   Each of the three NVO3 encapsulation headers includes a 2-bit Version
   field, which is currently defined to be zero.

   The Geneve and VXLAN-GPE headers include reserved fields; 14 bits in
   the Geneve header, and 27 bits in the VXLAN-GPE header are reserved.

S. Boutros & D. Eastlake  Expires January 2022                 [Page 22]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

A.4. Comparison Summary

   The following table summarizes the comparison between the three NVO3
   |                |     Geneve     |      GUE       |   VXLAN-GPE    |
   | Outer transport|     UDP/IP     |     UDP/IP     |     UDP/IP     |
   | Base header    |    8 octets    |    4 octets    |    8 octets    |
   | length         |                |                |  (16 octets    |
   |                |                |                |   using NSH)   |
   | Extensibility  |Variable length |Extension fields| No native ext- |
   |                |    options     |                | ensibility.    |
   |                |                |                | Extensible     |
   |                |                |                | using NSH.     |
   | Extension      |   TLV-based    |   Flag-based   |   TLV-based    |
   | parsing method |                |                |(using NSH with |
   |                |                |                |   MD Type 2)   |
   | Extension      |    Variable    |     Fixed      |    Variable    |
   | order          |                |                |  (using NSH)   |
   | Length field   |       +        |       +        |       -        |
   | Max Header     |   260 octets   |   128 octets   |    8 octets    |
   | Length         |                |                |(264 using NSH) |
   | Critical exte- |       +        |       -        |       -        |
   | nsion bit      |                |                |                |
   | VNI field size |    24 bits     |    32 bits     |    24 bits     |
   |                |                |  (extension)   |                |
   | Next protocol  |    16 bits     |     8 bits     |     8 bits     |
   | field          |   Ethertype    | Internet prot- |  New registry  |
   |                |   registry     | ocol registry  |                |
   | Next protocol  |       -        |       -        |       +        |
   | indicator      |                |                |                |
   | OAM / control  |    OAM bit     |  Control bit   |    OAM bit     |
   | field          |                |                |                |
   | Version field  |    2 bits      |    2 bits      |    2 bits      |
   | Reserved bits  |    14 bits     |       -        |    27 bits     |

S. Boutros & D. Eastlake  Expires January 2022                 [Page 23]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

                 Figure 1: NVO3 Encapsulations Comparison

S. Boutros & D. Eastlake  Expires January 2022                 [Page 24]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021


   The following co-authors have contributed to this document:

      Ilango Ganga    Intel     Email: ilango.s.ganga@intel.com

      Pankaj Garg     Microsoft Email: pankajg@microsoft.com

      Rajeev Manur    Broadcom  Email: rajeev.manur@broadcom.com

      Tal Mizrahi     Marvell   Email: talmi@marvell.com

      David Mozes               Email: mosesster@gmail.com

      Erik Nordmark   ZEDEDA    Email: nordmark@sonic.net

      Michael Smith   Cisco     Email: michsmit@cisco.com

      Sam Aldrin      Google    Email: aldrin.ietf@gmail.com

      Ignas Bagdonas  Equinix   Email: ibagdona.ietf@gmail.com

S. Boutros & D. Eastlake  Expires January 2022                 [Page 25]

Internet-Draft      NVO3 Encapsulation Considerations          July 2021

Authors' Addresses

      Sami Boutros (editor)

      Email: sboutros@ciena.com

      Donald E. Eastlake, 3rd (editor)
      Futurewei Technologies
      2386 Panoramic Circle
      Apopka, FL 32703

      Tel: +1-508-333-2270
      Email: d3e3e3@gmail.com

S. Boutros & D. Eastlake  Expires January 2022                 [Page 26]