Skip to main content

OAM for use in GENEVE

Document Type Active Internet-Draft (nvo3 WG)
Authors Greg Mirsky , Sami Boutros , David L. Black , Santosh Pallagatti
Last updated 2024-04-19
Replaces draft-mmbb-nvo3-geneve-oam
RFC stream Internet Engineering Task Force (IETF)
Intended RFC status (None)
Additional resources Mailing list discussion
Stream WG state WG Consensus: Waiting for Write-Up
Revised I-D Needed - Issue raised by WGLC
Document shepherd Sam Aldrin
IESG IESG state I-D Exists
Consensus boilerplate Unknown
Telechat date (None)
Responsible AD (None)
Send notices to
NVO3 Working Group                                             G. Mirsky
Internet-Draft                                                  Ericsson
Intended status: Standards Track                              S. Boutros
Expires: 21 October 2024                                           Ciena
                                                                D. Black
                                                                Dell EMC
                                                           S. Pallagatti
                                                           19 April 2024

                         OAM for use in GENEVE


   This document lists a set of general requirements for active OAM
   protocols in the Geneve overlay network.  Based on the requirements,
   IP encapsulation for active Operations, Administration, and
   Maintenance protocols in Geneve protocol is defined.  Considerations
   for using ICMP and UDP-based protocols are discussed.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 21 October 2024.

Copyright Notice

   Copyright (c) 2024 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights

Mirsky, et al.           Expires 21 October 2024                [Page 1]
Internet-Draft                OAM in GENEVE                   April 2024

   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Conventions used in this document . . . . . . . . . . . .   3
       1.1.1.  Terminology . . . . . . . . . . . . . . . . . . . . .   3
       1.1.2.  Requirements Language . . . . . . . . . . . . . . . .   3
       1.1.3.  Acronyms  . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Active OAM Protocols in Geneve Networks . . . . . . . . . . .   4
     2.1.  Defect Detection and Troubleshooting in Geneve Network with
           Active OAM  . . . . . . . . . . . . . . . . . . . . . . .   5
     2.2.  OAM Encapsulation in Geneve . . . . . . . . . . . . . . .   7
   3.  Echo Request and Echo Reply in Geneve Tunnel  . . . . . . . .   8
   4.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   8
   5.  Security Considerations . . . . . . . . . . . . . . . . . . .   8
   6.  Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .   8
   7.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   8
     7.1.  Normative References  . . . . . . . . . . . . . . . . . .   8
     7.2.  Informative References  . . . . . . . . . . . . . . . . .   9
   Appendix A.  Additional Considerations for OAM Encapsulation Method
           in Geneve . . . . . . . . . . . . . . . . . . . . . . . .  10
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  11

1.  Introduction

   Geneve [RFC8926] is intended to support various scenarios of network
   virtualization.  In addition to carrying multi-protocol payload,
   e.g., Ethernet, IPv4/IPv6, the Geneve message includes metadata.
   Operations, Administration, and Maintenance (OAM) protocols support
   fault management and performance monitoring functions necessary for
   comprehensive network operation.  Active OAM protocols, as defined in
   [RFC7799], use specially constructed packets that are injected into
   the network.  To ensure that the measured performance metric or the
   detected failure of the transport layer are related to a particular
   Geneve flow, it is critical that these test packets share fate with
   overlay data packets for that flow when traversing the underlay

   A set of general requirements for active OAM protocols in the Geneve
   overlay network is listed in Section 2.  IP encapsulation conforms to
   these requirements and is a suitable encapsulation of active OAM
   protocols in a Geneve overlay network.  Active OAM in a Geneve
   overlay network are exchanged between two Geneve tunnel endpoints,
   which may be an NVE (Network Virtualization Edge) or another device

Mirsky, et al.           Expires 21 October 2024                [Page 2]
Internet-Draft                OAM in GENEVE                   April 2024

   acting as a Geneve tunnel endpoint.  For simplicity, NVE is used to
   represent the Geneve tunnel endpoint.  Please refer to [RFC7365] and
   [RFC8014] for detailed definitions and descriptions of NVE.  The IP
   encapsulation of Geneve OAM defined in this document applies to an
   overlay service by introducing a Management Virtual Network
   Identifier (VNI) that could be used in combination with various
   values of the Protocol Type field in the Geneve header, i.e.,
   Ethertypes for IPv4 or IPv6.  Analysis and definition of other types
   of OAM encapsulation in Geneve are outside the scope of this

1.1.  Conventions used in this document

1.1.1.  Terminology

   The term "BIER OAM" used in this document interchangeably with longer
   version "set of OAM protocols, methods, and tools for BIER layer".

   *  In-band OAM is an active OAM or hybrid OAM method ([RFC7799]) that
      traverses the same set of links and interfaces receiving the same
      QoS treatment as the monitored object, i.e., a Geneve tunnel as a
      whole or a particular tenant flow within given Geneve tunnel.

1.1.2.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

1.1.3.  Acronyms

   Geneve Generic Network Virtualization Encapsulation

   NVO3 Network Virtualization Overlays

   OAM Operations, Administration, and Maintenance

   VNI Virtual Network Identifier

Mirsky, et al.           Expires 21 October 2024                [Page 3]
Internet-Draft                OAM in GENEVE                   April 2024

2.  Active OAM Protocols in Geneve Networks

   OAM protocols, whether part of fault management or performance
   monitoring, are intended to provide reliable information that can be
   used to detect a failure, identify the defect and localize it, thus
   helping to identify and apply corrective actions to minimize the
   negative impact on service.  Several OAM protocols are used to
   perform these functions; these protocols require demultiplexing at
   the receiving instance of Geneve.  To improve the accuracy of the
   correlation between the condition experienced by the monitored Geneve
   tunnel and the state of the OAM protocol the OAM encapsulation is
   required to comply with the following requirements:

      REQ#1: Geneve OAM test packets MUST share the fate with data
      traffic of the monitored Geneve tunnel, i.e., be in-band
      (Section 1.1.1) with the monitored traffic, follow the same
      overlay and transport path as packets with data payload, in the
      forward direction, i.e. from ingress toward egress endpoint(s) of
      the OAM test.

   An OAM protocol MAY be used to monitor the particular Geneve tunnel
   as a whole.  In that case, test packets could be in-band relative to
   a sub-set of tenant flows transported over the Geneve tunnel.  If the
   goal is to monitor the condition experienced by the flow of a
   particular tenant, the test packets MUST be in-band with that
   specific flow in the Geneve tunnel.  Both scenarios are discussed in
   detail in Section 2.1.

      REQ#2: Encapsulation of OAM control message and data packets in
      underlay network MUST be indistinguishable from the underlay
      network IP forwarding point of view.

      REQ#3: Presence of OAM control message in Geneve packet MUST be
      unambiguously identifiable to Geneve functionality, e.g., at
      endpoints of Geneve tunnels.

      REQ#4: OAM test packets MUST NOT be forwarded to a tenant system.

   A test packet generated by an active OAM protocol, either for a
   defect detection or performance measurement, according to REQ#1, MUST
   be in-band (Section 1.1.1) with the tunnel or data flow being
   monitored.  In an environment where multiple paths through the domain
   are available, underlay transport nodes can be programmed to use
   characteristic information to balance the load across known paths.
   It is essential that test packets follow the same route, i.e.,
   traverses the same set of nodes and links, as a data packet of the
   monitored flow.  Thus, the following requirement to support OAM
   packet fate-sharing with the data flow:

Mirsky, et al.           Expires 21 October 2024                [Page 4]
Internet-Draft                OAM in GENEVE                   April 2024

      REQ#5: It MUST be possible to express entropy for underlay Equal
      Cost Multipath in the Geneve encapsulation of OAM packets.

2.1.  Defect Detection and Troubleshooting in Geneve Network with Active

   Figure 1 presents an example of a Geneve domain.  In this section, we
   consider two scenarios of active OAM being used to detect and
   localize defects in the Geneve network.

       +--------+                                             +--------+
       | Tenant +--+                                     +----| Tenant |
       | VNI 28 |  |                                     |    | VNI 35 |
       +--------+  |          ................           |    +--------+
                   |  +----+  .              .  +----+   |
                   |  | NVE|--.              .--| NVE|   |
                   +--| A  |  .              .  | B  |---+
                      +----+  .              .  +----+
                      /       .              .
                     /        .     Geneve   .
       +--------+   /         .    Network   .
       | Tenant +--+          .              .
       | VNI 35 |             .              .
       +--------+             ................
                                   | NVE|
                                   | C  |
                             |               |
                         +--------+      +--------+
                         | Tenant |      | Tenant |
                         | VNI 28 |      | VNI 35 |
                         +--------+      +--------+

               Figure 1: An example of a Geneve domain

   In the first case, consider when a communication problem between
   Network Virtualization Edge (NVE) device A and NVE C exists.  Upon
   the investigation, the operator discovers that the forwarding in the
   underlay, e.g., the IP network, is working well.  Still, the Geneve
   connection is unstable for all NVE A and NVE C tenants.  Detection,
   troubleshooting, and localization of the problem can be done
   irrespective of the VNI value.

Mirsky, et al.           Expires 21 October 2024                [Page 5]
Internet-Draft                OAM in GENEVE                   April 2024

   In the second case, traffic on VNI 35 between NVE A and NVE B has no
   problems, as on VNI 28 between NVE A and NVE C.  But traffic on VNI
   35 between NVE A and NVE C experiences problems, for example,
   excessive packet loss.

   The first case can be detected and investigated using any VNI value,
   whether it connects tenant systems or not; however, to conform to
   REQ#4 (Section 2) OAM test packets SHOULD be transmitted on a VNI
   that doesn't have any tenants.  Such a Geneve tunnel is dedicated to
   carrying only control and management data between the tunnel
   endpoints, hence it is referred to as a Geneve control channel and
   that VNI is referred to as the Management VNI.  A configured VNI MAY
   be used to identify the control channel, but it is RECOMMENDED that
   the default value 1 be used as the Management VNI.  Encapsulation of
   test packets using the Management VNI is discussed in Section 2.2.

   The control channel of a Geneve tunnel MUST NOT carry tenant data.
   As no tenants are connected using the control channel, a system that
   supports this specification, MUST NOT forward a packet received over
   the control channel to any tenant.  A packet received over the
   control channel MAY be forwarded if and only if it is sent onto the
   control channel of the a concatenated Geneve tunnel.  The Management
   VNI SHOULD be terminated on the tenant-facing side of the Geneve
   encap/decap functionality, not the DC-network-facing side (per
   definitions in Section 4 of [RFC8014]) so that Geneve encap/decap
   functionality is included in its scope.  This approach causes an
   active OAM packet, e.g., an ICMP echo request, to be decapsulated in
   the same fashion as any other received Geneve packet.  In this
   example, the resulting ICMP packet is handed to NVE's local
   management functionality for the processing which generates an ICMP
   echo reply.  The ICMP echo reply is encapsulated in Geneve as
   specified in Section 2.2. for forwarding back to the NVE that sent
   the echo request.  One advantage of this approach is that a repeated
   ping test could detect an intermittent problem in Geneve encap/decap
   hardware, which would not be tested if the Management VNI were
   handled as a "special case" at the DC-network-facing interface.

   The second case is when a test packet is transmitted using the VNI
   value associated with the monitored service flow.  By doing that, the
   test packet experiences network treatment as the tenant's packets.
   Details of that use case are outside the scope of this specification.

Mirsky, et al.           Expires 21 October 2024                [Page 6]
Internet-Draft                OAM in GENEVE                   April 2024

2.2.  OAM Encapsulation in Geneve

   Active OAM over a Management VNI in the Geneve network uses an IP
   encapsulation.  Protocols such as BFD [RFC5880] or STAMP [RFC8762]
   use UDP transport.  The destination UDP port number in the inner UDP
   header (Figure 2) identifies the OAM protocol.  This approach is
   well-known and has been used, for example, in MPLS networks
   [RFC8029].  To use IP encapsulation for an active OAM protocol the
   Protocol Type field of the Geneve header MUST be set to the IPv4
   (0x0800) or IPv6 (0x86DD) value.

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    |                                                               |
    ~                        Outer IPvX Header                      ~
    |                                                               |
    |                                                               |
    ~                        Outer UDP Header                       ~
    |                                                               |
    |                                                               |
    ~                          Geneve Header                        ~
    |                                                               |
    |                                                               |
    ~                        Inner IPvX Header                      ~
    |                                                               |
    |                                                               |
    ~                        Inner UDP Header                       ~
    |                                                               |
    |                                                               |
    ~                        Active OAM Packet                      ~
    |                                                               |

      Figure 2: An Example of Geneve IP/UDP Encapsulation of an Active
                                 OAM Packet

   Inner IP header:

      Destination IP: The IP address MUST be set to the loopback address for IPv4, or the loopback address ::1/128 for IPv6

Mirsky, et al.           Expires 21 October 2024                [Page 7]
Internet-Draft                OAM in GENEVE                   April 2024

      Source IP: IP address of the NVE.

      TTL or Hop Limit: MUST be set to 255 per [RFC5082].

3.  Echo Request and Echo Reply in Geneve Tunnel

   ICMP and ICMPv6 ([RFC0792] and [RFC4443] respectively) provide
   required on-demand defect detection and failure localization.  ICMP
   control messages immediately follow the inner IP header encapsulated
   in Geneve.  ICMP extensions for Geneve networks use mechanisms
   defined in [RFC4884].

4.  IANA Considerations

   This document has no requirements for IANA.  This section can be
   removed before the publication.

5.  Security Considerations

   As part of a Geneve network, active OAM inherits the security
   considerations discussed in [RFC8926].  Additionally, a system MUST
   provide control to limit the rate of Geneve OAM packets punted to the
   Geneve control plane for processing in order to avoid overloading
   that control plane.

   OAM in GENEVE packets uses the General TTL Security Mechanism
   [RFC5082] and any packet received with an inner TTL / Hop Count other
   than 255 MUST be discarded.

6.  Acknowledgments

   The authors express their appreciation to Donald E.  Eastlake 3rd for
   his suggestions that improved the readability of the document.

7.  References

7.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,

   [RFC4385]  Bryant, S., Swallow, G., Martini, L., and D. McPherson,
              "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for
              Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385,
              February 2006, <>.

Mirsky, et al.           Expires 21 October 2024                [Page 8]
Internet-Draft                OAM in GENEVE                   April 2024

   [RFC5586]  Bocci, M., Ed., Vigoureux, M., Ed., and S. Bryant, Ed.,
              "MPLS Generic Associated Channel", RFC 5586,
              DOI 10.17487/RFC5586, June 2009,

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <>.

   [RFC8926]  Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed.,
              "Geneve: Generic Network Virtualization Encapsulation",
              RFC 8926, DOI 10.17487/RFC8926, November 2020,

7.2.  Informative References

   [RFC0792]  Postel, J., "Internet Control Message Protocol", STD 5,
              RFC 792, DOI 10.17487/RFC0792, September 1981,

   [RFC4291]  Hinden, R. and S. Deering, "IP Version 6 Addressing
              Architecture", RFC 4291, DOI 10.17487/RFC4291, February
              2006, <>.

   [RFC4443]  Conta, A., Deering, S., and M. Gupta, Ed., "Internet
              Control Message Protocol (ICMPv6) for the Internet
              Protocol Version 6 (IPv6) Specification", STD 89,
              RFC 4443, DOI 10.17487/RFC4443, March 2006,

   [RFC4884]  Bonica, R., Gan, D., Tappan, D., and C. Pignataro,
              "Extended ICMP to Support Multi-Part Messages", RFC 4884,
              DOI 10.17487/RFC4884, April 2007,

   [RFC5082]  Gill, V., Heasley, J., Meyer, D., Savola, P., Ed., and C.
              Pignataro, "The Generalized TTL Security Mechanism
              (GTSM)", RFC 5082, DOI 10.17487/RFC5082, October 2007,

   [RFC5880]  Katz, D. and D. Ward, "Bidirectional Forwarding Detection
              (BFD)", RFC 5880, DOI 10.17487/RFC5880, June 2010,

   [RFC7365]  Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y.
              Rekhter, "Framework for Data Center (DC) Network
              Virtualization", RFC 7365, DOI 10.17487/RFC7365, October
              2014, <>.

Mirsky, et al.           Expires 21 October 2024                [Page 9]
Internet-Draft                OAM in GENEVE                   April 2024

   [RFC7799]  Morton, A., "Active and Passive Metrics and Methods (with
              Hybrid Types In-Between)", RFC 7799, DOI 10.17487/RFC7799,
              May 2016, <>.

   [RFC8014]  Black, D., Hudson, J., Kreeger, L., Lasserre, M., and T.
              Narten, "An Architecture for Data-Center Network
              Virtualization over Layer 3 (NVO3)", RFC 8014,
              DOI 10.17487/RFC8014, December 2016,

   [RFC8029]  Kompella, K., Swallow, G., Pignataro, C., Ed., Kumar, N.,
              Aldrin, S., and M. Chen, "Detecting Multiprotocol Label
              Switched (MPLS) Data-Plane Failures", RFC 8029,
              DOI 10.17487/RFC8029, March 2017,

   [RFC8762]  Mirsky, G., Jun, G., Nydell, H., and R. Foote, "Simple
              Two-Way Active Measurement Protocol", RFC 8762,
              DOI 10.17487/RFC8762, March 2020,

Appendix A.  Additional Considerations for OAM Encapsulation Method in

   Several other options for OAM encapsulation were considered.  Those
   are listed in the Appendix solely for informational purposes.  These
   options were discarded in favor of the approach described in the main
   body of this document.

   A Protocol Type field might be used to demultiplex active OAM
   protocols directly.  Such method avoids the use of additional
   intermediate header but requires that each active OAM protocol be
   assigned unique identifier from the Ether Types registry maintained
   by IANA.

   The alternative to using the Protocol Type directly is to use a shim
   that, in turn, identifies the OAM Protocol and, optionally, includes
   additional information.  [RFC5586] defines how the Generic Associated
   Channel Label (GAL) can be used to identify that the Associated
   Channel Header (ACH), defined in [RFC4385], immediately follows the
   Bottom-of-the-Stack label.  Thus, the MPLS Generic Associated Channel
   can be identified, and protocols are demultiplexed based on the
   Channel Type field's value.  Number of channel types, e.g., for
   continuity check and performance monitoring, already have been
   defined and are listed in IANA MPLS Generalized Associated Channel
   Types (including Pseudowire Associated Channel Types) registry.  The
   value of the Protocol Type field in the Geneve header MUST be set to
   MPLS to use this approach.  The Geneve header MUST be immediately

Mirsky, et al.           Expires 21 October 2024               [Page 10]
Internet-Draft                OAM in GENEVE                   April 2024

   followed by the GAL label with the S flag set to indicate that GAL is
   the Bottom-of-the-stack label.  Then ACH MUST follow the GAL label
   and the value of the Channel Type identifies which of active OAM
   protocols being encapsulated in the packet.

Authors' Addresses

   Greg Mirsky

   Sami Boutros

   David Black
   Dell EMC
   176 South Street
   Hopkinton, MA,  01748
   United States of America

   Santosh Pallagatti

Mirsky, et al.           Expires 21 October 2024               [Page 11]