NVO3 Fault Management
draft-tissa-nvo3-oam-fm-01

The information below is for an old version of the document
Document Type Active Internet-Draft (individual)
Last updated 2015-01-27
Stream (None)
Intended RFC status (None)
Formats pdf htmlized bibtex
Stream Stream state (No stream defined)
Consensus Boilerplate Unknown
RFC Editor Note (None)
IESG IESG state I-D Exists
Telechat date
Responsible AD (None)
Send notices to (None)
NV03 Working Group                                    Tissa Senevirathne
Internet Draft                                        Samer Salam
Intended Status: Standard Track                       Deepak Kumar
                                                      Norman Finn

                                                                   Cisco
                                                      Donald Eastlake
                                                      Sam Aldrin
                                                                  Huawei
Expires July 2015                                     January 27, 2015

                         NVO3 Fault Management
                     draft-tissa-nvo3-oam-fm-01.txt

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79. 

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on July 31, 2015.

Copyright Notice

   Copyright (c) 2015 IETF Trust and the persons identified as the
   document authors.  All rights reserved.   

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
 

Senevirathne et al.      Expires July 31, 2015                  [Page 1]
Internet Draft           NVO3 Fault Management          January 27, 2015

   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License. 

Abstract          

   This document specifies Fault Management solution for network
   virtualization overlay networks. Methods in this document follow the
   IEEE 802.1 CFM framework and reuse OAM tools where possible.
   Additional messages and TLVs are defined for IETF overlay OAM
   specific applications or where extensions beyond IEEE 802.1 CFM are
   required.

Table of Contents
   1. Introduction  . . . . . . . . . . . . . . . . . . . . . . . . .  4
   2. Conventions used in this document . . . . . . . . . . . . . . .  5
   3. NV03 OAM Layers . . . . . . . . . . . . . . . . . . . . . . . .  6
   4. General Format of NV03 OAM frames . . . . . . . . . . . . . . .  6
     4.1. NV03 Shim . . . . . . . . . . . . . . . . . . . . . . . . .  8
     4.2. Identification of OAM frames  . . . . . . . . . . . . . . .  8
   5. Maintenance Associations (MA) in NVO3 . . . . . . . . . . . . .  9
   6. MEP Addressing  . . . . . . . . . . . . . . . . . . . . . . . . 10
     6.1. Use of MIP in NV03  . . . . . . . . . . . . . . . . . . . . 12
   7. Continuity Check Message (CCM)  . . . . . . . . . . . . . . . . 14
   8. NVO3 OAM Message Channel  . . . . . . . . . . . . . . . . . . . 16
     8.1. NVO3 OAM Message header . . . . . . . . . . . . . . . . . . 16
     8.2. IETF Overlay OAM Opcodes  . . . . . . . . . . . . . . . . . 17
     8.3. Format of IETF Overlay OAM TLV  . . . . . . . . . . . . . . 17
     8.4. IETF Overlay OAM TLVs . . . . . . . . . . . . . . . . . . . 18
       8.4.1. Common TLVs between 8201Q CFM and IETF Overlay OAM  . . 18
       8.4.2. IETF Overaly OAM Specific TLVs  . . . . . . . . . . . . 18
       8.4.3. OAM Application Identifier TLV  . . . . . . . . . . . . 19
       8.4.4. Out Of Band Reply Address TLV . . . . . . . . . . . . . 20
       8.4.5. Diagnostics Label TLV . . . . . . . . . . . . . . . . . 21
       8.4.6. Original Data Payload TLV . . . . . . . . . . . . . . . 21
       8.4.7. Flow Identifier (flow-id) TLV . . . . . . . . . . . . . 22
       8.4.8. Reflector Entropy TLV . . . . . . . . . . . . . . . . . 22
   9. Loopback Message  . . . . . . . . . . . . . . . . . . . . . . . 23
     9.2. Theory of Operation . . . . . . . . . . . . . . . . . . . . 24
       9.2.1. Actions by Originator . . . . . . . . . . . . . . . . . 24
       9.2.2. Intermediate Devices  . . . . . . . . . . . . . . . . . 24
       9.2.3. Destination Device  . . . . . . . . . . . . . . . . . . 24
 

Senevirathne et al.      Expires July 31, 2015                  [Page 2]
Internet Draft           NVO3 Fault Management          January 27, 2015

   10. Path Trace Message . . . . . . . . . . . . . . . . . . . . . . 25
     10.1. Theory of Operation  . . . . . . . . . . . . . . . . . . . 26
       10.1.1. Action by Originator Device  . . . . . . . . . . . . . 26
       10.1.2. Intermediate Device  . . . . . . . . . . . . . . . . . 27
       10.1.3. Destination Device . . . . . . . . . . . . . . . . . . 27
   11. Application of Continuity Check Message (CCM) in NVO3  . . . . 28
     11.1. CCM Error Notification . . . . . . . . . . . . . . . . . . 28
     11.2. Theory of Operation  . . . . . . . . . . . . . . . . . . . 29
       11.2.1. Actions by Originator Device . . . . . . . . . . . . . 30
       11.2.2. Intermediate Devices . . . . . . . . . . . . . . . . . 30
       11.2.3. Destination Device . . . . . . . . . . . . . . . . . . 30
   12. Security Considerations  . . . . . . . . . . . . . . . . . . . 31
   13. IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 31
   14. References . . . . . . . . . . . . . . . . . . . . . . . . . . 31
     14.1. Normative References . . . . . . . . . . . . . . . . . . . 31
     14.2. Informative References . . . . . . . . . . . . . . . . . . 31
   15. Acknowledgments  . . . . . . . . . . . . . . . . . . . . . . . 32
   Appendix A.                 Base Mode for NVO3 OAM . . . . . . . . 32

1. Introduction

   Conceptually, NVO3 architecture contains four separate layers,
   namely, Service (customer) Layer, Overlay Layer, Transport or
   Underlay Layer and Media Layer. Figure 1 below depicts the
   relationship between each of these layers.

   Fault Monitoring, Fault Verification, Fault Isolation, Loss and delay
   measurements are integral part of NVO3 [nvo3oamReq]. These need to
   cover both unicast and multicast traffic streams.

   For effective fault isolation, the overlay OAM solution should
   complement the OAM functions at adjacent layers, thereby leading  to
   nested OAM model. Nested OAM allows operators to quickly and
   effectively troubleshoot and isolate data plane failures using a
   common OAM framework.

   Common OAM message format and infrastructure makes it easier to
   accomplish nested OAM. IEEE Connectivity Fault Management (CFM)
   [8021Q] is widely used in the Ethernet world. The same technology has
   been extended by ITU-T [Y1731], Metro Ethernet Forum (MEF) and TRILL
   [TRILLFM].

   A common OAM message channel can be shared between different
   technologies. This consistency between different OAM technologies
   promotes nested fault monitoring and isolation between technologies
   that share the same OAM framework.
 

Senevirathne et al.      Expires July 31, 2015                  [Page 3]
Internet Draft           NVO3 Fault Management          January 27, 2015

   This document uses the message format defined in IEEE 802.1ag
   Connectivity Fault Management (CFM) [8021Q] as the basis for the OAM
   messages.

   This document presents the NVO3 OAM message structure, NVO3 frame
   identification and NV03 OAM tools. The NVO3 OAM Management
   Information Base (MIB) will be presented in a separate document.

   The ITU-T Y.1731 [Y1731] standard utilizes the same messaging format
   as [8021Q] and for OAM messages where applicable. This document takes
   a similar stance and reuses [8021Q] and TRILL OAM [TRILLFM]. It is
   assumed that readers are familiar with [8021Q] and [Y1731].

            |T |--|NVE|--|R|-|B|--|B|-|R|-|NVE|-|T|

       ---    X----o-----------------------o-----X [Service Layer]
      | O |        |                       |
      | A |        |                       |
      | M |        X------o-----------o----X      [NVo3 Layer]
      |   |               |           |
      | M |               |           |
      | I |               X--o-----o--X           [Transport Layer]
      | B |                  |     |
      |   |                  |     |
       ---                   X--o--X              [Link/Circuit Layer]

      X - Maintenance End Point (MEP)
      o - Maintenance Intermediate Point (MIP)

      T - Tenant System   NVE - Network Virtualization Edge
      R - Router           B   - Bridge

                        Figure 1 Layered OAM Architecture

2. Conventions used in this document

      The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", 
     "SHALL NOT","SHOULD", "SHOULD NOT", "RECOMMENDED", 
 

Senevirathne et al.      Expires July 31, 2015                  [Page 4]
Internet Draft           NVO3 Fault Management          January 27, 2015

     "MAY", and "OPTIONAL" in this
      document are to be interpreted as described in RFC-2119 [RFC2119].

      Acronyms used in the document include the following:

         MP - Maintenance Point [8021Q]

         MEP - Maintenance End Point [8021Q]

         MIP - Maintenance Intermediate Point [8021Q]

         MA - Maintenance Association [8021Q]

         MD - Maintenance Domain [8021Q]

         CCM - Continuity Check Message [8021Q]

         LBM - Loop Back Message [8021Q]

         PTM - Path Trace Message

         MTV - Multi-destination Tree Verification Message

         ECMP - Equal Cost Multipath

         ISS  - Internal Sub Layer Service [8021Q]

         VNI  - Virtual Network Instance [NVO3FRM]

         NVE - Network Virtual Edge [NVO3FRM]

         SAP - Service Access Point [8021Q]

3. NV03 OAM Layers

      Figure 1 above depicts different layers within NVO3. Each of these
      layers has a unique scope within the common framework. In this
      section, we define functionality of each of these layers

      Service Layer: Service Layer carries customer or user traffic. 
      It is originated at Tenant systems and terminates at other 
      Tenant system(s) within the same NVO3 context (VNI).

      NVO3 Layer: NVO3 Layer carries service Layer traffic 
      encapsulated in NVO3 format. NVO3 Layer originates at an 
      NVE and terminates at another NVE.

      Transport Layer: Transport Layer carries NVO3 Layer traffic
 

Senevirathne et al.      Expires July 31, 2015                  [Page 5]
Internet Draft           NVO3 Fault Management          January 27, 2015

      encapsulated in its data format. The transport Layer 
      can be IP, MPLS or any other protocol.

      Link Layer: This is also known as circuit layer. 
      It carries Transport Layer traffic in a specific manner 
      from one device to another.
      Ethernet is an example of link layer.

4. General Format of NV03 OAM frames

   For accurate monitoring and/or diagnostics, OAM Messages are required
   to follow the same data path as corresponding user packets.
   Additionally, NVEs are required to identify NVO3 frames and act on
   them, in addition to preventing the overlay OAM packets from leaking
   outside of the NVO3 domain [NVO3OAMREQ]. This document defines the
   format of the NVO3 overlay OAM messages.

      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                               |
      .    Circuit  Header               . (variable)
      |                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                               |
      +    Transport Header               Technology dependent
      |                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |       NVO3 Shim               |   8 bytes
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                               |
      .   Payload fragment            .   96 bytes (optional)
      .                               .
      |                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   OAM Ether Type              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                               |
      .   OAM Message Channel         . Variable
      .                               .
      |                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |     Link Trailer              | Variable
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 

Senevirathne et al.      Expires July 31, 2015                  [Page 6]
Internet Draft           NVO3 Fault Management          January 27, 2015

                  Figure 2 Format of NVO3 Overlay OAM Messages

   Link Header: Media-dependent header. For Ethernet, this includes
   Destination MAC, Source MAC, VLAN (optional) and EtherType fields.

   Transport Header: Header of the Transport Layer e.g. IP , MPLS, TRILL
   etc.

   NVO3 Shim: This is a fixed sized field (size TBD 8 bytes [vxLAN])
   that carries NVO3 specific information. Fields in this header will be
   utilized to identify OAM frames and other NVO3 specific operations.
   Please see section 4.2 below of NVO3 OAM specific operations.

   Payload Fragment: This is an optional field. When included it has a
   fixed size of 96 bytes. The least significant bits of the field MUST
   be padded with zeros, up to 96 bytes, when the payload fragment is
   less than 96 bytes. The Payload Fragment field starts with the
   Inner.MacDA.

   OAM Ether Type: OAM Ether Type is 16-bit EtherType that identifies
   the OAM Message channel that follows. This document specifies using
   the EtherType 0x8902 allocated for [8021Q] for this purpose.
   Identifying the OAM Message Channel with a dedicated EtherType allows
   the easy identification of the beginning of the OAM message channel
   across multiple standards.

   OAM Message Channel: This is a variable size section that carries OAM
   related information. The message format defined in [8021Q] will be
   reused.

   Link Trailer: Media-dependent trailer. For Ethernet, this is the FCS
   (Frame Check Sequence).

4.1. NV03 Shim

   NVO3 Shim is an 8 octet vector that carries series of flags and
   additional information [vxLAN]. Each of the flags identifies a
   specific operation.

      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |R|R|R|R|I|R|R|R|            Reserved                           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                VXLAN Network Identifier (VNI) |   Reserved    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                               Figure 3 NVO3 Shim
 

Senevirathne et al.      Expires July 31, 2015                  [Page 7]
Internet Draft           NVO3 Fault Management          January 27, 2015

4.2. Identification of OAM frames

   Implementations that comply with this document MUST utilize "O" flag
   to identify NVO3 OAM frames. The "O" flag MUST NOT BE utilized for
   forwarding decisions such as the selection of which ECMP path to use.

      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |O|F|R|R|I|R|R|R|            Reserved                           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                VXLAN Network Identifier (VNI) |   Reserved    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                      Figure 4 NVO3 shim with the "O" Flag

   O (1 bit) - Indicates this is a possible OAM frame and is subject to
   specific handling as specified in this document.

   F (1 bit) - When set to 1 indicates that 96 octets of payload
   fragments is included immediately after the NVO3 shim. When set 0
   indicates that OAM Ethertype 0x8902 immediately follows the NVO3
   shim.

   All other fields carry the same meaning as defined in [vXLAN].

5. Maintenance Associations (MA) in NVO3

   [8021Q] defines a maintenance association as a logical relationship
   between a group of nodes. Each Maintenance Association (MA) is
   identified with a unique MAID of 48 bytes [8021Q]. CCM and other
   related OAM functions operate within the scope of an MA. The
   definition of MA is technology independent. MA is encoded in the
   technology independent part of the OAM message Hence the MAID, as
   defined in [8021Q], can be utilized for NVo3 OAM, without
   modifications. This also allows us to utilize CCM and LBM messages
   defined in [8021Q], as is.

   In NVO3, an MA may contain two or more NVEs (MEPs). For unicast, it
   is likely that the MA contains exactly two MEPs that are the two end-
   points of the flow. For multicast, the MA may contain two or more
   MEPs.

   For NVO3, in addition to all of the standard CFM MIB [8021Q]
   definitions, each MEP's MIB contains one or more flow definitions
   corresponding to the set of flows that the MEP monitors. Flow entropy
   specifies the VNI within the NVE.

   MEPs can be created per VNI within the NVE.
 

Senevirathne et al.      Expires July 31, 2015                  [Page 8]
Internet Draft           NVO3 Fault Management          January 27, 2015

   We propose to augment the [8021Q] MIB to add NVO3 specific
   information. Figure 5, below depicts the augmentation of the CFM MIB
   to add the NVO3 specific Flow Entropy.

                MA---
               |
                --- MEP
               |
               . - Remote MEP List
                      .
                      |
                       --- MEP-A
                      |
                       --- MEP-B
                      .

               |
               . - Flow List { Augments IEEE8021-CFM-MIB}

                      |
                       --- (Flow-1)
                      |
                       --- (Flow-2)
                      |
                      . --- (Flow-n)
              |
               Other MIB entries

                   Figure 5 Correlation of NVO3 augmented MIB

 

Senevirathne et al.      Expires July 31, 2015                  [Page 9]
Internet Draft           NVO3 Fault Management          January 27, 2015

   The flow list contains multiple flows. Each flow contains a VNI and
   optional data payload. There can be more than one flow for a given
   VNI with different data payloads e.g. unicast vs. multicast or
   unicast with different data payloads.

6. MEP Addressing

   In IEEE 802.1ag [8021Q], OAM messages address the target MEP by
   utilizing a unique MAC address.  In NVO3, MEPs are created per VNI,
   MEPs are addressed by combination of Transport Layer address of the
   NVE and the VNI.

   At the MEP, OAM packets go through a hierarchy of op-code de-
   multiplexers. The op-code de-multiplexers channel the incoming OAM
   packets to the appropriate message processor (e.g. LBM) The reader
   may refer to Figure 6 below for a visual depiction of these different
   de-multiplexers.

   1. Identify the packets that need OAM processing at the Local Device
   as specified in Section 4.2.

        a.  Identify the MEP that is associated with the VNI.

   2. The MEP then validates the MD-LEVEL

        a.  Redirect to MD-LEVEL De-multiplexer

   3. MD-LEVEL de-multiplexer compares the MD-Level of the packet 
   against the MD level of the local MEPs. (Note: there can be more 
   than one MEP at the same MD-Level but belonging to different MAs)

        a.  If the packet MD-LEVEL is equal to the configured MD-LEVEL  
       of the MEP, then pass to the Opcode de-multiplexer

        b.  If the packet MD-LEVEL is less than the configured MD-LEVEL 
        of the MEP, discard the packet

        c.  If the packet MD-LEVEL is greater than the configured MD-   
      LEVEL of the MEP, then pass on to the next higher MD-LEVEL      
   de-multiplexer, if available. Otherwise, if no such higher       MD-
   LEVEL de-multiplexer exists, then forward the packet as       normal
   data.

   4. Opcode De-multiplexer compares the opcode in the packet with the 
   supported opcodes

        a.  If Op-code is CCM, LBM, LBR, PTM, PTR, MTVM, MTVR, then pass
 

Senevirathne et al.      Expires July 31, 2015                 [Page 10]
Internet Draft           NVO3 Fault Management          January 27, 2015

         on to the correct Processor

        b.  If Op-code is Unknown, then discard.

                                  |
                                  .CCM   LBM   PTM    MTV
                                  |      |    |      |
                                +-+-+-+-+-+-+-+-+-+-+-+-+
                                |        OP Code DE-Mux |--- Unknown
                                +-+-+-+-+-+-+-+-+-+-+-+-+
                                  ^       ^          ^
                        MD==Li    |       |          |
                               +-+-+   +-+-+      +-+-+
                               | L |-->|L2 |-.-   |Ln |---- >
                               +-+-+   +-+-+      +-+-+      |
                                |  ^    |          |         |
                        MD<LI Drop |    Drop       Drop      |
                                   |                         |
                        MD not --- |NVO3 OAM need local      |
                        Present    | Processing              |
                                   |                         |
                      NVO3 Data   ----  NVO3 Data          ----
                         ------->| T  |----------------- >|  M |--- >
                      + NVO3 OAM  ----  + pass through OAM ----

               Figure 6 OAM De-Multiplexers at MEP for active SAP

        T : Denotes Tap, that identifies OAM frames that need local
        processing. These are the packets with OAM flag set AND OAM
        Ether type is present after the flow entropy of the packet

 

Senevirathne et al.      Expires July 31, 2015                 [Page 11]
Internet Draft           NVO3 Fault Management          January 27, 2015

        M : Is the post processing merge, merges data and OAM messages
        that are passed through. Additionally, the Merge component
        ensures, as explained earlier, that OAM packets are not
        forwarded out as native frames.

        L : Denotes MD-Level processing. Packets with MD-Level less than
        the Level will be dropped. Packets with equal MD-Level are
        passed on to the opcode de-multiplexer. Others are passed on to
        the next level MD processors or eventually to the merge point
        (M).

        NOTE: LBM, MTV and PT are not subject to MA de-multiplexers.
        These packets do not have an MA encoded in the packet. Adequate
        response can be generated to these packets, without loss of
        functionality, by any of the MEPs.

6.1. Use of MIP in NV03

    Maintenance Intermediate Points (MIP) are mainly used for fault
    isolation. Link Trace Messages in [8021Q] utilize a well-known
    multicast MAC address and MIPs generate responses to Link Trace
    messages. Response to Link Trace messages or lack thereof can be
    used for fault isolation in NVO3.

    As explained in section 10. below, a TTL expiry approach will be
    utilized for fault isolation and path tracing. The approach is very
    similar to the well-known IP trace-route approach. Hence, explicit
    addressing of MIPs is not required for the purpose of fault
    isolation.

    Any given NVO3 device can have multiple MIPs located within the
    device. As such, a mechanism is required to identify which MIP
    should respond to an incoming OAM message.

    A similar approach to that presented above for MEPs can be used for
    MIP processing. It is important to note that "M", the merge block of
    a MIP, does not prevent OAM packets leaking out as native frames. On
    edge interfaces, MEPs MUST be configured to prevent the leaking of
    overlay OAM packets out of the NVO3 domain.

                                                 PT     MTV
                                                  |      |
                                 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                 |     OP Code De-Mux          |-> Unknown
                                 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
                                   ^       ^          ^
                         MD==Li    |       |          |
 

Senevirathne et al.      Expires July 31, 2015                 [Page 12]
Internet Draft           NVO3 Fault Management          January 27, 2015

                                +-+-+   +-+-+      +-+-+
                                | L |- >|L2 |-.-   |Ln |------+
                                +-+-+   +-+-+      +-+-+      |
                                    ^                         |
                                    |                         |
                         Drop       |                         |
                         MD not --- |NVO3 OAM                 |
                         Present    |                         |
                                    |                         v
                       NVO3 Data   ----  NVO3 Data            -----
                          ------- >| T  |------------------ >|  M  |--->
                       + NVO3  OAM  ----                      ----

                Figure 7 OAM De-Multiplexers at MIP for active SAP

    T: TAP processing for MIP. All packets with OAM flag set are
    captured.

    L : MD Level Processing, Packet with matching MD Level are "copied"
    to the Opcode de-multiplexer and original packet is passed on to the
    next MD level processor. Other packets are simply passed on to the
    next MD level processor, without copying to the OP code de-
    multiplexer.

    M : Merge processor, merge OAM packets to be forwarded along with
    the data flow.

    Packets that carry Path Trace Message (PTM) or Multi-destination
    Tree Verification (MTV) OpCodes are passed on to the respective
    processors.

    Packets with unknown OpCodes are counted and discarded.

7. Continuity Check Message (CCM)

    CCMs are used to monitor connectivity and configuration errors.
    [8021Q] monitors connectivity by having a MEP listening to periodic
    CCM messages received from its remote MEP partners in the MA. An
    [8021Q] MEP identifies cross-connect errors by comparing the MAID in
    the received CCM message with the MEP's local MAID. The MAID [8021Q]
    is a 48-byte field that is technology independent. Similarly, the
    MEPID is a 2-byte field that is independent of the technology. Given
    this generic definition of CCM fields, CCM as defined in [8021Q] can
    be utilized in NVO3 with no changes. NVO3 specific information may
    be carried in CCMs when encoded using IETF overlay specific TLVs or
    sub- TLVs. This is possible since CCMs are capable of carrying
 

Senevirathne et al.      Expires July 31, 2015                 [Page 13]
Internet Draft           NVO3 Fault Management          January 27, 2015

    optional TLVs.

    Unlike classical Ethernet environments with spanning tree, NVO3
    supports multipath forwarding. The path taken by a packet depends on
    the Transport header and other parts of the packet. The Maintenance
    Association identifies the interested end-points (MEPs) of a given
    monitored path. For unicast there are only two MEPs per MA. For
    multicast there can be two or more MEPs in the MA. The Flow (i.e VNI
    and other parameters) values of the monitored flows are defined
    within the MA. CCM transmit logic will utilize these flow entropy
    values when constructing the CCM packets. Please see below for the
    theory of operation of CCM.

    The CFM MIB of [8021Q] will be augmented with the definition of
    flow- entropy. Please see [TBDMIBD] for these and other NVO3 related
    OAM MIB definitions. The figure below depicts the correlation
    between MA, CCM and the flow.

              MA---
             |
              --- MEP
             |
             . - Remote MEP List
                    .
                    |
                     --- MEP-A
                    |
                     --- MEP-B
                    .

             |
             . - Flow List {Augments IEEE8021-CFM-MIB}

                    |
                     --- (Flow-1) {note we have to define
                    |                      destination NVE with
                     --- (Flow-2)  the flow entropy discuss}
                    |
                    . ---(Flow-n)
            |
            . - CCM
                   |
                    --- (standard 8021ag entries)
                   |
 

Senevirathne et al.      Expires July 31, 2015                 [Page 14]
Internet Draft           NVO3 Fault Management          January 27, 2015

                    --- (TTL) { Augments IEEE8021-CFM-MIB}
                   |
                    --- (Other TBD NVO3 OAM specific entries)
                                                    {Augmented}
            |
            .
            |
             - Other MIB entries

                  Figure 8 Augmentation of CCM MIB in NVO3

    NOTE: Flow entropy field contain VNI and flow specific information
    that affect the path selection and forwarding.

    In a multi-pathing environment, a Flow - by definition - is
    unidirectional. A question may arise as to what flow entropy should
    be used in the response. CCMs are unidirectional and have no
    explicit reply; as such, the issue of the response flow entropy does
    not arise. In the transmitted CCM, each MEP reports local status
    using the Remote Defect Indication (RDI) flag. Additionally, a MEP
    may raise SNMP TRAPs [TBDMIB] as Alarms when a connectivity failure
    occurs.

8. NVO3 OAM Message Channel

    The NVO3 OAM Message Channel can be divided into two parts: NVO3 OAM
    Message header and NVO3 OAM Message TLVs. Every OAM Message MUST
    contain a single OAM message header and a set of one or more
    specified OAM Message TLVs.

8.1. NVO3 OAM Message header

    As discussed earlier, a common messaging framework between [8021Q],
    NVO3, and other similar standards such as Y.1731 can be accomplished
    by re-using the OAM message header defined in [8021Q].

    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |MD-L | Version | OpCode        |  Flags        |FirstTLVOffset |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                                                               |
    |                                                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                                                               |
    |                                                               |
 

Senevirathne et al.      Expires July 31, 2015                 [Page 15]
Internet Draft           NVO3 Fault Management          January 27, 2015

    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                         Figure 9 OAM Message Format

      o  MD-L: Maintenance Domain Level (3 bits). Identifies the    
    maintenance domain level. For NVO3, in general, this field is    
    set to a single value. If using Base Mode operations as defined    
    in Appendix B, this field is set to 3. However, future    
    extensions of NVO3, for example to support hierarchy, may create    
    different MD-LEVELs and MD-L field must be appropriately set in    
    those scenarios. (Please refer to [8021Q] for the definition of    
    MD-Level)  o  Version: Indicates the version (5 bits), as specified
    in     [8021Q]. This document does not require changing the Version 
       defined in [8021Q].

      o  Flags: Includes operational flags (1 byte). The definition of  
      flags is Opcode-specific and is covered in the applicable    
    sections.

      o  FirstTLVOffset: Defines the location of the first TLV, in    
    bytes, starting from the end of the FirstTLVOffset field (1    
    byte). (Refer to [8021Q] for the definition of the    
    FirstTLVOffset.)

    MD-L, Version, Opcode, Flags and FirstTLVOffset fields collectively
    are referred to as the OAM Message Header.

    The Opcode specific information section of the OAM Message may
    contain Session Identification number, time-stamp, etc.

8.2. IETF Overlay OAM Opcodes

    The following Opcodes are defined for IETF Overlay OAM. Each of the
    Opcodes indicates a separate type of OAM message. Details of the
    messages are presented in the related sections.

    IETF OAM Message Opcodes:

    TBD-64 : Path Trace Reply TBD-65 : Path Trace Message TBD-66 :
    Multicast Tree Verification Reply TBD-67 : Multicast Tree
    Verification Message

8.3. Format of IETF Overlay OAM TLV

    The same TLV format as defined in section 21.5.1 of [8021Q] is used
    for IETF Overlay OAM. The following figure depicts the general
 

Senevirathne et al.      Expires July 31, 2015                 [Page 16]
Internet Draft           NVO3 Fault Management          January 27, 2015

    format of a IETF Overlay OAM TLV:

     0                   1                   2
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |    Type       |        Length                 |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                                               |
    |                                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                           Figure 10 NVO3 OAM TLV

    Type (1 octet): Specifies the Type of the TLV (see sections 8.4. 
    for TLV types).

    Length (2 octets): Specifies the length of the 'Value' field in
    octets. Length of the 'Value' field can be either zero or more
    octets.

    Value (variable): The length and the content of this field depend on
    the type of the TLV. Please refer to applicable TLV definitions for
    the details.

    Semantics and usage of Type values allocated for TRILL OAM purpose
    are defined by this document and other future related documents.

8.4. IETF Overlay OAM TLVs

    Overlay related TLVs are defined in this section. Previously defined
    [8021Q] TLVs are reused, where applicable. Block of 32 TLVs are
    allocated for the purpose of IETF defined standards

8.4.1. Common TLVs between 8201Q CFM and IETF Overlay OAM

    The following TLVs are defined in [8021Q]. We propose to re-use them
    where applicable. The format and semantics of the TLVs are as
    defined in [8021Q].

    Type    Name of TLV in [8021Q]
    ----   -------------
      0    End TLV
      1    Sender ID TLV
      2    Port Status TLV
      3    Data TLV
      4    Interface Status TLV
      5    Reply Ingress TLV
 

Senevirathne et al.      Expires July 31, 2015                 [Page 17]
Internet Draft           NVO3 Fault Management          January 27, 2015

      6    Reply Egress TLV
      7    LTM Egress Identifier TLV
      8    LTR Egress Identifier TLV
      9-30 Reserved
      31   Organization Specific TLV

8.4.2. IETF Overaly OAM Specific TLVs

    As indicated above, a block of 32 TLVs will be requested to be
    reserved for IETF OAM purposes. Listed below is a summary of the
    IETF Overlay OAM TLVs and their corresponding codes. Format and
    semantics of OAM TLVs are defined in subsequent sections.

      Type           TLV Name
    -----------    ----------------------
     TBD-TLV-64     OAM Application Identifier
     TBD-TLV-65     Out of Band IP Address
     TBD-TLV-66     Original Payload
     TBD-TLV-67     Diagnostic VLAN
     TBD-TLV-68     scope
     TBD-TLV-69     Previous Device address
     TBD-TLV-70     Next Hop Device List (ECMP)
     TBD-TLV-71     Multicast Receiver Availability
     TBD-TLV-72     Flow Identifier
     TBD-TLV-73     Reflector Entropy
     TBD-TLV-74 to TBD-TLV-95  Reserved

8.4.3. OAM Application Identifier TLV

    OAM Application Identifier TLV carries Overlay OAM application
    specific information. The Overlay OAM Application Identifier TLV
    MUST always be present and MUST be the first TLV in the OAM
    messages. Messages that do not include the OAM Application
    Identifier TLV as the first TLV MUST be discarded.

        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |    Type       | Length                        | Version       |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |  Return Code  |Return sub-code| Protocol      |Rsvd   |F|C|O|I|
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                     Figure 11 OAM Application Identifier TLV

    Type (1 octet) = TBD-TLV-64 indicate that this is the IETF Overlay
    OAM Application Identifier TLV

 

Senevirathne et al.      Expires July 31, 2015                 [Page 18]
Internet Draft           NVO3 Fault Management          January 27, 2015

    Length (2 octets) = 6

    IETF Overlay OAM Version (1 Octet), currently set to zero. Indicates
    the Overlay OAM version. IETF Overlay OAM version can be different
    than the [8021Q] version.

    Return Code (1 Octet): Set to zero on requests. Set to an
    appropriate value in response messages.

    Return sub-code (1 Octet): Return sub-code is set to zero on
    transmission of request message. Return sub-code identifies
    categories within a specific Return code. Return sub-code MUST be
    interpreted within a Return code.

    Protocol: This indicates the overlay protocol on which the OAM is
    applied. In this document we cover NVO3

       0 : TRILL

       1 : NVO3

       2 - 255 : reserved

    F (1 bit): Final flag, when set, indicates this is the last
    response.

    C (1 bit): Label error, if set indicates that the label (VLAN) in
    the flow entropy is different than the label included in the
    diagnostic TLV.  This field is ignored in request messages and MUST
    only be interpreted in response messages.

    O (1 bit): If set, indicates, OAM out-of-band response requested.

    I (1 bit): If set, indicates, OAM in-band response requested.

    NOTE: When both O and I bits are set to zero, indicates that no
    response is required (silent mode). User MAY specify both O and I or
    one of them or none. When both O and I bits are set response is sent
    both in-band and out-of-band.

8.4.4. Out Of Band Reply Address TLV

       Out of Band Reply Address TLV specifies the address to which an
    out   of band OAM reply message MUST be sent. When O bit in the
    Application   Identifier TLV is not set, Out of Band Reply Address
    TLV is ignored.

                            1                   2                   3
 

Senevirathne et al.      Expires July 31, 2015                 [Page 19]
Internet Draft           NVO3 Fault Management          January 27, 2015

        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |    Type       | Length                        | Address Type  |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       | Addr Length   |                                               |
       +-+-+-+-+-+-+-+-+                                               |
       |                                                               |
       .       Reply Address                                     .
       |                                                               |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                       Figure 12 Out of Band IP Address TLV

    Type (1 octet) = TBD-TLV-65

    Length (2 octets) = Variable. Minimum length is 2.

    Address Type (1 Octet): 0 - IPv4. 1 - IPv6. All other values
    reserved.

    Addr Length (1 Octet). 4 - IPv4. 16 - IPv6,

    Reply Address (variable): Address where the reply needed to be sent.
    Length depends on the address specification.

8.4.5. Diagnostics Label TLV

    Diagnostic label specifies the VNI which the OAM messages are
    generated. Receiving device MUST compare the VNI of the NVO3 shim to
    that of Diagnostic TLV. Label Error Flag in the response MUST be set
    when the two VLANs do not match.

     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |    Type       | Length                        | L-Type        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    | Reserved      |                       Label(VLAN)             |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                           Figure 13 Diagnostic TLV

     Type (1 octet) = TBD-TLV-66 indicates that this is the Diagnostic
    TLV

     Length (2 octets) = 5
 

Senevirathne et al.      Expires July 31, 2015                 [Page 20]
Internet Draft           NVO3 Fault Management          January 27, 2015

     L-Type (Label type, 1 octet)

        0- indicate 802.1Q 12 bit VLAN.

           2 - 24 bit ISID

           3 - VNI 24 bits

     Label (24 bits): Either 12 bit VLAN or 24 bit ISID or 24 bit VNI.

     Devices should perform Label error checking when Label TLV is not
    included in the OAM message.

8.4.6. Original Data Payload TLV

     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |    Type       | Length                        |               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+               +
    |                                                               |
    |                                                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                     Figure 14 Original Data Payload TLV

    Type (1 Octet) = TBD-TLV-67

    Length (2 octets) =  variable

8.4.7. Flow Identifier (flow-id) TLV

    Flow Identifier (flow-id) uniquely identifies a specific flow. The
    flow-id value is unique per MEP and needs to be interpreted as such.

                         1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |    Type       | Length                        | Reserved      |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |  MEP-ID                       |     flow-id                   |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                       Figure 15 Flow Identifier  TLV

    Type (1 octet) = TBD-TLV-72 Length (2 octets) = 5.

 

Senevirathne et al.      Expires July 31, 2015                 [Page 21]
Internet Draft           NVO3 Fault Management          January 27, 2015

    Reserved (1 octet) set to 0 on transmission and ignored on
    reception.

    MEP-ID (2 octets) = MEP-ID of the originator [8021Q].

    Flow-id (2 octets) = uniquely identifies the flow per MEP. Different
    MEPs may allocate the same flow-id value. The {MEP-ID, flow-id} pair
    is globally unique.

    Inclusion of the MEP-ID in the flow-id TLV allows inclusion of MEP-
    ID for messages that do not contain MEP-ID in OAM header.
    Applications may use MEP-ID information for different types of
    troubleshooting.

8.4.8. Reflector Entropy TLV

    Reflector Entropy TLV is an optional TLV. This TLV, when present,
    tells the responder to utilize the Reflector Entropy specified
    within the TLV as the flow-entropy of the response message.

                            1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |    Type       | Length                        | Reserved      |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                                                               |
       .               Reflector Entropy                               .
       |                                                               |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                         Figure 16 Reflector Entropy  TLV

    Type (1 octet) =TBD-TLV-73 Reflector Entropy TLV.

     Length (1 octet) =97.

     Reserved (1 octet) = set to zero on transmission and ignored by the
    recipient.

     Reflector Entropy (96-octet) = Flow Entropy to be used by the
    responder. May be padded with zero if the desired flow entropy is
    less than 96 octets.

9. Loopback Message

    9.1. Loopback OAM Message format

 

Senevirathne et al.      Expires July 31, 2015                 [Page 22]
Internet Draft           NVO3 Fault Management          January 27, 2015

                            1                   2                   3
        0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |MD-L | Version | OpCode        |  Flags        |FirstTLVOffset |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                   Loopback Transaction Identifier             |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
       |                                                               |
       .         TLVs                                                  .
       |                                                               |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                       Figure 17 Loopback OAM Message Format

    The above figure depicts the format of the Loopback Request and
    response messages as defined in [8021Q]. The Opcode for Loopback
    Message is set to 65 and the Opcode for the Reply Message is set to
    64. The Session Identification Number is a 32-bit integer that
    allows the requesting Device to uniquely identify the corresponding
    session. Responding Device, without modification, MUST echo the
    received "Loopback Transaction Identifier" number..

9.2. Theory of Operation

9.2.1. Actions by Originator

    Identifies the destination NVE address based on user specification
    or based on the specified destination, VNI and MAC

    Constructs the flow entropy based on user specified parameters or
    implementation specific default parameters.

    Constructs the OAM header: sets the opcode to Loopback message type
    (3). Assign applicable Loopback Transaction Identifier number for
    the request.

    The Overlay OAM Version TLV MUST be included and with the flags set
    to applicable values.

    Include following OAM TLVs, where applicable

      o  Out-of-band Reply address TLV

      o  Diagnostic Label TLV

 

Senevirathne et al.      Expires July 31, 2015                 [Page 23]
Internet Draft           NVO3 Fault Management          January 27, 2015

      o  Sender ID TLV

    Specify the Transport Layer parameters based on user inputs or
    default settings.

    Dispatch the OAM frame for transmission.

    Devices may continue to retransmit the request at periodic
    intervals, until a response is received or the re-transmission count
    expires. At each transmission Session Identification number MUST be
    incremented.

9.2.2. Intermediate Devices

    Intermediate Devices forward the frame as a normal data frame and no
    special handling is required.

9.2.3. Destination Device

    If the Loopback message is addressed to the local Device and
    satisfies the OAM identification criteria specified in section 4.2.
    then, the Device data plane forwards the message to the CPU for
    further processing.

    The OAM application layer further validates the received OAM frame
    by checking for the presence of OAM-Ethertype and the MD Level.
    Frames that do not contain OAM-Ethertype MUST be discarded.

    Construction of the OAM response:

    OAM application encodes the received Transport header, NVO3 shim and
    payload fragment (when present) in the Original payload TLV and
    includes it in the OAM message.

    Set the Return Code and Return sub code to applicable values. Update
    the OAM opcode to 2 (Loopback Message Reply)

    Optionally, if the VNI identifier value of the received differs from
    the value specified in the diagnostic Label, set the Label Error
    Flag on OAM Application Identifier TLV.

    Include the sender ID TLV (1)

    If in-band response was requested, dispatch the frame to theNVO3
    data plane with request-originator Transport Layer address as the
    destination address.

    If out-of-band response was requested, dispatch the frame to the IP
 

Senevirathne et al.      Expires July 31, 2015                 [Page 24]
Internet Draft           NVO3 Fault Management          January 27, 2015

    forwarding process.

10. Path Trace Message

    The primary use of the Path Trace Message is for fault isolation. It
    may also be used for plotting the path taken from a given Device to
    another Device.

    [8021Q] accomplishes the objectives of the NVO3 Path Trace Message
    using Link Trace Messages. Link Trace Messages utilize a well-known
    multicast MAC address. However, in NOV3 the transport Layer can be
    different technologies such as IP or MPLS etc. Hence, a definition
    of a new Path Trace message format is required for Overlay OAM. The
    Path Trace message is defined for the purpose.

    The Path Trace Message has the same format as Loopback Message, but
    utilizes a different opcode set. Opcode for Path Trace Reply Message
    is TBD-64 and Request Message is TBD-65

    Operation of the Path Trace message is identical to the Loopback
    message except that it is first transmitted with a TTL field value
    of 1. The sending device expects a Time Expiry Return-Code from the
    next hop or a successful response. If a Time Expiry Return-code is
    received as the response, the originator Device records the
    information received from intermediate node that generated the Time
    Expiry message and resends the message by incrementing the previous
    TTL value by 1. This process is continued until, a response is
    received from the destination Device or Path Trace process timeout
    occur or TTL reaches a configured maximum value.

10.1. Theory of Operation

10.1.1. Action by Originator Device Identifies the destination NVE
    address based on user specification or based on the specified
    destination, VNI and MAC

    Constructs the NVO3 shim and the flow based on user specified
    parameters or implementation specific default parameters.

    Construct the OAM header: Set the opcode to Path Trace Request
    message type (TBD-65). Assign an applicable Session Identification
    number for the request. Return-code and sub-code MUST be set to
    zero.

 

Senevirathne et al.      Expires July 31, 2015                 [Page 25]
Internet Draft           NVO3 Fault Management          January 27, 2015

    The OAM Application Identifier TLV MUST be included and set the
    flags to applicable values.

    Include following OAM TLVs, where applicable

      o  Out-of-band IP address TLV

      o  Diagnostic Label TLV

      o  Include the Sender ID TLV

    Specify the TTL of the transport header as 1 for the first request.

    Dispatch the OAM frame to the data plane for transmission.

    A Device (MEP) may continue to retransmit the request at periodic
    intervals, until a response is received or the re-transmission count
    expires. At each new re-transmission, the Session Identification

    number MUST be incremented. Additionally, for responses received
    from intermediate devices (MIP), the device address and interface
    information MUST be recorded.

10.1.2. Intermediate Device

    Path Trace Messages transit through Intermediate devices
    transparently, unless Hop-count has expired. OAM application layer
    further validates the received OAM frame by examining the presence
    of NVO3 OAM Flag and OAM-Ethertype and by examining the MD Level.
    Frames that do not contain OAM-Ethertype MUST be discarded.

    Construction of the OAM response:

    OAM application encodes the received Transport header and flow
    entropy in the Original payload TLV and include it in the OAM
    message.

    Set the Return Code to (2) "Time Expired" and Return sub code to
    zero (0). Update the OAM opcode to 64 (Path Trace Message  Reply).

    If the VNI identifier value of the received OAM message differs from
    the value specified in the diagnostic Label, set the Label Error
    Flag on OAM Application Identifier TLV.

    Include following TLVs
 

Senevirathne et al.      Expires July 31, 2015                 [Page 26]
Internet Draft           NVO3 Fault Management          January 27, 2015

    Reply Ingress TLV (5)

    Reply Egress TLV (6)

    Interface Status TLV (4)

    Sender ID TLV (1)

    If Label error detected, set C flag (Label error detected) in the
    version.

    If in-band response was requested, dispatch the frame to the NVO3
    data plane with request-originator Transport address as the
    destination address.

    If out-of-band response was requested, dispatch the frame to the
    standard IP forwarding process.

10.1.3. Destination Device

    Processing is identical to Loop back response processing in section
    9.2.3. with the exception that OAM Opcode is set to Path Trace Reply
    (64).

11. Application of Continuity Check Message (CCM) in NVO3

    Section 7. provides an overview of CCM Messages defined in [8021Q]
    and how they can be used within the NVO3 OAM. This section, presents
    the application and Theory of Operations of CCM within the NVO3 OAM
    framework. Readers are referred to [8021Q] for CCM message format
    and applicable TLV definitions and usages. Only the NVO3 specific
    aspects are explained below.

    In NVO3, between any two given MEPs there can be multiple potential
    paths. Whereas in [8021Q], there is always a single path between any
    two MEPs at any given time. It is important that solutions to have
    the ability to monitor continuity over one or more paths.

    CCM Messages are uni-directional, such that there is no explicit
    response to a received CCM message. Connectivity fault status is
    indicated by setting the applicable flags (e.g. RDI) of the CCM
    messages transmitted by an MEP.

    It is important that the solution presented in this document
    accomplishes the requirements specified in [NVO3OAMREQ] within the
    framework of [8021Q] in a straightforward manner and with minimum
    changes. Section 8 above defines multiple flows within the CCM
    object, each corresponding to a flow that a given MEP wishes to
 

Senevirathne et al.      Expires July 31, 2015                 [Page 27]
Internet Draft           NVO3 Fault Management          January 27, 2015

    monitor.

    Receiving MEPs do not cross check whether a received CCM belongs to
    a specific flow from the originating MEP. Any attempt to track
    status of individual flows may explode the amount of state
    information that any given MEP has to maintain.

    The obvious question arises: How does the originating device know
    which flow or flows are at fault?

    This is accomplished with a combination of the RDI flag in the CCM
    header, flow-id TLV, and SNMP Notifications (Traps). Section 11.1
    below discusses the procedure.

11.1. CCM Error Notification

    Each MEP transmits 4 CCM messages per each flow. ([8021Q] detects
    CCM fault when 3 consecutive CCM messages are lost). Each CCM
    Message has a unique sequence number and unique flow-identifier. The
    flow identifier is included in the OAM message via flow-id TLV.

    When an MEP notices a CCM timeout from a remote MEP ( MEP-A), it
    sets the RDI flag on the next CCM message it generates.
    Additionally, it logs and sends SNMP notification that contain the
    remote MEP Identification, flow-id and the Sequence Number of the
    last CCM message it received and if available, the flow-id and the
    Sequence Number of the first CCM message it received after the
    failure. Each MEP maintains a unique flow-id per each flow, hence
    the operator can easily identify flows that correspond to the
    specific flow-id.

    The following example illustrates the above.

    Assume there are two MEPs, MEP-A and MEP-B.

    Assume there are 3 flows between MEP-A and MEP-B.

    Let's assume MEP-A allocates sequence numbers as follows

    Flow-1 Sequence={1,2,3,4,13,14,15,16,.. } flow-id=(1)

    Flow-2 Sequence={5,6,7,8,17,18,19,20,.. } flow-id=(2)

    Flow-3 Sequence={9,10,12,11,21,22,23,24,.. } flow-id=(3)

    Let's Assume Flow-2 is at fault.

    MEP-B, receives CCM from MEP-A with sequence numbers 1,2,3,4, but
 

Senevirathne et al.      Expires July 31, 2015                 [Page 28]
Internet Draft           NVO3 Fault Management          January 27, 2015

    did not receive 5,6,7,8. CCM timeout is set to 3 CCM intervals in
    [8021Q]. Hence MEP-B detects the error at the 8'th CCM message. At
    this time the sequence number of the last good CCM message MEP-B has
    received from MEP-A is 4 and flow-id of the last good CCM Message is
    (1). Hence MEP-B will generate a CCM error SNMP notification with
    MEP-A and Last good flow-id (1) and sequence number 4.

    When MEP-A switches to flow-3 after transmitting flow-2, MEP-B will
    start receiving CCM messages. In the foregoing example it will be
    CCM message with Sequence Numbers 9,10,11,12,21 and so on. When in
    receipt of a new CCM message from a specific MEP, after a CCM
    timeout, the NVO3 OAM will generate an SNMP Notification of CCM
    resume with remote MEP-ID and the first valid flow-id and the
    Sequence number after the CCM timeout. In the foregoing example, it
    is MEP-A, flow-id (3) and Sequence Number 9.

    The remote MEP list under the CCM MIB Object is augmented to contain
    "Last Sequence Number", flow-id and "CCM Timeout" variables. Last
    Sequence Number and flow-id are updated every time a CCM is received
    from a remote MEP. CCM Timeout variable is set when the CCM timeout
    occurs and is cleared when a CCM is received.

11.2. Theory of Operation

11.2.1. Actions by Originator Device

    Derive the VNI and entropy based on flow entropy specified in the
    CCM Management object.

    Construct the NVO3 CCM OAM header as specified in [8021Q].

    OAM Version TLV MUST be included as the first TLV and set the flags
    to applicable values.

    Include other TLVs specified in [8021Q]

    Include the following optional OAM TLVs, where applicable

      o  Sender ID TLV

    Specify the TTL of the NVO3 data frame per user specification or
    utilize an applicable Hop count value.

    Dispatch the OAM frame to the NVO3 data plane for transmission.

    An MEP transmits a total of 4 requests, each at CCM retransmission
    interval. At each transmission the Session Identification number
    MUST be incremented by one.
 

Senevirathne et al.      Expires July 31, 2015                 [Page 29]
Internet Draft           NVO3 Fault Management          January 27, 2015

    At the 5'th retransmission interval, flow entropy of the CCM packet
    is updated to the next flow entropy specified in the CCM Management
    Object. If current flow entropy is the last flow entropy specified,
    move to the first flow entropy specified and continue the process.

11.2.2. Intermediate Devices

       Intermediate devices forward the frame as a normal data frame and
    no   special handling is required.

11.2.3. Destination Device

    If the CCM Message is addressed to the local MEP or multicast and
    satisfies OAM identification methods specified in sections 4.2. 
    then the data plane forwards the message to the CPU for further
    processing.

    The OAM application layer further validates the received OAM frame
    by examining the presence of OAM-Ethertype at the end of the flow
    entropy. Frames that do not contain OAM-Ethertype at the end of the
    flow entropy MUST be discarded.

    Validate the MD-LEVEL and pass the packet to the Opcode de-
    multiplexer. The Opcode de-multiplexer delivers CCM packets to the
    CCM process.

    The CCM Process performs processing specified in [8021Q].

    Additionally the CCM process updates the CCM Management Object with
    the sequence number of the received CCM packet. Note: The last
    received CCM sequence number and CCM timeout are tracked per each
    remote MEP.

    If the CCM timeout is true for the sending remote MEP, then clear
    the CCM timeout in the CCM Management object and generate the SNMP
    notification as specified above.

12. Security Considerations

    Specific security considerations related methods presented in this
    document are currently under investigation.

13. IANA Considerations

    IANA is requested to allocate 4 CFM Op-codes and 10 CFM TLV types
    and enter them into the IANA "CFM OAM IETF Op-codes" and "CFM OAM
    IETF TLV Types" sub-reigistries:

 

Senevirathne et al.      Expires July 31, 2015                 [Page 30]
Internet Draft           NVO3 Fault Management          January 27, 2015

    TBD values

14. References

14.1. Normative References

    [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate       
      Requirement Levels", BCP 14, RFC 2119, March 1997.

    [8021Q] IEEE, "Media Access Control (MAC) Bridges and Virtual       
      Bridged Local Area Networks", IEEE Std 802.1Q-2011, August,       
      2011.

14.2. Informative References

    [RFC4379] Kompella, K. et.al, "Detecting Multi-Protocol Label       
      Switched (MPLS) Data Plane Failures", RFC 4379, February         
    2006.

    [RFC6291] Andersson, L., et.al., "Guidelines for the use of the
    "OAM"          Acronym in the IETF" RFC 6291, June 2011.

    [Y1731] ITU, "OAM functions and mechanisms for Ethernet based       
      networks", ITU-T G.8013/Y.1731, July, 2011.

    [NVO3FRM] Lasserre, M., et.al., "Framework for DC Network         
    Virtualization", Work in Progress, January, 2014.

    [NVO3ARC] Black, D., et.al., "Architecture for Overlay Networks     
        (NVO3)", Work in Progress, December, 2013.

    [NVO3OAMREQ] Ashwood-Smith, P., "NVO3 Operations, Administrations
    and          Maintenance Requirements", work in progress, January,
    2014.

    [NV03DPREQ] Bitar, N., "NVO3 Data Plane Requirements", Work in      
       Progress, November, 2013.

15. Acknowledgments

       This document was prepared using 2-Word-v2.0.template.dot.

 

Senevirathne et al.      Expires July 31, 2015                 [Page 31]
Internet Draft           NVO3 Fault Management          January 27, 2015

Appendix A.                 Base Mode for NVO3 OAM

    CFM, as defined in [8021Q], requires configuration of several
    parameters before the protocol can be used. These parameters include
    MAID, Maintenance Domain Level (MD-LEVEL) and MEPIDs. The Base Mode
    for NVO3 OAM defined here facilitates ease of use and provides out
    of the box plug-and-play capabilities.

    All NVO3 compliant devices that support OAM specified in this
    document MUST support Base Mode operation.

    All NVO3 compliant devices that support OAM MUST create a default MA
    with MAID as specified herein.

    MAID [8021Q] has a flexible format and includes two parts:
    Maintenance Domain Name and Short MA name. In the Based Mode of
    operation, the value of the Maintenance Domain Name must be the
    character string "NVO3BaseMode" (excluding the quotes "). In Base
    Mode operation Short MA Name format is set to 2-octet integer format
    (value 3 in Short MA Format field) and Short MA name set to 65532
    (0xFFFC).

    The Default MA belongs to MD-LEVEL 3.

    In the Base Mode of operation, each NVO3 device creates a single UP
    MEP associated with a virtual OAM port located within the CPU with
    no physical layer (NULL PHY). The MEPID associated with this MEP is
    the 2-octet VNI Nickname.

    By default, with the base mode OAM in place, all NVO3 devices
    operating in the Base Mode for NVO3 OAM are able to initiate LBM, PT
    and other OAM tools with no configuration.

    Implementation MAY provide default flow to be included in OAM
    messages. Content of the default flow is outside the scope of this
    document.

    Figure 18, below depicts encoding of MAID within the CCM messages.

       +-+-+-+-+-+-+-+-+-+-+-+-+-+
 

Senevirathne et al.      Expires July 31, 2015                 [Page 32]
Internet Draft           NVO3 Fault Management          January 27, 2015

       |Field Name     |Size     |
       |               |         |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+
       |Maintenance    | 1       |
       |Domain Format  |         |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+
       |Maintenance    | 2       |
       |Domain Length  |         |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+
       |Maintenance    | variable|
       |Domain Name    |         |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+
       |Short MA       | 1       |
       |Name   Format  |         |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+
       |Short MA       | 2       |
       |Name  Length   |         |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+
       |Short MA       | variable|
       |Name           |         |
       +-+-+-+-+-+-+-+-+-+-+-+-+-+
       |Padding        | Variable|
       +-+-+-+-+-+-+-+-+-+-+-+-+-+

                  Figure 18 MAID structure as defined in [8021Q]

    Maintenance Domain Name Format is set to Value: 4

    Maintenance Domain Name Length is set to value: 13

    Maintenance Domain Name is set to: NVO3BaseMode

    Short MA Name Format is set to value: 3

    Short MA Name Length is set to value: 2

    Shirt MA Name is set to : FFFC

    Padding : set of zero up to 48 octets of total length of the MAID.

    Please refer to [8021Q] for details.

    Authors' Addresses
 

Senevirathne et al.      Expires July 31, 2015                 [Page 33]
Internet Draft           NVO3 Fault Management          January 27, 2015

    Tissa Senevirathne
    CISCO Systems
    375 East Tasman Drive.
    San Jose, CA 95134
    USA.

    Phone: +1 408-853-2291
    Email: tsenevir@cisco.com

    Norman Finn
    CISCO Systems
    510 McCarthy Blvd
    Milpitas, CA 95035
    USA

    Email: nfinn@cisco.com

    Samer Salam
    CISCO Systems
    595 Burrard St. Suite 2123
    Vancouver, BC V7X 1J1, Canada

    Email: ssalam@cisco.com

    Deepak Kumar
    CISCO Systems
    510 McCarthy Blvd,
    Milpitas, CA 95035, USA

    Phone : +1 408-853-9760
    Email: dekumar@cisco.com

    Donald Eastlake
    Huawei Technologies
    155 Beaver Street
    Milford, MA 01757

    Phone: +1-508-333-2270
    Email: d3e3e3@gmail.com

    Sam Aldrin
    Huawei Technologies
    2330 Central Express Way
    Santa Clara, CA 95951
 

Senevirathne et al.      Expires July 31, 2015                 [Page 34]
Internet Draft           NVO3 Fault Management          January 27, 2015

    USA

    Email: aldrin.ietf@gmail.com

Senevirathne et al.      Expires July 31, 2015                 [Page 35]