Network Working Group                         Thomas D. Nadeau
Internet Draft                                Monique Morrow
Expires: November  2003                       George Swallow
                                              Cisco Systems, Inc.

                                              David Allan
                                              Nortel Networks

                                              Satoru Matsushima
                                              Japan Telecom

                                              June 2003

           OAM Requirements for MPLS Networks

Status of this Memo

   This document is an Internet-Draft and is in full
   conformance with all provisions of Section 10 of RFC 2026

   Internet-Drafts are working documents of the Internet
   Engineering Task Force (IETF), its areas, and its working
   groups.  Note that other groups may also distribute working
   documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of
   six months and may be updated, replaced, or obsoleted by
   other documents at any time.  It is inappropriate to use
   Internet-Drafts as reference material or to cite them other
   than as "work in progress."

   The list of current Internet-Drafts can be accessed at

   The list of Internet-Draft Shadow Directories can be
   accessed at


   As transport of diverse traffic types such as voice, frame
   relay, and ATM over MPLS become more common, the ability to detect,
   handle and diagnose control and data plane defects becomes

   Detection and specification of how to handle those defects is not
   only important because such defects may not only affect the
   fundamental operation of an MPLS network, but also because they
   may impact SLA commitments for customers of that network.

MPLS Working Group             Expires April 2004            [Page 1]

Internet Draft           MPLS OAM Requirements           October 2003

   This Internet draft describes requirements for user and data
   plane operations and management (OAM) for Multi-Protocol
   Label Switching (MPLS). These requirements have been gathered
   from network operators who have extensive experience deploying
   MPLS networks, similarly some of these requirements have
   appeared in other documents [Y1710]. This draft specifies OAM
   requirements for MPLS, as well as for applications of MPLS such
   as pseudowire voice and VPN services. Those interested in specific
   issues relating to instrumenting MPLS for OAM purposes are directed

   Table of Contents

   Security Considerations..........................................8
   Authors' Addresses..............................................10
   Intellectual Property Rights Notices............................11
   Full Copyright Statement........................................11

1. Introduction

   This Internet draft describes requirements for user and data
   plane operations and management (OAM) for Multi-Protocol
   Label Switching (MPLS). These requirements have been gathered
   from network operators who have extensive experience deploying
   MPLS networks. This draft specifies OAM requirements
   for MPLS, as well as for applications of MPLS such as
   pseudowire [PWE3FRAME] voice, and VPN services.

   No specific mechanisms are proposed to address these
   requirements at this time.  The goal of this draft is to
   identify a commonly applicable set of requirements for MPLS
   OAM. Specifically, a set of requirements that apply to
   the most common set of MPLS networks deployed by service
   provider organizations today. These requirements can then be used
   as a base for network management tool development and to guide
   the evolution of currently specified tools, as well as the
   specification of OAM functions that are intrinsic to protocols
   used in MPLS networks.

   Comments should be made directly to the MPLS mailing list

   This memo does not, in its draft form, specify a standard

MPLS Working Group             Expires April 2004            [Page 2]

Internet Draft           MPLS OAM Requirements           October 2003

   for the Internet community.

2. Terminology

   CE:     Customer Edge

   Defect:   Any error condition that prevents an LSP
             functioning correctly. For example, loss of an
             IGP path will most likely also result in an LSP
             not being able to deliver traffic to its
             destination. Another example is the breakage of
             a TE tunnel.  These may be due to physical
             circuit failures or failure of switching nodes
             to operate as expected.

             Multi-vendor/multi-provider network operation typically
             requires agreed upon definitions of defects (when it is
             broken and when it is not) such that both recovery
             procedures and SLA impacts can be specified.

   ECMP:  Equal Cost Multipath

   LSP:   Label Switch Path

   LSR:   Label Switch Router

   OAM:   Operations and Management

   PE:    Provider Edge

   PW:    Pseudowire

   SLA:   Service Level Agreement

   VCC:   Virtual Channel Connection

   VPC:   Virtual Path Connection

3   Motivations

   MPLS OAM has been tackled in numerous Internet drafts.
   However all existing drafts focus on single provider
   solutions or focus on a single aspect of the MPLS architecture
   or application of MPLS. For example, the use of RSVP or LDP
   signaling and defects may be covered in some deployments,
   and a corresponding SNMP MIB module exists to manage this
   application; however, the handling of defects and specification
   of which types of defects are interesting to operational

MPLS Working Group             Expires April 2004            [Page 3]

Internet Draft           MPLS OAM Requirements           October 2003

   networks may not have been created in concert with those for
   other applications of MPLS such as L3 VPN.  This leads to
   inconsistent and inefficient applicability across the MPLS
   architecture, and/or requires significant modifications to
   operational procedure and systems in order to provide consistent
   and useful OAM functionality. As MPLS matures relationships
   between providers has become more complex. Furthermore, the
   deployment of multiple concurrent applications of MPLS is common
   place. This has led to a need to consider deployments that span
   arbitrary networking arrangements and boundaries;
   so that broader and more uniform applicability to the MPLS
   architecture for OAM is possible.

3. Requirements

   The following sections enumerate the OAM requirements
   gathered from service providers. Each requirement is
   further specified in detail to further clarify its

   3.1 Detection of Broken Label Switch Paths

   The ability to detect a broken Label Switch Path (LSP)
   should not require manual hop-by-hop troubleshooting of
   each LSR used to switch traffic for that LSP. For example,
   it is not desirable to manually visit each LSR along the data
   plane path used to transport an LSP; instead,this function
   should be automated and performed from the origination of that LSP.
   Furthermore, the automation of path liveliness is desired in
   cases where large amounts of LSPs might be tested. For example,
   automated PE-to-PE  LSP testing functionality is desired.
   The goal is to detect LSP problems before customers do, and
   this requires detection of problems in a "reasonable" amount of

   One useful definition  of reasonable is both predictable and

   If the time to detect defects is specified and tools designed
   accordingly then a harmonized operational framework can be
   established both within MPLS levels, and with MPLS applications.
   If the time to detect is known, then automated responses can be
   specified both w.r.t.with regard to resiliency and SLA
   reporting. One consequence is that ambiguity in maintenance
   procedures MUST be minimized as ambiguity in test results impacts
   detection time.

   Although ICMP-based ping can be sent through an LSP,
   the use of this tool to verify the LSP path liveliness has the

MPLS Working Group             Expires April 2004            [Page 4]

Internet Draft           MPLS OAM Requirements           October 2003

   potential for returning erroneous results (both positive and
   negative) given the nature of MPLS LSPs. For example, failures can
   be may occur where inconsistencies exist between the IP and MPLS
   forwarding tables, inconsistencies in the MPLS control and data
   plane or problems with the reply path (i.e.: a reverse MPLS
   path does not exist). Detection tools should have minimal
   dependencies on network components that do not implement the LSP.

   The OAM packet MUST follow exactly the customer data path in order
   to reflect path liveliness used by customer data. Particular
   cases of interest are forwarding mechanisms such as equal cost
   multipath (ECMP) scenarios within the operator's network whereby
   flows are load-shared across parallel (i.e.: equal IGP cost) paths.
   Where the customer traffic may be spread over multiple paths, it
   is required to be able to detect failures on any of the path
   permutations.  Where the spreading mechanism is payload specific,
   payloads need to have forwarding that is common with the traffic
   under test. Satisfying these requirements introduces complexity
   into ensuring that ECMP connectivity permutations are exercised,
   and that defect detection occurs in a reasonable amount of time.
   [GUIDELINES] discusses some of the issues and offers suggestions
   for ensuring mutual compatibility of ECMP and maintenance
   functions (both detection and diagnostic).

  3.2 Diagnosis of a Broken Label Switch Path

   The ability to diagnose a broken LSP and to isolate the failed
   resource in the path is required. This is particularly true for
   misbranching defects which are particularly difficult to specify
   recovery actions in an LDP network.
   Experience suggests that this is best accomplished via a path
   trace function that can return the entire list of LSRs and links
   used by a certain LSP (or at least the set of LSRs/links up to the
   location of the defect) is required. The tracing capability should
   include the ability to trace recursive paths, such as when nested
   LSPs are used, or when LSPs enter and exit traffic-engineered
   tunnels [TUNTRACE]. This path trace function must also be
   capable of diagnosing LSP mis-merging by permitting comparison
   of expected vs. actual forwarding behavior at any LSR in the path.
   The path trace capability should be capable of being
   executed from both the head end Label Switch Router (LSR) and any
   mid-point LSR.
   Additionally, the path trace function MUST have the ability to
   support equal cost multipath scenarios as described above in
   section 3.1.

  3.3 Path characterization

   The ability of a path trace function to reveal details of LSR
   forwarding operations relevant to OAM functionality. This would

MPLS Working Group             Expires April 2004            [Page 5]

Internet Draft           MPLS OAM Requirements           October 2003

   include but not be limited to:
     - use of pipe or uniform TTL models by an LSR
     - externally visible aspects of load spreading (such as
       ECMP), including type of algorithm used
       examples of how algorithm will spread traffic
     - data/control plane OAM capabilities of the LSR
     - stack operations performed by the LSR (pushes and pops)

3.4 Service Level Agreement Measurement

   Mechanisms are required to measure diverse aspects of Service
   Level Agreements:
     - availability - in which the service is considered to be
       available and the other aspects of performance measurement
       listed below have meaning, or unavailable and other aspects
       of performance measurement do not.
     - latency - amount of time required for traffic to transit
       the network
     - packet loss
     - jitter - measurement of latency variation

   Such measurements can be made independently of the user traffic
   or via a hybrid of user traffic measurement and OAM probing.

   At least one mechanism is required to measure the quantity
   (i.e.: number of packets) of OAM packets. In addition, the
   ability to measure the qualitative aspects of OAM probing must
   be available to specifically compute the latency of OAM packets
   generated and received at each end of a tested LSP. Latency is
   considered in this context as a measurable parameter for SLA
   reporting. There is no assumption that bursts of OAM packets are
   required to characterize the performance of an LSP, but it is
   suggested that any method considered be capable of measuring the
   latency of an LSP with minimal impact on network resources.

3.5 Frequency of OAM Execution

   The operator MUST be have the flexibility to configure OAM
   parameters and the frequency of the execution of any OAM
   functions provided that there is some synchronization possible
   of tool usage for availability metrics. The motivation for this
   is to permit the network to function as a system of harmonious
   OAM functions consistent across the entire network.

   To elaborate, there are defect conditions (specifically
   misbranching or misdirection of traffic) for probe based detection
   mechanisms combined with automated network response requires
   harmonization of probe insertion rates and probe handling across
   the network in order to avoid flapping.

MPLS Working Group             Expires April 2004            [Page 6]

Internet Draft           MPLS OAM Requirements           October 2003

   One observation would be that commoditization of MPLS, common
   optimized implementation of monitoring tools and the need for inter-
   carrier harmonization of defect and SLA handling will drive
   specification of OAM parameters to commonly agreed on values and
   such values will have to be harmonized with the surrounding
   technologies (e.g. SONET/SDH, ATM etc.) in order to be useful.
   This will become particularly important as networks scale
   and misconfiguration can result in churn, alarm flapping etc.

3.5 Alarm Suppression and layer coordination

   Devices must provide alarm suppression functionality that
   prevents the generation of superfluous generation of alarms.
   When viewed in conjuction with requirement 3.6 below, this
   typically requires fault notification to the LSP egress, that
   may have specific time constraints if the client PW independently
   implements path continuity testing (for example ATM I.610
   Continuity check (CC)[I610]).

   This would also be true for LSPs that have client LSPs that are
   monitored. MPLS arbitrary hierarchy introduces the opportunity to
   have multiple MPLS levels attempt to respond to defects
   simultaneously. Mechanisms are required to coordinate network
   response to defects.

3.6 Support for OAM Interworking for Fault Notification

   An LSR supporting OAM functions for pseudo-wire functions that
   join one or more networking technologies over MPLS must be
   able to translate an MPLS defect into the native technology's
   error condition. For example, errors occurring over the MPLS
   transport LSP that supports an emulated ATM VC must translate
   errors into native ATM OAM AIS cells at the edges of the pseudo-
   wire. The mechanism SHOULD consider possible bounded detection
   time parameters, e.g., a "hold off" function before reacting as
   to harmonize with the client OAM. One goal would be alarm
   suppression in the psuedo-wire's client layer. As observed in
   section 3.5, this requires that the MPLS layer perform detection
   in a bounded timeframe in order to initiate alarm suppression
   prior to the psuedo-wire client layer independently detecting the

3.7 Error Detection and Recovery.

    Mechanisms are needed to detect an error, react to it (ideally
    in some form of automated response by the network), recover from
    it and alert the network operator prior to the customer informing

MPLS Working Group             Expires April 2004            [Page 7]

Internet Draft           MPLS OAM Requirements           October 2003

    the network operator of the error condition. The ideal situation
    would be where the network is resilient and can restore service
    prior any significant impact on the customer perception of the
    service. There are also defects that by virtue of available network
    resources or topology that cannot be recovered automatically.

    It is however, sometimes a requirement that the customer be
    notified of the defect condition at the same time that the network
    operator is made aware of the defect (as in the example of alarm
    suppression for PW clients discussed above). In these situations,
    the customer network may be capable of processing automated
    responses based on notification of a defect condition.  It is
    preferred that the format of these notifications be made
    consistent (i.e.: standardized) as to increase the applicability
    of such messages. Depending on the device's capabilities, the
    device may be programmed to take automatic corrective actions as
    a result of detection of defect conditions. These actions may be
    user or operator-specified, or may simply be inherent to the
    underlying transport technology (i.e.: MPLS Fast-Reroute,
    graceful restart or high-availability functionality).

3.8 The commoditization of MPLS will require common information
    modeling of management and control of OAM functionality. This
    will be reflected in the the integration of standard MPLS-related
    MIBs (e.g. [LSRMIB][TEMIB][LBMIB][FTNMIB]) for fault, statistics
    and configuration management. These standard interfaces
    provide operators with common programmatic interface access to
    operations and management functions and their status.

3.9 Detection of Denial of Service attacks as part of security

3.10 Per-LSP Accounting Requirements

     In an MPLS network, SPs can measure traffic from an LSR to the
     of the MPLS network using some MPLS related MIBs, for example.
     This means that it is reasonable wish to know how much traffic is
     traveling from where to where (i.e.: a traffic matrix) by analyzing
     the flow of traffic.
     Therefore, traffic accounting in an MPLS network can be summarized
     the following three items.

     (1) Collecting information to design network

         For the purpose of optimized network design, SP offers that the
         traffic information regarding among POP and/or router.
         Optimizing network design needs this information.

     (2) Providing high-level SLA

MPLS Working Group             Expires April 2004            [Page 8]

Internet Draft           MPLS OAM Requirements           October 2003

         Due to the progress of the recent [MPLS-TE] technology,
         SPs and their customers may need to verify high-level SLAs. The
         resource optimization and high-speed restoration by [FRR] is
         being offered; therefore, continuous improvement of the service
         is expected.  Moreover, bandwidth guaranteed service can be
         achieved by resource management based on [DS-TE].

         To provide services based on these applications, the SP
         needs to perform traffic accounting to monitor their

     (3) Inter-AS environment

         SPs which offer inter-as services [Inter-AS TE][Inter-AS VPN]
         require accounting of the service.

     These three motivations need to satisfy the following.

        - In (1) and (2), collection of information on a per-LSP basis
          is a minimum level of granularity of collecting accounting
          information at both of ingress and egress of an LSP.

        - In (3), SP's ASBR carry out interconnection functions as an
          intermediate LSR. Therefore, identifying a pair of ingress
          and egress LSRs using each LSP is needed to determine the
          cost of the service that a customer is using.

3.10.1 Requirements

     Accounting on a per-LSP basis encompasses the following set of

      (1) At an ingress LSR accounting of traffic through LSPs
          beginning at each egress in question.

      (2) At an intermediate LSR, accounting of traffic through
          LSPs for each pair of ingress to egress.

      (3) At egress LSR, accounting of traffic through LSPs
          for each ingress.

      (4) All LSRs that contain LSPs that are being measuremented
          need to have a common key to distinguish each LSP.
          The key must be unique to each LSP, and its mapping to
          LSP should be provided from whether manual or automatic

3.10.2 Scalability

     It is not realistic to perform the just described operations by

MPLS Working Group             Expires April 2004            [Page 9]

Internet Draft           MPLS OAM Requirements           October 2003

     LSRs in a network and on all LSPs which exist in a network.
     At a minimum, per-LSP based accounting should be performed on the
     edges of the network -- at the edges of both LSPs and the MPLS

4. Security Considerations

   LSP mis-merging has security implications beyond that of simply
   being a network defect. LSP mis-merging can happen due to a number
   of potential sources of failure, some of which (due to MPLS label
   stacking) are new to  MPLS.

   The performance of diagnostic functions and path characterization
   involve extracting a significant amount of information about
   network construction which the network operator may consider
   Mechanisms are required to prevent unauthorized use of either those
   tools or protocol features.

5. Acknowledgments

    The authors wish to acknowledge and thank the following
    individuals for their valuable comments to this document:
    Adrian Smith, British Telecom; Chou Lan Pok, SBC; Mr.
    Ikejiri, NTT Communications and Mr.Kumaki of KDDI.
    Hari Rakotoranto, Miya Khono, Cisco Systems; Luyuan Fang, AT&T;
    Danny McPherson, TCB; Dr.Ken Nagami, Ikuo Nakagawa, Intec Netcore.

6. References

   [TUNTRACE]    Bonica, R., Kompella, K., Meyer, D.,
                 "Tracing Requirements for Generic Tunnels",
                 Internet Draft <draft-bonica-tunneltrace-
                 02.txt>, November 2001.

   [LSRMIB]      Srinivasan, C., Viswanathan, A. and T.
                 Nadeau, "MPLS Label Switch Router Management
                 Information Base Using SMIv2", Internet
                 Draft <draft-ietf-mpls-lsr-mib-07.txt>,
                 January 2001.

   [TEMIB]       Srinivasan, C., Viswanathan, A. and T.
                 Nadeau, "MPLS Traffic Engineering Management
                 Information Base Using SMIv2", Internet
                 Draft <draft-ietf-mpls-te-mib-07.txt>,
                 August 2001.

   [FTNMIB]      Nadeau, T., Srinivasan, C., and A.
                 Viswanathan, "Multiprotocol Label Switching

MPLS Working Group            Expires April 2004            [Page 10]

Internet Draft           MPLS OAM Requirements           October 2003

                 (MPLS) FEC-To-NHLFE (FTN) Management
                 Information Base", Internet Draft <draft-
                 ietf-mpls-ftn-mib-03.txt>, August 2001.

   [LBMIB]       Dubuc, M., Dharanikota, S., Nadeau, T., J.
                 Lang, "Link Bundling Management Information
                 Base Using SMIv2", Internet Draft <draft-
                 ietf-mpls-bundle-mib-00.txt>, September

    [MPLS-TE]  Awduche et. al., "RSVP-TE: Extensions to RSVP for LSP
               Tunnels", RFC 3209, December 2001

    [FRR]  Pan, "Fast Reroute Extensions to RSVP-TE for LSP
           Tunnels", Internet draft,
           <draft-ietf-mpls-rsvp-lsp-fastreroute-03.txt>, December 2003

    [DS-TE] Le Faucheur & Lai., "Requirements for Diff-Serv-aware TE",
            RFC3564, July 2003

    [Inter-AS TE] Zhang, Vasseur, et. al., "MPLS Inter-AS Traffic
                  Engineering requirements", Internet draft
                  <draft-ietf-tewg-interas-mpls-te-req-00.txt>, May 2003

    [Inter-AS VPN] Rosen, et al., "BGP/MPLS IP VPNs", Internet draft,
                   <draft-ietf-l3vpn-rfc2547bis-01.txt>, September 2003

   [PWE3FRAME]   Pate, P., Xiao, X., White., C., Kompella.,
                 K., Malis, A., Johnson, T., and T. Nadeau,
                 "Framework for Pseudo Wire Emulation Edge-to-
                 Edge (PWE3)", Internet Draft <draft-ietf-
                 pwe3-framework-00.txt>, September, 2001.

   [RFC2026]     S. Bradner, "The Internet Standards Process
                 -- Revision 3", RFC 2026, October 1996.

   [Y1710]       ITU-T Recommendation Y.1710, "Requirements for
                 OAM Functionality In MPLS Networks"

   [GUIDELINES]  Allan, D., "Guidelines for MPLS load
                 balancing", Internet draft,
                 <draft-allan-mpls-loadbal-01.txt>, February

   [I610]      ITU-T Recommendation I.610, "B-ISDN operations and
               maintenance principles and functions", February 1999

   [FRAMEWORK] Allan "A Framework for MPLS OAM", Internet
               draft <draft-allan-mpls-oam-frmwk-04.txt>, February 2003

MPLS Working Group            Expires April 2004            [Page 11]

Internet Draft           MPLS OAM Requirements           October 2003

7. Authors' Addresses

  Thomas D. Nadeau
  Cisco Systems, Inc.
  300 Beaver Brook Road
  Boxboro, MA 01719
  Phone: +1-978-936-1470

  Monique Jeanne Morrow
  Cisco Systems, Inc.
  Glatt-Com, 2nd Floor
  Voice:  (0)1 878-9412

  George Swallow
  Cisco Systems, Inc.
  300 Beaver Brook Road
  Boxboro, MA 01719
  Voice: +1-978-936-1398

  David Allan
  Nortel Networks
  3500 Carling Ave.
  Ottawa, Ontario, CANADA
  Voice: 1-613-763-6362

  Satoru Matsushima
  Japan Telecom
  4-7-1, Hatchobori, Chuo-ku
  Tokyo, 104-8508 Japan
  Phone: +81-3-5540-8214

8. Full Copyright Statement

   Copyright (C) The Internet Society (2001). All Rights

   This document and translations of it may be copied and
   furnished to others, and derivative works that comment on
   or otherwise explain it or assist in its implementation may
   be prepared, copied, published and distributed, in whole or

MPLS Working Group            Expires April 2004            [Page 12]

Internet Draft           MPLS OAM Requirements           October 2003

   in part, without restriction of any kind, provided that the
   above copyright notice and this paragraph are included on
   all such copies and derivative works.  However, this
   document itself may not be modified in any way, such as by
   removing the copyright notice or references to the Internet
   Society or other Internet organizations, except as needed
   for the purpose of developing Internet standards in which
   case the procedures for copyrights defined in the Internet
   Standards process must be followed, or as required to
   translate it into languages other than English.

   The limited permissions granted above are perpetual and
   will not be revoked by the Internet Society or its
   successors or assigns. This document and the information
   contained herein is provided on an "AS IS" basis and THE

9. Intellectual Property Rights Notices

   The IETF takes no position regarding the validity or scope of any
   intellectual property or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; neither does it represent that it
   has made any effort to identify any such rights.  Information on the
   IETF's procedures with respect to rights in standards-track and
   standards-related documentation can be found in BCP-11.  Copies of
   claims of rights made available for publication and any assurances
   of licenses to be made available, or the result of an attempt made
   to obtain a general license or permission for the use of such
   proprietary rights by implementers or users of this specification
   can be obtained from the IETF Secretariat.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights which may cover technology that may be required to practice
   this standard.  Please address the information to the IETF Executive

MPLS Working Group Expires April 2004 [Page 13]