Network Working Group Thomas D. Nadeau
Internet Draft Monique Morrow
Expires: November 2003 George Swallow
Cisco Systems, Inc.
David Allan
Nortel Networks
Satoru Matsushima
Japan Telecom
June 2003
OAM Requirements for MPLS Networks
draft-ietf-mpls-oam-requirements-02.txt
Status of this Memo
This document is an Internet-Draft and is in full
conformance with all provisions of Section 10 of RFC 2026
[RFC2026].
Internet-Drafts are working documents of the Internet
Engineering Task Force (IETF), its areas, and its working
groups. Note that other groups may also distribute working
documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of
six months and may be updated, replaced, or obsoleted by
other documents at any time. It is inappropriate to use
Internet-Drafts as reference material or to cite them other
than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be
accessed at http://www.ietf.org/shadow.html.
Abstract
As transport of diverse traffic types such as voice, frame
relay, and ATM over MPLS become more common, the ability to detect,
handle and diagnose control and data plane defects becomes
critical.
Detection and specification of how to handle those defects is not
only important because such defects may not only affect the
fundamental operation of an MPLS network, but also because they
may impact SLA commitments for customers of that network.
MPLS Working Group Expires April 2004 [Page 1]
Internet Draft MPLS OAM Requirements October 2003
This Internet draft describes requirements for user and data
plane operations and management (OAM) for Multi-Protocol
Label Switching (MPLS). These requirements have been gathered
from network operators who have extensive experience deploying
MPLS networks, similarly some of these requirements have
appeared in other documents [Y1710]. This draft specifies OAM
requirements for MPLS, as well as for applications of MPLS such
as pseudowire voice and VPN services. Those interested in specific
issues relating to instrumenting MPLS for OAM purposes are directed
to [FRAMEWORK]
Table of Contents
Introduction.....................................................2
Terminology......................................................2
Motivations......................................................3
Requirements.....................................................4
Security Considerations..........................................8
Acknowledgments..................................................9
References.......................................................9
Authors' Addresses..............................................10
Intellectual Property Rights Notices............................11
Full Copyright Statement........................................11
1. Introduction
This Internet draft describes requirements for user and data
plane operations and management (OAM) for Multi-Protocol
Label Switching (MPLS). These requirements have been gathered
from network operators who have extensive experience deploying
MPLS networks. This draft specifies OAM requirements
for MPLS, as well as for applications of MPLS such as
pseudowire [PWE3FRAME] voice, and VPN services.
No specific mechanisms are proposed to address these
requirements at this time. The goal of this draft is to
identify a commonly applicable set of requirements for MPLS
OAM. Specifically, a set of requirements that apply to
the most common set of MPLS networks deployed by service
provider organizations today. These requirements can then be used
as a base for network management tool development and to guide
the evolution of currently specified tools, as well as the
specification of OAM functions that are intrinsic to protocols
used in MPLS networks.
Comments should be made directly to the MPLS mailing list
at mpls@uu.net.
This memo does not, in its draft form, specify a standard
MPLS Working Group Expires April 2004 [Page 2]
Internet Draft MPLS OAM Requirements October 2003
for the Internet community.
2. Terminology
CE: Customer Edge
Defect: Any error condition that prevents an LSP
functioning correctly. For example, loss of an
IGP path will most likely also result in an LSP
not being able to deliver traffic to its
destination. Another example is the breakage of
a TE tunnel. These may be due to physical
circuit failures or failure of switching nodes
to operate as expected.
Multi-vendor/multi-provider network operation typically
requires agreed upon definitions of defects (when it is
broken and when it is not) such that both recovery
procedures and SLA impacts can be specified.
ECMP: Equal Cost Multipath
LSP: Label Switch Path
LSR: Label Switch Router
OAM: Operations and Management
PE: Provider Edge
PW: Pseudowire
SLA: Service Level Agreement
VCC: Virtual Channel Connection
VPC: Virtual Path Connection
3 Motivations
MPLS OAM has been tackled in numerous Internet drafts.
However all existing drafts focus on single provider
solutions or focus on a single aspect of the MPLS architecture
or application of MPLS. For example, the use of RSVP or LDP
signaling and defects may be covered in some deployments,
and a corresponding SNMP MIB module exists to manage this
application; however, the handling of defects and specification
of which types of defects are interesting to operational
MPLS Working Group Expires April 2004 [Page 3]
Internet Draft MPLS OAM Requirements October 2003
networks may not have been created in concert with those for
other applications of MPLS such as L3 VPN. This leads to
inconsistent and inefficient applicability across the MPLS
architecture, and/or requires significant modifications to
operational procedure and systems in order to provide consistent
and useful OAM functionality. As MPLS matures relationships
between providers has become more complex. Furthermore, the
deployment of multiple concurrent applications of MPLS is common
place. This has led to a need to consider deployments that span
arbitrary networking arrangements and boundaries;
so that broader and more uniform applicability to the MPLS
architecture for OAM is possible.
3. Requirements
The following sections enumerate the OAM requirements
gathered from service providers. Each requirement is
further specified in detail to further clarify its
applicability.
3.1 Detection of Broken Label Switch Paths
The ability to detect a broken Label Switch Path (LSP)
should not require manual hop-by-hop troubleshooting of
each LSR used to switch traffic for that LSP. For example,
it is not desirable to manually visit each LSR along the data
plane path used to transport an LSP; instead,this function
should be automated and performed from the origination of that LSP.
Furthermore, the automation of path liveliness is desired in
cases where large amounts of LSPs might be tested. For example,
automated PE-to-PE LSP testing functionality is desired.
The goal is to detect LSP problems before customers do, and
this requires detection of problems in a "reasonable" amount of
time.
One useful definition of reasonable is both predictable and
consistent.
If the time to detect defects is specified and tools designed
accordingly then a harmonized operational framework can be
established both within MPLS levels, and with MPLS applications.
If the time to detect is known, then automated responses can be
specified both w.r.t.with regard to resiliency and SLA
reporting. One consequence is that ambiguity in maintenance
procedures MUST be minimized as ambiguity in test results impacts
detection time.
Although ICMP-based ping can be sent through an LSP,
the use of this tool to verify the LSP path liveliness has the
MPLS Working Group Expires April 2004 [Page 4]
Internet Draft MPLS OAM Requirements October 2003
potential for returning erroneous results (both positive and
negative) given the nature of MPLS LSPs. For example, failures can
be may occur where inconsistencies exist between the IP and MPLS
forwarding tables, inconsistencies in the MPLS control and data
plane or problems with the reply path (i.e.: a reverse MPLS
path does not exist). Detection tools should have minimal
dependencies on network components that do not implement the LSP.
The OAM packet MUST follow exactly the customer data path in order
to reflect path liveliness used by customer data. Particular
cases of interest are forwarding mechanisms such as equal cost
multipath (ECMP) scenarios within the operator's network whereby
flows are load-shared across parallel (i.e.: equal IGP cost) paths.
Where the customer traffic may be spread over multiple paths, it
is required to be able to detect failures on any of the path
permutations. Where the spreading mechanism is payload specific,
payloads need to have forwarding that is common with the traffic
under test. Satisfying these requirements introduces complexity
into ensuring that ECMP connectivity permutations are exercised,
and that defect detection occurs in a reasonable amount of time.
[GUIDELINES] discusses some of the issues and offers suggestions
for ensuring mutual compatibility of ECMP and maintenance
functions (both detection and diagnostic).
3.2 Diagnosis of a Broken Label Switch Path
The ability to diagnose a broken LSP and to isolate the failed
resource in the path is required. This is particularly true for
misbranching defects which are particularly difficult to specify
recovery actions in an LDP network.
Experience suggests that this is best accomplished via a path
trace function that can return the entire list of LSRs and links
used by a certain LSP (or at least the set of LSRs/links up to the
location of the defect) is required. The tracing capability should
include the ability to trace recursive paths, such as when nested
LSPs are used, or when LSPs enter and exit traffic-engineered
tunnels [TUNTRACE]. This path trace function must also be
capable of diagnosing LSP mis-merging by permitting comparison
of expected vs. actual forwarding behavior at any LSR in the path.
The path trace capability should be capable of being
executed from both the head end Label Switch Router (LSR) and any
mid-point LSR.
Additionally, the path trace function MUST have the ability to
support equal cost multipath scenarios as described above in
section 3.1.
3.3 Path characterization
The ability of a path trace function to reveal details of LSR
forwarding operations relevant to OAM functionality. This would
MPLS Working Group Expires April 2004 [Page 5]
Internet Draft MPLS OAM Requirements October 2003
include but not be limited to:
- use of pipe or uniform TTL models by an LSR
- externally visible aspects of load spreading (such as
ECMP), including type of algorithm used
examples of how algorithm will spread traffic
- data/control plane OAM capabilities of the LSR
- stack operations performed by the LSR (pushes and pops)
3.4 Service Level Agreement Measurement
Mechanisms are required to measure diverse aspects of Service
Level Agreements:
- availability - in which the service is considered to be
available and the other aspects of performance measurement
listed below have meaning, or unavailable and other aspects
of performance measurement do not.
- latency - amount of time required for traffic to transit
the network
- packet loss
- jitter - measurement of latency variation
Such measurements can be made independently of the user traffic
or via a hybrid of user traffic measurement and OAM probing.
At least one mechanism is required to measure the quantity
(i.e.: number of packets) of OAM packets. In addition, the
ability to measure the qualitative aspects of OAM probing must
be available to specifically compute the latency of OAM packets
generated and received at each end of a tested LSP. Latency is
considered in this context as a measurable parameter for SLA
reporting. There is no assumption that bursts of OAM packets are
required to characterize the performance of an LSP, but it is
suggested that any method considered be capable of measuring the
latency of an LSP with minimal impact on network resources.
3.5 Frequency of OAM Execution
The operator MUST be have the flexibility to configure OAM
parameters and the frequency of the execution of any OAM
functions provided that there is some synchronization possible
of tool usage for availability metrics. The motivation for this
is to permit the network to function as a system of harmonious
OAM functions consistent across the entire network.
To elaborate, there are defect conditions (specifically
misbranching or misdirection of traffic) for probe based detection
mechanisms combined with automated network response requires
harmonization of probe insertion rates and probe handling across
the network in order to avoid flapping.
MPLS Working Group Expires April 2004 [Page 6]
Internet Draft MPLS OAM Requirements October 2003
One observation would be that commoditization of MPLS, common
optimized implementation of monitoring tools and the need for inter-
carrier harmonization of defect and SLA handling will drive
specification of OAM parameters to commonly agreed on values and
such values will have to be harmonized with the surrounding
technologies (e.g. SONET/SDH, ATM etc.) in order to be useful.
This will become particularly important as networks scale
and misconfiguration can result in churn, alarm flapping etc.
3.5 Alarm Suppression and layer coordination
Devices must provide alarm suppression functionality that
prevents the generation of superfluous generation of alarms.
When viewed in conjuction with requirement 3.6 below, this
typically requires fault notification to the LSP egress, that
may have specific time constraints if the client PW independently
implements path continuity testing (for example ATM I.610
Continuity check (CC)[I610]).
This would also be true for LSPs that have client LSPs that are
monitored. MPLS arbitrary hierarchy introduces the opportunity to
have multiple MPLS levels attempt to respond to defects
simultaneously. Mechanisms are required to coordinate network
response to defects.
3.6 Support for OAM Interworking for Fault Notification
An LSR supporting OAM functions for pseudo-wire functions that
join one or more networking technologies over MPLS must be
able to translate an MPLS defect into the native technology's
error condition. For example, errors occurring over the MPLS
transport LSP that supports an emulated ATM VC must translate
errors into native ATM OAM AIS cells at the edges of the pseudo-
wire. The mechanism SHOULD consider possible bounded detection
time parameters, e.g., a "hold off" function before reacting as
to harmonize with the client OAM. One goal would be alarm
suppression in the psuedo-wire's client layer. As observed in
section 3.5, this requires that the MPLS layer perform detection
in a bounded timeframe in order to initiate alarm suppression
prior to the psuedo-wire client layer independently detecting the
defect.
3.7 Error Detection and Recovery.
Mechanisms are needed to detect an error, react to it (ideally
in some form of automated response by the network), recover from
it and alert the network operator prior to the customer informing
MPLS Working Group Expires April 2004 [Page 7]
Internet Draft MPLS OAM Requirements October 2003
the network operator of the error condition. The ideal situation
would be where the network is resilient and can restore service
prior any significant impact on the customer perception of the
service. There are also defects that by virtue of available network
resources or topology that cannot be recovered automatically.
It is however, sometimes a requirement that the customer be
notified of the defect condition at the same time that the network
operator is made aware of the defect (as in the example of alarm
suppression for PW clients discussed above). In these situations,
the customer network may be capable of processing automated
responses based on notification of a defect condition. It is
preferred that the format of these notifications be made
consistent (i.e.: standardized) as to increase the applicability
of such messages. Depending on the device's capabilities, the
device may be programmed to take automatic corrective actions as
a result of detection of defect conditions. These actions may be
user or operator-specified, or may simply be inherent to the
underlying transport technology (i.e.: MPLS Fast-Reroute,
graceful restart or high-availability functionality).
3.8 The commoditization of MPLS will require common information
modeling of management and control of OAM functionality. This
will be reflected in the the integration of standard MPLS-related
MIBs (e.g. [LSRMIB][TEMIB][LBMIB][FTNMIB]) for fault, statistics
and configuration management. These standard interfaces
provide operators with common programmatic interface access to
operations and management functions and their status.
3.9 Detection of Denial of Service attacks as part of security
management.
3.10 Per-LSP Accounting Requirements
In an MPLS network, SPs can measure traffic from an LSR to the
egress
of the MPLS network using some MPLS related MIBs, for example.
This means that it is reasonable wish to know how much traffic is
traveling from where to where (i.e.: a traffic matrix) by analyzing
the flow of traffic.
Therefore, traffic accounting in an MPLS network can be summarized
as
the following three items.
(1) Collecting information to design network
For the purpose of optimized network design, SP offers that the
traffic information regarding among POP and/or router.
Optimizing network design needs this information.
(2) Providing high-level SLA
MPLS Working Group Expires April 2004 [Page 8]
Internet Draft MPLS OAM Requirements October 2003
Due to the progress of the recent [MPLS-TE] technology,
SPs and their customers may need to verify high-level SLAs. The
resource optimization and high-speed restoration by [FRR] is
being offered; therefore, continuous improvement of the service
is expected. Moreover, bandwidth guaranteed service can be
achieved by resource management based on [DS-TE].
To provide services based on these applications, the SP
needs to perform traffic accounting to monitor their
services.
(3) Inter-AS environment
SPs which offer inter-as services [Inter-AS TE][Inter-AS VPN]
require accounting of the service.
These three motivations need to satisfy the following.
- In (1) and (2), collection of information on a per-LSP basis
is a minimum level of granularity of collecting accounting
information at both of ingress and egress of an LSP.
- In (3), SP's ASBR carry out interconnection functions as an
intermediate LSR. Therefore, identifying a pair of ingress
and egress LSRs using each LSP is needed to determine the
cost of the service that a customer is using.
3.10.1 Requirements
Accounting on a per-LSP basis encompasses the following set of
functions:
(1) At an ingress LSR accounting of traffic through LSPs
beginning at each egress in question.
(2) At an intermediate LSR, accounting of traffic through
LSPs for each pair of ingress to egress.
(3) At egress LSR, accounting of traffic through LSPs
for each ingress.
(4) All LSRs that contain LSPs that are being measuremented
need to have a common key to distinguish each LSP.
The key must be unique to each LSP, and its mapping to
LSP should be provided from whether manual or automatic
configuration.
3.10.2 Scalability
It is not realistic to perform the just described operations by
MPLS Working Group Expires April 2004 [Page 9]
Internet Draft MPLS OAM Requirements October 2003
LSRs in a network and on all LSPs which exist in a network.
At a minimum, per-LSP based accounting should be performed on the
edges of the network -- at the edges of both LSPs and the MPLS
domain.
4. Security Considerations
LSP mis-merging has security implications beyond that of simply
being a network defect. LSP mis-merging can happen due to a number
of potential sources of failure, some of which (due to MPLS label
stacking) are new to MPLS.
The performance of diagnostic functions and path characterization
involve extracting a significant amount of information about
network construction which the network operator may consider
private.
Mechanisms are required to prevent unauthorized use of either those
tools or protocol features.
5. Acknowledgments
The authors wish to acknowledge and thank the following
individuals for their valuable comments to this document:
Adrian Smith, British Telecom; Chou Lan Pok, SBC; Mr.
Ikejiri, NTT Communications and Mr.Kumaki of KDDI.
Hari Rakotoranto, Miya Khono, Cisco Systems; Luyuan Fang, AT&T;
Danny McPherson, TCB; Dr.Ken Nagami, Ikuo Nakagawa, Intec Netcore.
6. References
[TUNTRACE] Bonica, R., Kompella, K., Meyer, D.,
"Tracing Requirements for Generic Tunnels",
Internet Draft <draft-bonica-tunneltrace-
02.txt>, November 2001.
[LSRMIB] Srinivasan, C., Viswanathan, A. and T.
Nadeau, "MPLS Label Switch Router Management
Information Base Using SMIv2", Internet
Draft <draft-ietf-mpls-lsr-mib-07.txt>,
January 2001.
[TEMIB] Srinivasan, C., Viswanathan, A. and T.
Nadeau, "MPLS Traffic Engineering Management
Information Base Using SMIv2", Internet
Draft <draft-ietf-mpls-te-mib-07.txt>,
August 2001.
[FTNMIB] Nadeau, T., Srinivasan, C., and A.
Viswanathan, "Multiprotocol Label Switching
MPLS Working Group Expires April 2004 [Page 10]
Internet Draft MPLS OAM Requirements October 2003
(MPLS) FEC-To-NHLFE (FTN) Management
Information Base", Internet Draft <draft-
ietf-mpls-ftn-mib-03.txt>, August 2001.
[LBMIB] Dubuc, M., Dharanikota, S., Nadeau, T., J.
Lang, "Link Bundling Management Information
Base Using SMIv2", Internet Draft <draft-
ietf-mpls-bundle-mib-00.txt>, September
2001.
[MPLS-TE] Awduche et. al., "RSVP-TE: Extensions to RSVP for LSP
Tunnels", RFC 3209, December 2001
[FRR] Pan et.al., "Fast Reroute Extensions to RSVP-TE for LSP
Tunnels", Internet draft,
<draft-ietf-mpls-rsvp-lsp-fastreroute-03.txt>, December 2003
[DS-TE] Le Faucheur & Lai., "Requirements for Diff-Serv-aware TE",
RFC3564, July 2003
[Inter-AS TE] Zhang, Vasseur, et. al., "MPLS Inter-AS Traffic
Engineering requirements", Internet draft
<draft-ietf-tewg-interas-mpls-te-req-00.txt>, May 2003
[Inter-AS VPN] Rosen, et al., "BGP/MPLS IP VPNs", Internet draft,
<draft-ietf-l3vpn-rfc2547bis-01.txt>, September 2003
[PWE3FRAME] Pate, P., Xiao, X., White., C., Kompella.,
K., Malis, A., Johnson, T., and T. Nadeau,
"Framework for Pseudo Wire Emulation Edge-to-
Edge (PWE3)", Internet Draft <draft-ietf-
pwe3-framework-00.txt>, September, 2001.
[RFC2026] S. Bradner, "The Internet Standards Process
-- Revision 3", RFC 2026, October 1996.
[Y1710] ITU-T Recommendation Y.1710, "Requirements for
OAM Functionality In MPLS Networks"
[GUIDELINES] Allan, D., "Guidelines for MPLS load
balancing", Internet draft,
<draft-allan-mpls-loadbal-01.txt>, February
2003
[I610] ITU-T Recommendation I.610, "B-ISDN operations and
maintenance principles and functions", February 1999
[FRAMEWORK] Allan et.al. "A Framework for MPLS OAM", Internet
draft <draft-allan-mpls-oam-frmwk-04.txt>, February 2003
MPLS Working Group Expires April 2004 [Page 11]
Internet Draft MPLS OAM Requirements October 2003
7. Authors' Addresses
Thomas D. Nadeau
Cisco Systems, Inc.
300 Beaver Brook Road
Boxboro, MA 01719
Phone: +1-978-936-1470
Email: tnadeau@cisco.com
Monique Jeanne Morrow
Cisco Systems, Inc.
Glatt-Com, 2nd Floor
CH-8301
Switzerland
Voice: (0)1 878-9412
Email: mmorrow@cisco.com
George Swallow
Cisco Systems, Inc.
300 Beaver Brook Road
Boxboro, MA 01719
Voice: +1-978-936-1398
Email: swallow@cisco.com
David Allan
Nortel Networks
3500 Carling Ave.
Ottawa, Ontario, CANADA
Voice: 1-613-763-6362
Email: dallan@nortelnetworks.com
Satoru Matsushima
Japan Telecom
4-7-1, Hatchobori, Chuo-ku
Tokyo, 104-8508 Japan
Phone: +81-3-5540-8214
Email: satoru@ft.solteria.net
8. Full Copyright Statement
Copyright (C) The Internet Society (2001). All Rights
Reserved.
This document and translations of it may be copied and
furnished to others, and derivative works that comment on
or otherwise explain it or assist in its implementation may
be prepared, copied, published and distributed, in whole or
MPLS Working Group Expires April 2004 [Page 12]
Internet Draft MPLS OAM Requirements October 2003
in part, without restriction of any kind, provided that the
above copyright notice and this paragraph are included on
all such copies and derivative works. However, this
document itself may not be modified in any way, such as by
removing the copyright notice or references to the Internet
Society or other Internet organizations, except as needed
for the purpose of developing Internet standards in which
case the procedures for copyrights defined in the Internet
Standards process must be followed, or as required to
translate it into languages other than English.
The limited permissions granted above are perpetual and
will not be revoked by the Internet Society or its
successors or assigns. This document and the information
contained herein is provided on an "AS IS" basis and THE
INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE
DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR
PURPOSE.
9. Intellectual Property Rights Notices
The IETF takes no position regarding the validity or scope of any
intellectual property or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; neither does it represent that it
has made any effort to identify any such rights. Information on the
IETF's procedures with respect to rights in standards-track and
standards-related documentation can be found in BCP-11. Copies of
claims of rights made available for publication and any assurances
of licenses to be made available, or the result of an attempt made
to obtain a general license or permission for the use of such
proprietary rights by implementers or users of this specification
can be obtained from the IETF Secretariat.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights which may cover technology that may be required to practice
this standard. Please address the information to the IETF Executive
Director.
MPLS Working Group Expires April 2004 [Page 13]
