Internet Draft David Allan, editor Document: draft-allan-mpls-oam-frmwk-02.txt Mina Azad Nortel Networks Enrique Cuevas Neil Harrison AT&T British Telecom Sanford Goldfless Eric Davalo Lucent Maple Optical Arun Punj Marcus Brunner Marconi NEC Chou Lan Pok Wesam Alanqar SBC TRI Sprint April 2002 A Framework for MPLS User Plane OAM Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Copyright Notice Copyright(C) The Internet Society (2001). All Rights Reserved. Abstract This Internet draft discusses many of the issues associated with user plane OAM for MPLS. The goal being to provide tools to perform "in service" maintenance of LSPs. Included in this discussion is some of the implications of MPLS architecture on the ability to support fault, diagnostic and performance management OAM applications, potential solutions for distinguishing user plane OAM, and a summary of what the authors believe can be achieved. Allan et.al Expires November 2002 Page 1 A Framework for MPLS User Plane OAM May 2002 This framework is predicated on requirements described in [HARRISON- REQ]. Sub-IP ID Summary WHERE DOES IT FIT IN THE PICTURE OF THE SUB-IP WORK Fits in the MPLS box. WHY IS IT TARGETED AT THIS WG MPLS WG is currently looking at MPLS-specific error detection and recovery mechanisms. The mechanisms and design aspects discussed in this document are for packet-based MPLS LSPs independent of payload. JUSTIFICATION The WG should consider this document, as it discusses the design aspects of error detection and measurement for packet based MPLS LSPs. Table of Contents 1. Conventions used in this document...............................3 2. Changes since the last version..................................3 3. Motivations.....................................................3 4. Requirements....................................................3 5. Domain Concepts.................................................3 6. Different deployment scenarios..................................4 7. MPLS architecture implications for OAM..........................5 7.1 Topology variations within an MPLS level....................5 7.1.1 Implications for fault management.........................7 7.1.2 Implications for performance management...................8 7.2 LSP Creation Method.........................................9 7.3 Lack of Fixed Hierarchy....................................10 7.4 Use of time to live (TTL)..................................11 7.5 Other design issues........................................12 8. OAM Applications...............................................12 9. Ease of Implementation.........................................13 10. OAM Messaging.................................................14 11. Distinguishing OAM user plane flows...........................15 11.1 Adding an LSP level with arbitrary label..................15 11.2 Adding an LSP level with a reserved label value...........15 11.3 Header modification.......................................15 12. The OAM return path...........................................16 13. Security Considerations.......................................18 14. A summary of what can be achieved.............................18 15. References....................................................19 16. Author's Addresses............................................19 Allan et. al. Expires November 2002 Page 2 A Framework for MPLS User Plane OAM May 2002 1. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 [1]. 2. Changes since the last version Addition of Sub-IP summary. Additional implications of fault management for LDP based MPLS applications. Discussion on implementation complexity according to both MPLS application and OAM function. Minor editorial changes. 3. Motivations MPLS OAM and survivability have been tackled in numerous Internet drafts. However all existing drafts focus on single provider solutions or focus on a single aspect of the MPLS architecture such as the use of RSVP or LDP signalling. This leads to inconsistent and inefficient applicability across the MPLS architecture, and/or requires significant modifications to operational procedure in order to provide OAM functionality. As MPLS matures and relationships between providers become more complex, there is a need to consider deployments that span arbitrary networking arrangements and have a broader and more uniform applicability to the MPLS architecture. 4. Requirements MPLS user-plane OAM specific requirements and a summary of requirements that have appeared in numerous PPVPN, PWE3, and MPLS documents appear in [HARRISON-REQ]. This Internet draft discusses the implications of extending OAM across the MPLS architecture, and adds additional user-plane OAM requirements and capabilities for managing multi-provider networks. This document also broadens the scope of the requirements discussion in identifying where certain OAM applications simply cannot be implemented without modifications to current practice/architecture. 5. Domain Concepts MPLS introduces a richness in layering which renders traditional definitions inadequate. In particular, it is noted that MPLS has no fixed layered hierarchy (this is a unique property that no other technology has offered before). Allan et. al. Expires November 2002 Page 3 A Framework for MPLS User Plane OAM May 2002 A provider may have MPLS peer providers, use MPLS transit from serving providers (and require MPLS or non-MPLS client transport), and offer MPLS transit to MPLS or non-MPLS clients). Further, the same provider may use a hierarchy of LSPs within their own network. Hence this Internet Draft defines the concept of an "Operations Domain" (to cover OAM capabilities operated by a single provider) that may only be a partition of the end-end LSP. Operations Domain functions are an interdependent mix of control-plane, user-plane (a.k.a. data-plane), and management-plane functions. An LSP "of level m" may span numerous Operational Domains (contiguous user plane) while the control and management planes may be disjoint. The goal is to provide OAM functionality for each LSP "of level m" regardless of "m", and independent of the LSP creation mechanism or payload. It is possible to have a hierarchy of operators (e.g. carriers of carriers), where overlay Operational Domains are opaque to the serving Domain. Therefore it is required that each LSP "of level m" implement its own OAM functionality, and the OAM applications are confined to the Operational Domains traversed at level "m". Note that this concept has subtle differences with concepts of horizontal and vertical hierarchy as defined in [HIERARCHY]. Vertical hierarchy usually refers to networking layer boundaries distinguished by technology. An operational domain may refer to an operator specific hierarchical subset of the LSP levels within the MPLS network and/or a horizonal partitioning within a specific LSP. Similarly there is a further way to consider the concept of operational domain and horizontal hierarchy. An operational domain may be hierarchically partitioned (e.g. OSPF "areas") but may be operationally integrated and contiguous. 6. Different deployment scenarios At the present time there are a number of deployment scenarios proposed for MPLS each with a number of subtleties from a user plane OAM perspective. Each can be viewed as a characteristic of an operational domain: The sparse model: This can be in conjunction with control plane signalling (e.g. MPLS based traffic engineering applied to an IP network) or with simple provisioned LSPs (no control plane signalling). The key feature being that the MPLS operational domain will most likely not have any-to-any connectivity at the MPLS layer within the operational domain due to the sparse use of LSPs to augment the served layer connectivity. This has operational and scalability implications as OAM connectivity must be explicitly added to the model, or the operator may be obliged to depend on "layer violations" embedded in OAM mechanisms which are strictly only relevant to a different higher layer network (e.g. [ICMP]) to generate a return path. Allan et. al. Expires November 2002 Page 4 A Framework for MPLS User Plane OAM May 2002 The ubiquitous model: This model generally combines MPLS, integrated routing and control to produce ubiquitous any-to-any connectivity within an operational domain. This may be combined with a hierarchy of LSPs to modify the topology presented to the client layer. This offers providers the option of utilizing the resources inherent to all planes of the Operational Domain in designing OAM functionality. These two models of MPLS connectivity can be stacked or concatenated to support numerous manners of peering and overlay networking arrangements between providers and users. A direct inference being that an operational domain will not necessarily have knowledge of the domains above and/or below it, and in the general case far less knowledge of (and certainly less control over) its peer domains. OAM applications for LSPs of a specific level are confined to an operational domain and its user plane peers. More recently there is a tendency to overlay a L2 or L3 VPN service level on the user-plane of an operational domain, with it's own identifiers and addressing, while tunneling control information across the control plane of the operational domain using BGP-4 [2547][KOMPELLA] or extended LDP adjacencies [MARTINI][HEINANEN]. From a user plane OAM perspective, we would consider this to be a separate operational domain, and anticipate that it is only a matter of time before such service levels evolve to span multiple operational domains (for example, an L2 or L3 VPN that spans multiple providers, or the introduction of tandem points at the user plane of the service level). 7. MPLS architecture implications for OAM 7.1 Topology variations within an MPLS level There are a number of topology variations in the MPLS architecture that have OAM implications. These are: - Uni-directional and bi-directional LSPs. A uni-directional LSP only provides connectivity in one direction, and if return path connectivity exists, it is an attribute of the operational domain (e.g. signalling, management or client layers), and not a unique attribute of the LSP. Bi-directional LSPs or specific return path (e.g. [CHANG]) have inherent symmetrical connectivity as an attribute of the LSP. - Multipoint-to-point (mp2p) LSPs are where a single LSP uses "merge" LSR transfer functions to provide connectivity between multiple ingress LSRs and a single egress LSR (sufficient information being present in the payload to permit higher layer demultiplexing at the egress). There are a number of problems inherent to mp2p topological constructs that cannot be addressed by traditional p2p mechanisms. One issue being that for some OAM applications (e.g. user plane fault propagation) OAM flows may require visibility at merge-points to limit the impact of partial failures or congestion. Allan et. al. Expires November 2002 Page 5 A Framework for MPLS User Plane OAM May 2002 "Best effort" mp2p LSPs may have fairness issues with some packet schedulers. This may complicate obtaining consistent measurements under congestion conditions. Explicitly routed mp2p LSPs with associated resource reservations are significantly more complex. The resource reservations required will be cumulative at merge points, and the ability to provide differentiated handling for specific ingresses disappears. One opinion would be that the complexity and difficulty in the configuration/maintenance of ER-mp2p LSPs significantly outweighs the alleged scalability benefits, and would not likely be deployed. - Penultimate Hop label Popping (PHP), is an optimization in the architecture in which the last LSR prior to the egress removes the (supposedly) redundant current MPLS label from the label stack. Therefore the ability to infer LSP specific context (OAM and other) is lost prior to reaching the final destination. MPLS does not provide for protocol multiplexing via payload identification (with the exception of the explicit IPV4 and IPV6 labels). PHP requires that the final hop have a common protocol payload (typically IP) or is able to map to lower layer protocol multiplexing capability (e.g. PPP Protocol Field or Ethernet ethertype) as the ability to infer payload from LSP label is lost. When a specific queuing discipline is associated with the LSP, such as reserved resources, the outgoing interface at the PHP hop must be able to provide the differentiated packet handling. Another scenario where PHP is employed is when the egress LSR is not actually MPLS user plane capable. This has user-plane OAM implications in that MPLS specific flows may need to terminate at the PHP LSR. This would include the potential requirement that the PHP LSR proxies OAM functions on behalf of the egress LSR. This will introduce complexity when any type of consequent actions are required. - E-LSPs [MPLSDIFF] in which a single LSP supports multiple queuing disciplines to support multiple QoS behavior aggregates. Ability to perform OAM functions on a "per behavior aggregate" basis is critical to managing E-LSPs. - Management plane provisioned LSPs, vs. control plane signalled LSPs. In many scenarios associated with a control plane, the topology of the LSP varies over time. This can be due to many reasons, implicit routing, dynamic set up of local repair tunnels etc. etc. - The potential existence of multiple LSPs between an ingress and an egress LSR. This can be for many reasons, L-LSPs, equal cost multipath routing etc. etc. Allan et. al. Expires November 2002 Page 6 A Framework for MPLS User Plane OAM May 2002 - The potential existence of multiple next hop label forwarding entries (NHLFEs) for a single incoming label. This is the scenario whereby the incoming label map (ILM) for an incoming label switch hop (LSH) maps to an inverse multiplex of NHLFEs which may be re- merged into a common egress or have multiple egress points. The mechanism for selecting the NHLFE to use may be proprietary and is performed on a packet by packet basis. Similarly such a construct can partially degrade. OAM tools not specifically aware of this construct may produce random results (insufficient frequency of failure to trigger threshold detection), or pathologically may only test a portion of the NHLFEs. Similarly performance monitoring is problematic as packets in flight cannot accurately be accounted for. - Use (and abuse) of per-platform label space. A per-platform label has significance at a nodal level and not just an interface level. Some of the more interesting applications being the ability to create unsignalled backup LSPs in "bypass tunnels" [SWALLOW]. Traffic arriving on multiple interfaces and/or LSP tunnels may use a common per-platform label and will have a common ILM and NHLFEs. This can have implications similar to mp2p and PHP depending on how it is used; packet origin information is not conserved when multiple sources use a common label. - p2mp and mp2mp LSPs (a.k.a. MPLS Multicast) is for further study. At the present time what placeholders exist in the architecture for multicast treat it as a separate protocol from "unicast" MPLS. These topological variations introduce complexity when attempting to instrument OAM applications such as performance management, fault detection, fault isolation/diagnosis, fault handling (e.g. consequent actions taken to avoid raising unnecessary alarms in client layers) and fault notification. 7.1.1 Implications for fault management mp2p, E-LSPs and PHP have implications for fault management, specifically if an LSR is required to have knowledge of both the ingress LSR and the specific LSP that an OAM message arrived on, or is expected to have knowledge of, and maintain state about the set of ingress LSRs for an LSP. OAM messaging needs to carry sufficient information to distinguish both the ingress LSR and the specific LSP. (This ability is expressed on these terms as LSPs are typically not given globally unique identifiers, more frequently some locally administered LSR-ID is used). Frequently it will not be possible to infer the ingress LSR and specific LSP simply via association with the LSP probes arrive on as such information is lost at merge points in mp2p LSPs or due to a PHP. This is true for both OAM messaging, and normal user plane payloads. There may be numerous reasons why an ingress-egress pair may have a plurality of LSPs between them, so the ability to Allan et. al. Expires November 2002 Page 7 A Framework for MPLS User Plane OAM May 2002 distinguish the source and purpose of specific probes beyond mere knowledge of the originating LSR is a hard requirement. 7.1.2 Implications for performance management Many performance management functions can be performed by obtaining and comparing measurements taken at different points in the network. Comparing ingress and egress statistics being the simplest and most obvious example (but is usually restricted to within a single domain). The key issue is ensuring that "apples-to-apples" comparison of measurements is possible. This means that all measurement points need to be able to similarly classify what they are measuring, and that the measurements are synchronized in time and compensate for traffic in flight between the measurement points. For example, a relatively simple technique for establishing key performance metrics would be to compare what was sent with what was received. For example the PPP line quality monitoring (LQM) function the ingress periodically sends statistics to the egress for comparison subject to the same queuing discipline as the user plane traffic, such that traffic in flight is properly accounted for. (Note that re-ordering will introduce errors but is not expected to be frequent, examples of re-ordering situations would be routing changes (e.g. due to protection-switching), or E-LSPs encountering congestion). It is also important to distinguish, and be able to measure, what constitutes the up and down states of an LSP. This also needs to be standardized so that there is unified treatment. A key observation here is that and QoS metrics (like loss, errored packets, delay, etc) are strictly only relevant to when the LSP is in the up-state; and so any collection of QoS measurements is suspended when the LSP enters the down-state. This requires specification of the state transitions to achieve measurement consistency, a pre-requisite to QoS assessment. This is a particularly important metric to operators, since customers will be expecting operators to be able to offer both QoS and availability SLAs, and so these must be differentiated and uniquely measurable. Returning to the measurement of QoS metrics, such a simple ingress/egress comparison is not always possible, there is not necessarily the ability to similarly classify what is being measured at the ingress and egress of an LSP. mp2p LSPs and PHP do not have a 1:1 relationship between the ingress and the egress. LSPs containing ILMs that map to multiple NHLFEs introduce measurement inaccuracy as not all packets share a common queuing discipline and where this results in multiple egress points from the network, there is an inability to synchronize measurements. Further, partial failure of an mp2p LSP will result in the inability to successfully collect statistics So, in addition to having to define up/down-state transitions, for successful PM the 1:1 relationship needs to be restored by either: Allan et. al. Expires November 2002 Page 8 A Framework for MPLS User Plane OAM May 2002 - The mp2p/PHP LSP is modeled as a collection of "ingress" LSPs for measurement. This means that the egress needs to be able maintain statistics by ingress and appropriately classify traffic measurements. In which case the measurement result of common LSP segments could be misleading. - The mp2p/PHP LSP is modeled as one LSP for measurement. This means that measurements performed at ingress points need to be synchronized and adjusted for common LSP segments such that the results are all presented to the egress simultaneously (again correcting for traffic in flight), a technique dependent on such a high degree of synchronization would be impossible to perfect, hence prone to a degree of error. Neither of the above is achievable at the present time without modifying existing operational procedures, such as overlaying p2p connectivity on top of a merge/PHP based transport level. The existence of E-LSPs adds a wrinkle to the problem of measurement synchronization. An E-LSP may implement multiple diffserv PHBs and incorporate multiple queuing disciplines. An aggregate measurement for the entire LSP sent from ingress to egress would frequently have a small margin of error when compared with an aggregate measurement taken at the egress. Separate measurement comparisons for each supported EXP code point would be required to eliminate the error. The situation is slightly different for p2p LSPs containing ILMs that map to multiple NHLFEs. If all the NHLFEs are merged back into a single entity prior to the egress, then the upshot is that there will inherently be a degree of measurement error that modifications to operational procedure cannot correct. However there is no guarantee that this will be the case, and any individual ingress measurement may be compared with only one of several egress measurement points (either random or pathological). 7.2 LSP Creation Method The ability to usefully audit the constituent components of an LSP is dependent on the technique used to create the LSP.Presently defined are provisioning, LDP, CR_LDP, RSVP-TE, and BGP. LSP creation techniques that are currently defined fall at two relative extremes: At one extreme is explicitly routed point-to-point connection between fixed ingress and egress points in the network. Explicitly routed (ER) LSPs (today created via provisioning, CR-LDP, RSVP-TE or BGP) have a significant degree of testability as the path across the network and the egress point is fixed and knowable to a testing entity. Similarly explicit pairwise and stateful Allan et. al. Expires November 2002 Page 9 A Framework for MPLS User Plane OAM May 2002 testing/measurement relationships can be set up (e.g. connectivity verification) and strict criteria for failure established. In the middle is static mp2p constructs typically signalled via BGP (e.g. RFC 2547). At the other extreme is when LSP construction is topology driven (such as dynamic "shortest path first" routing combined with LDP), whereby the details of path construction between the ingress and egress points in the network will vary over time and may involve several stages of multiplexing with traffic from other sources. The details of path construction at any given instant are not necessarily knowable to an auditing entity so any attempt to interpret the results of an audit may generate spurious results. Further, the MPLS network may only be a portion of the operational domain, and the egress point from the network for an FEC may vary over time. The testable unit in an LDP network is the FEC not the LSP, and the potential existence of a many to many relationship of ingress and egress points limits the testability of the FEC, or at least may limit the frequency of using such tests. The connectivity instantiated in a specific LSP created by a topology driven control plane signalling mechanism will recover from many defects in the network. The quality of recovery typically being a function of how the network is engineered. Problems are typically detected by having MPLS connectivity fate share with the constituent physical links and routing adjacencies, and topology driven path re-arrangement will restore the connectivity (with some interruption and other side effects occurring between the initial failure and re-convergence of the network). However exclusive dependence on fate sharing for failure detection means that LSP components may have unique failure modes from which the network will not recover and can only be diagnosed reactively. As can be inferred from the above, what is required for topology driven LSPs is a test mechanism that audits forwarding policy as this is the metric by which some aspects of network performance can be measured. 7.3 Lack of Fixed Hierarchy MPLS supports an arbitrary hierarchy in the form of label stacking. This is a facility that can be leveraged for OAM purposes. As an example, the section on implications for performance management has already outlined how p2p topology for PM can be overlaid on an arbitrary merged topology to add manageability of services. Similarly functions requiring sectionalization of an LSP or ability to isolate partial failure of a complex construct can be achieved by constructing the LSP as an overlay upon a concatenation of operationally significant shorter LSPs. By operationally significant we would refer to LSPs that spanned useful portions of the whole Allan et. al. Expires November 2002 Page 10 A Framework for MPLS User Plane OAM May 2002 construct (e.g. a branch of an mp2p LSP, or bypassed LSRs that did not have OAM capability). This could simplify the instrumentation of level specific OAM by ensuring only e2e functions were required (as opposed to functions originating or terminating at arbitrary points in the network), while driving up the complexity of LSP establishment due to the resultant inter-level configuration issues when creating multi-level constructs with the desired manageability. 7.4 Use of time to live (TTL) Experience within the IP world has suggested that TTL was a serendipitous feature that can most likely be similarly leveraged by MPLS. However in the MPLS world, TTL suffers from inconsistent implementation depending on the link layer technology spanned by the target LSP. The existence of non-TTL capable links (e.g. MPLS/ATM) has impact on the utility of using TTL to augment the MPLS OAM toolkit. For example, use of TTL as an aid in fault sectionalization can only isolate a fault to the granularity of a non-TTL capable span of LSH or LSP segments. There are other variations in TTL handling that suggest interpreting results of TTL based tests may be problematic. As outlined in [TTL] there are two models of TTL handling with different implications: - the uniform model, in which decrement of TTL is independent of the MPLS level. At the ingress point to an MPLS level, the current TTL is copied into the new top label, and at egress is copied back to the revealed top level. - the pipe and short pipe models, whereby MPLS tunnels (aka LSPs) are used to hide the intermediate MPLS nodes between LSP Ingress and Egress from a TTL perspective. The uniform model originates with preserving IP TTL semantics when IP traffic transits an MPLS subnetwork. The uniform model will reduce the resource consumption of routing loops, but in a correctly operating network may lead to premature discard of packets outside the operational domain they originated from (due to the existence of an arbitrary number of serving MPLS levels). Similarly when a routing loop occurs, diagnosing the MPLS level that is the source of the problem will be difficult as there is no method to correlate it with the level where the exhaustion event occurred. The pipe model is more consistent with the operational domain model in that TTL exhaustion will only occur at a specified level and the initial values used at LSP ingress are more likely to be reflective of detecting what would genuinely constitute a routing loop. Allan et. al. Expires November 2002 Page 11 A Framework for MPLS User Plane OAM May 2002 A reasonable expectation is that the uniform model would not be used outside of an operational domain. A separate issue is that it is also possible that an LSR may decrement TTL by an amount other than one as a matter of policy. This means that the results obtained via any tools that use TTL exhaustion will require some interpretation. 7.5 Other design issues It is desirable to make the user plane OAM implementations independent of LSP specifics. It would be desirable to have common mechanisms across p2p and mp2p LSPs, PHP or no-PHP and independent of payload and the method of LSP creation. Via specification of common mechanisms extremely complex relationships (in terms of fault specification/handling) when a fault spanned more than one mode of OAM type can be avoided. The OAM application originator should not need (as far as is practical) any knowledge of the details of LSP construction. PM may impose certain operational procedures such as the implementation of many OAM applications only being possible for p2p LSPs and will most likely be segregated into only being possible for a select group of levels (e.g. overlaid service labels as per [KOMPELLA] or [MARTINI]). Fault management must be applicable across the spectrum of all label levels and LSR transfer functions. Finally, the possibility of re-ordering of OAM messaging must be considered. The design of OAM applications and messaging must be tolerant of out of order delivery. For some applications the originator/termination will require a means to uniquely correlate requests with probe responses (including responses to mis-directed probes) or verify in sequence receipt. 8. OAM Applications The purpose of having user plane LSP specific OAM transactions is to support useful OAM operator required applications. Examples of such applications include: Fault management - On demand verification: the ability to perform connectivity tests that exercise the specific LSP and the provisioning at the ingress and egress. On demand suggests that verification may be performed on an ad-hoc basis. - Fault detection: Operators cannot expect customers to act as fault detectors, and so the ability to perform automated detection of the failure of a specific LSP is a "must have" feature (although when one reviews the section on LSP creation above, one realizes it will Allan et. al. Expires November 2002 Page 12 A Framework for MPLS User Plane OAM May 2002 not be ubiquitously used). Some MPLS deployment scenarios may not have a control plane or may have LSP processing components not in common with the control plane, so fault detection procedures may need to be augmented with LSP specific methods. - Fault sectionalization: The ability to efficiently determine where a failure has occurred in an LSP. Sectionalization must be able to be performed from an arbitrary LSR along the path of the LSP. - Fault Propagation: specific MPLS deployment scenarios may not have a control-plane to propagate LSP failure information. Fault propagation has numerous forms and there are variations depending on whether the failure is in the serving layer/level or : i) Northbound from the failed level to the management plane. ii) Within the failed level. iii) From the failed level to its clients. iv) Within the client level to the LSP ingress and egress either via the user or control planes. And in all cases it is the termination of a layer that performs the function. Performance management - The ability to determine whether an LSP meets certain goals with respect to latency, packet loss etc. - The ability to collect information to facilitate network engineering decisions. Of the above applications, verification, detection and sectionalization explicitly need to exercise all components of the forwarding path of the target LSP, otherwise there will be failure scenarios that cannot be detected or properly sectionalized. These applications cannot be supported properly if there are differences in handling between user traffic and OAM probes at intermediate LSRs. 9. Ease of Implementation The ease of implementation is ultimately reflected in ease of adding hardware assist. Complex functions are typically implemented in software and are not capable of handling line rate messaging. Implementations benefit from assistance in the form of rate-limiting or similar load management techniques to avoid vulnerabilities to DOS attacks or simple mis-use by incompetent craftspersons. Mechanisms used for proactive fault detection gain the most benefit when they can run at line rate such that there are no concerns about processing capacity. Such tests will generate predicable results (or at least not have results degraded when network elements are under stress) and automated procedures can be designed around such mechanisms. Allan et. al. Expires November 2002 Page 13 A Framework for MPLS User Plane OAM May 2002 Messaging mechanisms to perform diagnostic tests (once a fault has been authoritatively established) tend to be more complex and software intensive. Diagnostic tests are frequently used by craftspersons, and can be more tolerant of things like discard due to rate limiting. 10.OAM Messaging OAM should be decoupled from user behavior to ensure consistent OAM functional behavior (under any traffic conditions) and avoid the use of customers as guinea pigs. We consider it to be self evident that providers will require a toolkit that includes some form of user plane OAM messaging. At the specific LSP level, support of OAM applications require messages that flow between three entities, the LSP ingress, the intervening network and the LSP egress. As an LSP is unidirectional, it should be self evident that OAM applications that require feedback in the reverse direction will have such communication occur either at the specific LSP level, or some user plane LSP level in the operational domain, or one of the other planes (control or management) of the operational domain. The set of possible individual transactions (plus examples of their utility) is as follows: LSP specific user-plane transactions: - ingress to egress applicability: verification, fault detection, performance management - ingress to network message will terminate at an intermediate LSR traversed by the LSP. Applicability: sectionalization from source - network to egress message is inserted into the LSP at an intermediate node and terminates at the LSP egress LSR. Applicability: sectionalization from arbitrary point in an LSP. - Network to network Applicability: sectionalization from arbitrary point in an LSP. Feedback transactions - egress to ingress applicability: verification, fault detection. - egress to network flow originates at the LSP egress and terminates at an intermediate node traversed by the LSP. Applicability: sectionalization from arbitrary point in an LSP. - network to ingress Allan et. al. Expires November 2002 Page 14 A Framework for MPLS User Plane OAM May 2002 flow will originate at an intermediate LSR traversed by the LSP and terminate at the LSP source. Applicability: sectionalization from ingress. - network to network Applicability: sectionalization from arbitrary point in an LSP. 11.Distinguishing OAM user plane flows MPLS does not currently provide for protocol multiplexing at a specific LSP level. However a requirement still exists to distinguish per-LSP OAM messaging from user payload. The options for addressing the identification of OAM flows are: 11.1 Adding an LSP level with arbitrary label OAM flows could be identified by adding an LSP level to the existing LSP using an arbitrary label value that by convention (negotiated at LSP establishment) carries the OAM payload. Adding an arbitrarily labeled LSP level to multiplex OAM flows will add complexity and additional failure modes that make it an undesirable solution. 11.2 Adding an LSP level with a reserved label value Adding an LSP level to the existing LSP via the use of a stacked reserved label value that explicitly identifies OAM flows. Note that this approach was proposed in [HARRISON-MECH]. Adding an LSP level using a reserved label has a number of virtues: - LSP level OAM flows will explicitly exercise the components of the forwarding path. - LSP level OAM flows will not be erroneously forwarded outside the LSP they have been inserted into. The LSP itself may be defective or misrouted (but that is a separate issue) but we have not introduced an additional LSP level for OAM with its own set of possible defects. - LSP level e2e OAM flows are transparent to non-compliant/legacy equipment. However hop-by-hop OAM flows are not possible (although the approach could be augmented via the use of MPLS TTL or the router alert label to gain SOME of the benefits of hop-by-hop messaging). By losing hop-by-hop, the ability to provide flows with preferential treatment when required is also lost. 11.3 Header modification Modifying the MPLS header or stealing a bit from the label space to permit OAM payload to be uniquely distinguished by all LSRs traversed at the specific LSP level similarly has a number of advantages and disadvantages. We gain hop-by-hop visibility of messaging but at the expense of: Allan et. al. Expires November 2002 Page 15 A Framework for MPLS User Plane OAM May 2002 - Losing backwards compatibility with legacy equipment. - Losing the ability to ensure we are fully exercising the forwarding path for verification and sectionalization. Similarly stealing a label bit for PHP and E-LSPs becomes problematic as the OAM identifier is lost when the label is popped (in the original MPLS architecture, the label was considered to have no informational value past the next to egress LSR). Adding special handling for OAM packets to specifically avoid PHP would no longer exercise all components of the forwarding path. The use of a reserved label under the top label would appear to be the approach that has the most utility and least impact on current deployments. 12.The OAM return path The ability to use OAM applications such as single-ended monitoring of both directions from one end, or to support applications such as protection switching in a 1/N:M case, requires the existence of a return path to the LSP ingress. This enhances the scalability and reliability of some OAM applications as initiation need only occur at a single LSR, all further coordination of LSRs exercised by the application being performed by user plane messaging inherent to the OAM application. A specific example being use of a loopback where the only place state and timing need be maintained is at the loopback originator. This requires a return path to complete the loop between the "target LSP" and the OAM application originator. This will permit reliable transaction flows to be implemented that impose minimal state on the network. For the few OAM applications that require a return path, the return path can be tolerant of being topologically disjoint with the target LSP (providing the differential delays are small, ie <<1s), reachability of the application originator being the only hard requirement. Similarly, different OAM applications will have different return path requirements, and a hybrid of using all the planes of the operational domain (according to the application) may be significantly simpler and more operationally tractable than significant modifications to current usage to fill in connectivity gaps at the specific label level. This is a key point, LSPs are currently by definition uni- directional (bi-directional to date being a construct of multiple uni-directional LSPs), so for any non-ubiquitous deployment of MPLS connectivity, some modification of operational procedure to provide for OAM messaging will be required for the few applications that need it. Strict symmetry of connectivity at a specific label level is not guaranteed. Allan et. al. Expires November 2002 Page 16 A Framework for MPLS User Plane OAM May 2002 In any type of sparse usage scenario (e.g. provisioned LSPs or use exclusively for TE) there will not be an inherent any-to-any connectivity in the user plane, and there may not be a control plane signalling system. Additional artificial constructs such as a "reverse notification tree" [CHANG] have been proposed to address this although these introduce additional operational complexity and a requirement for OAM for the OAM connectivity. In an implicit MPLS topology (e.g. LDP DU), any to any connectivity will typically exist, or will be easily available with minor alterations to operational procedure (LSRs advertise selves as FECs). This would continue to be true for an integrated model in which TE and an implicit topology were combined. In any type of multi-provider MPLS topology, the scenario is more complex, as for numerous reasons a provider may not wish to provision/advertise external connectivity to their LSRs. Similarly, for security reasons, providers may wish to apply some degree of policy or filtering of OAM traffic at operational domain boundaries. User plane OAM messaging should be designed to leverage as much "free connectivity" as can be obtained in the network, while ensuring the constructs have sufficient extensibility to ensure the corner cases are covered. Within the operational domain of a single provider, it is relatively easy to envision that a combination of user-plane, and control plane functionality will ensure that a user-plane return path is frequently available (although it may be topologically disjoint from the target LSP). This is less so for inter provider scenarios. Here there are a number of potential obstacles such as: - disjoint control plane - disjoint addressing plan - requirements for policy enforcement and security - impacts to scalability of ubiquitous visibility of individual LSRs across multiple operational domains. There are a number of approaches to providing inter-domain OAM connectivity, the following is a brief commentary on each: 1) Reverse Notification Tree (a.k.a using bi-directional LSP) In this method, each LSP has a dedicated reverse path - i.e. the reverse path is established and associated with the LSP at the LSP setup time. This requires binding the reverse path to each LSR that is traversed by the LSP. This method is not scaleable, as it requires doubling the number of LSPs in the network. Moreover each reverse path requires its own OAM. 2) Global OAM capability Similar to IP v4 to IP v6 migration methodology, this method proposes use of a global operations domain with control-plane, user- plane, and management-plane that interact with control-plane, user- Allan et. al. Expires November 2002 Page 17 A Framework for MPLS User Plane OAM May 2002 plane, and management-plane of individual operations domains. This method requires commitment and buy-in from all network operators. 3) Inter-domain OAM gateway This method proposes use of a gateway like functions at LSRs that are at operations domain boundaries. OAM gateway like functions includes capabilities to correlate OAM information from one operations domain to another and permit inter-carrier sectionalization problems to be resolved. Specification of an inter-domain OAM gateway capability would appear to be the most realistic solution. 13.Security Considerations Support for intra-provider user plane OAM messaging does not introduce any new security concerns to the MPLS architecture. Though it does actually address some that already exist, i.e. through rigorous defect handling operator's can offer their customers a greater degree of integrity protection that their traffic will not be misdelivered (for example by being able to detect leaking LSP traffic from a VPN). Support for inter-provider user plane OAM messaging introduces a number of security concerns as by definition, portions of LSPs will not be in trusted space, the provider has no control over who may inject traffic into the LSP. This creates opportunity for malicious or poorly behaved users to disrupt network operations. Attempts to introduce filtering on target LSP OAM flows may be problematic if flows are not visible to intermediate LSRs. However it may be possible to interdict flows on the return path between providers (as faithfulness to the forwarding path is not a return path requirement) to mitigate aspects of this vulnerability. 14. A summary of what can be achieved. This draft identifies useful MPLS OAM capability that potentially could be provided via user plane OAM functions. In particular with respect to automatic fault detection and failure handling. This draft suggests that it may be possible to provide this capability for any level in the label stack and across the full set of topological constructs available in the MPLS architecture. This "any level"/"any construct" applicability is a key requirement. This draft also identifies that many aspects of performance management are problematic without modifications to operational procedure. Any type of comparative measurement between the ingress and egress requires a 1:1 cardinality, or the ability of the egress to uniquely determine the ingress for each measured unit of communication, something that LSP merge, PHP and possible use of per platform label space at the measured LSP level undermine. Services requiring performance management functionality will not be able to Allan et. al. Expires November 2002 Page 18 A Framework for MPLS User Plane OAM May 2002 utilize the full set of constructs in the MPLS architecture at the service level. 15. References [CHANG] Owens et.al., "A Path Protection/Restoration Mechanism for MPLS Networks", draft-chang-mpls-path-protection-02.txt, IETF work in progress, November 2000. [HEINANEN] Heinanen, J., "Directory/LDP Based Ethernet VPNs", draft-heinanen-dirldp-eth-vpns-01.txt, IETF work in progress, November 2001 [HIERARCHY] Lai et.al. " Network Hierarchy and Multilayer Survivability", draft-ietf-tewg-restore-hierarchy-00.txt, IETF Work in Progress, September 2001 [ICMP] Bonica et. al. "ICMP Extensions for MultiProtocol Label Switching", draft-ietf-mpls-icmp-02.txt, IETF Work in Progress, August 2000. [KOMPELLA] Kompella et.al. "MPLS-based Layer 2 VPNs", draft-kompella-mpls-l2vpn-02.txt, IETF Work in Progress, December 2000 [MARTINI]Martini et.al. "Transport of Layer 2 Frames Over MPLS", draft-martini-l2circuit-trans-mpls-06.txt, IETF Work in Progress, May 2001 [MPLSDIFF] Le Faucheur et.al. "MPLS Support of Differentiated Services", draft-ietf-mpls-diff-ext-09.txt, IETF Work in Progress, April 2001 [2547] Rosen, E. Rekhter, Y., "BGP/MPLS VPNs", IETF RFC 2547, March 1999 [HARRISON-MECH] Harrison et.al. "OAM Functionality for MPLS Networks", draft-harrison-mpls-oam-00.txt, February 2001 [HARRISON-REQ] Harrison et.al. "Requirements for OAM in MPLS Networks", draft-harrison-mpls-oam-req-01.txt, November 2001 [SWALLOW] Swallow, G. and Goguen, R., "RSVP Label Allocation for Backup Tunnels", draft-swallow-rsvp-bypass-label-01.txt, November 2000 [TTL] Agarwal, P., and Akyol, B., "TTL Processing in MPLS Networks", draft-agarwal-mpls-ttl-01, October 2001 16. Author's Addresses Allan et. al. Expires November 2002 Page 19 A Framework for MPLS User Plane OAM May 2002 David Allan Nortel Networks Phone: 1-613-763-6362 3500 Carling Ave. Email: dallan@nortelnetworks.com Ottawa, Ontario, CANADA Mina Azad Nortel Networks 3500 Carling Ave. phone: 1-613-763-2044 Ottawa, Ontario, CANADA Email: mazad@nortelnetworks.com Enrique G. Cuevas AT&T Room D3-2B25 Phone: +1 732 420 3252 200 S. Laurel Avenue E-mail: ecuevas@att.com Middletown, NJ 07748 USA Neil Harrison Phone: 44-1604-845933 British Telecom Email: neil.2.Harrison@bt.com Sanford Goldfless Lucent Technologies 55 Fairbanks Rd. Marlborough, MA 01752 USA Email: sgoldfless@lucent.com Eric Davalo Maple Optical Systems 3200 North First Street Phone: 408 545 3110 San Jose CA 95134 Email: edavalo@mapleoptical.com Arun Punj Marconi Communications 1000 Marconi Drive, Warrandale - PA - 15086 Email: Arun.Punj@marconi.com Marcus Brunner Network Laboratories - NEC Europe Ltd. Adenauerplatz 6 Phone: +49 (0)6221/ 9051129 D-69115 Heidelberg, Germany Email: brunner@ccrle.nec.de Chou Lan Pok SBC Technology Resources, Inc. 4698 Willow Road, Phone: 925-598-1229 Pleasanton, CA 94583 Email: pok@tri.sbc.com Wesam Alanqar Sprint 9300, Metcalf Ave, Phone: +1-913-534-5623 Overland Park, KS 66212 wesam.alanqar@mail.sprint.com Allan et. al. Expires November 2002 Page 20