Traffic Engineering WG Stephen Shew
Internet Draft Nortel Networks
Document: <draft-shew-lsp-restoration-00.txt> October 1999
Fast Restoration of MPLS Label Switched Paths
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026 [1].
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts. Internet-Drafts are draft documents valid for a maximum of
six months and may be updated, replaced, or obsoleted by other
documents at any time. It is inappropriate to use Internet- Drafts
as reference material or to cite them other than as "work in
progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
1. Abstract
This document describes a mechanism for fast detection of MPLS LSP
failure when a link fails and scales to all LSPs affected by the
failure. Fast detection enables ingress LERs to quickly recover
onto backup LSPs. A performance improvement in the reliability of
LSPs is expected. The mechanism described relies on a node and
network architecture that integrates L1/L2/L3 technology.
2. Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in
this document are to be interpreted as described in RFC-2119 [2].
Acronyms from the MPLS Architecture document [3] are used here.
Specifically, we use:
LSR - Label Switching Router
LSP - Label Switched Path
Additionally,
LER - Label Edge Router - an LSR which originates/terminates an LSP.
Shew Expires April 1999 [Page 1]
Internet Draft draft-shew-lsp-restoration-00.txt October 1999
3. Introduction
MPLS provides a useful mechanism for placing traffic in IP networks
which is a key capability for traffic engineering. Label Switched
Paths can be defined independent of L3 shortest paths and this
enables explicit engineering of traffic loads.
In an MPLS network supported by an underlying optical network,
increasing reliance on large LSPs is a problem because the impact of
an LSP failure could be extensive. Traditional SONET layer
protection could be used for the L2 segments in an MPLS network but
the efficiency of bandwidth usage is an issue.
The potential for MPLS mechanisms to provide recovery performance
similar to SONET has been mentioned [5] and a framework for
protection and recovery in MPLS networks is described in [6]. A
goal for MPLS protection then is to have better recovery performance
when there is an L1 link failure in the network and also be
efficient with bandwidth (i.e., no reserved protection bandwidth).
This document describes a mechanism for fast LSP failure detection
that is needed before recovery procedures can be executed. It also
describes an integrated network architecture in which reserved
protection bandwidth is optional.
4. Problem Space
In this draft, we want to consider the problems of engineering
reliability of router-router links and fast recovery of MPLS LSPs.
Specifically, the problem of fast failure detection and notification
of affected MPLS LSPs is addressed.
4.1 LSP failure detection
Fast recovery in MPLS is hampered by the fact that detecting an LSP
failure at the ingress LER can take a long time. After a break in
an LSP hop, Notification messages are propagated along the LSP
intermediate nodes back to the ingress LER. Message processing
occurs at each hop and this adds delay in informing the ingress LER
that the LSP has failed.
ATM has a similar problem with VC failure detection in that Release
messages also have to be processed at each intermediate switch on
the way back to the source node. I.610 and I.630 are attempts to
standardize fast detection and recovery methods in ATM but this
relies on support of OAM cells flowing along the VC.
In IP connectionless networks, failures affecting TCP sessions can
also take a long time to detect since the end-systems must decide if
the session went down. This is a consequence of the connectionless
paradigm where all you care about is maintaining connectivity.
Because connectionless recovery is dependent on IP routing,
detecting loss of connectivity can take seconds.
Shew Expires April 2000 [Page 2]
Internet Draft draft-shew-lsp-restoration-00.txt October 1999
The fastest detection occurs at the local end of a link failure.
Schemes that try to mend connections at the point of failure are
known as "local repair" schemes. In ATM PNNI, the connection
signalling procedure can crankbank to a earlier intermediate point
and then try to establish connectivity toward the destination.
Local repair has performance advantages in maintaining connectivity
but at the expense of efficiency (more hops, more bandwidth, more
end-to-end delay).
4.2 Scaling of LSP Failure Notification
A second problem with single L2 link failure is that multiple LSPs
can be affected and many (hundreds) ingress points must be informed.
This is computationally expensive if MPLS signalling (LDP or RSVP)
is used for each LSP. Because many LSP can be affected by a single
link failure, the magnitude of failure notification is an important
issue.
4.3 Magnitude of L1 Failure
Just as a single L2 failure can affect multiple LSPs, a single L1
failure can affect multiple L2 links. Here, failure detection is
not as much of a problem if L2 restoral mechanisms exist (e.g., ATM
I.630), but the effect of multiple simultaneous router-router link
failures is large on the stability of an IP network.
When an IP network operator leases lines for router-router links,
physical link diversity is a consideration. It can be difficult to
ensure that a physical link failure does not affect two or more
leased lines due to the multiplexing complexities of L1 and more
recently, L0 networks.
If an IP network runs over an L2 network (e.g., ATM), there can be
similar difficulty in ensuring minimum impact on router-router links
when an L2 link fails. Even if both IP and L2 networks are
controlled by the same organization, engineering for router-router
link reliability over shared L2 links is complex.
5. Solution Motivation
This section describes several concepts that help motivate the
solution presented later.
5.1 Overlay vs Integrated Networks
MPLS networks are less complex than an IP network overlayed on an L2
switched network. One of the reasons for this is that the L2/L3
topology is aligned and there is a single routing protocol that can
take action when an L2 link fails. This was noted in [4].
In an overlayed network, an L2 link could be part of two switched
connections that are actually router-router links. If that link
Shew Expires April 1999 [Page 3]
Internet Draft draft-shew-lsp-restoration-00.txt October 1999
fails, multiple router-router connections are affected which trigger
an IP routing protocol to update the topological and forwarding
views. If the L2 network also has a routing system (e.g., ATM PNNI)
then both L3 and L2 routing systems are run. The L3 routing system
will be affected by the L2 routing system in that the L2 routing
system may attempt to reroute or re-establish connections. This can
increase the detection time for a link failure at L3 because the L2
control layer has to declare the connection to be down first, and
the L2 connection tear down procedure may have to be executed across
multiple switches
A single routing system minimizes the link failure detection time
for L3 since there is no L2 control that precedes the L2 connection
down event.
5.2 L1 Detection
As noted earlier, L1 failure detection is fast due to physical
methods (loss of light, loss of carrier signal). This is an
attractive property. Further, in a TDM, optical mux (SONET), or
optical cross connect network, when a link fails all of the paths
(at that layer) which use the link go down.
Unlike higher layers, the endpoints of those paths detect the
failure quickly because the signalling of the failure is very fast
(e.g., AIS signals in SONET) and because the signalling is sent to
each channel of the failed link. So in L1 networks, the detection
of a failed connection is fast and scales well for all connections
on the failed link.
To be fair, the number of L1 connections on a link is not as high as
the number of say ATM VCs on a link. However, these detection
properties are highly desirable for L2 connections on L2 links.
Fast detection is possible with ATM that has hardware that can
handle inband OAM cells (I.630), but is not really tractable for
MPLS LSPs. This is because of the variety of L2 media (esp.
Ethernet and PPP) and the amount of packets that would have to be
sent to get fast (<100 millisecond) detection. Also, I.630 is
slower due to timers in some of its mechanisms (e.g., 3.5 seconds
for Loss of Continuity, up to 500ms for AIS injection).
6. L1/L2/L3 Integration Solution
A key to the solution for fast detection is the alignment of L1, L2,
and L3 capabilities into a single node. This architecture and its
impacts on the ability to detect LSP failure are now described.
6.1 L1/L2/L3 Integration
As was noted earlier, in MPLS LSRs, the alignment of the L3 and L2
topology brings some advantages in the speed at which the network
can react to a link failure. This integration is extended to
Shew Expires April 1999 [Page 4]
Internet Draft draft-shew-lsp-restoration-00.txt October 1999
encompass L1 components in order to realize further speed
advantages.
We define an L1/L2/L3 switch as an LSR combined with an L1 cross
connect switch. This could be a SONET Add/Drop Mux, an optical
cross connect, or traditional TDM switch. The integrated switch is
able to originate and terminate IP traffic from the L1 cross
connect. Conceptually, this is done over dedicated L1 channels
between the L1 cross connect and the pure IP router function of the
integrated switch.
The switch can also tandem L1 traffic through the L1 cross connect
component. This is similar to the way in which LSRs can tandem L2
data in their established tandem LSPs. In the L2 case, this is a
label swap. In the L1 case, it is for example, a mapping from one
time slot to another time slot on an outgoing interface.
Two L1/L2/L3 nodes are connected by a physical L1 link. A channel
in that link is used as a router-router IP link. For example, an
OC-3 channel of an OC-48 link with PPP over SONET for the framing.
This is analogous to the L2 control channel between two MPLS
switches connected over an ATM interface.
A key difference between this type of network and L2/L3 networks
which are overlayed on L1 networks, is that the L1/L2/L3 network
does not have any L1 paths which act as router-router links. In an
integrated network, the L3 routing protocol has a view of both the
L2 and L1 topology since those layers are now aligned.
Consider a SONET ring in Fig. 1 with 4 Add/Drop Muxes (ADMs) and an
LSR attached to each ADM. A typical configuration with an L1
overlay is to fully mesh the LSRs. Of the six router-router links,
four of them are one-hop channels between ADMS, and two of them are
actually SONET paths which bypass an ADM. If the ring is protected,
then if there is a fiber cut, all router-router links will be
preserved as the affected L1 paths would be L1 rerouted over the
protection bandwidth.
Converting such a network to an L1/L2/L3 network would involve the
elimination of the two-hop L1 paths that act as router-router links.
The protection bandwidth could also be fully used and as a
consequence of the failure recovery method of section 6.3, the L1
protection function is not needed.
LSR1
|
+---ADM1---+
| |
LSR4--ADM4 ADM2--LSR2
| |
+---ADM3---+
|
LSR3 Fig. 1 SONET ring with 4 LSRs
Shew Expires April 1999 [Page 5]
Internet Draft draft-shew-lsp-restoration-00.txt October 1999
Note that an L1/L2/L3 network can also be built with LSRs and TDM
switches.
6.2 L1 Cut-through Paths
In earlier IP over ATM work (e.g., MPOA, LANE, NHRP, use of OSPF
ARA), the notion of an "L2 cut-through" was defined. This is a VC
which is set up to directly connect two IP routers/hosts for a known
IP flow between those entities. MPLS re-uses the "L2 cut-through"
in a different manner. Instead of a separate L2 network around
which the L3 nodes are connected over L2 cut-throughs, MPLS combines
or integrates the L2 cut-throughs in the same L2/L3 network. That
is, every LSR is capable of L2 switching and L3 forwarding.
Cut-through paths are distinguished from L2 paths which are used as
L3 links. When an L2 path is an L3 link, it carries L3 routing
control traffic and is equivalent to a PPP link between IP routers.
Cut-throughs are not router-router links and don't carry routing
control traffic. Thus they don't need to appear in the L3 topology
database.
LSRs are used by inserting them as next hop entries in the IP
forwarding table of ingress LERs. If incoming IP connectionless
traffic matches a Forwarding Equivalence Class, the traffic is sent
to the corresponding LSP. Once on the LSP, the traffic is label
switched along the path to the end of the LSP and is independent of
how L3 forwarding would have directed it.
Existing L2 overlays on L1 networks exhibit the same separation as
earlier IP over ATM work. An L1 path is configured for one L2 link
between two L2 switches. This L1 path is a series of channels that
are connected by L1 cross connects. Typically, the service offered
over L1 paths is a leased line.
The same concept of MPLS integration can be applied to L1 paths.
Here, in an L1/L2/L3 network, an L1 path has an LSR at every cross
connect point. To use an L1 path, treat it as if were an LSP, or
overlay an LSP onto this path. That is, consider the L1 path as a
cut-through. When an incoming IP packet is matched to a Forwarding
Equivalence Class associated with this L1 cut-through, the IP
forwarding table entry points to the start of this L1 path. As with
L2 cut-throughs, an L2 header is added. The packet is sent to this
path and is then L1 switched until it reaches the end of the path.
At the termination point, the packet could be L2 switched or L3
forwarded.
Like existing LSPs, packets traversing an L1 cut-through are
independent of how L3 forwarding would have directed them. Also, L1
cut-throughs are not router-router links.
6.3 Fast Recovery
6.3.1 LSP Recovery
Shew Expires April 1999 [Page 6]
Internet Draft draft-shew-lsp-restoration-00.txt October 1999
Using L1 cut-throughs in an L1/L2/L3 network enables fast detection
of LSP failure. Consider two LSPs that are L1 cut-throughs:
LSR1-LSR2-LSR3-LSR4 and
LSR5-LSR2-LSR3-LSR6
If L1 link LSR2-LSR3 goes down, all nodes in both LSRs can detect
the path failure based on L1 physical methods. For example, loss of
light (Alarm Indication Signal in SONET) or carrier signal (TDM). In
particular, the LSP endpoints can determine that the LSP is down
much faster than the protocol based method in LDP of Notification
messages which is processed at each LSR on the paths back to the
ingress and egress. For example, propagation of the physical
failure is about 5 microseconds per kilometer.
Not only is the failure detection fast, but it scales for all LSPs
that are affected by a single L1 failure. In the example above, two
LSPs are notified, but if there were 192 paths in an OC192 link,
then all of their endpoints could detect the link failure within a
short period of time (a few milliseconds).
When an LSP failure is detected, the LSR can reroute the traffic to
a backup LSP. This backup LSP could be pre-defined to be link
disjoint from the primary LSP, and could also be set up in advance.
To avoid wasting dedicated bandwidth (i.e., a dedicated backup L1
cut-through), the backup LSP for the L1 cut-through could be an LSP
created over L2 connections which share bandwidth (e.g., ATM UBR
VC).
Assuming that a backup LSP is already set up, restoration of a
failed LSP that is overlayed on an L1 cut-through could be
implemented with similar performance to SONET Line and Ring
restoration.
For LSR which provide L3 connectionless forwarding, traffic from the
failed LSP could also be immediately handled by L3 forwarding if a
backup path LSP is not provided.
6.3.2 Router Link Recovery
In an L1/L2/L3 network, when a physical link goes down only one
router-router link is affected. This is in contrast to an overlayed
network where multiple router-router links could be affected by a
single L1 link failure. The alignment then of the layers can reduce
the magnitude of the L1 failure in the L3 topology.
Note that the IP routing update process executes in parallel with
the fast LSR recovery scheme. It will however, be much slower due
to protocol processing and topology database maintenance.
6.4 Applicability and Limitations
This scheme obviously requires either control of an L1 network,
and/or information from an L1 service provider (leased line) on L1
topology. Co-ordination of L1 changes would be important in the
Shew Expires April 1999 [Page 7]
Internet Draft draft-shew-lsp-restoration-00.txt October 1999
latter case. The MPLS network operator needs to be able to
configure L1 paths or have them configured so that matching LSPs can
be overlayed.
It is recognized that the number of channels in optical or TDM
multiplexing is less than the number of labels available in L2
networks (e.g., VPI/VCI space on an ATM interface) so the number of
LSPs is limited to hundreds, not tens of thousands.
LSPs overlayed onto L1 cut-throughs will have fixed bandwidths
unlike LSPs that share a common L2 link. They will also be
bidirectional since L1 facilities come this way.
Despite the above 2 limitations, the solution may be a good fit in
high performance IP backbones whose LSPs are 'core' LSPs which
contain stacked LSPs inside them.
The solution adheres to the principle of end-system control, in this
case, the LER contains the intelligence to use the L1 cut-through
and the recovery procedure.
7. Intellectual Property Considerations
Nortel Networks may seek patent or other intellectual property
protection for some or all of the technologies disclosed in this
document. If any standards arising from this document are or become
protected by one or more patents assigned to Nortel Networks, Nortel
intends to disclose those patents and licence them on reasonable and
non-discriminatory terms.
8. Security Considerations
In addition to security issues raised in [3], if the MPLS network
leases L1 services from another organization, then maintaining the
alignment of L1 switches with LSRs requires that the MPLS network
operator be notified of any changes in the L1 network. Otherwise,
L1 cut-throughs may not be correctly set up.
9. References
[1] Bradner, S., "The Internet Standards Process -- Revision 3",
BCP 9, RFC 2026, October 1996.
[2] Bradner, S., "Key words for use in RFCs to Indicate Requirement
Levels", BCP 14, RFC 2119, March 1997
[3] E. Rosen, A. Viswanathan, R. Callon, "Multiprotocol Label
Switching Architecture", Work in Progress, August 1999.
[4] C. Semeria, J. Stewart, "Optimized Routing Software for Reliable
Internet Growth", Juniper Networks white paper, July 1998
Shew Expires April 1999 [Page 8]
Internet Draft draft-shew-lsp-restoration-00.txt October 1999
[5] W.F. Maton, "MPLS and CA*Net 2/3", MPLS'99 Conference, June 1999
[6] S. Makam, V. Sharma, K. Owens, C. Huang, "Protection/Restoration
of MPLS Networks", <draft-makam-mpls-protection-00.txt>, work in
progress, June 1999.
10. Acknowledgments
The author would like to thank Ken Hayward for discussing many of
the ideas and issues in this draft.
11. Author's Addresses
Stephen Shew
Nortel Networks
PO Box 3511 Station C
Ottawa, ON
Canada K1Y 4H7
Phone: 613-763-2462
Email: sdshew@nortelnetworks.com
Shew Expires April 1999 [Page 9]
Internet Draft draft-shew-lsp-restoration-00.txt October 1999
Full Copyright Statement
"Copyright (C) The Internet Society (date). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implmentation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph
are included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.