Network Working Group C. Filsfils, Ed.
Internet-Draft S. Previdi, Ed.
Intended status: Informational Cisco Systems, Inc.
Expires: April 29, 2015 J. Mitchell
B. Black
Microsoft Corporation
D. Afanasiev
Yandex
S. Ray
K. Patel
Cisco Systems, Inc.
October 26, 2014
BGP-Prefix Segment in large-scale data centers
draft-filsfils-spring-segment-routing-msdc-00
Abstract
This document describes a practical use case where BGP segment
routing can be used in a large-scale data center.
Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on April 29, 2015.
Filsfils, et al. Expires April 29, 2015 [Page 1]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Reference Diagram . . . . . . . . . . . . . . . . . . . . 3
2. BGP Prefix Segment . . . . . . . . . . . . . . . . . . . . . 5
3. Segment Routing Design . . . . . . . . . . . . . . . . . . . 5
3.1. Control Plane . . . . . . . . . . . . . . . . . . . . . . 6
3.2. Data Plane . . . . . . . . . . . . . . . . . . . . . . . 7
3.3. Network Design Variation . . . . . . . . . . . . . . . . 8
4. Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.1. MPLS Dataplane with operational simplicity . . . . . . . 9
4.2. Minimizing the FIB table . . . . . . . . . . . . . . . . 9
4.3. Egress Peer Engineering . . . . . . . . . . . . . . . . . 10
4.4. Capacity Optimization . . . . . . . . . . . . . . . . . . 10
4.5. Incremental Deployments . . . . . . . . . . . . . . . . . 11
4.6. Anycast . . . . . . . . . . . . . . . . . . . . . . . . . 12
5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 12
6. Manageability Considerations . . . . . . . . . . . . . . . . 12
7. Security Considerations . . . . . . . . . . . . . . . . . . . 12
8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 12
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 13
9.1. Normative References . . . . . . . . . . . . . . . . . . 13
9.2. Informative References . . . . . . . . . . . . . . . . . 13
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 14
1. Introduction
Segment Routing (SR), as described in
[I-D.filsfils-spring-segment-routing] leverages the source routing
paradigm. A node steers a packet through an ordered list of
instructions, called segments. A segment can represent any
instruction, topological or service-based. A segment can have a
local semantic to an SR node or global within an SR domain. SR
Filsfils, et al. Expires April 29, 2015 [Page 2]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
allows to enforce a flow through any topological path and service
chain while maintaining per-flow state only at the ingress node to
the SR domain. Segment Routing can be applied to the MPLS and IPv6
dataplanes.
The use-case described in this document focuses on SR applied to the
MPLS dataplane. In this context, a segment is encoded as an MPLS
label. An ordered list of segments is encoded as a stack of labels.
The segment to process is on the top of the stack. Upon completion
of a segment, the related label is popped from the stack. No
forwarding change is required to the MPLS dataplane.
The use-case described in this document should be considered in the
context of the BGP-based large-scale data-center (DC) design
described in [I-D.ietf-rtgwg-bgp-routing-large-dc] where eBGP3107
described in [RFC3107] is used instead of eBGP.
1.1. Reference Diagram
We reuse the 5-stage topology diagram from
[I-D.ietf-rtgwg-bgp-routing-large-dc] while adapting the device
naming to simplify the text.
Tier-3
+-----+
|NODE |
+->| 5 |--+
| +-----+ |
Tier-2 | | Tier-2
+-----+ | +-----+ | +-----+
+------------>|NODE |--+->|NODE |--+--|NODE |-------------+
| +-----| 3 |--+ | 6 | +--| 9 |-----+ |
| | +-----+ +-----+ +-----+ | |
| | | |
| | +-----+ +-----+ +-----+ | |
| +-----+---->|NODE |--+ |NODE | +--|NODE |-----+-----+ |
| | | +---| 4 |--+->| 7 |--+--| 10 |---+ | | |
| | | | +-----+ | +-----+ | +-----+ | | | |
| | | | | | | | | |
+-----+ +-----+ | +-----+ | +-----+ +-----+
|NODE | |NODE | Tier-1 +->|NODE |--+ Tier-1 |NODE | |NODE |
| 1 | | 2 | | 8 | | 11 | | 12 |
+-----+ +-----+ +-----+ +-----+ +-----+
| | | | | | | |
A O B O <- Servers -> Z O O O
Figure 1: 5-stage Clos topology
Filsfils, et al. Expires April 29, 2015 [Page 3]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
Briefly, we remind the salient points of the eBGP-3107 large-scale DC
design ([I-D.ietf-rtgwg-bgp-routing-large-dc])
o Each node is its own AS:
For simple and efficient route propagation filtering, Nodes 5,
6, 7 and 8 share the same AS, Nodes 3 and 4 share the same AS,
nodes 9 and 10 share the same AS.
For efficient usage of the scarce 2-byte private AS pool,
different tier-1 nodes might share the same AS.
Without loss of generality, we will simplify these details in
this document and assume that each node has its own AS.
o Each node peers with its neighbors via eBGP3107 session.
o Each node originates its loopback into BGP and announces it to its
neighbors.
o The forwarding plane at Tier-2 and Tier-1 is MPLS.
o The forwarding plane at Tier-3 is either IP2MPLS (if the host
sends IP traffic) or MPLS2MPLS (if the host sends MPLS-
encapsulated traffic).
For illustration purpose, we assume that:
o The AS of Node X is AS X.
o The loopback of Node X is 1.1.1.x/32.
In this document, we also refer to the Tier-3, Tier-2 and Tier-1
switches respectively as Spine, Leaf and ToR (top of rack) switches.
When a ToR switch acts as a gateway to the "outside world", we call
it a border switch.
Filsfils, et al. Expires April 29, 2015 [Page 4]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
+-----+ +-----+ +-----+
+---------->|NODE | |NODE | |NODE |
| | 4 |--+->| 7 |--+--| 10 |---+
| +-----+ +-----+ +-----+ |
| |
+-----+ +-----+
|NODE | |NODE |
| 1 | | 11 |
+-----+ +-----+
| |
A <- Servers -> Z
Figure 2: Path from A to Z via nodes 1, 4, 7, 10 and 11
2. BGP Prefix Segment
A BGP-Prefix Segment is a segment associated with a BGP prefix. A
BGP-Prefix Segment is a network-wide instruction to forward the
packet along the ECMP-aware best path to the related prefix
[I-D.keyupate-idr-bgp-prefix-sid].
In this document, we make the network design decision to assume that
all the nodes are allocated the same SRGB, e.g. [16000, 23999]. This
is important to fulfill the requirement for operational
simplification as explained in [I-D.filsfils-spring-segment-routing]
and [I-D.filsfils-spring-segment-routing-use-cases].
Note well that the use of a common SRGB in all nodes is not a
requirement, one could use a different SRGB at every node. However,
this would make the operation of the DC fabric more complex as the
label allocated to the loopback of a remote switch is then different
at every node.
For illustration purpose, we assume that the segment index allocated
to prefix 1.1.1.x/32 is X.
As a result, a local label 1600x is allocated for prefix 1.1.1.x/32
by each node throughout the DC fabric.
3. Segment Routing Design
Referring to Figure 1 and Figure 2 and assuming the IP address, AS
and index allocation previously described, this section details the
control plane operation and the data plane states for the prefix
1.1.1.11/32 (loopback of node 11).
Filsfils, et al. Expires April 29, 2015 [Page 5]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
3.1. Control Plane
Node 11 originates 1.1.1.11/32 in BGP and allocates to it the BGP-
Prefix Segment attribute (index11).
Node 11 sends the following eBGP3107 update to Node 10:
. NLRI: 1.1.1.11/32
. Label: Implicit-Null
. Next-hop: Node11's interface address on the link to Node10
. AS Path: {11}
. BGP-Prefix Attribute: Index 11
Node 10 receives the above update. As it is SR capable, Node10 is
able to interpret the BGP-Prefix Attribute and hence allocates the
label 16011 to the NLRI (instead of asking a "random/local" label
from its label manager). The implicit-null label in the update
signals to Node 10 that it is the penultimate hop and MUST pop the
top label on the stack before forwarding traffic for this prefix to
Node 11.
Then, Node 10 sends the following eBGP3107 update to Node 7:
. NLRI: 1.1.1.11/32
. Label: 16011
. Next-hop: Node10's interface address on the link to Node7
. AS Path: {10, 11}
. BGP-Prefix Attribute: Index 11
Node 7 receives the above update. As it is SR capable, Node 7 is
able to interpret the BGP-Prefix Attribute and hence allocates the
label 16011 to the NLRI (instead of asking a "random/local" label
from its label manager).
Node 7 sends the following eBGP3107 update to Node 4:
. NLRI: 1.1.1.11/32
. Label: 16011
. Next-hop: Node7's interface address on the link to Node4
. AS Path: {7, 10, 11}
. BGP-Prefix Attribute: Index 11
Node 4 receives the above update. As it is SR capable, Node 4 is
able to interpret the BGP-Prefix Attribute and hence allocates the
label 16011 to the NLRI (instead of asking a "random/local" label
from its label manager).
Node 4 sends the following eBGP3107 update to Node 1:
Filsfils, et al. Expires April 29, 2015 [Page 6]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
. NLRI: 1.1.1.11/32
. Label: 16011
. Next-hop: Node4's interface address on the link to Node1
. AS Path: {4, 7, 10, 11}
. BGP-Prefix Attribute: Index 11
Node 1 receives the above update. As it is SR capable, Node 1 is
able to interpret the BGP-Prefix Attribute and hence allocates the
label 16011 to the NLRI (instead of asking a "random/local" label
from its label manager).
3.2. Data Plane
Referring to figure 1, and assuming all nodes apply the same
advertisement rules described above, here are the IP/MPLS forwarding
tables for prefix 1.1.1.11/32 at nodes 1, 4, 7 and 10.
-----------------------------------------------
Incoming label | outgoing label | Outgoing
or IP destination | | Interface
------------------+----------------+-----------
16011 | 16011 | ECMP{3, 4}
1.1.1.11/32 | 16011 | ECMP{3, 4}
------------------+----------------+-----------
Figure 3: Node 1 Forwarding Table
-----------------------------------------------
Incoming label | outgoing label | Outgoing
or IP destination | | Interface
------------------+----------------+-----------
16011 | 16011 | ECMP{7, 8}
1.1.1.11/32 | 16011 | ECMP{7, 8}
------------------+----------------+-----------
Figure 4: Node-4 Forwarding Table
-----------------------------------------------
Incoming label | outgoing label | Outgoing
or IP destination | | Interface
------------------+----------------+-----------
16011 | 16011 | 10
1.1.1.11/32 | 16011 | 10
------------------+----------------+-----------
Figure 5: Node-7 Forwarding Table
Filsfils, et al. Expires April 29, 2015 [Page 7]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
-----------------------------------------------
Incoming label | outgoing label | Outgoing
or IP destination | | Interface
------------------+----------------+-----------
16011 | POP | 11
1.1.1.11/32 | N/A | 11
------------------+----------------+-----------
Node-10 Forwarding Table
3.3. Network Design Variation
A network design choice could consist of switching all the traffic
through tier-2 and tier-3 as MPLS traffic. In this case, one could
filter away the IP entries at nodes 4, 7 and 10. This might be
beneficial in order to optimize the forwarding table size.
A network design choice could consist in allowing the hosts to send
MPLS-encapsulated traffic (based on EPE use-case,
[I-D.filsfils-spring-segment-routing-central-epe]). For example,
Node 1 would receive Node11-destined MPLS-encapsulated traffic from
its attached host A and would switch this traffic on the basis of the
MPLS entry for 16011 (instead of classically receiving IP traffic
from A and performing an IPtoMPLS switching operation).
4. Benefits
The network design illustrated in this document retains all the
benefits explained in [I-D.ietf-rtgwg-bgp-routing-large-dc], namely:
o Bandwidth and traffic patterns
o Capex minimization
o Opex minimization
o Traffic Engineering
o Fast routing convergence
o Anycast for extra availability and load-balancing
Furthermore, it introduces the following benefits:
o MPLS dataplane with operational simplicity
o Minimization of the FIB table size
Filsfils, et al. Expires April 29, 2015 [Page 8]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
o Egress Peer Engineering
o Capacity Optimization
o Incremental Deployment
In the following sections, we detail the anycast benefit and the five
"additional" benefits introduced by the BGP-Prefix Segment
([I-D.keyupate-idr-bgp-prefix-sid]).
4.1. MPLS Dataplane with operational simplicity
As required by [I-D.ietf-rtgwg-bgp-routing-large-dc], no new
signaling protocol is introduced. The Prefix Segment is a
lightweight extension to BGP3107 [RFC3107]. LDP and RSVP-TE are not
used.
Thanks to the BGP-Prefix Segment extension
([I-D.keyupate-idr-bgp-prefix-sid]) and the design decision to use
the same SRGB at each node in the DC fabric, the troubleshooting of
the network is drastically simplified. At every node in the fabric,
the same label is associated to each remote prefix/switch.
When a controller (e.g. EPE controller in
[I-D.filsfils-spring-segment-routing-central-epe]) programs a host A
to send its traffic to host Z via the normal BGP multipath, the
controller uses label 16011 associated with the ToR switch connected
to the server Z. Specifically, the controller does not need to pick
the label based on the source ToR that the source host is connected
to.
In a classic BGP3107 design applied to the DC fabric illustrated in
Figure 1, the ToR switch 1 connected to server A would most likely
allocate a different label for 1.1.1.11/32 than the one allocated by
ToR switch 2. As a consequence, the controller would need to adapt
the SR policy to each host, based on the ToR switch that they are
connected to. This adds state maintenance and synchronization
problems. All this unnecessary complexity is eliminated thanks to
the BGP-Prefix Segment extension. Again, both the BGP-Prefix Segment
and the design decision to use a common SRGB on all nodes have made
this possible.
4.2. Minimizing the FIB table
The designer may decide to switch all the traffic at tier2 and
tier3's based on MPLS, hence drastically decreasing the IP table size
at these nodes.
Filsfils, et al. Expires April 29, 2015 [Page 9]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
This is easily accomplished by encapsulating the traffic directly at
the host, or at the source ToR switch by pushing the BGP-Prefix
Segment of the destination ToR for intra-DC traffic or border switch
for inter-DC or DC-to-outside-world traffic.
4.3. Egress Peer Engineering
It is straightforward to combine the design illustrated in this
document with the EPE use-case
[I-D.filsfils-spring-segment-routing-central-epe].
In such case, the operator is able to engineer its outbound traffic
on a per host-flow basis, without incurring any additional state at
intermediate points in the DC fabric.
For example, the controller only needs to inject a per-flow state on
the host A to force it to send its traffic destined to a specific
internet destination D via a selected border switch (say 12 in
Figure 1instead of another border switch 13) and a specific egress
peer of border switch 12 (say peer AS 9999 of local PeerNode segment
9999 at border switch 12 instead of any other peer which provides a
path to the destination D). Any packet matching this state at host A
would be encapsulated with SR segment list (i.e.: label stack)
{16012, 9999}. 16012 would steer the flow through the DC fabric,
leveraging any ECMP, along the best path to border switch 12. Once
the flow gets to border switch 12, the active segment is 9999. This
EPE PeerNode segment forces border switch 12 to forward the packet to
peer AS 9999, without any IP lookup at the border switch. There is
no per-flow state for this engineered flow in the DC fabric. The
per-flow state is only required at the source (source routing
benefits).
Note as well, that on top of allowing full engineering control, such
a design also offer FIB table minimization benefits as the internet-
scale IP lookup at border switch 12 might be avoided.
4.4. Capacity Optimization
It is straightforward to combine the centralized capacity
optimization process described in
[I-D.filsfils-spring-segment-routing-use-cases] with the design
introduced in this document.
For example, in Figure 1, the controller may detect a hot spot on
node 5. One way to alleviate the load is to deploy a set of per-
destination flow states at a set of ToR switches such that they send
they traffic via fabric paths that avoid Node 5.
Filsfils, et al. Expires April 29, 2015 [Page 10]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
For example, host A could be forced to go to host Z via Node 4. This
is conveniently programmed by the controller as a flow state for Z at
host A which pushes the segment list {16004, 16011}. 16004 steers the
traffic to node 4 via any ECMP path (e.g. multiple parallel links
from Node 1 to Node 4). 16011 then steers the traffic from node 4 to
node 11 load-balancing the traffic via nodes 7 and 8, and any ECMP
along that path. This flow is thus avoiding Node 5 while still
leveraging the maximum number of available ECMP paths. This is
realized without any intermediate per-flow state.
Another alternative state at A could be {16008, 16011}. In this case,
this flow would use any ECMP path up to node 8 and then any ECMP path
up to node 11.
While traffic-engineering within a DC has been rarely used in the
past, it is expected to eventually be required as Clos topologies get
optimized for higher scale [DRAGONFLY].
4.5. Incremental Deployments
Referring to , let us assume that node 7 does not support the BGP-
Prefix Segment attribute.Figure 2, let us assume that node 7 does not
support the BGP-Prefix Segment attribute.
From a signaling viewpoint, nothing would change as even if Node6
does not understand the BGP-Prefix Segment attribute, it does
propagate it unmodified to its neighbors.
From a label allocation viewpoint, the only difference is that Node7
would allocate a dynamic label to the prefix 1.1.1.11/32 (e.g.
12345) and would advertise that label to its neighbor Node4.
Let's highlight that Node4 does understand the BGP-Prefix Segment
attribute and hence allocates the indexed label in the SRGB (16011)
for 1.1.1.11/32.
As a result, all the dataplane entries across the network would be
unchanged except the entries at Node7 and its neighbor Node4 as shown
in the figures below.
------------------------------------------
Incoming label | outgoing | Outgoing
or IP destination | label | Interface
-------------------+----------------------
12345 | 16011 | 10
Figure 7: Node 7 Forwarding Table
Filsfils, et al. Expires April 29, 2015 [Page 11]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
------------------------------------------
Incoming label | outgoing | Outgoing
or IP destination | label | Interface
-------------------+----------------------
16011 | 12345 | 7
Figure 8: Node 4 Forwarding Table
The BGP-Prefix Segment functionality can thus be deployed
incrementally one node at a time.
Where it is deployed, the operator enjoys its benefits without any
dependency on the deployment state at any other node.
4.6. Anycast
The design presented in this document preserves the availability and
load-balancing properties of the base design presented in
[I-D.filsfils-spring-segment-routing].
For example, one could assign an anycast loopback 1.1.1.20/32 to the
border switches 11 and 12 (on top of their node-specific loopbacks).
Doing so, the EPE controller could express a default "go-to-the-
internet via any border switch" policy as segment list {16020}.
Indeed, from any host in the DC fabric, from any ToR switch, 16020
steers the packet towards the border switches 11 or 12 leveraging any
ECMP along the best paths to these switches.
5. IANA Considerations
TBD
6. Manageability Considerations
TBD
7. Security Considerations
TBD
8. Acknowledgements
TBD
Filsfils, et al. Expires April 29, 2015 [Page 12]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
9. References
9.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC3107] Rekhter, Y. and E. Rosen, "Carrying Label Information in
BGP-4", RFC 3107, May 2001.
9.2. Informative References
[DRAGONFLY]
Kim, J., Dally, W., Scott, S., and D. Abts, "Cost-
Efficient Dragonfly Topology for Large-Scale Systems",
2009.
[I-D.filsfils-spring-segment-routing]
Filsfils, C., Previdi, S., Bashandy, A., Decraene, B.,
Litkowski, S., Horneffer, M., Milojevic, I., Shakir, R.,
Ytti, S., Henderickx, W., Tantsura, J., and E. Crabbe,
"Segment Routing Architecture", draft-filsfils-spring-
segment-routing-04 (work in progress), July 2014.
[I-D.filsfils-spring-segment-routing-central-epe]
Filsfils, C., Previdi, S., Patel, K., Aries, E.,
shaw@fb.com, s., Ginsburg, D., and D. Afanasiev, "Segment
Routing Centralized Egress Peer Engineering", draft-
filsfils-spring-segment-routing-central-epe-02 (work in
progress), July 2014.
[I-D.filsfils-spring-segment-routing-use-cases]
Filsfils, C., Francois, P., Previdi, S., Decraene, B.,
Litkowski, S., Horneffer, M., Milojevic, I., Shakir, R.,
Ytti, S., Henderickx, W., Tantsura, J., Kini, S., and E.
Crabbe, "Segment Routing Use Cases", draft-filsfils-
spring-segment-routing-use-cases-01 (work in progress),
October 2014.
[I-D.ietf-rtgwg-bgp-routing-large-dc]
Lapukhov, P., Premji, A., and J. Mitchell, "Use of BGP for
routing in large-scale data centers", draft-ietf-rtgwg-
bgp-routing-large-dc-00 (work in progress), August 2014.
[I-D.keyupate-idr-bgp-prefix-sid]
Patel, K., Ray, S., Previdi, S., and C. Filsfils, "Segment
Routing Prefix SID extensions for BGP", 2014.
Filsfils, et al. Expires April 29, 2015 [Page 13]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
Authors' Addresses
Clarence Filsfils (editor)
Cisco Systems, Inc.
Brussels
BE
Email: cfilsfil@cisco.com
Stefano Previdi (editor)
Cisco Systems, Inc.
Via Del Serafico, 200
Rome 00142
Italy
Email: sprevidi@cisco.com
Jon Mitchell
Microsoft Corporation
One Microsoft Way
Redmond, WA 98052
United States
Email: Jon.Mitchell@microsoft.com
Benjamin Black
Microsoft Corporation
One Microsoft Way
Redmond, WA 98052
United States
Email: benblack@microsoft.com
Dmitry Afanasiev
Yandex
RU
Email: fl0w@yandex-team.ru
Filsfils, et al. Expires April 29, 2015 [Page 14]
Internet-Draft BGP-Prefix SID in large-scale DCs October 2014
Saikat Ray
Cisco Systems, Inc.
170, West Tasman Drive
San Jose, CA 95134
US
Email: sairay@cisco.com
Keyur Patel
Cisco Systems, Inc.
US
Email: keyupate@cisco.com
Filsfils, et al. Expires April 29, 2015 [Page 15]