Network Working Group                                 Huajin Jeng (AT&T)
Internet Draft                                       James Uttaro (AT&T)
Expiration Date: March 2012                         Luay Jalil (Verizon)
Intended Status: Proposed Standard       Bruno Decraene (France Telecom)
                                        Yakov Rekhter (Juniper Networks)
                                       Rahul Aggarwal (Juniper Networks)
                                                       September 6, 2011

                 Virtual Hub-and-Spoke in BGP/MPLS VPNs

                 draft-rekhter-l3vpn-virtual-hub-02.txt


Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups. Note that other
   groups may also distribute working documents as Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.


Copyright Notice

   Copyright (c) 2010 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document. Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.



                                                              [Page 1]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


Abstract

   With BGP/MPLS VPNs any-to-any connectivity among sites of a given
   Virtual Private Network would require each Provider Edge router that
   has one or more of these sites connected to it to hold all the routes
   of that Virtual Private Network. The approach described in this
   document allows to reduce the number of Provider Edge routers that
   have to maintain all these routes by requiring only a subset of these
   routers to maintain all these routes.


Specification of Requirements

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].



1. Overview

   With BGP/MPLS VPNs [rfc4364] providing any-to-any connectivity among
   sites of a given Virtual Private Network (VPN) is usually
   accomplished by requiring each Provider Edge router (PE) that has one
   or more of these sites connected to it to hold all the routes of that
   VPN.  The approach described in this document allows to reduce the
   number of PEs that have to maintain all these routes by requiring
   only a subset of such PEs to maintain all these routes.

   Consider a VPN that has its sites connected to a set of PEs. In the
   context of this VPN we designate a subset of these PEs as "Virtual
   Spoke" PEs (or just Virtual Spokes), while some other (non-
   overlapping) subset of these PEs will be "Virtual Hub" PEs (or just
   Virtual Hubs), with the rest of the PEs in the set will be "vanilla"
   PEs (PEs that implement the [rfc4364] procedures, but do not
   implement procedures specified in this document).

   For the sake of brevity we will use the term "V-hub" to denote a
   Virtual Hub, and "V-spoke" to denote a Virtual Spoke.

   For a given VPN, its set of V-hubs could include not only the PEs
   that have sites of that VPN connected to them, but also PEs that have
   no sites of that VPN connected to them.

   Note that while in the context of one VPN a given PE may act as a V-
   hub, in the context of another VPN the same PE may act as a V-spoke,
   and vice versa. Thus a given PE may act as a V-hub only for some, but
   not all the VPNs present on that PE. Likewise, a given PE may act as



                                                              [Page 2]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


   a V-spoke only for some, but not all the VPNs present on that PE.

   For a given VPN each V-spoke of that VPN is "associated" with two or
   more V-hubs of that VPN (we use two V-hubs for redundancy to avoid a
   single point of failure). Note that a given V-hub may have no V-
   spokes associated with it.

   A PE that acts as a V-hub of a given VPN maintains all the routes of
   that VPN. A PE that acts as a V-spoke of a given VPN needs to
   maintain only the routes originated by the sites of that VPN
   connected to that PE, plus two or more VPN-IP default routes whose
   Next Hops are the addresses of the V-hubs associated with that V-
   spoke. This way, only a subset of PEs that have sites of a given VPN,
   namely only the PEs acting as V-hubs of that VPN,  have to maintain
   all the routes of that VPN. PEs acting as V-spokes of that VPN need
   to maintain only a (small) subset of the routes of that VPN.


2. Routing Information Exchange

   Routing information exchange among all the PEs of a given VPN is
   subject to the following rules.

   A PE that has sites of a given VPN connected to it has to retain
   routing information received from the VPN sites connected to it.
   This is irrespective of whether this PE acts as a V-hub or a V-spoke
   of that VPN.

   A PE that has sites of a given VPN connected to it has to export (as
   VPN-IP routes) all the routes received from these sites.  This is
   irrespective of whether this PE acts as a V-hub or a V-spoke of that
   VPN.

   In addition, a V-hub of a given VPN has to export a VPN-IP default
   route for that VPN. This route SHOULD be exported to only the V-
   spokes of that VPN that are associated with that V-hub.

   To enable a V-spoke of a given VPN to share its outbound traffic load
   among the V-hubs associated with that V-spoke, each V-hub of that
   VPN, when originating a VPN-IP default route MUST use a distinct RD
   (per V-hub, per VPN).

   If a V-spoke imports several VPN-IP default routes, each originated
   by its own V-hub, and these routes have the same preference, then
   traffic from the V-spoke to other sites of that VPN would be load
   shared among the V-hubs.

   A V-hub of a given VPN has to import all the non-default VPN-IP



                                                              [Page 3]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


   routes originated by all other PEs that have sites of that VPN
   connected to them (irrespective of whether these other PEs act as V-
   hubs or V-spokes for that VPN).

   A V-hub of a given VPN MUST NOT import a VPN-IP default route unless
   this is the Internet VPN-IP default route (for the definition of
   "Internet VPN-IP default route" see section "Internet Connectivity").

   A V-spoke of a given VPN has to import a VPN-IP default route for
   that VPN that has been originated by the V-hubs of that VPN that are
   associated with that V-spoke.

   In addition, a V-spoke of a given VPN MAY import VPN-IP routes for
   that VPN that have been originated by some other V-spokes of that
   VPN.

   The above rules that govern the routing information exchange among
   all the PEs of a given VPN are realized using Route Target (RT)
   extended communities [rfc4360] and VRF export/import policies based
   on these RTs. One possible scheme that enables to realize such rules
   is described in Section "Deployment Considerations". This document
   does not preclude use of other schemes, as long as such schemes would
   support the above rules.


3. Forwarding Considerations

   This section describes changes/modifications to the forwarding
   procedures specified in [rfc4364].

   The MPLS label that the V-hub advertises with a VPN-IP default route
   MUST be the label that points to the VRF of the V-hub.  This way,
   whenever the V-hub receives packets carrying this label, the V-hub
   forwards the packets by performing IP lookup in the VRF identified by
   the label.

   When a V-hub of a given VPN originates a VPN-IP default route for
   that VPN, the V-hub MUST NOT install in its VRF of that VPN a default
   route, unless this route has been originated either (a) as a result
   of the V-hub receiving an IP default route from one of the CEs of
   that VPN connected to it, or (b) as a result of the V-hub receiving
   (and importing) a VPN-IP default route from some other PE, or (c) the
   VRF being provisioned with a default route pointing to the routing
   table on the same PE that maintains the Internet routes.

   In the absence of V-hubs and V-spokes, support for IBGP/EBGP load
   balancing in the presence of any-to-any connectivity uses the
   following rule. When a PE receives a packet from another PE, the PE



                                                              [Page 4]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


   that receives the  packet determines (using the MPLS label carried in
   the packet) the VRF that should be used for further forwarding the
   packet. If (using the information present in the VRF) the PE
   determines that the packet could be forwarded over one of the
   attachment circuits of that VRF (for the definition of "VRF
   attachment circuits" see [rfc4364]), then the PE MUST forward the
   packet over one of these circuits, and MUST NOT forward the packet to
   another PE.

   In order to support IBGP/EBGP load balancing on a V-hub the above
   rule is modified as follows. When a V-hub receives a packet from
   another PE scenario, the V-hub identifies (using the MPLS label
   carried in the packet) the VRF that should be used for further
   forwarding the packet. If the MPLS label carried in the packet is the
   label that the V-hub advertised in the VPN-IP default route, then the
   V-hub is not restricted to use only one of the VRF attachment
   circuits for forwarding the packet - the V-hub may forward the packet
   to another PE.

   To illustrate this consider the following example:

                 <- RD:0/0           RD:0/0->

                                  <- RD:10.2.1        <-10.2.1/24
   CE1----PE-S-------------PE-H----------------PE-S1-------------CE2
                          /
                          |    |
                          |    |  10.2.1/24
                          |    |
                         CE4   CE3

   A multi-homed site (not shown in the above figure) is connect via CE2
   and CE3. Thus both CE2 and and CE3 advertise a route to 10.2.1/24.
   CE2 advertises this route to PE-S1, which in turn originates a VPN-IP
   route RD:10.2.1.  CE3 advertises this route to PE-H.

   PE-H is a V-hub, while PE-S and PE-S1 are V-spokes associated with
   that V-hub. Thus PE-H originates a VPN-IP default route (RD:0/0), and
   both PE-S and PE-S1 import that route.

   PE-H receives from PE-S1 a VPN-IP route to RD:10.2.1 and from CE3 a
   plain IP route to 10.2.1. Thus the VRF entry on PE-H has two possible
   next hops for 10.2.1: CE3 and PE-S1 (the latter is an indirect next
   hop).

   Now consider what happens when CE1 originates a packet destined to
   10.2.1.1. When PE-S receives this packet, PE-S (following the VPN-IP
   default route) forwards the packet to PE-H. In the absence of the



                                                              [Page 5]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


   modification specified above, when PE-H receives this packet, PE-H
   will be forced to forward the packet to CE3 (as PE-H can reach CE3
   via a VRF attachment circuit).  However, that would prevent IBGP/EBGP
   load balancing, as it would preclude the possibility of forwarding
   this packet via PE-S1 and CE2.

   With the modified rule specified above, PE-H determines that the MPLS
   label in the packet is the MPLS label that PE-H advertised to PE-S in
   the VPN-IP default route. Thus PE-H may forward the packet either via
   CE3 or via PE-S1 (with PE-S1 subsequently forwarding the packet to
   CE2), resulting in preserving IBGP/EBGP load balancing.

   Likewise, if CE4 originates a packet destined to 10.2.1.1, PE-H may
   forward the packet either via CE3 or via PE-S1 (with PE-S1
   subsequently forwarding the traffic to CE2), resulting in preserving
   IBGP/EBGP load balancing.

   Note however, that if there is some other CE, CE5, connected to PE-
   S1, and that CE5 sends a packet to 10.2.1.1, then (due to the IP
   longest match rule) PE-S1 will always forward this packet to CE2.
   Thus IBGP/EBGP load balancing will not be available for the
   destinations that have been learned locally by a V-spoke.

   Moreover, if CE3 advertises 10.2.1.0/25 and 10.2.1/24, while CE2
   advertises 10.2.1.128/25 and 10.2.1/24 (which is yet another form of
   load balancing for a multi-homed site), then when CE5 sends a packet
   to 10.2.1.1, then (due to the IP longest match rule) PE-S1 will
   always forward this packet to CE2, even though the VPN customer would
   expect this traffic to flow via CE3.

   This document proposes two options to address this, as well as to
   make the IBGP/EBGP load balancing available for the destinations that
   have been learned locally by a V-spoke. One option is to provision
   PEs that have multi-homed sites connected to them as V-hubs. Another
   option is for the V-spoke, when it receives an IP route from a CE,
   not to install this route in its forwarding table, but just re-
   advertise this route as a VPN-IP route, together with an MPLS label.
   The packet's next hop of the Next Hop Label Forwarding Entry (NHLFE)
   [rfc3031] associated with that label MUST point to the CE that
   advertised the IP route. As a result, when the PE receives data that
   carries that label, the PE performs no IP lookup on the data, and
   just forwards the data to the CE. Note that doing this would result
   in forcing the traffic between a pair of sites connected to the same
   V-spoke to go through the V-hub of that V-spoke.







                                                              [Page 6]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


4. Internet Connectivity

   In this document we consider two possible scenarios of providing
   Internet connectivity for a given VPN.

   The first scenario is when one or more sites of a given VPN are
   connected to a PE, and that PE also maintains Internet routes. In
   this case the Internet connectivity for that VPN is provided by
   provisioning in the VPN's VRF on that PE a default route pointing to
   the routing table on that PE that maintains the Internet routes.
   This PE MUST act as a V-hub for that VPN, and therefore MUST
   originate a VPN-IP default route.

   The second scenario is when a site of a given VPN has connection to
   the Internet, and a CE of that site advertises an IP default route to
   the PE connected to that CE. This scenario has two sub-cases: (a) PE
   is provisioned as a V-hub, and (b) PE is provisioned as a V-spoke.

   If a PE is provisioned as a V-hub, then the PE re-advertises this IP
   default route as a VPN-IP default route, and install in its VRF an IP
   default route with the next hop pointing to the CE(s) that advertise
   the IP default route to the PE.

   If a PE is provisioned as a V-spoke, then the PE MUST NOT install in
   its VRF an IP default route. The PE MUST originate a VPN-IP default
   route with a (non-null) MPLS label. The packet's next hop of the Next
   Hop Label Forwarding Entry (NHLFE) [rfc3031] associated with that
   label MUST point to the CE that advertises the IP default route. As a
   result, when the PE receives data that carries that label, the PE
   performs no IP lookup on the data, and just forwards the data to the
   CE. Note that in this case the VRF on the PE is going to have an IP
   default route, but this route would be created as a result of
   receiving a VPN-IP default route from one of the V-hubs of that PE
   (and not as a result of receiving the IP default route from the CE).
   Note also that if this PE has other sites of that VPN connected to
   it, then traffic from these sites to the Internet would go to that
   PE, then to the V-hub selected by the PE, then from that V-hub back
   to the PE, and then to the CE that advertises IP default route to the
   PE.

   If a PE is provisioned as a V-spoke of a given VPN, and if a CE of
   that VPN advertises an IP default route to the PE (as the CE belongs
   to the site that provides the Internet connectivity for the VPN),
   then the PE can not advertise an IP default route back to that CE.
   Yet, the CE has to point to the PE as the next hop for all the
   traffic to other sites of that VPN. This document proposes the
   following options to accomplish this. The first option is to require
   the V-spoke to implement procedures of section "Further refinements".



                                                              [Page 7]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


   The second option is for the V-spoke to advertise to the CE 10/8,
   172.16/12, and 192.168/16. Note that this option is applicable only
   if the service provider knows that the VPN customer uses only the
   [rfc1918] addresses. The third option is to re-provision this PE as a
   V-hub.

   In all of the above we refer to the originated VPN-IP default route
   as the "Internet VPN-IP default route".

   Note that if a V-hub originates the Internet VPN-IP default route for
   a given VPN, the V-hub does not originate any other VPN-IP default
   routes for that VPN.

   If a V-hub originates the Internet VPN-IP default route, then all the
   V-spokes associated with that V-hub MUST import that route. In
   addition (and in contrast with a non-Internet VPN-IP default route),
   other V-hubs MAY import that route.

   A V-hub MAY also import the Internet VPN-IP default routes originated
   by V-spoke(s).

   A V-spoke MUST NOT import the Internet VPN-IP default route
   originated by any other V-spoke. Such a route MAY be imported only by
   V-hubs.

   The above rules that govern exchange of the Internet VPN-IP default
   route among all the PEs of a given VPN are realized using Route
   Target (RT) extended communities [rfc4360] and VRF export/import
   policies based on these RTs. One possible scheme that enables to
   realize such rules is described in Section "Deployment
   Considerations".  This document does not preclude use of other
   schemes, as long as such schemes would support the above rules.

   If the Internet VPN-IP default route originated by a V-hub has the
   same preference as the (non Internet) VPN-IP default route originated
   by some other V-hub, then a V-spoke that imports VPN-IP default
   routes originated by both of these V-hubs would load share the
   outgoing Internet traffic between these two V-hubs (and thus some of
   the outgoing Internet traffic from that V-spoke will first be routed
   to the V-hub that does not originate the Internet VPN-IP default
   route, and then from that V-hub to the V-hub that does originate the
   Internet VPN-IP default route).

   If taking an extra-hub hop for the Internet traffic is viewed as
   undesirable, then it is RECOMMENDED that the Internet VPN-IP default
   route be of higher preference than a (non-Internet) VPN-IP default
   route originated by some other V-hub.  However, in this case the
   traffic from the V-spokes to other sites of that VPN will not be load



                                                              [Page 8]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


   shared between these two V-hubs.


5. Deployment Considerations

   For a given VPN a V-hub and a set of V-spokes associated with that V-
   hub should be chosen in a way that minimizes the additional network
   distance/latency penalty, given that VPN geographic footprint.

   For a given VPN some/all of its V-spokes could be grouped into
   geographically-based clusters with any-any connectivity within each
   cluster. Note that the V-spokes within a given cluster need not be
   associated with the same V-hub(s). Likewise, not all V-spokes
   associated with a given V-hub need to be in the same cluster.  A use
   case for this would be VPN for large retail chain in which data
   traffic is hub/spoke between each store and centralized data-centers
   but there is need for direct VoIP traffic between stores within same
   geographical area.

   The following describes one possible scheme that enables to realize
   the rules for exchanging routing information among PEs, as specified
   in Section "Routing Information Exchange".

   Consider a "vanilla" any-to-any VPN. All the PEs of such VPN are
   provisioned with the same export and import RT - we will refer to
   this RT as RT-VPN.

   To evolve this VPN into V-hubs and V-spokes we keep the same export
   RT-VPN on all the PEs of that VPN (both V-hubs and V-spokes). This
   RT-VPN is attached to all the VPN-IP routes that PEs originate as a
   result of receiving corresponding IP routes from their CEs.  We also
   keep the same import RT-VPN on all the V-hubs.

   In addition, each V-hub is provisioned with its own import RT, called
   RT-VH.  This RT-VH MUST be different from the export RT provisioned
   on that V-hub. For a given PE this RT-VH has to be distinct for every
   V-hub present on that PE. To avoid the situation where within a given
   VPN all the V-spokes would be associated with every V-hub (in other
   words, to partition V-spokes among V-hubs), different V-hubs within
   that VPN MAY use different RT-VHs. At one extreme every V-hub may use
   a distinct RT-VH. However, it is also possible for several V-hubs to
   use the same RT-VH, in which case all these V-hubs are used by the
   same set of V-spokes. Use of IP-address specific RTs may be an
   attractive option for RT-VHs.

   When a V-hub originates a (non-Internet) VPN-IP default route, then
   the V-hub attaches RT-VH to that route. Thus this route is imported
   by all the V-spokes associated with the V-hub.



                                                              [Page 9]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


   If a V-hub originates the Internet VPN-IP default route, then the V-
   hub attaches both RT-VH and RT-VPN to that route. Thus this route is
   imported by all the V-spokes associated with the V-hub, as well as by
   other V-hubs.

   If a V-spoke originates the Internet VPN-IP default route, then the
   V-spoke attaches only RT-VPN to that route. Thus this route is not
   imported by any of the V-spokes, but is imported by V-hubs.

   A given V-spoke is provisioned to import only VPN-IP routes that
   carry RT-VHs of the V-hubs associated with that V-spoke (thus
   associating a new V-spoke with a given V-hub requires provisioning
   only on that V-spoke - no provisioning changes are required on the V-
   hub).

   A V-spoke may be provisioned to export VPN-IP routes not just to the
   V-hubs, but also to the V-spokes that import the same VPN-IP default
   route(s) as the V-spoke itself. The V-spoke accomplishes this by
   adding its import RT-VH(s) to the VPN-IP routes exported by the V-
   spoke.

   Use of RT Constrains [rfc4684] may further facilitate/optimize
   routing exchange in support of V-hubs and V-spokes.


6. Multicast Considerations

   This section describes procedures for supporting MVPN in the presence
   of Virtual Hub-and-Spoke. The procedures rely on MVPN specifications
   as defined in [MVPN], [BGP-MVPN].

   The procedures assume that for the purpose of ensuring non-
   duplication both V-hubs and V-spokes can discard packets from a
   "wrong" PE, as specified in [MVPN]. This applies for all types of P-
   tunnels, including Ingress Replication.


6.1. Terminology

   When Ingress Replication is used to realize P-tunnels, a P-tunnel
   being "bound" to a particular I-PMSI/S-PMSI A-D route means the set
   of (unicast) tunnels used to carry traffic from the PE originating
   the route to the PEs that import the route.

   When Ingress Replication is used to realize P-tunnels, traffic
   received by a PE on a P-tunnel "bound" to particular I-PMSI/S-PMSI A-
   D received by the PE means the traffic received on the (unicast)
   tunnel that the PE originating the route uses to send traffic to the



                                                             [Page 10]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


   PE that receives the route.


6.2. Eligible Upstream Multicast Hop (UMH) routes

   On a V-spoke the set of Eligible UMH routes consists of all the
   unicast VPN-IP routes received by the V-spoke, including the default
   VPN-IP routes received from its V-hub(s). Note that such routes may
   include VPN-IP routes received from other V-spokes.


6.3. Originating VPN-IP default route by a V-hub

   A V-hub, when originating a VPN-IP default route, follows the
   procedures of [BGP-MVPN]. Specifically the V-hub must add the VRF
   Route Import extended community that embeds the V-hub's IP address.
   The route also must include the Source AS extended community.


6.4. Handling C-multicast routes

   Origination of C-multicast routes follow the procedures of [BGP-MVPN]
   (this is irrespective of whether these routes are originated by a V-
   hub or by a V-spoke). Note that for a given V-spoke its set of
   eligible UMH routes may contains not only the VPN-IP default routes
   advertised by its V-hubs, but also VPN-IP routes advertised by some
   other V-spokes.

   When a V-hub receives a C-multicast route, the V-hub determines
   whether the C-RP/C-S of the route is reachable via one of its VRF
   interfaces, and if yes, then the V-hub follows the [BGP-MVPN]
   procedures. Otherwise, the C-RP/C-S of the route is reachable via
   some other PE, in which case the V-hub uses the type (Source Tree
   Join vs Shared Tree Join), the Multicast Source, and Multicast Group
   from the received route to construct a new route. The hub constructs
   the rest of the new route following procedures specified in "11.1.3.
   Constructing the rest of the C-multicast route" of [BGP-MVPN]. The
   hub also creates the appropriate (C-*, C-G)/(C-S, C-G) state in its
   MVPN-TIB.


6.5. Originating I-PMSI/S-PMSI/SA A-D routes by V-spoke

   When a V-spoke originates an I-PMSI/S-PMSI/SA A-D route, the V-spoke
   follows the procedures of [BGP-MVPN], including the procedures for
   constructing RT(s) carried by the route. Note that as a result, such
   a route will be imported by the V-hubs. The P-tunnel bound to this
   route is used to carry to these V-hubs traffic originated by the



                                                             [Page 11]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


   sites connected to the V-spoke.

   This route may also be imported by some other V-spokes - the V-spokes
   that import the (unicast) VPN-IP routes originated by the V-spoke
   that originates the I-PMSI/S-PMSI/SA A-D route. In this case the
   route would carry not one, but two RTs: the first one results in
   importing the route by V-hubs, the second results in importing the
   route by the V-spokes (the second RT is the RT that V-spoke use for
   importing the VPN-IP default route). In this case the P-tunnel bound
   to this route is also used to carry traffic originated by the sites
   connected to the V-spoke that originates the route to these other V-
   spokes.


6.6. Originating I-PMSI/S-PMSI/SA A-D routes by V-hub

   When a V-hub originates an I-PMSI/S-PMSI/SA A-D route, the V-hub
   follows the procedures of [BGP-MVPN], except that in addition to the
   RT(s) constructed following these procedures, the route also carries
   the RT of the VPN-IP default route advertised by the V-hub.  Note
   that as a result, such a route will be imported by other V-hubs, and
   also by the V-spokes, but only by the V-spokes that import the VPN-IP
   default route originated by the V-hub.  The P-tunnel bound to this
   route is used to carry to these other V-hubs and V-spokes the traffic
   originated by the sites connected to the V-hub that originates the
   route.

   In addition, a V-hub originates another I-PMSI A-D route - we'll
   refer to this route as a "Default I-PMSI A-D route". The RT carried
   by this route is the RT that is carried in the VPN-IP default route
   advertised by the V-hub (RT-VH). Therefore, this route will be
   imported only by the V-spokes that import the VPN-IP default route
   advertised by this V-hub. The P-tunnel bound to this route is used to
   carry to these V-spokes traffic originated by either (a) other V-
   hubs, or (b) V-spokes, including the V-spokes that import the VPN-IP
   default route from the V-hub. More details on the use of this P-
   tunnel is described later.

   As a result, a V-hub originates not one, but two I-PMSI A-D routes.
   Each of these routes MUST have a distinct RD.

   When a V-hub receives traffic from one of the sites connected to the
   V-hub, and the V-hub determines (using some local policies) that this
   traffic should be transmitted using an I-PMSI, the V-hub forwards
   this traffic on the P-tunnel bound to the first (non Default) I-PMSI
   A-D route, but not on the P-tunnel bound to the second, the Default
   I-PMSI A-D route.




                                                             [Page 12]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


6.7. Receiving I-PMSI/S-PMSI/SA A-D routes by V-spoke

   When a V-spoke receives an I-PMSI/S-PMSI/SA A-D route, the V-spoke
   follows the procedures of [BGP-MVPN]. As a result, a V-spoke that
   imports the VPN-IP default route originated by a given V-hub will
   also import I-PMSI/S-PMSI/SA A-D routes originated by that V-hub.
   Specifically, the V-spoke will import both the non Default I-PMSI A-D
   route and the Default I-PMSI A-D route originated by the V-hub.

   In addition, if a V-spoke imports the (unicast) VPN-IP routes
   originated by some other V-spokes, then the V-spoke will also import
   I-PMSI/S-PMSI/SA A-D routes originated by these other V-spokes.


6.8. Receiving I-PMSI/S-PMSI/SA A-D routes by V-hub

   The following describe procedures that a V-hub must follow when it
   receives an I-PMSI/S-PMSI/SA A-D route.


6.8.1. Case 1

   This is the case where the received I-PMSI/S-PMSI/SA A-D route was
   originated by a V-spoke that imports the VPN-IP default route
   originated by the V-hub, and the received route is imported not just
   by the V-hub (as well as by all other V-hubs), but also by other V-
   spokes, but only by the V-spokes that import the VPN-IP default route
   originated by the V-hub. This is also the case where the received
   route was originated by a V-hub that uses the same RT as the received
   V-hub for advertising the VPN-IP default route.

   In this case the received I-PMSI/S-PMSI/SA A-D route carries more
   than one RT. One of these RTs results in importing this route by the
   V-hubs. Another of these RTs is the RT that the V-hub uses when
   advertising its VPN-IP default route (RT-VH). This RT results in
   importing the received I-PMSI/S-PMSI/SA A-D route o all the V-spokes
   that import the VPN-IP default route originated by the V-hub.

   In handling such I-PMSI/S-PMSI/SA A-D route the V-hub follows the
   [BGP-MVPN] procedures. Specifically, in contrast to Case 2 below, the
   V-hub MUST NOT re-advertise this route.

   The V-hub may forward traffic received on a P-tunnel bound to this I-
   PMSI/S-PMSI A-D route to only the sites connected to that V-hub.  The
   V-hub MUST NOT forward the traffic received on this P-tunnel to any
   other V-hubs or V-spokes, including the V-spokes that import the VPN-
   IP default route originated by the V-hub. Specifically, the V-hub
   MUST NOT forward the traffic received on the P-tunnel advertised in



                                                             [Page 13]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


   the received I-PMSI A-D route over the P-tunnel that the V-hub binds
   to its Default I-PMSI A-D route.


6.8.2. Case 2

   This is the case where the received I-PMSI/S-PMSI/SA A-D route was
   originated by either some other V-hub, or by a V-spoke. The I-PMSI/S-
   PMSI/SA A-D route is imported by the V-hub (as well as by other V-
   hubs), but not by any of the V-spokes that import the VPN-IP default
   route originated by the V-hub.

   In this case the received I-PMSI/S-PMSI/SA A-D route does not carry
   the RT that the receiving V-hub uses when advertising its VPN-IP
   default route (RT-VH), although the route may carry more than one RT.

   In this case the V-hubs follows the procedures of [BGP-MVPN] with the
   following additions.

   Once a V-hub accepts an I-PMSI A-D route, when the V-hub receives
   data on the P-tunnel bound to that I-PMSI A-D route, the V-hub
   follows procedures in [MVPN], [MVPN-BGP] to determine whether to
   accept the data. If the data is accepted, then the V-hub further
   forwards the data over the P-tunnel bound to the Default I-PMSI A-D
   route originated by the V-hub. Note that in deciding whether to
   forward the data over the P-tunnel bound to the Default I-PMSI A-D
   route originated by the V-hub, the V-hub SHOULD take into account the
   (multicast) state present in its MVPN-TIB that has been created as a
   result receiving C-multicast routes from the V-spokes associated with
   the V-hub.

   Whenever a V-hub accepts an S-PMSI/SA A-D route, the V-hub, in
   contrast to Case 1 above, MUST re-advertise this route to its V-
   spokes. To accomplish this, the V-hub changes the RT carried by the
   route to the RT that the V-hub uses when originating its VPN-IP
   default route (RT-VH), and sets Next Hop to self. In addition, for S-
   PMSI/SA A-D routes the V-hub changes the RD of the route to the RD
   that the hub uses when originating its VPN-IP default route, and for
   S-PMSI A-D routes the V-hub changes the Originating Router's IP
   address in the MCAST-VPN NLRI of the route to its own IP address.

   Moreover, before re-advertising S-PMSI A-D routes to V-spokes, the V-
   hub modifies the PMSI Tunnel attribute of these routes as appropriate
   (e.g., by replacing the P-tunnel rooted at the originator of these
   routes with a P-tunnel rooted at the V-hub).

   Whenever the V-hub receives data on the P-tunnel bound to the
   received S-PMSI A-D route, the V-hub follows procedures of [MVPN],



                                                             [Page 14]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


   [MVPN-BGP] to determine whether to accept the data. If the data is
   accepted, then the V-hub further forwards it over the P-tunnel bound
   to the S-PMSI A-D route that has been re-advertised by the V-hub.

   If multiple S-PMSIs received by the V-hub have been aggregated into
   the same P-tunnel, then the V-hub, prior to forwarding to the V-
   spokes the data received on this P-tunnel may de-aggregate and then
   re-aggregate (in a different way) this data using the state present
   in its MVPN-TIB that has been created as a result of receiving C-
   multicast routes from the V-spokes. Even if S-PMSIs received by the
   V-hub each has its own P-tunnel, the V-hub, prior to forwarding to
   the V-spokes the data received on these P-tunnels may aggregate these
   S-PMSIs using the state present in its MVPN-TIB that has been created
   as a result of receiving C-multicast routes from the V-spokes.



6.9. Use of Ingress Replication with I-PMSI A-D routes

   The existing procedures for S-PMSI A-D routes are sufficient to
   discard packets coming from a "wrong" PE for any type of P-tunnels,
   including Ingress Replication.

   The following modifications to originating/receiving I-PMSI A-D
   routes enable to discard packets coming from a "wrong" PE when
   Ingress Replication is used for I-PMSI P-tunnels (for other types of
   P-tunnels the procedures specified in [MVPN], [BGP-MVPN] are
   sufficient).

   If Ingress Replication is used with I-PMSI A-D routes, then when a PE
   advertises such routes, the Tunnel Type in the PMSI Tunnel attribute
   is set to Ingress Replication; the Leaf Information Required flag is
   set to 1; the the attribute carries no MPLS labels.

   A PE that receives such I-PMSI A-D route MUST respond with a Leaf A-D
   route.  The PMSI Tunnel attribute of that Leaf A-D route is
   constructed as follows. The Tunnel Type is set to Ingress
   Replication.  The Tunnel Identifier MUST carry a routable address of
   the PE that originates the Leaf A-D route. The PMSI Tunnel attribute
   MUST carry a downstream assigned MPLS label that is used to
   demultiplex the traffic received over a unicast tunnel by the PE.










                                                             [Page 15]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


7. An example of RT provisioning

   Consider a VPN A that consists of 9 sites - site-1 through site-9.
   Each site is connected to its own PE - PE-1 through PE-9.

   We designate PE-3, PE-6, and PE-9 as V-hubs.

   To simplify the presentation the following example assumes that each
   V-spoke is associated with just one V-hub. However, as mentioned
   earlier, in practice each V-spoke should be associated with two or
   more V-hubs.

   PE-1 and PE-2 are V-spokes associated with PE-3. PE-4 and PE-5 are V-
   spokes associated with PE-6. PE-7 and PE-8 are V-spokes associated
   with PE-9.


7.1. Unicast routing

   All the PEs (both V-hubs and V-spokes) are provisioned to export
   routes using RT-A (just as it would be with "vanilla" any-to-any
   VPN).

   All the V-hubs (PE-3, PE-6, and PE-9) are provisioned to import
   routes with RT-A (just as it would be with "vanilla" any-to-any VPN).

   In addition, PE-3 is provisioned to originate a VPN-IP default route
   with RT-A-VH-1 (but not with RT-A), while PE-1 and PE-2 are
   provisioned to import routes with RT-A-VH-1.

   Likewise, PE-6 is provisioned to originate a VPN-IP default route
   with RT-A-VH-2 (but not with RT-A), while PE-4 and PE-5 are
   provisioned to import routes with RT-A-VH-2.

   Finally, PE-9 is provisioned to originate a VPN-IP default route with
   RT-A-VH-3 (but not with RT-A), while PE-7 and PE-8 are provisioned to
   import routes with RT-A-VH-3.

   Now let's modify the above a bit by assuming that site-3 has Internet
   connectivity. Thus site-3 advertises an IP default route to PE-3.
   PE-3, in turn originates a VPN-IP default route. In this case, the
   VPN-IP default route carries RT-A and RT-A-VH-1 (rather than just RT-
   A-VH-1, as before), which results in importing this route to PE-6 and
   PE-9, as well as to PE-1 and PE-2.

   If PE-7 and PE-8, in addition to importing VPN-IP default route from
   PE-9, also want to import each other VPN-IP routes, then PE-7 and
   PE-8 export their VPN-IP routes with two RTs: RT-A and RT-A-VH-3.



                                                             [Page 16]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


7.2. Multicast routing

   All the PEs designated as V-spokes (PE-1, PE-2, PE-4, PE-5, PE-7, and
   PE-8) are provisioned to export their I-PMSI/S-PMSI/SA A-D routes
   using RT-A (just as it would be with "vanilla" any-to-any MVPN). Thus
   these routes could be imported by all the V-hubs (PE-3, PE-6, and
   PE-9).

   The V-hub on PE-3 is provisioned to export its I-PMSI/S-PMSI/SA A-D
   routes with two RTs: RT-A and RT-A-VH-1. Thus these routes could be
   imported by all the other V-hubs (PE-6 and PE-9), and also by the V-
   spokes, but only by the V-spokes associated with the V-hub on PE-3
   (PE-1 and PE-2). In addition, the V-hub on PE-3 originates the
   Default I-PMSI A-D route with RT-A-VH-1. This route could be imported
   only by the V-spokes associated with the V-hub on PE-3 (PE-1 and
   PE-2).

   The V-hub on PE-6 is provisioned to export its I-PMSI/S-PMSI/SA A-D
   routes with two RTs: RT-A and RT-A-VH-2. Thus these routes could be
   imported by all the other V-hubs (PE-3 and PE-9), and also by the V-
   spokes, but only by the V-spokes associated with the V-hub on PE-6
   (PE-4 and PE-5). In addition, the V-hub on PE-6 originates the
   Default I-PMSI A-D route with RT-A-VH-2. This route could be imported
   only by the V-spokes associated with the V-hub on PE-6 (PE-4 and
   PE-5).

   The V-hub on PE-9 is provisioned to export its I-PMSI/S-PMSI/SA A-D
   routes with two RTs: RT-A and RT-A-VH-3. Thus these routes could be
   imported by all the other V-hubs (PE-3 and PE-6), and also by the V-
   spokes, but only by the V-spokes associated with the V-hub on PE-9
   (PE-7 and PE-8). In addition, the V-hub on PE-9 originates the
   Default I-PMSI A-D route with RT-A-VH-3. This route could be imported
   only by the V-spokes associated with the V-hub on PE-9 (PE-7 and
   PE-8).

   If PE-7 and PE-8, in addition to importing VPN-IP default route from
   PE-9, also want to import each other VPN-IP routes, then PE-7 and
   PE-8 export their I-PMSI/S-PMSI/SA A-D routes with two RTs: RT-A and
   RT-A-VH-3.

   If the V-hub on PE-9 receives an S-PMSI/SA A-D route originated by
   either some other V-hub (PE-3 or PE-6), or by a V-spoke that is not
   associated with this V-hub (PE-1, or PE-2, or PE-4, or PE-5), the V-
   hub, before re-advertising this route replaces the RT(s) carried in
   this route with just one RT - RT-A-VH-3. Thus, the re-advertised
   route could be imported only by the V-spokes associated with the V-
   hub on PE-9 (PE-7 and PE-8).




                                                             [Page 17]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


8. Further refinements

   In some cases a VPN customer may not want to rely solely on an (IP)
   default route being advertised from a V-spoke to a CE, but may want
   CEs to receive all the VPN routes (e.g., for the purpose of faster
   detection of VPN connectivity failures, and activating some backup
   connectivity).

   In this case one possible approach would be to install in the V-
   spoke's data plane only the default route (following the Virtual Hub
   and Spoke model, as described above), but keep all the VPN-IP routes
   in the V-spoke's control plane (and thus being able to advertise
   these routes from the V-spoke to the CEs).  Granted, this would not
   change control plane resource consumption, but would (significantly)
   reduce resource consumption on the data plane.


9. IANA Considerations

   This document introduces no new IANA Considerations.


10. Security Considerations

   This document introduces no new Security Considerations, above and
   beyond what is already specified in [rfc4364].


11. Acknowledgements

   We would like to acknowledge Han Nguyen (AT&T) for his contributions
   to this document. We would also like to thank Jeffrey (Zhaohui) Zhang
   (Juniper) for his review and comments.


12. Normative References

   [rfc1918] Rekhter, Y., et.al., "Address Allocation for Private
   Internets", RFC 1918, February 1996.

   [rfc3031] Rosen, E., et. al., "MPLS Architecture", RFC3031, January
   2001.

   [rfc4360] Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended
   Communities Attribute", RFC 4360, February 2006.

   [rfc4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
   Networks (VPNs)", RFC 4364, February 2006.



                                                             [Page 18]


Internet Draft   draft-rekhter-l3vpn-virtual-hub-02.txtSeptember 6, 2011


   [rfc4684] Pedro Marques, et al., "Constrained Route Distribution for
   Border Gateway Protocol/MultiProtocol Label Switching (BGP/MPLS)
   Internet Protocol (IP) Virtual Private Networks (VPNs)", RFC4684,
   November 2006

   [MVPN] Eric C. Rosen, Rahul Aggarwal, et. al., "Multicast in MPLS/BGP
   IP VPNs", draft-ietf-l3vpn-2547bis-mcast-10, Work In Progress.

   [BGP-MVPN] R. Aggarwal, E. Rosen, T. Morin, Y. Rekhter, "BGP
   Encodings and Procedures for Multicast in MPLS/BGP IP VPNs", draft-
   ietf-l3vpn-2547bis-mcast-bgp-08.txt, Work In Progress.


13. Non-normative References



14. Authors' Addresses


   Huajin Jeng
   AT&T
   e-mail: hj2387@att.com

   James Uttaro
   AT&T
   e-mail: ju1738@att.com

   Luay Jalil
   Verizon
   e-mail: luay.jalil@verizon.com

   Bruno Decraene
   France Telecom
   e-mail: bruno.decraene@orange-ftgroup.com

   Rahul Aggarwal
   e-mail: raggarwa_1@yahoo.com

   Yakov Rekhter
   Juniper Networks, Inc.
   e-mail: yakov@juniper.net









                                                             [Page 19]