Skip to main content

EVPN Interworking with IPVPN
draft-ietf-bess-evpn-ipvpn-interworking-07

Document Type Active Internet-Draft (bess WG)
Authors Jorge Rabadan , Ali Sajassi , Eric C. Rosen , John Drake , Wen Lin , Jim Uttaro , Adam Simpson
Last updated 2022-07-21 (Latest revision 2022-07-06)
Replaces draft-rabadan-sajassi-bess-evpn-ipvpn-interworking
Stream Internet Engineering Task Force (IETF)
Intended RFC status Proposed Standard
Formats
Stream WG state In WG Last Call
Revised I-D Needed - Issue raised by WG
Document shepherd Stephane Litkowski
IESG IESG state I-D Exists
Consensus boilerplate Yes
Telechat date (None)
Responsible AD (None)
Send notices to slitkows.ietf@gmail.com
draft-ietf-bess-evpn-ipvpn-interworking-07
BESS Workgroup                                           J. Rabadan, Ed.
Internet-Draft                                                     Nokia
Intended status: Standards Track                         A. Sajassi, Ed.
Expires: January 7, 2023                                           Cisco
                                                                E. Rosen
                                                              Individual
                                                                J. Drake
                                                                  W. Lin
                                                                 Juniper
                                                               J. Uttaro
                                                                    AT&T
                                                              A. Simpson
                                                                   Nokia
                                                            July 6, 2022

                      EVPN Interworking with IPVPN
               draft-ietf-bess-evpn-ipvpn-interworking-07

Abstract

   EVPN is used as a unified control plane for tenant network intra and
   inter-subnet forwarding.  When a tenant network spans not only EVPN
   domains but also domains where BGP VPN-IP or IP families provide
   inter-subnet forwarding, there is a need to specify the interworking
   aspects between BGP domains of type EVPN, VPN-IP and IP, so that the
   end to end tenant connectivity can be accomplished.  This document
   specifies how EVPN interworks with VPN-IPv4/VPN-IPv6 and IPv4/IPv6
   BGP families for inter-subnet forwarding.  The document also
   addresses the interconnect of EVPN domains for Inter-Subnet
   Forwarding routes.  In addition, this specification defines a new BGP
   Path Attribute called D-PATH (Domain PATH) that protects gateways
   against control plane loops.  D-PATH modifies the BGP best path
   selection for multiprotocol BGP routes of SAFI 1, 128 and EVPN IP
   Prefix routes, and therefore this document updates [RFC4271].

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any

Rabadan, et al.          Expires January 7, 2023                [Page 1]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on January 7, 2023.

Copyright Notice

   Copyright (c) 2022 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction and Problem Statement  . . . . . . . . . . . . .   3
   2.  Conventions used in this document . . . . . . . . . . . . . .   4
   3.  Terminology and Interworking PE Components  . . . . . . . . .   4
   4.  Domain Path Attribute (D-PATH)  . . . . . . . . . . . . . . .   9
   5.  BGP Path Attribute Propagation across ISF SAFIs . . . . . . .  14
     5.1.  No-Propagation-Mode . . . . . . . . . . . . . . . . . . .  14
     5.2.  Uniform-Propagation-Mode  . . . . . . . . . . . . . . . .  15
     5.3.  Aggregation of Routes and Path Attribute Propagation  . .  16
   6.  Route Selection Process for ISF Routes  . . . . . . . . . . .  17
   7.  Composite PE Procedures . . . . . . . . . . . . . . . . . . .  18
   8.  Gateway PE Procedures . . . . . . . . . . . . . . . . . . . .  20
   9.  Interworking Use-Cases  . . . . . . . . . . . . . . . . . . .  23
   10. BGP Error Handling on Interworking PEs  . . . . . . . . . . .  24
   11. Conclusion  . . . . . . . . . . . . . . . . . . . . . . . . .  26
   12. Security Considerations . . . . . . . . . . . . . . . . . . .  26
   13. IANA Considerations . . . . . . . . . . . . . . . . . . . . .  27
   14. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .  27
   15. Contributors  . . . . . . . . . . . . . . . . . . . . . . . .  27
   16. References  . . . . . . . . . . . . . . . . . . . . . . . . .  27
     16.1.  Normative References . . . . . . . . . . . . . . . . . .  27
     16.2.  Informative References . . . . . . . . . . . . . . . . .  29
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  29

Rabadan, et al.          Expires January 7, 2023                [Page 2]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

1.  Introduction and Problem Statement

   EVPN is used as a unified control plane for tenant network intra and
   inter-subnet forwarding.  When a tenant network spans not only EVPN
   domains but also domains where BGP VPN-IP or IP families provide
   inter-subnet forwarding, there is a need to specify the interworking
   aspects between the different families, so that the end to end tenant
   connectivity can be accomplished.  This document specifies how EVPN
   should interwork with VPN-IPv4/VPN-IPv6 and IPv4/IPv6 BGP families
   for inter-subnet forwarding.  The document also addresses the
   interconnect of an EVPN domain to another EVPN domain for Inter-
   Subnet Forwarding routes.  In addition, this specification defines a
   new BGP Path Attribute called D-PATH (Domain PATH) that protects
   gateways against control plane loops.  Loops are created when two (or
   more) redundant gateway PEs interconnect two domains and exchange
   inter-subnet forwarding routes.  For instance, if PE1 and PE2 are
   redundant gateway PEs interconnecting an IPVPN and an EVPN domain,
   gateway PE1 receives a VPN-IP route to prefix P and propagates the
   route into an EVPN IP Prefix to P.  If gateway PE2 receives the EVPN
   IP Prefix route, it cannot propagate the route back to the IPVPN
   domain, or it would create a loop for prefix P.

   D-PATH modifies the BGP best path selection for multiprotocol BGP
   routes of SAFI 1, 128 and EVPN IP Prefix routes, and therefore this
   document updates [RFC4271].

   EVPN supports the advertisement of IPv4 or IPv6 prefixes in two
   different route types:

   o  Route Type 2 - MAC/IP route (only for /32 and /128 host routes),
      as described by [RFC9135].

   o  Route Type 5 - IP Prefix route, as described by [RFC9136].

   When interworking with other BGP address families (AFIs/SAFIs) for
   inter-subnet forwarding, the IP prefixes in those two EVPN route
   types must be propagated to other domains using different SAFIs.
   Some aspects of that propagation must be clarified.  Examples of
   these aspects or procedures across BGP families are: route selection,
   loop prevention or BGP Path attribute propagation.  The Interworking
   PE concepts are defined in Section 3, and the rest of the document
   describes the interaction between Interworking PEs and other PEs for
   end-to-end inter-subnet forwarding.

Rabadan, et al.          Expires January 7, 2023                [Page 3]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

2.  Conventions used in this document

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

3.  Terminology and Interworking PE Components

   This section summarizes the terminology related to the "Interworking
   PE" concept that will be used throughout the rest of the document.

      +-------------------------------------------------------------+
      |                                                             |
      |              +------------------+           Interworking PE |
      | Attachment   | +------------------+                         |
      | Circuit(AC1) | |  +----------+    |                MPLS/NVO tnl
    ----------------------*Bridge    |    |                    +------
      |              | |  |Table(BT1)|    |    +-----------+  / \     \
   MPLS/NVO tnl +-------->|          *---------*           |<--> | Eth |
     -------+   |    | |  |Eth-Tag x +    |IRB1|           |  \ /     /
    / Eth  / \<-+    | |  +----------+    |    |           |   +------
   |      |   |      | |     ...          |    |  IP-VRF1  |        |
    \      \ /<-+    | |  +----------+    |    |  RD2/RT2  |MPLS/NVO tnl
     -------+   |    | |  |Bridge    |    |    |           |   +------
      |         +-------->|Table(BT2)|    |IRB2|           |  / \     \
      |              | |  |          *---------*           |<--> | IP  |
    ----------------------*Eth-Tag y |    |    +-----*-----+  \ /     /
      |  AC2         | |  +----------+    |       AC3|         +------
      |              | |    MAC-VRF1      |          |              |
      |              +-+    RD1/RT1       |          |              |
      |                +------------------+          |  SAFIs       |
      |                                              |  1     +---+ |
    -------------------------------------------------+  128   |BGP| |
      |                                                 EVPN  +---+ |
      |                                                             |
      +-------------------------------------------------------------+

                   Figure 1: EVPN-IPVPN Interworking PE

   o  ISF SAFI: Inter-Subnet Forwarding (ISF) SAFI is a MP-BGP Sub-
      Address Family that advertises reachability for IP prefixes and
      can be used for inter-subnet forwarding within a given tenant
      network.  The ISF SAFIs are 1 (including IPv4 and IPv6 AFIs), 128
      (including IPv4 and IPv6 AFIs) and 70 (EVPN, including only AFI
      25).  This document uses the following terms interchangeably: ISF
      SAFI 1 or BGP IP, ISF SAFI 128 or IPVPN, ISF SAFI 70 or EVPN.

Rabadan, et al.          Expires January 7, 2023                [Page 4]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   o  ISF route: a route for a given prefix, whose ISF SAFI may change
      as it transits different domains.  BGP IP routes as in [RFC4760]
      [RFC8950], IPVPN routes as in [RFC4364], [RFC4659], or EVPN IP
      Prefix routes as in [RFC9136], are considered ISF routes in this
      document.

   o  IP-VRF: an IP Virtual Routing and Forwarding table, as defined in
      [RFC4364].  Route Distinguisher and Route Target(s) are required
      properties of an IP-VRF.  An IP-VRF is programmed with ISF routes.

   o  MAC-VRF: a MAC Virtual Routing and Forwarding table, as defined in
      [RFC7432].  It is also the instantiation of an EVI (EVPN Instance)
      in a PE.  Route Distinguisher and Route Target(s) are required
      properties and they are normally different than the ones defined
      in the associated IP-VRF (if there is an associated IP-VRF linked
      to a BT of the MAC-VRF, via IRB interface).

   o  BT: a Bridge Table, as defined in [RFC7432].  A BT is the
      instantiation of a Broadcast Domain in a PE.  When there is a
      single Broadcast Domain in a given EVI, the MAC-VRF in each PE
      will contain a single BT.  When there are multiple BTs within the
      same MAC-VRF, each BT is associated to a different Ethernet Tag.
      The EVPN routes specific to a BT, will indicate which Ethernet Tag
      the route corresponds to.

      Example: In Figure 1, MAC-VRF1 has two BTs: BT1 and BT2.  Ethernet
      Tag x is defined in BT1 and Ethernet Tag y in BT2.

   o  AC: Attachment Circuit or logical interface associated to a given
      BT or IP-VRF.  To determine the AC on which a packet arrived, the
      PE will examine the combination of a physical port and VLAN tags
      (where the VLAN tags can be individual c-tags, s-tags or ranges of
      both).

      Example: In Figure 1, AC1 is associated to BT1, AC2 to BT2 and AC3
      to IP-VRF1.

   o  IRB: Integrated Routing and Bridging interface.  It refers to the
      logical interface that connects a BT to an IP-VRF and allows to
      forward packets with destination in a different subnet.

   o  MPLS/NVO tnl: It refers to a tunnel that can be MPLS or NVO-based
      (Network Virtualization Overlays) and it is used by MAC-VRFs and
      IP-VRFs.  Irrespective of the type, the tunnel may carry an
      Ethernet or an IP payload.  MAC-VRFs can only use tunnels with
      Ethernet payloads (setup by EVPN), whereas IP-VRFs can use tunnels
      with Ethernet (setup by EVPN) or IP payloads (setup by EVPN or

Rabadan, et al.          Expires January 7, 2023                [Page 5]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

      IPVPN).  IPVPN-only PEs have IP-VRFs but they cannot send or
      receive traffic on tunnels with Ethernet payloads.

      Example: Figure 1 shows an MPLS/NVO tunnel that is used to
      transport Ethernet frames to/from MAC-VRF1.  The PE determines the
      MAC-VRF and BT the packets belong to based on the EVPN label (MPLS
      or VNI).  Figure 1 also shows two MPLS/NVO tunnels being used by
      IP-VRF1, one carrying Ethernet frames and the other one carrying
      IP packets.

   o  RT-2: Route Type 2 or MAC/IP route, as per [RFC7432].

   o  RT-5: Route Type 5 or IP Prefix route, as per [RFC9136].

   o  NVE: Network Virtualization Edge router.

   o  Domain: Two PEs are in the same domain if they are attached to the
      same tenant and the packets between them do not require a data
      path IP lookup (in the tenant space) in any intermediate router.
      A gateway PE is always configured with multiple DOMAIN-IDs.

      Example 1: Figure 2 depicts an example where Tenant Systems TS1
      and TS2 belong to the same tenant, and they are located in
      different Data Centers that are connected by gateway PEs (see the
      gateway PE definition later).  These gateway PEs use IPVPN in the
      WAN.  When TS1 sends traffic to TS2, the intermediate routers
      between PE1 and PE2 require a tenant IP lookup in their IP-VRFs so
      that the packets can be forwarded.  In this example there are
      three different domains.  The gateway PEs connect the EVPN domains
      to the IPVPN domain.

                           GW1------------GW3
                         +------+       +------+
           +-------------|IP-VRF|       |IP-VRF|-------------+
          PE1            +------+       +------+            PE2
        +------+   DC1      |     WAN      |     DC2     +------+
    TS1-|IP-VRF|   EVPN     |    IPVPN     |     EVPN    |IP-VRF|-TS2
        +------+           GW2            GW4            +---+--+
           |             +------+       +------+             |
           +-------------|IP-VRF|       |IP-VRF|-------------+
                         +------+       +------+
                            +--------------+
               DOMAIN 1         DOMAIN 2       DOMAIN 3
           <---------------> <------------> <---------------->

                   Figure 2: Multiple domain DCI example

Rabadan, et al.          Expires January 7, 2023                [Page 6]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

     Example 2: Figure 3 illustrates a similar example, but PE1 and PE2
     are now connected by a BGP-LU (BGP Labeled Unicast) tunnel, and
     they have a BGP peer relationship for EVPN.  Contrary to Example 1,
     there is no need for tenant IP lookups on the intermediate routers
     in order to forward packets between PE1 and PE2.  Therefore, there
     is only one domain in the network and PE1/PE2 belong to it.

                                EVPN
           <------------------------------------------------->
                                BGP-LU
           <------------------------------------------------->

                          ASBR------------ASBR
                         +------+       +------+
           +-------------|      |       |      |-------------+
          PE1            +------+       +--+---+            PE2
        +------+   DC1      |     WAN      |     DC2     +------+
    TS1-|IP-VRF|   EVPN     |              |     EVPN    |IP-VRF|-TS2
        +------+          ASBR            ASBR           +---+--+
           |             +------+       +------+             |
           +-------------|      |       |      |-------------+
                         +------+       +------+
                            +--------------+

           <--------------------DOMAIN-1--------------------->

                    Figure 3: Single domain DCI example

   o  Regular Domain: a domain in which a single control plane ISF SAFI,
      i.e., BGP IP, IPVPN or EVPN, is used.  A Regular Domain is
      composed of regular PEs, see below.  In Figure 2 and Figure 3,
      above, all domains are regular domains.

   o  Composite Domain: a domain in which multiple control plane ISF
      SAFIs, i.e., BGP IP, IPVPN and/or EVPN, are used and which is
      composed of regular PEs, see below, and composite PEs, see below.

   o  Regular PE: a PE that is attached to a domain, either regular or
      composite, and which uses one of the control plane protocols (BGP
      IP, IPVPN or EVPN) operating in the domain.

   o  Interworking PE: a PE that may advertise a given prefix with an
      EVPN ISF route (RT-2 or RT-5) and/or an IPVPN ISF route and/or a
      BGP IP ISF route.  An interworking PE has one IP-VRF per tenant,
      and zero, one or multiple MAC-VRFs per tenant.  Each MAC-VRF may
      contain one or more BTs, where each BT may be attached to that IP-
      VRF via IRB.  There are two types of Interworking PEs: composite
      PEs and gateway PEs.  Both PE functions can be independently

Rabadan, et al.          Expires January 7, 2023                [Page 7]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

      implemented per tenant and they may both be implemented for the
      same tenant.

      Example: Figure 1 shows an interworking PE of type gateway, where
      ISF SAFIs 1, 128 and 70 are enabled.  IP-VRF1 and MAC-VRF1 are
      instantiated on the PE, and together provide inter-subnet
      forwarding for the tenant.

   o  Composite PE: an interworking PE that is attached to a composite
      domain and advertises a given prefix to an IPVPN peer with an
      IPVPN ISF route, to an EVPN peer with an EVPN ISF route, and to a
      route reflector with both an IPVPN and EVPN ISF route.  A
      composite PE performs the procedures of Section 7.

      Example: Figure 4 shows an example where PE1 is a composite PE
      since PE1 has EVPN and another ISF SAFI enabled to the same route-
      reflector, and PE1 advertises a given IP prefix IPn/x twice, one
      using EVPN and another one using ISF SAFI 128.  PE2 and PE3 are
      not composite PEs.

                                   +---+
                                   |PE2|
                                   +---+
                                    ^
               Interworking         |EVPN
                       PE    EVPN   v
                      +---+  IPVPN +--+       +---+
                      |PE1| <----> |RR| <---> |PE3|
                      +---+        +--+ IPVPN +---+
                    Composite

                Figure 4: Interworking composite PE example

   o  Gateway PE: an interworking PE that is attached to two (or more)
      domains, each either regular or composite, and which, based on
      configuration, does one of the following:

      -  Propagates ISF routes of the same ISF SAFI, i.e., BGP IP, IPVPN
         or EVPN, between the two domains.

      -  Propagates an ISF route received with an ISF SAFI to a domain
         that uses a different ISF SAFI.  E.g., it propagates a received
         EVPN ISF route as an IPVPN ISF route in the other domain and
         vice versa.  A gateway PE performs the procedures of Section 8.

      A gateway PE is always configured with multiple DOMAIN-IDs.  The
      DOMAIN-ID is encoded in the Domain Path Attribute (D-PATH), and

Rabadan, et al.          Expires January 7, 2023                [Page 8]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

      advertised along with ISF SAFI routes.  Section 4 describes the
      D-PATH attribute.

      Example: Figure 5 illustrates an example where PE1 is a gateway PE
      since the EVPN and IPVPN SAFIs are enabled on different BGP peers,
      and a given local IP prefix IPn/x is sent to both BGP peers for
      the same tenant.  PE2 and PE1 are in one domain and PE3 and PE1
      are in another domain.

                               Interworking PE
                       +---+ EVPN   +---+ IPVPN  +---+
                       |PE2| <----> |PE1| <----> |PE3|
                       +---+        +---+        +---+
                                   Gateway

                 Figure 5: Interworking gateway PE example

   o  Composite/Gateway PE: an interworking PE that is both a composite
      PE and a gateway PE that is attached to two domains, one regular
      and one composite, and which does the following:

      -  Propagates an ISF route from the regular domain into the
         composite domain.  Within the composite domain it acts as a
         composite PE.

      -  Propagates an ISF route from the composite domain into the
         regular domain.  Within the regular domain it is propagated as
         an ISF route using the ISF SAFI for that domain.

      This is particularly useful when a tenant network uses multiple
      ISF SAFIs (BGP IP, IPVPN and EVPN domains) and any-to-any
      connectivity is required.  In this case end-to-end control plane
      consistency, when possible, is desired.

4.  Domain Path Attribute (D-PATH)

   The BGP Domain Path (D-PATH) attribute is an optional and transitive
   BGP path attribute.

   Similar to AS_PATH, D-PATH is composed of a sequence of Domain
   segments.  Each Domain segment is comprised of <domain segment
   length, domain segment value>, where the domain segment value is a
   sequence of one or more Domains, as illustrated in Figure 6.  Each
   domain is represented by <DOMAIN-ID:ISF_SAFI_TYPE>.

Rabadan, et al.          Expires January 7, 2023                [Page 9]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   Octets
   0               1                    8                         n
   +---------------+----------------//--+----//-------------------+
   |Domain Segment |   Last Domain      |        Domain of Origin |
   |    Length     |                    |                         |
   +---------------+----------------//--+----//-------------------+
                    \__________________/
                                |
              Octets            v
              0                         6                7
              +------------------//-----+----------------+
              |    DOMAIN-ID            | ISF_SAFI_TYPE  |
              +------------------//-----+----------------+
              \________________________/
                           |
           Octets          v
           0     1     2     3     4     5     6
           +-----------------------+-----------+
           |        Global         |  Local    |
           |        Admin          |  Admin    |
           +-----------------------+-----------+

                      Figure 6: D-PATH Domain Segment

   o  The domain segment length field is a 1-octet field, containing the
      number of domains in the segment.

   o  DOMAIN-ID is a 6-octet field that represents a domain.  It is
      composed of a 4-octet Global Administrator sub-field and a 2-octet
      Local Administrator sub-field.  The Global Administrator sub-field
      MAY be filled with an Autonomous System Number (ASN, Public or
      Private), an IPv4 address, or any value that guarantees the
      uniqueness of the DOMAIN-ID (when the tenant network is connected
      to multiple Operators) and helps troubleshooting and debugging of
      D-PATH in ISF routes.

   o  ISF_SAFI_TYPE is a 1-octet field that indicates the Inter-Subnet
      Forwarding SAFI type in which a route was received, before the
      route is re-exported into a different domain.  The ISF_SAFI_TYPE
      field is informational and does not have any impact on the loop
      detection or BGP Path selection procedures.  The following types
      are assigned by this document:

Rabadan, et al.          Expires January 7, 2023               [Page 10]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

                     Value Type
                     ----- --------------------------
                     0     Gateway PE local ISF route
                     1     SAFI 1
                     70    EVPN
                     128   SAFI 128

   About the BGP D-PATH attribute:

   a.  Identifies the sequence of domains, each identified by a <DOMAIN-
       ID:ISF_SAFI_TYPE> through which a given ISF route has passed.

       -  This attribute list MAY contain one or more segments.

       -  The first entry in the list (leftmost) is the <DOMAIN-
          ID:ISF_SAFI_TYPE> from which a gateway PE is propagating an
          ISF route.  The last entry in the list (rightmost) is the
          <DOMAIN-ID:ISF_SAFI_TYPE> from which a gateway PE received an
          ISF route without a D-PATH attribute (the Domain of Origin).
          Intermediate entries in the list are domains that the ISF
          route has transited.

       -  As an example, an ISF route received with a D-PATH attribute
          containing a domain segment of {length=2,
          <6500:2:IPVPN>,<6500:1:EVPN>} indicates that the ISF route was
          originated in EVPN domain 6500:1, and propagated into IPVPN
          domain 6500:2.

   b.  It is added/modified by a gateway PE when propagating an update
       to a different domain (which runs the same or different ISF
       SAFI):

       -  A gateway PE's IP-VRF, that connects two domains, belongs to
          two DOMAIN-IDs, e.g. 6500:1 for EVPN and 6500:2 for IPVPN.

       -  Whenever a prefix arrives at a gateway PE in a particular ISF
          SAFI route, if the gateway PE needs to export that prefix to a
          BGP peer, the gateway PE MUST prepend a <DOMAIN-
          ID:ISF_SAFI_TYPE> to the list of domains in the received
          D-PATH, as long as the gateway PE works in Uniform-
          Propagation-Mode, as explained in Section 5.2 .

       -  For instance, in an IP-VRF configured with DOMAIN-IDs 6500:1
          for EVPN and 6500:2 for IPVPN, if an EVPN route for prefix P
          is received and P installed in the IP-VRF, the IPVPN route for
          P that is exported to an IPVPN peer will prepend the domain
          <6500:1:EVPN> to the previously received D-PATH attribute.
          Likewise, IP-VRF prefixes that are received from IP-VPN, will

Rabadan, et al.          Expires January 7, 2023               [Page 11]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

          be exported to EVPN peers with the domain <6500:2:IPVPN> added
          to the segment.

       -  In the above example, if the EVPN route is received without
          D-PATH, the gateway PE will add the D-PATH attribute with one
          segment {length=1, <6500:1:EVPN>} when re-advertising to
          domain 6500:2.

       -  Within the Domain of Origin, the update does not contain a
          D-PATH attribute because the update has not passed through a
          gateway PE yet.

   c.  For a local ISF route, i.e., a configured route or a route
       learned from a local attachment circuit, a gateway PE has three
       choices:

       1.  It MAY advertise that ISF route without a D-PATH attribute
           into one or more of its configured domains, in which case the
           D-PATH attribute will be added by the other gateway PEs in
           each of those domains.

       2.  It MAY advertise that ISF route with a D-PATH attribute into
           one or more of its configured domains, in which case the
           D-PATH attribute in each copy of the ISF route is initialized
           with an ISF_SAFI_TYPE of 0 and the DOMAIN-ID of the domain
           with which the ISF route is associated.

       3.  It MAY advertise that ISF route with a D-PATH attribute that
           contains a configured domain specific to its local ISF routes
           into one or more of its configured domains, in which case the
           D-PATH attribute in each copy of the ISF route is initialized
           with a ISF_SAFI_TYPE of 0 and the DOMAIN-ID for the local ISF
           routes.  This DOMAIN-ID MUST be globally unique and MAY be
           shared by two or more gateway PEs.

   d.  An ISF route received by a gateway PE with a D-PATH attribute
       that contains one or more of its locally associated DOMAIN-IDs
       for the IP-VRF is considered to be a looped ISF route.  The ISF
       route in this case MUST be flagged as "looped" and be installed
       in the IP-VRF only in case there is no better route after the
       best path selection (Section 6).  The ISF_SAFI_TYPE is irrelevant
       for the purpose of loop detection of an ISF route.  In other
       words, an ISF route is considered as a looped route if it
       contains a D-PATH attribute with at least one DOMAIN-ID matching
       a local DOMAIN-ID, irrespective of the ISF_SAFI_TYPE of the
       DOMAIN-ID.

Rabadan, et al.          Expires January 7, 2023               [Page 12]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

       For instance, in the example of Figure 2, gateway GW1 receives
       TS1 prefix in two different ISF routes:

       -  In an EVPN RT-5 with next-hop PE1 and no D-PATH attribute.

       -  In a SAFI 128 route with next-hop GW2 and D-PATH = {length=1,
          <6500:1:EVPN>}, assuming that DOMAIN-ID for domain 1 is
          6500:1.

       Gateway GW1 flags the SAFI 128 route as "looped" (since 6500:1 is
       a local DOMAIN-ID in GW1) and it will not install it in the
       tenant IP-VRF, since the route selection process selects the EVPN
       RT-5 due to a shorter D-PATH attribute (Section 6).  Gateway GW1
       identifies the route as "looped" even if the ISF_SAFI_TYPE value
       is unknown to GW1, i.e., any value different from the ones
       specified in this document).

   e.  A DOMAIN-ID value on a gateway PE MAY be assigned for a peering
       domain or MAY be scoped for an individual tenant IP-VRF.

       -  If allocated for a peering domain, the DOMAIN-ID applies to
          all tenant IP-VRFs for that domain.

       -  If allocated for a specific tenant IP-VRF, the processing of
          the received D-PATH and its propagation is in the context of
          the IP-VRF DOMAIN-ID.  Route leaking is a use-case where a
          per-IP-VRF DOMAIN-ID assignment is necessary.  Suppose
          gateways PE1 and PE2 are attached to two different tenant IP-
          VRFs, IP-VRF-1 and IP-VRF-2.  ISF SAFI routes advertised by
          gateway PE1 for IP-VRF-1 are received on gateway PE2 with
          DOMAIN-ID 6500:1.  If the routes are leaked from IP-VRF-1 into
          IP-VRF-2 on PE2, and re-advertised back to PE1 in the context
          of IP-VRF-2, PE1 will not identify the route as a looped
          route.  This is because PE1 processes the route in the context
          of IP-VRF-2, where DOMAIN-ID 6500:1 is not a local DOMAIN-ID.

   f.  The number of domains in the D-PATH attribute indicates the
       number of gateway PEs that the ISF route update has transited.
       If one of the transit gateway PEs leaks a given ISF route between
       two local IP-VRFs, it MAY prepend a domain with a ISF_SAFI_TYPE
       of 0 for the leaked route when the route is exported into an ISF
       SAFI.  In that case, the number of domains in the D-PATH
       attribute indicates the number of tenant IP-VRFs that the ISF
       route update has transited.

   g.  The following error-handling rules apply to the D-PATH attribute:

Rabadan, et al.          Expires January 7, 2023               [Page 13]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

       1.  A received D-PATH attribute is considered malformed if it
           contains a malformed Domain Segment.

       2.  A Domain Segment is considered malformed in any of the
           following cases:

           *  The Domain Segment length of the last Domain Segment
              causes the D-PATH attribute length to be exceeded.

           *  After the last successfully parsed Domain Segment there
              are less than eight octets remaining.

           *  The Domain Segment has a Domain Segment Length of zero.

       3.  A PE receiving an UPDATE message with a malformed D-PATH
           attribute SHALL apply "treat-as-withdraw" [RFC7606].

       4.  Domains in the D-PATH attribute with unknown ISF_SAFI_TYPE
           values are accepted and not considered an error.

       5.  In case a PE receives more than one D-PATH attribute with a
           route, the PE SHALL process the first one in the list and not
           store and propagate the others.

5.  BGP Path Attribute Propagation across ISF SAFIs

   Based on its configuration, a gateway PE is required to propagate an
   ISF route between two domains that use the same or different ISF
   SAFI.  This requires a definition of what a gateway PE has to do with
   BGP Path Attributes attached to the ISF route that the gateway PE is
   propagating.  This section specifies the BGP Path Attribute
   propagation modes that a gateway PE may follow when receives an ISF
   route with ISF SAFI-x, installs the route in the IP-VRF and exports
   the ISF route into ISF SAFI-y.  ISF SAFI-x and SAFI-y values MAY be
   the same values.

5.1.  No-Propagation-Mode

   This is the default mode of operation for gateway PEs that re-export
   ISF routes from a domain into another domain.  In this mode, the
   gateway PE will simply re-initialize the BGP Path Attributes when
   propagating an ISF route, as though it would for direct or local IP
   prefixes.  This model may be enough in those use-cases where, e.g.,
   the EVPN domain is considered an "abstracted" CE and remote IPVPN/IP
   PEs don't need to consider the original EVPN Attributes for path
   calculations.

Rabadan, et al.          Expires January 7, 2023               [Page 14]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   Since this mode of operation does not propagate the D-PATH attribute
   either, redundant gateway PEs are exposed to routing loops.  Those
   loops may be resolved by policies and the use of other attributes,
   such as the Route Origin extended community [RFC4360], however not
   all the loop situations may be identified.

5.2.  Uniform-Propagation-Mode

   In this mode, the gateway PE simply keeps accumulating or mapping
   certain key commonly used BGP Path Attributes when propagating an ISF
   route.  This mode is typically used in networks where EVPN and IPVPN
   SAFIs are used seamlessly to distribute IP prefixes.

   The following rules MUST be observed by the gateway PE when
   propagating BGP Path Attributes:

   1.  The gateway PE imports an ISF route in the IP-VRF and stores the
       original Path Attributes.  The following set of Path Attributes
       SHOULD be propagated by the gateway PE to other ISF SAFIs (other
       BGP Path Attributes SHOULD NOT be propagated):

       -  AS_PATH

       -  D-PATH

       -  IBGP-only Path Attributes: LOCAL_PREF, ORIGINATOR_ID,
          CLUSTER_ID

       -  MED

       -  AIGP

       -  Communities, Extended Communities and Large Communities,
          except for the EVPN extended communities, Route Target
          extended communities and BGP Encapsulation extended
          communities.

   2.  When propagating an ISF route to a different IBGP peer, the
       gateway PE SHOULD keep the AS_PATH of the originating ISF route
       and add it to the destination ISF SAFI without any modification.
       When re-advertising to an EBGP peer, the gateway PE SHOULD keep
       the AS_PATH of the originating ISF route and prepend the IP-VRF's
       AS before sending the route.

   3.  When propagating an ISF route to IBGP peers, the gateway PE
       SHOULD keep the IBGP-only Path Attributes from the originating
       route to the re-advertised route.

Rabadan, et al.          Expires January 7, 2023               [Page 15]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   4.  As discussed in point 1, Communities, Extended Communities and
       Large Communities SHOULD be preserved from the originating ISF
       route by the gateway PE.  Exceptions of Extended Communities that
       SHOULD NOT be propagated are:

       A.  BGP Encapsulation extended communities [RFC9012].

       B.  Route Target extended communities.  Route Targets are always
           initialized when readvertising an ISF route into a different
           domain, i.e., they are not propagated.  The initialized Route
           Target in the re-advertised ISF route may or may not have the
           same value as the Route Target of the originating ISF route.

       C.  All the extended communities of type EVPN.

       The gateway PE SHOULD NOT copy the above extended communities
       from the originating ISF route to the re-advertised ISF route.

   5.  For a given ISF route, only the BGP Path Attributes of the best
       path can be propagated to another ISF route.  If multiple paths
       are received for the same route in an ISF SAFI, the BGP best path
       selection will determine what the best path is, and therefore the
       set of Path Attributes to be propagated.  Even if Equal Cost
       Multi-Path (ECMP) is enabled on the IP-VRF by policy, only the
       BGP Path Attributes of the selected best path are propagated.

5.3.  Aggregation of Routes and Path Attribute Propagation

   Instead of propagating a high number of (host) ISF routes between
   domains, a gateway PE that receives multiple ISF routes from a domain
   MAY choose to propagate a single ISF aggregate route into a different
   domain.  In this document, aggregation is used to combine the
   characteristics of multiple ISF routes in such way that a single
   aggregate ISF route can be propagated to the destination domain.
   Aggregation of multiple ISF routes of one ISF SAFI into an aggregate
   ISF route is only done by a gateway PE.

   Aggregation on gateway PEs may use either the No-Propagation-Mode or
   the Uniform-Propagation-Mode explained in Section 5.1 and
   Section 5.2, respectively.

   When using Uniform-Propagation-Mode, Path Attributes of the same type
   code MAY be aggregated according to the following rules:

   o  AS_PATH is aggregated based on the rules in [RFC4271].  The
      gateway PEs SHOULD NOT receive AS_PATH attributes with path
      segments of type AS_SET [RFC6472].  Routes received with AS_PATH
      attributes including AS_SET path segments MUST NOT be aggregated.

Rabadan, et al.          Expires January 7, 2023               [Page 16]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   o  ISF routes that have different attributes of the following type
      codes MUST NOT be aggregated: D-PATH, LOCAL_PREF, ORIGINATOR_ID,
      CLUSTER_ID, MED or AIGP.

   o  The Community, Extended Community and Large Community attributes
      of the aggregate ISF route MUST contain all the Communities/
      Extended Communities/Large Communities from all of the aggregated
      ISF routes, with the exceptions of the extended communities listed
      in Section 5.2 that are not propagated.

   Assuming the aggregation can be performed (the above rules are
   applied), the operator should consider aggregation to deal with
   scaled tenant networks where a significant number of host routes
   exists.  For example, large Data Centers.

6.  Route Selection Process for ISF Routes

   A PE may receive an IP prefix in ISF routes with different ISF SAFIs,
   from the same or different BGP peer.  It may also receive the same IP
   prefix (host route) in an EVPN RT-2 and RT-5.  A route selection
   algorithm across all ISF SAFIs is needed so that:

   o  Different gateway and composite PEs have a consistent and
      deterministic view on how to reach a given prefix.

   o  Prefixes advertised in EVPN and other ISF SAFIs can be compared
      based on path attributes commonly used by operators across
      networks.

   o  Equal Cost Multi-Path (ECMP) is allowed across EVPN and other ISF
      SAFI routes.

   For a given prefix advertised in one or more non-EVPN ISF routes, the
   BGP best path selection procedure will produce a set of "non-EVPN
   best paths".  For a given prefix advertised in one or more EVPN ISF
   routes, the BGP best path selection procedure will produce a set of
   "EVPN best paths".  To support EVPN/non-EVPN ISF interworking in the
   context of the same IP-VRF receiving non-EVPN and EVPN ISF routes for
   the same prefix, it is then necessary to run a tie-breaking selection
   algorithm on the union of these two sets.  This tie-breaking
   algorithm begins by considering all EVPN and other ISF SAFI routes,
   equally preferable routes to the same destination, and then selects
   routes to be removed from consideration.  The process terminates as
   soon as only one route remains in consideration.

   The route selection algorithm must remove from consideration the
   routes following the rules and the order defined in [RFC4271], with
   the following exceptions and in the following order:

Rabadan, et al.          Expires January 7, 2023               [Page 17]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   1.  Immediately after removing from consideration all routes that are
       not tied for having the highest Local Preference, any routes that
       do not have the shortest D-PATH are also removed from
       consideration.  Routes with no D-PATH are considered to have a
       zero-length D-PATH.

   2.  Then regular [RFC4271] selection criteria is followed.

   3.  At the end of the selection algorithm, if at least one route
       still under consideration is an RT-2 route, remove from
       consideration any RT-5 routes.

   4.  If Steps 1-3 leave Equal Cost Multi-Paths (ECMP) between non-EVPN
       and EVPN paths, the EVPN path MUST be considered (and the non-
       EVPN path removed from consideration).  However, if ECMP across
       ISF SAFIs is enabled by policy, and one EVPN path and one non-
       EVPN path remain at the end of step 3, both path types MUST be
       used.

   The above process modifies the [RFC4271] selection criteria for
   multiprotocol BGP routes with SAFIs 1, 128 and EVPN IP Prefix routes
   to include the shortest D-PATH so that operators minimize the number
   of Gateways and domains through which packets need to be routed.

   Example 1 - PE1 receives the following routes for IP1/32, that are
   candidate to be imported into IP-VRF-1:

   {SAFI=EVPN, RT-2, Local-Pref=100, AS-Path=(100,200)}
   {SAFI=EVPN, RT-5, Local-Pref=100, AS-Path=(100,200)}
   {SAFI=128, Local-Pref=100, AS-Path=(100,200)}

   Selected route: {SAFI=EVPN, RT-2, Local-Pref=100, AS-Path=100,200]
   (due to step 3, and no ECMP).

   Example 2 - PE1 receives the following routes for IP2/24, that are
   candidate to be imported into IP-VRF-1:

   {SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS-Path=(100,200), MED=10}
   {SAFI=128, D-PATH=(6500:1:EVPN,6500:2:IPVPN), AS-Path=(200), MED=200}

   Selected route: {SAFI=EVPN, RT-5, D-PATH=(6500:3:IPVPN), AS-
   Path=(100,200), MED=10} (due to step 1).

7.  Composite PE Procedures

   As described in Section 3, composite PEs are typically used in tenant
   networks where EVPN and IPVPN are both used to provide inter-subnet
   forwarding within the same composite domain.

Rabadan, et al.          Expires January 7, 2023               [Page 18]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   Figure 7 depicts an example of a composite domain, where PE1/PE2/PE4
   are composite PEs (they support EVPN and IPVPN ISF SAFIs on their
   peering to the Route Reflector), and PE3 is a regular IPVPN PE.

                +-----------------------------------+
                |                                   |
                |        MPLS                  IPVPN PE3
                |        Network              +----------+ IP3/24
                |                     IPVPN   |+------+  |   +---+
                |                      +----->||IP-VRF|------|CE3|
           Composite PE1               |      |+------+  |   +---+
          +---------------+            |      +----------+
          |      +------+ |  EVPN      v             |
          |      |IP-VRF| |  IPVPN   +--+            |
          | +----|      | | <------> |RR|            |
   +---+  | |    +------+ |          +--+         Composite PE4
   |CE2|----|MAC-VRF|     |          ^  ^         +---------+ IP4/24
   +---+  | +-------+     |    EVPN  |  | EVPN    |+------+ |   +---+
          +---|-----------+    IPVPN |  | IPVPN   ||IP-VRF|-----|CE4|
              |  |              +----+  +-------->|+------+ |   +---+
       IP1/24 |  |              v                 +---------+
       +---+  |  |    +---------------+              |
       |CE1|--+  +----|      +------+ +--------------+
       +---+          |      |IP-VRF| |
         |            | +----|      | |
         |            | |    +------+ |
         +--------------|MAC-VRF|     |
                      | +-------+     |
                      +---------------+
                         Composite PE2

                      Figure 7: Composite PE example

   In a composite domain with composite and regular PEs:

   1.  The composite PEs MUST advertise the same IP prefixes in each ISF
       SAFI to the RR.  For example, in Figure 7, the prefix IP1/24 is
       advertised by PE1 and PE2 to the RR in two separate NLRIs, one
       for AFI/SAFI 1/128 and another one for EVPN.

   2.  As an informative note, the RR does not forward EVPN routes to
       neighbors on which the EVPN SAFI is not enabled, and similarly,
       the RR does not forward IPVPN routes to neighbors on which the
       IPVPN SAFI is not enabled.  For example, the RR does not forward
       EVPN routes to PE3 (since the RR does not have the EVPN SAFI
       enabled on its BGP session to PE3), whereas the IPVPN routes are
       forwarded to all the PEs.

Rabadan, et al.          Expires January 7, 2023               [Page 19]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   3.  IPVPN PEs process and import IPVPN routes, as in [RFC4364].  As
       an example, PE3 receives only the IPVPN route for IP1/24 and
       resolves the BGP next-hop to an MPLS tunnel (with IP payload) to
       PE1 and/or PE2.

   4.  Composite PEs MUST process routes for the same prefix coming from
       different ISF SAFI routes, and perform route selection.

       -  As an example, PE4 receives IP1/24 encoded in EVPN and another
          ISF SAFI route (EVPN RT-5 and IPVPN).  The route selection
          follows the procedures in Section 6.

       -  Assuming an EVPN route is selected, PE4 resolves the BGP next-
          hop to an MPLS tunnel (with Ethernet or IP payload) to PE1
          and/or PE2.  As described in Section 3, two EVPN PEs may use
          tunnels with Ethernet or IP payloads to connect their IP-VRFs,
          depending on the [RFC9136] model implemented.

       -  The other composite PEs (PE1 and PE2) receive also the same IP
          prefix via EVPN and IPVPN SAFIs and they also follow the route
          selection in Section 6.

   5.  When a given route has been selected as the route for a
       particular packet, the transmission of the packet MUST be done
       according to the rules for that route's AFI/SAFI.

   6.  As an informative note, in composite domains, such as the one in
       Figure 7, the EVPN advanced forwarding features will only be
       available to composite and EVPN PEs (assuming they select an RT-5
       to forward packets for a given IP prefix), and not to IPVPN PEs.
       For example, assuming PE1 sends IP1/24 in an EVPN and an IPVPN
       route and the EVPN route is the best one in the selection, the
       recursive resolution of the EVPN RT-5s can only be used in PE2
       and PE4 (composite PEs), and not in PE3 (IPVPN PE).  As a
       consequence of this, the indirection provided by the RT5's
       recursive resolution and its benefits in a scaled network, will
       not be available in all the PEs in the network.

8.  Gateway PE Procedures

   Section 3 defines a gateway PE as an Interworking PE that is attached
   to two (or more) domains and propagates ISF routes between those
   domains.  Examples of gateway PEs are Data Center gateways connecting
   domains that make use of EVPN and other ISF SAFIs for a given tenant.
   The gateway PE procedures in this document provide an interconnect
   solution for ISF routes and complement the gateway definition of
   [RFC9014], which focuses on the interconnect solution for Layer 2.
   This section applies to the interconnect of two domains that use

Rabadan, et al.          Expires January 7, 2023               [Page 20]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   different ISF SAFIs (e.g., EVPN to IPVPN), as well as the
   interconnect of two domains of the same ISF SAFI (e.g., EVPN to
   EVPN).  Figure 8 illustrates a gateway PE use-case, in which PE1 and
   PE2 (and PE3/PE4) are gateway PEs interconnecting domains for the
   same tenant.

      <----EVPN---->    <----------IPVPN--------->   <----EVPN---->
        6500:1:EVPN             6500:2:IPVPN           6500:3:EVPN
   <DOMAIN-ID:ISF_SAFI_TYPE>
                         +-----------------------+
                  Gateway PE1              Gateway PE3
                  +----------+             +----------+
      +-----------|+------+  |  MPLS tnls  |+------+  |------------+
      |           ||IP-VRF|  |             ||IP-VRF|  |            |
    PE5           |+------+  |             |+------+  |           PE6
   +------+       +----------+             +----------+         +------+
   |IP-VRF| NVO tnls |   |                       |  | NVO tnls  |IP-VRF|
   |      |          |   |                       |  |           |      |
   +------+       +----------+             +----------+         +------+
   IP1/24-->      |+------+  |             |+------+  |            |
      |           ||IP-VRF|  |             ||IP-VRF|  |            |
      +-----------|+------+  |             |+------+  |------------+
                  +----------+             +----------+
                  Gateway PE2   +------+   Gateway PE4
                        +-------|IP-VRF|---------+
                                |      |
                                +------+
                                  PE7

                       Figure 8: Gateway PE example

   The procedures for a gateway PE enabled for ISF SAFI-x and ISF SAFI-y
   on the same IP-VRF follow:

   1.  A gateway PE that imports an ISF SAFI-x route to prefix P in an
       IP-VRF, MUST export P in ISF SAFI-y if:

       A.  P is installed in the IP-VRF - which means the SAFI-x route
           is well-formed, valid and the best one for P - and

       B.  PE has a BGP peer for SAFI-y (enabled for the same IP-VRF)
           and

       C.  The advertisement is allowed by policy and

       D.  ISF SAFI-x and ISAF SAFI-y are any of the types defined in
           Section 3.  Note that SAFI-x and SAFI-y MAY have the same
           value.

Rabadan, et al.          Expires January 7, 2023               [Page 21]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

       In the example of Figure 8, gateway PE1 and PE2 receive an EVPN
       RT-5 with IP1/24, install the prefix in the IP-VRF and re-
       advertise it using SAFI 128.

   2.  A gateway PE that receives an ISF SAFI-x route to prefix P in an
       IP-VRF MUST NOT export P in ISF SAFI-y if:

       A.  The ISF SAFI-x route is not well-formed or valid.  Rules to
           determine if a route is well-formed or valid for a given ISF
           SAFI are defined by the specification of each ISF SAFI.  As
           an example, an EVPN IP Prefix route received with non-zero
           ESI and GW IP values, at the same time, is not valid as per
           [RFC9136], section 3.2.

       B.  The ISF SAFI-x route contains a D-PATH attribute with one or
           more of the gateway PE's locally associated domains for the
           IP-VRF.  In this case the route is considered to be a looped
           ISF route, as described in Section 4 and hence MUST NOT be
           exported in ISF SAFI-y.

   Once the gateway PE determines that P must be exported, P will be
   advertised using ISF SAFI-y as follows:

   a.  The D-PATH attribute MUST be included, so that loops can be
       detected in remote gateway PEs.  When a gateway PE propagates an
       ISF route between domains, it MUST prepend a <DOMAIN-
       ID:ISF_SAFI_TYPE> to the received D-PATH attribute.  The DOMAIN-
       ID and ISF_SAFI_TYPE fields refer to the domain over which the
       gateway PE received the IP prefix and the ISF SAFI of the route,
       respectively.  If the received IP prefix route did not include
       any D-PATH attribute, the gateway IP MUST add the D-PATH when
       readvertising.  The D-PATH in this case will have only one
       segment on the list, the <DOMAIN-ID:ISF_SAFI_TYPE> of the
       received route.

       In the example of Figure 8, gateway PE1/PE2 receive the EVPN RT-5
       with no D-PATH attribute since the route is originated at PE5.
       Therefore PE1 and PE2 will add the D-PATH attribute including
       <DOMAIN-ID:ISF_SAFI_TYPE> = <6500:1:EVPN>.  Gateways PE3/PE4 will
       propagate the route again, now prepending their <DOMAIN-
       ID:ISF_SAFI_TYPE> = <6500:2:IPVPN>.  PE6 receives the EVPN RT-5
       routes with D-PATH = {<6500:2:IPVPN>,<6500:1:EVPN>} and can use
       that information to make BGP path decisions.

   b.  The gateway PE MAY use the Route Distinguisher of the IP-VRF to
       readvertise P in the ISF SAFI-y.

Rabadan, et al.          Expires January 7, 2023               [Page 22]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   c.  The label allocation used by each gateway PE is a local
       implementation matter.  The IP-VRF advertising IP prefixes for
       EVPN and another ISF SAFI may use a label per-VRF, per-prefix,
       etc.

   d.  The gateway PE MUST be able to use the same or different set of
       Route Targets per domain on the same IP-VRF.  In particular, if
       different domains use different set of Route Targets for the same
       tenant, the gateway PE MUST be able to import and export routes
       with the different sets.

   e.  Even though Figure 8 only shows two domains per gateway PE, the
       gateway PEs may be connected to more than two domains.

   f.  There is no limitation of gateway PEs that a given IP prefix P
       can pass through until it reaches a given PE.

   g.  If the gateway PE uses Uniform-Propagation-Mode for BGP Path
       Attribute propagation, the rules in Section 5.2 are followed.

   h.  As an informative note, if P was originated in an EVPN domain but
       traversed a different ISF SAFI domain (or domains), it will lose
       EVPN-specific attributes that are used in advanced EVPN
       procedures.  For example, even if PE1 advertises IP1/24 along
       with a given non-zero ESI (for recursive resolution to that ESI),
       when PE6 receives the IP prefix in an EVPN route, the ESI value
       will be zero.  This is because the route traverses an ISF SAFI
       domain that is different than EVPN.

9.  Interworking Use-Cases

   While Interworking PE networks may well be similar to the examples
   described in Section 7 and Section 8, in some cases a combination of
   both functions may be required.  Figure 9 illustrates an example
   where the gateway PEs are also composite PEs, since not only they
   need to propagate ISF routes between domains (from EVPN SAFI to IPVPN
   and/or EVPN SAFIs), but they also need to interwork with IPVPN-only
   PEs in a domain with a mix of composite and IPVPN-only PEs.

Rabadan, et al.          Expires January 7, 2023               [Page 23]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

                         +-----------------------------------+
                         |                                   |
                         |        MPLS                 IPVPN PE3
                         |        Network              +---------+
                         |                     IPVPN   |+------+ |
                         |                      +----->||IP-VRF|---TS3
                  (GW+composite) PE1            |      |+------+ |
                   +---------------+            |      +---------+
                   |      +------+ |  EVPN      v            |
                   |      |IP+VRF| |  IPVPN  +--+            |
                   | +----|      | | <------>|RR|            |
          +--------| |    +------+ |         +--+       Composite PE4
          |        | |MAC+VRF|     |         ^  ^        +---------+
          |        | +-------+     |   EVPN  |  | EVPN   |+------+ |
       +----+      +---------------+   IPVPN |  | IPVPN  ||IP-VRF|---TS4
   TS1-|NVE1|             |             +----+  +------->|+------+ |
       +----+             |             v                +---------+
          |    EVPN DC    |    +---------------+             |
          |    NVO tnls   +----|      +------+ |-------------+
          |                    |      |IP+VRF| |
          |                    | +----|      | |
          |                    | |    +------+ |
          |     +----+         | |MAC+VRF|     |
          +-----|NVE2|---------| +-------+     |
                +----+         +---------------+
                  |           (GW+composite) PE2
                 TS2

       Figure 9: Gateway and composite combined functions - example

   In the example above, PE1 and PE2 MUST follow the procedures
   described in Section 7 and Section 8.  Compared to the example in
   Section 8, PE1 and PE2 now need to also propagate ISF routes from
   EVPN to EVPN, in addition to propagating prefixes from EVPN to IPVPN.

   It is worth noting that PE1 and PE2 will receive TS4's IP prefix via
   IPVPN and EVPN RT-5 routes.  When readvertising to NVE1 and NVE2, PE1
   and PE2 will consider the D-PATH rules and attributes of the selected
   route for TS4 (Section 6 describes the Route Selection Process).

10.  BGP Error Handling on Interworking PEs

   An Interworking PE (acting as gateway PE or composite PE) observes
   the following error-handling procedures for ISF routes:

   o  An UPDATE message for an ISF route containing a D-PATH attribute
      MUST follow the error-handling rules for D-PATH, as specified in
      Section 4.

Rabadan, et al.          Expires January 7, 2023               [Page 24]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   o  Any received UPDATE for an ISF route complies with the procedures
      in [RFC7606].

   o  The Interworking PEs do not introduce any new error-handling rules
      for UPDATES received with NLRIs and BGP Path Attributes defined in
      other specifications.  Interworking PEs follow the error-handling
      defined in the specification for the specific NLRI or BGP Path
      Attribute.  In other words, UPDATES for BGP IP routes MUST follow
      the error-handling procedures of [RFC4760] [RFC8950], UPDATES for
      IPVPN routes MUST follow the error-handling rules of [RFC4364]
      [RFC4659], UPDATES for EVPN MAC/IP routes MUST follow the error-
      handling of [RFC7432] [RFC8365] and UPDATES for EVPN IP Prefix
      routes MUST follow the error-handling in [RFC9136].

   o  Received UPDATE messages to be programmed in IP-VRFs supporting
      Segment Routing for IPv6 data path (SRv6) follow the error-
      handling rules defined in [I-D.ietf-bess-srv6-services].

   If a gateway PE is set to propagate BGP Path Attributes for ISF
   routes across domains, the procedures in Section 5.2 guarantee that a
   BGP speaker does not receive UPDATES with well-formed but unexpected
   BGP Path Attributes.  If a gateway PE fails to follow the propagation
   rules in Section 5.2 and propagates some BGP Path Attributes
   erroneously, the receiving PEs follow the specifications for the
   specific ISF route type and BGP Path Attribute.  Some (but not all)
   examples follow:

   o  If the gateway PE erroneously propagates the Router's MAC Extended
      Community [RFC9135] from an EVPN domain to another EVPN domain,
      the receiving PE may find two EVPN Router's MAC extended
      communities in the same ISF route.  In this case, the PE follows
      [RFC9135] and processes the first one (ignoring the second
      extended community).

   o  If the gateway PE erroneously propagates the BGP Encapsulation
      Extended Community (or equivalent Encapsulation TLV in the Tunnel
      Encapsulation Attribute) [RFC9012] from an EVPN domain to another
      EVPN domain, the receiving PE may find two BGP Encapsulation
      Extended Communities with different values in the same ISF route.
      The PE in this case follows [RFC8365], which allows multiple
      encapsulations being signaled in the route.  As per [RFC9012],
      encapsulations advertised using the Tunnel Encapsulation attribute
      are considered equally with those advertised using the
      Encapsulation Extended Community.

   o  If the gateway PE erroneously propagates any EVPN extended
      community from an EVPN domain into an IPVPN domain, the receiving

Rabadan, et al.          Expires January 7, 2023               [Page 25]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

      IPVPN PE ignores the EVPN extended communities, since their
      semantics do not apply to the IPVPN SAFI.

   o  If the gateway PE erroneously propagates a BGP Prefix-SID
      attribute with SRv6 Service TLVs [I-D.ietf-bess-srv6-services] for
      an ISF route propagated between domains, the receiving PE follows
      [I-D.ietf-bess-srv6-services] in case multiple SRv6 TLV instances
      are received.

11.  Conclusion

   This document describes the procedures required in PEs that process
   and advertise ISF routes for a given tenant.  In particular, this
   document defines:

   o  A route selection algorithm so that a PE can determine what path
      to choose between EVPN paths and other ISF SAFI paths.

   o  A new BGP Path attribute called D-PATH that provides loop
      protection and visibility on the domains a particular route has
      traversed.

   o  The way BGP Path Attributes should be propagated between domains.

   o  The procedures that must be followed on Interworking PEs that
      behave as composite PEs, gateway PEs or a combination of both.

   The above procedures provide an operator with the required tools to
   build large tenant networks that may span multiple domains, use
   different ISF SAFIs to handle IP prefixes, in a deterministic way and
   with routing loop protection.

12.  Security Considerations

   In general, the security considerations described in [RFC9136] and
   [RFC4364] apply to this document.

   Section 4 introduces the use of the D-PATH attribute, which provides
   a security tool against control plane loops that may be introduced by
   the use of gateway PEs that propagate ISF routes between domains.  A
   correct use of the D-PATH will prevent control plane and data plane
   loops in the network, however an incorrect configuration of the
   DOMAIN-IDs on the gateway PEs may lead to the detection of false
   route loops and the blackholing of the traffic.  An attacker may
   benefit of this transitive attribute to propagate the wrong domain
   information across multiple domains.

Rabadan, et al.          Expires January 7, 2023               [Page 26]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   In addition, Section 5.2 introduces the propagation of BGP Path
   Attributes between domains on gateway PEs.  Without this mode of
   propagation, BGP Path Attributes are re-initialized when re-exporting
   ISF routes into a different domain, and the operator does not have
   the end-to-end visibility of a given ISF route path.  However, the
   Uniform Propagation mode introduces the capability of propagating BGP
   Path Attributes beyond the ISF SAFI scope.  While this is a useful
   tool to provide end-to-end visibility across multiple domains, it can
   also be used by an attacker to propagate wrong (although correctly
   formed) BGP Path Attributes that can influence the BGP path selection
   in remote domains.  An implementation can also choose Section 5.1
   (No-propagation mode) to minimize the risks derived from propagating
   incorrect attributes, but again, this mode of operation will prevent
   the receiver PE from seeing the attributes that the originator of the
   route intended to convey in the first place.

13.  IANA Considerations

   This document defines a new BGP path attribute known as the BGP
   Domain Path (D-PATH) attribute.

   IANA has assigned a new attribute code type from the "BGP Path
   Attributes" subregistry under the "Border Gateway Protocol (BGP)
   Parameters" registry:

   Path Attribute Value    Code                       Reference
   --------------------    ------------------------   ---------------
   36                      BGP Domain Path (D-PATH)   [This document]

14.  Acknowledgments

   The authors want to thank Russell Kelly, Dhananjaya Rao, Suresh
   Basavarajappa, Mallika Gautam, Senthil Sathappan, Arul Mohan Jovel,
   Naveen Tubugere, Mathanraj Petchimuthu, Eduard Vasilenko, Amit Kumar,
   Mohit Kumar and Lukas Krattiger for their review and suggestions.

15.  Contributors

16.  References

16.1.  Normative References

   [RFC7432]  Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
              Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
              Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
              2015, <https://www.rfc-editor.org/info/rfc7432>.

Rabadan, et al.          Expires January 7, 2023               [Page 27]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   [RFC8365]  Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
              Uttaro, J., and W. Henderickx, "A Network Virtualization
              Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
              DOI 10.17487/RFC8365, March 2018,
              <https://www.rfc-editor.org/info/rfc8365>.

   [RFC4271]  Rekhter, Y., Ed., Li, T., Ed., and S. Hares, Ed., "A
              Border Gateway Protocol 4 (BGP-4)", RFC 4271,
              DOI 10.17487/RFC4271, January 2006,
              <https://www.rfc-editor.org/info/rfc4271>.

   [RFC4364]  Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
              Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
              2006, <https://www.rfc-editor.org/info/rfc4364>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC7606]  Chen, E., Ed., Scudder, J., Ed., Mohapatra, P., and K.
              Patel, "Revised Error Handling for BGP UPDATE Messages",
              RFC 7606, DOI 10.17487/RFC7606, August 2015,
              <https://www.rfc-editor.org/info/rfc7606>.

   [RFC4760]  Bates, T., Chandra, R., Katz, D., and Y. Rekhter,
              "Multiprotocol Extensions for BGP-4", RFC 4760,
              DOI 10.17487/RFC4760, January 2007,
              <https://www.rfc-editor.org/info/rfc4760>.

   [RFC9136]  Rabadan, J., Ed., Henderickx, W., Drake, J., Lin, W., and
              A. Sajassi, "IP Prefix Advertisement in Ethernet VPN
              (EVPN)", RFC 9136, DOI 10.17487/RFC9136, October 2021,
              <https://www.rfc-editor.org/info/rfc9136>.

   [RFC9135]  Sajassi, A., Salam, S., Thoria, S., Drake, J., and J.
              Rabadan, "Integrated Routing and Bridging in Ethernet VPN
              (EVPN)", RFC 9135, DOI 10.17487/RFC9135, October 2021,
              <https://www.rfc-editor.org/info/rfc9135>.

Rabadan, et al.          Expires January 7, 2023               [Page 28]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   [I-D.ietf-bess-srv6-services]
              Dawra, G., Talaulikar, K., Raszuk, R., Decraene, B.,
              Zhuang, S., and J. Rabadan, "SRv6 BGP based Overlay
              Services", draft-ietf-bess-srv6-services-15 (work in
              progress), March 2022.

16.2.  Informative References

   [RFC4360]  Sangli, S., Tappan, D., and Y. Rekhter, "BGP Extended
              Communities Attribute", RFC 4360, DOI 10.17487/RFC4360,
              February 2006, <https://www.rfc-editor.org/info/rfc4360>.

   [RFC9012]  Patel, K., Van de Velde, G., Sangli, S., and J. Scudder,
              "The BGP Tunnel Encapsulation Attribute", RFC 9012,
              DOI 10.17487/RFC9012, April 2021,
              <https://www.rfc-editor.org/info/rfc9012>.

   [RFC6472]  Kumari, W. and K. Sriram, "Recommendation for Not Using
              AS_SET and AS_CONFED_SET in BGP", BCP 172, RFC 6472,
              DOI 10.17487/RFC6472, December 2011,
              <https://www.rfc-editor.org/info/rfc6472>.

   [RFC4659]  De Clercq, J., Ooms, D., Carugi, M., and F. Le Faucheur,
              "BGP-MPLS IP Virtual Private Network (VPN) Extension for
              IPv6 VPN", RFC 4659, DOI 10.17487/RFC4659, September 2006,
              <https://www.rfc-editor.org/info/rfc4659>.

   [RFC8950]  Litkowski, S., Agrawal, S., Ananthamurthy, K., and K.
              Patel, "Advertising IPv4 Network Layer Reachability
              Information (NLRI) with an IPv6 Next Hop", RFC 8950,
              DOI 10.17487/RFC8950, November 2020,
              <https://www.rfc-editor.org/info/rfc8950>.

   [RFC9014]  Rabadan, J., Ed., Sathappan, S., Henderickx, W., Sajassi,
              A., and J. Drake, "Interconnect Solution for Ethernet VPN
              (EVPN) Overlay Networks", RFC 9014, DOI 10.17487/RFC9014,
              May 2021, <https://www.rfc-editor.org/info/rfc9014>.

Authors' Addresses

   J. Rabadan (editor)
   Nokia
   520 Almanor Avenue
   Sunnyvale, CA  94085
   USA

   Email: jorge.rabadan@nokia.com

Rabadan, et al.          Expires January 7, 2023               [Page 29]
Internet-Draft         EVPN and IPVPN Interworking             July 2022

   A. Sajassi (editor)
   Cisco
   225 West Tasman Drive
   San Jose, CA  95134
   USA

   Email: sajassi@cisco.com

   E. Rosen
   Individual

   Email: erosen52@gmail.com

   J. Drake
   Juniper

   Email: jdrake@juniper.net

   W. Lin
   Juniper

   Email: wlin@juniper.net

   J. Uttaro
   AT&T

   Email: ju1738@att.com

   A. Simpson
   Nokia

   Email: adam.1.simpson@nokia.com

Rabadan, et al.          Expires January 7, 2023               [Page 30]