Skip to main content

EVPN Inter-Domain Option-B Solution
draft-rabadan-bess-evpn-inter-domain-opt-b-00

The information below is for an old version of the document.
Document Type
This is an older version of an Internet-Draft whose latest revision state is "Active".
Authors Jorge Rabadan , Senthil Sathappan , Ali Sajassi
Last updated 2023-03-10
RFC stream (None)
Formats
Stream Stream state (No stream defined)
Consensus boilerplate Unknown
RFC Editor Note (None)
IESG IESG state I-D Exists
Telechat date (None)
Responsible AD (None)
Send notices to (None)
draft-rabadan-bess-evpn-inter-domain-opt-b-00
BESS Workgroup                                           J. Rabadan, Ed.
Internet-Draft                                              S. Sathappan
Intended status: Informational                                     Nokia
Expires: 11 September 2023                                    A. Sajassi
                                                                   Cisco
                                                           10 March 2023

                  EVPN Inter-Domain Option-B Solution
             draft-rabadan-bess-evpn-inter-domain-opt-b-00

Abstract

   An EVPN Inter-Domain interconnect solution is required if two or more
   sites of the same Ethernet Virtual Private Network (EVPN) are
   attached to different IGP domains or Autonomous Systems (AS), and
   they need to communicate.  The Inter-Domain Option-B connectivity
   model is one of the most popular solutions for such EVPN
   connectivity.  While multiple documents refer to this type of
   interconnect solution and specify different aspects of it, there is
   no document that summarizes the impact of the Inter-Domain Option-B
   connectivity in the EVPN procedures.  This document does not specify
   new procedures but analyses the EVPN procedures in an Inter-Domain
   Option-B network and suggests potential solutions for the described
   issues.  Those solutions are based on either other specifications or
   based on local implementations that do not modify the end-to-end EVPN
   control plane.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 11 September 2023.

Rabadan, et al.         Expires 11 September 2023               [Page 1]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

Copyright Notice

   Copyright (c) 2023 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.1.  Terminology and Conventions . . . . . . . . . . . . . . .   4
   2.  EVPN Inter-Domain Option-B General Procedures . . . . . . . .   6
     2.1.  Border Router procedures on EVPN routes . . . . . . . . .   9
       2.1.1.  EVPN Labeled Routes . . . . . . . . . . . . . . . . .  10
       2.1.2.  EVPN Unlabeled Routes . . . . . . . . . . . . . . . .  13
   3.  EVPN Inter-Domain Option-B and Multi-Homing . . . . . . . . .  13
     3.1.  Mass Withdraw . . . . . . . . . . . . . . . . . . . . . .  14
       3.1.1.  The Originating PE Attribute Solution . . . . . . . .  15
       3.1.2.  The RD Administrator Subfield Solution  . . . . . . .  16
       3.1.3.  The EVPN Instance RD Solution . . . . . . . . . . . .  16
     3.2.  Aliasing and Backup Path Procedures . . . . . . . . . . .  17
     3.3.  Designated Forwarder Election and AC-Influenced
           Capability  . . . . . . . . . . . . . . . . . . . . . . .  17
     3.4.  Split Horizon Filtering . . . . . . . . . . . . . . . . .  18
   4.  Inter-Domain Option-B and Load Balancing Procedures . . . . .  19
     4.1.  Flow Label  . . . . . . . . . . . . . . . . . . . . . . .  20
     4.2.  Control Word  . . . . . . . . . . . . . . . . . . . . . .  20
     4.3.  Source UDP port . . . . . . . . . . . . . . . . . . . . .  20
   5.  Inter-Domain Option-B and Layer-2 MTU . . . . . . . . . . . .  21
   6.  E-Tree Considerations . . . . . . . . . . . . . . . . . . . .  21
     6.1.  E-Tree Composite Tunnels  . . . . . . . . . . . . . . . .  21
     6.2.  Egress Filtering of BUM Traffic Originated from a Leaf
           Attachment Circuit  . . . . . . . . . . . . . . . . . . .  21
       6.2.1.  Identication of the PE of Origin  . . . . . . . . . .  23
       6.2.2.  Domain-wide Common Block Leaf Labels  . . . . . . . .  23
       6.2.3.  Source MAC-based Egress Filtering . . . . . . . . . .  23
   7.  Inter-Domain Option-B and PBB-EVPN  . . . . . . . . . . . . .  24
   8.  Security Considerations . . . . . . . . . . . . . . . . . . .  24
   9.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  25
   10. Contributors  . . . . . . . . . . . . . . . . . . . . . . . .  25
   11. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .  25

Rabadan, et al.         Expires 11 September 2023               [Page 2]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   12. References  . . . . . . . . . . . . . . . . . . . . . . . . .  25
     12.1.  Normative References . . . . . . . . . . . . . . . . . .  25
     12.2.  Informative References . . . . . . . . . . . . . . . . .  27
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  28

1.  Introduction

   An EVPN Inter-Domain interconnect solution is required if two or more
   sites of the same Ethernet Virtual Private Network (EVPN)
   [I-D.ietf-bess-rfc7432bis] are attached to different IGP domains or
   Autonomous Systems (AS), and they need to communicate.  In general,
   there are different types of EVPN Inter-Domain models that are
   classified depending on the procedures implemented on the Border
   Routers interconnecting the domains.  The industry typically
   classifies the models into three groups:

   *  EVPN Service Interworking Solution: also referred to as the
      Service Gateway solution, since the Border Routers instantiate
      Virtual Routing and Forwarding tables (MAC-VRFs and/or IP-VRFs)
      and perform a lookup (after decapsulating the transport headers)
      on those tables so that packets are forwarded between domains.
      [RFC9014], [I-D.sr-bess-evpn-vpws-gateway] and
      [I-D.ietf-bess-evpn-ipvpn-interworking] specify the Service
      Gateway solution for EVPN ELAN, VPWS and Layer-3 services,
      respectively.

   *  Inter-Domain Option-B Solution: described in [RFC8365] section 10,
      this solution provides an interconnect solution for EVPN services
      by using Border Routers that re-write the EVPN BGP next hops and
      program a swap operation of the VNIs or MPLS labels (depending on
      whether the encapsulation is NVO-based or MPLS-based).  The
      "Option-B" term refers to the resemblance of this model with the
      Multi-AS "type B" interconnect for IP-VPN in [RFC4364], only that
      this document uses the model for the EVPN family.  This solution
      does not require the instantiation of Virtual Routing and
      Forwarding tables (VRFs) on the Border Routers.

   *  Inter-Domain Transport Solution: refers to any Inter-Domain
      solution that provides connectivity at the transport layer, and
      therefore does not instantiate VRFs or re-write EVPN BGP next hops
      or programs swap operations of the EVPN service identifiers (such
      as VNIs or MPLS service labels) on the Border Routers.  The Inter-
      AS Option-C model described in [RFC4364] section 10 subsection "c"
      (only that the procedures would be used for EVPN routes, as
      opposed to VPN-IPv4 and VPN-IPv6 routes in [RFC4364]) is an
      example of Inter-Domain Transport Solution.

Rabadan, et al.         Expires 11 September 2023               [Page 3]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   The Inter-Domain Option-B connectivity model is one of the most
   popular solutions for Inter-Domain EVPN connectivity, due to the fact
   that it provides isolation for each of the interconnected domains (it
   prevents the need to leak PE loopbacks between domains) while it does
   not require the instantiation of VRFs on the Border Routers.  While
   multiple documents refer to this type of interconnect solution and
   specify different aspects of it, there is no document that summarizes
   the impact of the Inter-Domain Option-B connectivity in the EVPN
   procedures.  This document does not specify new procedures but
   analyses the EVPN procedures in an Inter-Domain Option-B network for:

   *  Multi-Homing

   *  EVPN E-Tree

   *  BUM and IP Multicast forwarding using Ingress Replication or
      Point-to-Multi-Point tunnels

   *  Other EVPN services and including Network Virtualization Overlay
      (NVO) encapsulations or MPLS-based encapsulations

   and provide some guidelines for the described issues.  Those
   guidelines are based on either other specifications or based on local
   implementations that do not modify the end-to-end EVPN control plane.

1.1.  Terminology and Conventions

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

   *  BD: Broadcast Domain.  An EVI may be comprised of one BD (VLAN-
      based or VLAN Bundle services) or multiple BDs (VLAN-aware Bundle
      services).  This document makes use of the term "BD" as described
      in [I-D.ietf-bess-evpn-irb-mcast] section 1.1.4.

   *  BR: Border Router, router that provides connectivity between
      domains, typically an Area Border Router (ABR) or Autonomous
      System Border Router (ASBR).

   *  BUM traffic: Broadcast, Unknown unicast and Multicast traffic.

   *  CE: Customer Edge device, e.g., a host, router, or switch.

   *  EVI: An EVPN instance spanning the Provider Edge (PE) devices
      participating in that EVPN.

Rabadan, et al.         Expires 11 September 2023               [Page 4]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   *  MAC-VRF: A Virtual Routing and Forwarding table for Media Access
      Control (MAC) addresses on a PE.  In VLAN-based or VLAN Bundle
      modes [I-D.ietf-bess-rfc7432bis] a BD is equivalent to a MAC-VRF.

   *  MPLS and non-MPLS NVO tunnels: refer to Multi-Protocol Label
      Switching (or the absence of it) Network Virtualization Overlay
      tunnels.  Network Virtualization Overlay tunnels use an IP
      encapsulation for overlay frames, where the source IP address
      identifies the ingress PE (or ingress Border Router) and the
      destination IP address the egress PE (or egress Border Router).

   *  Ethernet Segment (ES): When a customer site (device or network) is
      connected to one or more PEs via a set of Ethernet links, then
      that set of links is referred to as an 'Ethernet segment'.

   *  Ethernet Segment Identifier (ESI): A unique non-zero identifier
      that identifies an Ethernet segment is called an 'Ethernet Segment
      Identifier'.

   *  IP-VRF: A VPN Routing and Forwarding table for IP routes on an PE.
      In this document, an IP-VRF is an instantiation of a layer 3 EVPN
      service in a PE as per [RFC9135][RFC9136].

   *  IRB: Integrated Routing and Bridging

   *  IRB Interface: Integrated Bridging and Routing Interface.  A
      virtual interface that connects the Bridge Table and the IP-VRF on
      an NVE.

   *  PE: Provider Edge device.  In this document a PE can be a Leaf
      node in a Data Center or a traditional Provider Edge router in an
      MPLS network.

   *  I-PE: Ingress PE.

   *  E-PE: Egress PE.

   *  DF and non-DF: Designated Forwarder and non Designated Forwarder.
      In an Ethernet Segment, the Designated Forwarder PE or Service
      Gateway forwards unicast and BUM traffic.  The non-Designated
      Forwarder PE or Service Gateway blocks BUM traffic (if working in
      All-Active redundancy mode) or unicast and BUM (if working in
      Single-Active redundancy mode).

Rabadan, et al.         Expires 11 September 2023               [Page 5]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   *  Single-Active Redundancy Mode: When only a single PE, among all
      the PEs attached to an Ethernet segment, is allowed to forward
      traffic to/from that Ethernet segment for a given BD, then the
      Ethernet segment is defined to be operating in Single-Active
      redundancy mode.

   *  All-Active Redundancy Mode: When all PEs attached to an Ethernet
      segment are allowed to forward known unicast traffic to/from that
      Ethernet segment for a given BD, then the Ethernet segment is
      defined to be operating in All-Active redundancy mode.

   *  SBD: Supplementary Broadcast Domain, a special BD that has an IRB
      interface to an IP-VRF and it is used in the Optimized Inter-
      Subnet Multicast model, as described in
      [I-D.ietf-bess-evpn-irb-mcast].

   *  VRF: A generic Virtual Routing and Forwarding table, used in this
      document to indicate the instantiation of an EVPN service onto a
      PE.  This service can be any supported EVPN service such as
      layer-2 multipoint services [I-D.ietf-bess-rfc7432bis], EVPN VPWS
      [RFC8214], EVPN E-Tree [RFC8317], PBB-EVPN [RFC7623], or Layer-3
      services as defined in [RFC9135] or [RFC9136].

   *  VPWS: EVPN Virtual Private Wire Service, as in [RFC8214].

   *  VNI: Virtual Network Identifier, a 24-bit label used in EVPN
      services that make use of non-MPLS NVO tunnels [RFC8365].

   *  SR-MPLS SID: Segment Routing MPLS Segment IDentifier.

   *  SRv6 SID: Segment Routing for IPv6 Segment IDentifier.

2.  EVPN Inter-Domain Option-B General Procedures

   The EVPN Inter-Domain Option-B procedures are applied in Border
   Routers that interconnect domains, and the Ingress and Egress PEs
   should be configured and operated in the same way they are when
   communicating with other PEs within their domain.  The typical
   deployments are illustrated in Figure 1 and Figure 2.  Figure 1
   illustrates an Inter-Domain example where each domain is an IGP
   instance.  The Border routers BR-1 and BR-2 show direct BGP EVPN
   neighboring between them, and also with the Ingress PE (I-PE) and the
   Egress PE (E-PE) respectively.  However, Route Reflectors may exist
   in each of the domains.  The procedures described in this document
   remain unchanged irrespective of the presence of Route Reflectors in
   each domain.  Note that in this document VRF is generically used, and
   may mean either MAC-VRF or IP-VRF, unless otherwise specified.

Rabadan, et al.         Expires 11 September 2023               [Page 6]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

         EVPN Route                                     EVPN Route
         Label L22       L22<-L33         L33<-L44      Label L44
         NHop BR-1       NHSelf           NHSelf        NHop E-PE
             <------------+ <--------------+ <-------------+
             +------------+ +--------------+ +-------------+
             |  AS 64500  | |   AS 64500   | |  AS 64500   |
           I-PE           BR-1             BR-2           E-PE
        +-------+       +-------+       +-------+       +-------+
        |+-----+|       |       |       |       |       |+-----+|
   CE1--|| VRF ||       |       |       |       |       || VRF ||-->CE2
        |+-----+|       |       |       |       |       |+-----+|
        +-------+       +-------+       +-------+       +-------+
             |            | |              | |              |
             +------------+ +--------------+ +--------------+
             <--Domain-1--> <---Domain-2---> <---Domain-3--->
           +-------+         +-------+       +-------+
           |Tunnel |         |Tunnel |       |Tunnel |
           +-------+         +-------+       +-------+
           |L22    | --->    |L33    | --->  |L44    | --->
           +-------+         +-------+       +-------+
           |Eth/IP |         |Eth/IP |       |Eth/IP |
           |Payload|         |Payload|       |Payload|
           +-------+         +-------+       +-------+

       Figure 1: EVPN Inter-Domain Option-B scenario for IGP domains

   This document describes also the Inter-Domain Option-B aspects in
   scenarios such as the one portrayed in Figure 2, where the Border
   Routers connect different Autonomous Systems.  As in the case in
   Figure 1 the procedures do not change in case the Domains use Route
   Reflectors.

Rabadan, et al.         Expires 11 September 2023               [Page 7]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

         EVPN Route                                     EVPN Route
         Label L22       L22<-L33         L33<-L44      Label L44
         NHop BR-1       NHSelf           NHSelf        NHop E-PE
             <------------+ <--------------+ <-------------+
             +------------+                  +-------------+
             |  AS 64500  |                  |  AS 64501   |
           I-PE           BR-1             BR-2           E-PE
        +-------+       +-------+       +-------+       +-------+
        |+-----+|       |       |       |       |       |+-----+|
   CE1--|| VRF ||       |       |-------|       |       || VRF ||-->CE2
        |+-----+|       |       |       |       |       |+-----+|
        +-------+       +-------+       +-------+       +-------+
             |            |                  |              |
             +------------+                  +--------------+
             <--Domain-1-->                  <---Domain-3--->
           +-------+                         +-------+
           |Tunnel |                         |Tunnel |
           +-------+         +-------+       +-------+
           |L22    | --->    |L33    | --->  |L44    | --->
           +-------+         +-------+       +-------+
           |Eth/IP |         |Eth/IP |       |Eth/IP |
           |Payload|         |Payload|       |Payload|
           +-------+         +-------+       +-------+

    Figure 2: EVPN Inter-Domain Option-B scenario for Multi-AS Backbones

   In either Figure 1 or Figure 2, this Inter-Domain Option-B solution
   involves the redistribution of EVPN routes from domain to domain by
   the Border Routers.  A Border Router learns all the EVPN routes of
   its own domain, typically via IBGP from the Egress PE or as a client
   from the domain's Route Reflector, and readvertises those routes to
   the neighboring Border Router(s), via EBGP or IBGP.  When
   redistributing EVPN routes to the adjacent Border Routers or Route
   Reflectors within the adjacent domain, the Border Router changes the
   Next Hop IP address to itself, and the EVPN label of the readvertised
   BGP MP_REACH_NLRI message to a new generated label.  In essence, this
   means that the Border Router programs a label swap operation in the
   data path for the EVPN label.  For example, packets received on BR-1
   with EVPN label L22 are looked up and switched to the interface to
   the next domain or Border Router, now with EVPN label L33.  The EVPN
   label in this document can be a 20-bit label (that is, an MPLS label
   or Segment Routing MPLS Segment Identifier) or a 24-bit label (that
   is, a VNI label for non-MPLS NVO tunnels).

   For EVPN routes with 20-bit EVPN labels, in case the Border Router
   receives the EVPN route via IBGP, the route is resolved to a
   transport MPLS or SR-MPLS tunnel that provides reachability to the
   Egress PE or the adjacent Border Router.  The imported EVPN route is

Rabadan, et al.         Expires 11 September 2023               [Page 8]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   considered valid and redistributed only in the case the Next Hop is
   resolved to such a transport tunnel.  In case the Border Router
   receives the EVPN route via single-hop EBGP, the next hop is resolved
   to a local interface associated to the next hop, and packets matching
   the Forwarding Information Base entry for that route are forwarded
   with a single label in the label stack, i.e. the swapped EVPN label.

   In Inter-Domain Option-B scenarios where the transport in the domains
   is based on NVO tunnels, the EVPN routes advertised from the egress
   PEs (and redistributed by the Border Routers) use 20-bit labels (in
   case of MPLS NVO tunnels, e.g., MPLSoGRE) or 24-bit labels (in case
   if non-MPLS NVO tunnels, e.g., VXLAN).  The Border Routers in this
   case not only swap the label (e.g., VNI) for the NVO packets that
   they route, but they change the source and destination IP address of
   the router IP header.  When the Border Router forwards packets into
   an adjacent domain, the outer source IP address of the packets is an
   IP address of the Border Router.  The outer destination IP address is
   given by the next hop of the EVPN route that created the Forwarding
   Information Base entry.

   The key attributes of the solution are that the Border Routers keep
   each domain isolated from each other, e.g.  BR-2 does not leak E-PE's
   loopback into other domains, and the Border Routers do not need to
   have VRFs explicitly configured.  The latter aspect also means that
   the Border Routers need to learn all the EVPN routes within their own
   domain(s) regardless of the Route Targets, as well as readvertise
   those to the adjacent domains, possibly selecting a subset of the
   EVPN routes to be redistributed, via RIB-IN or RIB-OUT policy.  The
   solution does no impose any changes or requirements on the Ingress or
   Egress PEs, or Route Reflectors.  The procedures are solely supported
   on the Border Routers and should be transparent for the Ingress and
   Egress PEs.

   [RFC8365] section 10.2 is the existing specification for Inter-Domain
   Option-B in case EVPN uses encapsulations with 20-bit or 24-bit
   labels, and, in particular for the scenario in Figure 2.  This
   document clarifies that the same procedures and issues apply to the
   scenario in Figure 1.  Although the generic operation of the Border
   Routers on the received EVPN routes is characterized above,
   Section 2.1 clarifies the expected behavior on each EVPN route type.

2.1.  Border Router procedures on EVPN routes

   The Border Router behavior described in Section 2 can be summarized
   in the following tasks performed on each received EVPN BGP UPDATE:

Rabadan, et al.         Expires 11 September 2023               [Page 9]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   *  The Border Router accepts any EVPN route from the Border Routers
      and PEs it is connected to (possibly filtering some of the routes
      via RIB-IN import policies).

   *  Extracts the EVPN label of each EVPN route, either from the NLRI
      (Network Layer Reachability Information) or from an attribute
      included in the BGP UPDATE.

   *  Programs an EVPN label swap operation in the data path, which
      switches the extracted EVPN label to a locally generated new EVPN
      label for the same EVPN route.

   *  Readvertises the EVPN route (assuming the operation is allowed by
      policy) with:

      a.  Next Hop Self, i.e., a new IP address owned by the Border
          Router itself

      b.  The locally generated EVPN label for the route

   However, there are some subtleties with some EVPN route types that
   are important to clarify in order to guarantee interoperability
   across implementations.  We differentiate between EVPN Labeled Routes
   and EVPN Unlabeled Routes.

2.1.1.  EVPN Labeled Routes

   EVPN Labeled Routes are those that carry EVPN Labels or
   demultiplexors in the NLRI or an attribute of the BGP UPDATE.  If
   those EVPN Labels are used in the Forwarding Information Base of the
   Border Router to forward packets between domains, the Label is
   extracted and added to the Forwarding Information Base associated to
   a swap operation.  If those EVPN Labels are not used to forward
   packets between domains, but they indicate certain properties of the
   route, e.g.,: ESI Labels or E-Tree Labels, then the Labels are not
   extracted, programmed or changed when the route is readvertised.  The
   previous statements MUST be applied to existing and future EVPN route
   types in Inter-Domain Option-B networks.  As an example:

   a.  Ethernet Auto-Discovery per Ethernet Segment Route (or route type
       1 per ES)

       Defined in [I-D.ietf-bess-rfc7432bis], this route signals the
       multi-homing mode information, as well as the value of the ESI
       label, encoded in the ESI Label extended community.  It is used
       for fast convergence in case of multi-homed PE failures, via the
       "Mass Withdraw per Ethernet Segment" procedure.  When used with
       an ESI of zero, the route is used to advertised a Leaf Label in

Rabadan, et al.         Expires 11 September 2023              [Page 10]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

       the E-Tree extended community [RFC8317].  The Leaf Label is used
       by the Ingress PE when forwarding BUM traffic generated from a
       Leaf Attachment Circuit.  Both labels, ESI label and Leaf label,
       are not used for packet forwarding at the Border Router and
       therefore the Border Router does not extract them.  The Border
       Router MUST preserve the content of the ESI label or the E-Tree
       extended community when readvertising the route to the adjacent
       domain.  Although the next hop self operation is performed on the
       route by the Border Router, none of the NLRI fields are changed
       when readvertising the route to the adjacent domain.

   b.  Ethernet Auto-Discovery per EVPN Instance Route (or route type 1
       per EVI)

       Defined in [I-D.ietf-bess-rfc7432bis], this route signals the
       forwarding information associated to the local EVPN-VPWS
       Attachment Circuit [RFC8214], and when used with a non-zero ESI,
       it also performs the Aliasing and Backup procedures for multi-
       homing in EVPN services.  The EVPN label encoded in the NLRI of
       this route is used when forwarding packets, hence the label must
       be extracted by the Border Router and programmed in the
       Forwarding Information Base for a swap operation.  Besides the
       next hop self operation and the new valid label to be encoded in
       the route, the Border Router does not change any other field of
       the route.  This includes the content of the EVPN Layer-2
       Attributes extended community advertised with the route.
       [RFC8214] section 4 discusses the Inter-domain Option-B solution
       for EVPN-VPWS.

   c.  MAC/IP Advertisement Route (or route type 2)

       Defined in [I-D.ietf-bess-rfc7432bis], this route advertises
       forwarding information for MAC and IP addresses that are used by
       the Ingress PE to populate the layer-2 Forwarding Information
       Base, the Address Resolution Protocol or Neighbor Discovery
       tables [RFC9161] or even the layer-3 Forwarding Information Base
       [RFC9135].  The route's NLRI contains a mandatory EVPN label,
       Label1, and an optional Label2.  In addition to the next hop self
       operation, a Border Router that receives a route type 2, with
       only Label1, needs to extract Label1 from the NLRI, program its
       value in the Forwarding Information Base, and generate a new
       valid label that is encoded in Label1 when redistributing the
       route to the adjacent domain.  If the received route type 2
       contains a value for both, Label1 and Label2, the Border Router
       needs to program two separate entries in the Forwarding
       Information Base (for the value in Label1 and the value in
       Label2) and generate two valid Label1 and Label2 values.  The
       rest of the information in the route, including EVPN extended

Rabadan, et al.         Expires 11 September 2023              [Page 11]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

       communities and Default Gateway extended community, is preserved
       by the Border Router when readvertising.  This method at the
       Border Router is applied irrespective of the Egress PE using an
       EVPN label per VRF, EVPN label per Ethernet Segment or EVPN label
       per MAC address.  However, using a label per VRF on the Egress
       PEs has the least impact on the Border Routers Forwarding
       Information Base scale, compared to label per MAC or label per
       Ethernet Segment.

   d.  Inclusive Multicast Ethernet Tag Route (or route type 3)

       Also defined in [I-D.ietf-bess-rfc7432bis], this route is used
       for the auto-discovery of the remote PEs attached to the same
       Broadcast domain, as well as the creation of the flooding tree
       used to forward BUM traffic by the PEs attached to the same
       Broadcast Domain.  The route type 3 does not contain any EVPN
       label in its NLRI.  The Provider Tunnel (P-Tunnel) identification
       is carried in the PMSI Tunnel Attribute.  When used for Ingress
       Replication or Assisted Replication tunnel types, the PMSI Tunnel
       Attribute contains an EVPN Label (downstream allocated) that is
       extracted by the Border Router and programmed in the Forwarding
       Information Base in the same way as for the EVPN labels in the
       routes above.  The Border Router generates a valid new label that
       is encoded in the PMSI Tunnel Attribute of the route readvertised
       to the adjacent domain.  In addition to the next hop self and
       label swap operation, the Border Router preserves all the fields
       in the NLRI (including the Originating Router's IP Address) and
       the attributes of the routes (including the Layer 2 Attributes
       extended community).  When the route type 3 uses a P-Tunnel
       different than Ingress Replication, the Border Router should
       carry out the segmentation procedures specified in
       [I-D.ietf-bess-evpn-bum-procedure-updates].

   e.  IP Prefix Route (or route type 5)

       Specified in [RFC9136], this route allows the Egress PEs to
       advertise the IPv4 or IPv6 prefixes that they have learned
       locally in their IP-VRF.  The route's NLRI contains an EVPN label
       that the Option-B Border Router needs to extract and program in
       the Forwarding Information Base, along with a label swap
       operation.  Besides the next hop self and generating a new valid
       EVPN label for the IP Prefix route readvertised to the adjacent
       domain, the Border Router does not change any of the fields in
       the NLRI and preserves all the attributes along with the route,
       including EVPN extended communities.

   f.  Per-Region I-PMSI A-D Route (or route type 9)

Rabadan, et al.         Expires 11 September 2023              [Page 12]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

       Used for P-Tunnel Segmentation on Border Routers, its definition
       and procedures are described in
       [I-D.ietf-bess-evpn-bum-procedure-updates].

   g.  S-PMSI A-D Route (or route type 10)

       Also defined in [I-D.ietf-bess-evpn-bum-procedure-updates], the
       Border Router should follow the same procedures as for the
       Inclusive Multicast Ethernet Tag Route above,

2.1.2.  EVPN Unlabeled Routes

   Examples or EVPN Unlabeled Routes are:

   *  Ethernet Segment Route (or route type 4)

   *  Selective Multicast Ethernet Tag Route (or route type 6)

   *  Multicast Membership Report Synch Route (or route type 7)

   *  Multicast Leave Synch Route (or route type 8)

   *  Leaf Auto-Discovery Route (or route type 11)

   The Border Router receiving these routes simply redistributes the
   routes to the adjacent domain with a next hop of itself, and
   preserving all the attributes that the routes contain.

3.  EVPN Inter-Domain Option-B and Multi-Homing

   This section summarizes the issues of the Inter-Domain Option-B
   associated to EVPN Multi-Homing.  Figure 3 illustrates the use of
   multi-homing in an Inter-Domain Option-B example.

Rabadan, et al.         Expires 11 September 2023              [Page 13]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

         MAC CE2                                        MAC CE2
         RD1                                            RD1 ESI1
         Label L22       NHSelf           NHSelf        Label L44
         NHop BR-1     L22<-L33         L33<-L44        NHop E-PE1
             <------------+ <--------------+ <-------------+
           <--------------+ <--------------+ <-----------+ AD EVI RD1/ESI1
           <--------------+ <--------------+ <-----------+ AD ES  RDx/ESI1

                                                         E-PE1
                                          ES RDa/ESI1 +-------+
                                         <---------+  |+-----+|
                                                  +---|| VRF ||---+
         I-PE3            BR-1            BR-2    |   |+-----+|   |
       +-------+       +-------+       +-------+  |   +-------+   |
       |+-----+|       |       |       |       |--+               |   +---+
CE1 ---|| VRF ||-------|       |-------|       |         E-PE2    +---|CE2|
       |+-----+|       |       |       |       |--+   +-------+   |   +---+
       +-------+       +-------+       +-------+  |   |+-----+|   |
                                                  +---|| VRF ||---+
                                          ES RDb/ESI1 |+-----+|
                                         <----------  +-------+
                        NHSelf           NHSelf
           <--------------+ <--------------+ <-----------+ AD EVI RD2/ESI1
           <--------------+ <--------------+ <-----------+ AD ES  RDy/ESI1

        Figure 3: EVPN Inter-Domain Option-B and multi-homing

   The Border Router rewriting the EVPN multi-homing routes next hop has
   an impact on the EVPN multi-homing procedures that follow:

   *  Mass Withdrawal

   *  Aliasing and Backup Path procedures

   *  Designated Forwarder Election and AC-Influenced Capability

   *  Split Horizon Filtering

3.1.  Mass Withdraw

   The limitations of the mass withdraw procedures when the multi-homed
   egress PEs and the ingress PEs are in different domains are explained
   in [RFC8365] section 10.2.2.

   As a refresher, suppose the example of Figure 3 in which CE2 is
   multi-homed to egress PE1 and PE2 (on Ethernet Segment ES1 with
   identifier ESI1), and the ingress PE3 sits in a different domain.  As
   illustrated, only E-PE1 advertises the MAC/IP route for MAC CE2,

Rabadan, et al.         Expires 11 September 2023              [Page 14]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   whereas both E-PE1 and E-PE2 advertise the A-D per ES and A-D per EVI
   routes for ESI1.  The fact that the Border Routers rewrite the next
   hops of all the routes, prevents I-PE3 from being able to correlate
   the MAC/IP Advertisement route with the A-D per ES route advertised
   from the same E-PE, since the only mechanism in
   [I-D.ietf-bess-rfc7432bis] to correlate A-D per ES and MAC/IP
   Advertisement routes advertised from the same E-PE is the route next-
   hop.  As an example, if the link from CE2 to E-PE1 fails, E-PE1 sends
   a MP_UNREACH_NLRI message for the A-D per ES route and A-D per EVI
   route for ESI1.  The messages get to I-PE3 and are processed,
   however, I-PE3 is unable to correlate the withdrawn A-D per ES route
   with the MAC/IP Advertisement route for CE2 and therefore it does not
   perform any mass withdraw of the MACs associated to ESI1, as long as
   at least one A-D per ES route for ESI1 exists.  Note that the route
   distinguishers of the MAC/IP Advertisement route and A-D per ES route
   advertised from E-PE1 are different, hence the routes cannot be
   associated.

   As also explained in [RFC8365] section 10.2.2, a "mass withdraw per
   EVI" is possible though, due to the fact that the A-D per EVI routes
   and MAC/IP Advertisement routes advertised from the same PE and ES
   can be correlated based on the route distinguisher.  In Figure 3, if
   the link between CE2 and E-PE1 fails, I-PE3 receives the A-D per EVI
   route withdrawal from E-PE1 and can withdraw all the MACs related to
   the MAC/IP Advertisement routes that match the route distinguisher of
   the A-D per EVI route, i.e., RD1 in the example, hence MAC CE2 is
   flushed on I-PE3.  Although the issue is explained for MAC address
   mass withdrawal, the same issue exists with IP Prefixes, as in
   [I-D.sajassi-bess-evpn-ip-aliasing].

   This document assumes that "mass withdraw per EVI" is the default
   behavior that all PEs and Border Routers MUST support.  The following
   subsections also suggest some potential solutions to overcome the
   mass withdraw (per ES) limitation imposed by the Border Routers in
   the Inter-Domain Option-B model.  All of them are based on finding a
   way to correlate the withdrawn A-D per ES route with the routes type
   2 and/or 5 advertised by the same egress PE, so that the
   corresponding MACs or IP Prefixes can be removed.

3.1.1.  The Originating PE Attribute Solution

   A way to solve the mass withdraw limitation imposed by the Border
   Routers (for MACs and IP Prefixes) is documented in
   [I-D.heitz-bess-evpn-option-b], which defines a transitive attribute
   called Originating PE (OPE) that removes the ambiguity to find the
   identity of the originator of the routes.  When the egress PE
   advertises the OPE attribute along with the A-D per ES routes and
   MAC/IP Advertisement or IP Prefix routes, the ingress PE is able to

Rabadan, et al.         Expires 11 September 2023              [Page 15]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   correlate the routes that are originally advertised from the same
   egress PE based on the same OPE value received on AD per ES and MAC/
   IP Advertisement (or IP Prefix) routes.  The use of OPE provides a
   solution to support mass withdrawal per ES in Inter-Domain Option-B
   networks.

3.1.2.  The RD Administrator Subfield Solution

   An alternative solution is also hinted by
   [I-D.heitz-bess-evpn-option-b] section 9.2, where the routes type 2
   and 5 can be correlated with the A-D per ES routes from the same PE
   based on the Administrator subfield of the route distinguisers (RDs).
   That is, in Figure 3, suppose E-PE1 advertises the A-D per ES route
   with route distinguisher RDx = <RD1:0> and the MAC/IP Advertisement
   route with <RD1:1>, with "RD1" being the Administrator subfield of
   the route distinguisher.  E-PE2 allocates "RD2" as Administrator
   subfield for A-D per ES and MAC/IP Advertisement routes.  Now, in
   case of a withdraw of the A-D per ES route from E-PE1, I-PE3 can
   perform a mass withdraw operation based on the assumption that all
   the MACs from the MAC/IP Advertisement routes with RD1 as
   Administrator subfield are advertised from the same E-PE1 that failed
   and withdrew the A-D per ES route.  The same solution is valid for
   the mass withdraw of IP Prefix routes.

3.1.3.  The EVPN Instance RD Solution

   This document suggests a third solution based on the E-PEs using the
   same route distinguisher on A-D per ES routes and routes type 2 or 5.
   The A-D per ES routes are normally advertised per <ES, EVI-set>,
   where an EVI-set is a group of EVPN Instances, each one represented
   by a different route target in the route.  Because of this, the A-D
   per ES route cannot use the route distinguisher of an existing VRF in
   the PE, but a unique route distinguisher not assigned to any EVPN
   Instance (instantiated in a VRF).  However, suppose each EVI-set is
   composed of a single EVI, hence the A-D per ES routes are advertised
   per <ES, EVI> and therefore there is a separate A-D per ES route per
   EVPN Instance (or VRF).  If that is the case, now the A-D per ES
   routes can use the route distinguisher assigned to the EVPN Instance
   (or VRF), which is the same one used by the routes type 2 or 5 for
   the EVI.  Since A-D per ES routes are - with this solution -
   advertised per <ES, EVI>, this is really a "mass withdraw per EVI"
   solution, similar to the one described in Section 3.1 in terms of
   efficiency.  However, the advantage of this solution is that the A-D
   per ES routes are REQUIRED, while A-D per EVI routes are OPTIONAL
   [I-D.ietf-bess-rfc7432bis] and may not be used in the EVI.

Rabadan, et al.         Expires 11 September 2023              [Page 16]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

3.2.  Aliasing and Backup Path Procedures

   The Aliasing and Backup Path procedures work in an Inter-Domain
   Option-B solution as per [RFC8365], section 10.2.  That is, since
   EVPN MAC/IP Advertisement routes and A-D per EVI routes are both
   advertised on a per Broadcast Domain basis and they use the same
   route distinguisher and route target, the receiving ingress PE can
   associate them together to determine the BGP paths available for the
   MAC (multiple aliasing paths in case of all-active mode, or one
   active and one backup in case of single-active mode).  Different
   paths can still be created without ambiguity even if they all go
   through the same Border Router.

   Although the Aliasing and Backup Path procedures per se are not
   affected, note that the ingress PE installs the MAC from an EVPN MAC/
   IP Advertisement route (with non-reserved ESI), only if the
   associated set of Ethernet A-D per ES routes are received from the
   same egress PE ( [I-D.ietf-bess-rfc7432bis], section 9.2.2).  Due to
   the same issues described in Section 3.1, the ingress PE cannot
   determine if the received MAC/IP Advertisement route and the received
   set of Ethernet A-D per ES routes are coming from the same egress PE.
   This document suggests two approaches to solve this resolution issue:

   1.  Use a "loose" resolution for the MAC/IP Advertisement route -
       that is, the ingress PE considers the MAC/IP Advertisement route
       (with a non-reserved ESI) resolved if (and only if) at least one
       Ethernet A-D per ES route has been received with the same ESI and
       same next hop as the MAC/IP Advertisement route (it is assumed
       that its route target set contains the route target of the MAC/IP
       Advertisement route).

   2.  Use any of the approaches in Section 3.1 to correlate MAC/IP
       Advertisement routes and A-D per ES routes, and then resolve the
       MAC/IP Advertisement route as in ( [I-D.ietf-bess-rfc7432bis].

3.3.  Designated Forwarder Election and AC-Influenced Capability

   On an all-active Ethernet Segment, the Designated Forwarder is the PE
   router responsible for sending Broadcast, Unknown Unicast, and
   Multicast (BUM) traffic to a multihomed Customer Edge (CE) device, in
   the <ES, Ethernet Tag> for which the PE is elected.  If the Ethernet
   Segment works in single-active mode or port-active mode, the
   Designated Forwarder is the PE router that sends all traffic to a
   multihomed CE [RFC8584].  When a CE is multihomed to two or more PEs
   sitting in different domains, the Designated Forwarder candidate list
   is still created normally.  The Designated Forwarder Election is
   unaffected by the Border Routers next hop self operation on the ES
   routes.  This is due to the fact that the candidate list is created

Rabadan, et al.         Expires 11 September 2023              [Page 17]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   out of the Originating Router's IP Address of the ES routes (which is
   not changed by the Border Routers) as opposed to the ES route next
   hops [RFC8584].  However, the Attachment Circuit Influenced
   Designated Forwarder (AC-Influenced DF Election) capability [RFC8584]
   is affected by the next hop self operation of the Border Routers.

   If the AC-Influenced DF Election capability is enabled on all the PEs
   attached to the Ethernet Segment, the Designated Forwarder candidate
   list needs to be pruned based on the presence of the A-D per ES and
   A-D per EVI routes for a given candidate.  That is, even if E-PE1's
   ES route is received Figure 3, E-PE2 cannot add E-PE1 to the
   Designated Forwader candidate list for <ES1, BD1> until the valid A-D
   per ES and A-D per EVI routes (for ES1 and BD1) are received and
   identified as originated from E-PE1.  However, because BR-2 changes
   the next hop of the A-D routes, E-PE2 cannot rely on the next hop to
   identify the routes as coming from E-PE1.  This issue is similar to
   the one discussed in Section 3.1 for mass withdraw, only that the PE
   now needs to correlate the ES route and A-D per ES/EVI routes coming
   from the same PE of origin.

   This document assumes that, in case the PEs attached to the same
   Ethernet Segment are located in different domains, the operator may
   choose one of the following alternatives:

   *  Disable the AC-Influenced Designated Forwarder capability in the
      PEs attached to the Ethernet Segment, or

   *  Enable the AC-Influenced Designated Forwarder capability in all
      the PEs attached to the Ethernet Segment, and correlate the
      received A-D per ES/EVI routes with their corresponding
      Originating Router's IP Address based on any of the three
      procedures of Section 3.1.

3.4.  Split Horizon Filtering

   The Split Horizon Filtering is a fundamental part of the EVPN multi-
   homing procedures to avoid BUM looped frames to go back to the multi-
   homed CE.  As described in [I-D.ietf-bess-evpn-mh-split-horizon]
   there are two Split Horizon Filtering Types: ESI label based and
   Local Bias.  Which one is applied depends on the transport tunnel
   being used by the EVPN BUM packets, and some transport tunnels may
   support both mechanisms.  If two or more PEs of the same Ethernet
   Segment are sitting in different domains, the procedures in the
   Border Router may have an impact on the Split Horizon Filtering
   mechanisms.  In particular:

   1.  If the multi-homed PEs use an ESI label based Split Horizon
       Filtering Type:

Rabadan, et al.         Expires 11 September 2023              [Page 18]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

       a.  Regardless of the PEs using upstream or downstream allocated
           ESI labels (for P2MP/MP2MP or Ingress Replication,
           respectively), the PEs in the Ethernet Segment need to
           correlate the identity of the PE advertising the ESI label
           with the Inclusive Multicast Ethernet Tag routes advertised
           by the same PE.  This brings us back to the same issue of
           identifying the origin of the A-D per ES route described in
           Section 3.1, only that this time the receiving PE needs to
           correlate A-D per ES routes with routes type 3, as opposed to
           types 2 or 5.  In this case, any of the solutions in
           Section 3.1 could be used.

       b.  The use of ESI labels allocated from a Domain-wide Common
           Block (DCB) and the same label used by all the PEs attached
           to the same Ethernet Segment may simplify the procedures.  If
           that is the case, the ingress PE can program the received ESI
           label without the need to correlate the received A-D per ES
           routes with the Inclusive Multicast Ethernet Tag routes.

       c.  In addition, the Border Routers need to preserve the ESI
           label when they route packets between domains.

   2.  If the multi-homed PEs use Local Bias as the Split Horizon
       Filtering Type:

       a.  The Border Router cannot change the outer source IP address
           of the IP tunnel, so that the egress PE can still identify
           the source PE.  Note this may not be possible in many
           implementations.

   The above considerations may influence Inter-Domain Option-B designs,
   so the capabilities of the Border Routers and PEs have to be analized
   before the operator deploys CEs that are multi-homed to PEs located
   in different domains.

4.  Inter-Domain Option-B and Load Balancing Procedures

   This section will cover the impact of Inter-Domain Option-B Border
   Router procedures in load balancing related mechanisms such as Flow
   Label or Control Word for MPLS tunnels (see
   [I-D.ietf-bess-rfc7432bis] section 18), or the source UDP port for
   NVO tunnels that is used for provide entropy when load balancing
   traffic on the core routers.  VXLAN [RFC7348] is an example of NVO
   tunnel type that uses the source UDP port to provide entropy.

Rabadan, et al.         Expires 11 September 2023              [Page 19]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

4.1.  Flow Label

   The use of Flow Label and its signaling is described in
   [I-D.ietf-bess-rfc7432bis] section 18.1.  The ingress PE pushes the
   Flow Label only on EVPN-encapsulated known unicast packets forwarded
   to egress PEs that previously advertised their Flow Label support on
   Inclusive Multicast Ethernet Tag routes with the F-bit set.  When
   programming the data path for a given MAC, the ingress PE needs
   therefore to program the use of Flow Label if the MAC/IP
   Advertisement route came from the same PE that advertised an
   Inclusive Multicast Ethernet Tag route with F-bit set.  The ingress
   PE correlates both, MAC/IP Advertisement route and Inclusive
   Multicast Ethernet Tag route based on the matching route
   distinguisher of the two.

   The Flow Label MUST be preserved by the Border Routers receiving
   EVPN-encapsulated packets containing a Flow Label, so that the EVPN
   packets for the same flow are forwarded following the same path
   within each domain.

4.2.  Control Word

   The signaling of the Control Word in the Inclusive Multicast Ethernet
   Tag routes (C-bit) is described in [I-D.ietf-bess-rfc7432bis] section
   7.11.  As in the case described in Section 4.2, when a Border Router
   rewrites the next hops of the MAC/IP Advertisement and Inclusive
   Multicast Ethernet Tag routes, the ingress PE needs to identify the
   egress PE based on the matching route distinguisher of the two
   routes.  Also, if included in the received EVPN-encapsulated packets,
   the Control Word MUST be preserved by the Border Routers so that no
   packet reordering happens for flows forwarded into an adjacent
   domain.

4.3.  Source UDP port

   If ingress and egress PEs use NVO tunnels [RFC8365], i.e., IP
   tunnels, the ingress PE typically encodes a per-flow hash value into
   the the outer tunnel source UDP port of the EVPN-encapsulated
   packets.  Examples of tunnel types that use the outer source UDP port
   as an entropy field are VXLAN, GENEVE, or MPLSoUDP.  The Border
   Routers between the ingress and egress PEs MUST preserve the value of
   the source UDP port so that EVPN-encapsulated packets for the same
   flow are forwarded following the same path within each domain.

Rabadan, et al.         Expires 11 September 2023              [Page 20]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

5.  Inter-Domain Option-B and Layer-2 MTU

   In the same way the support for Flow Label or Control Word is
   signaled, the egress PE's supported layer-2 MTU (Maximum Transfer
   Unit) is indicated in the Layer-2 MTU field of the EVPN Layer-2
   Attributes extended community advertised along with the Inclusive
   Multicast Ethernet Tag route ([I-D.ietf-bess-rfc7432bis], section
   7.11.1).  The Border Router(s) between ingress an egress PEs do not
   modify any of the advertised attributes, and therefore the layer-2
   MTU value is propagated end to end up to the ingress PE.  In general,
   the layer-2 MTU configured in all PEs attached to the same EVPN
   service SHOULD match, irrespective of the domain where they reside.

6.  E-Tree Considerations

   [RFC8317], or Ethernet-Tree in EVPN networks, describes two areas
   that are impacted by the presence of an Inter-Domain Option-B Border
   Router between ingress and egress PEs: the use of composite tunnels
   for BUM traffic and the egress PE filtering of BUM traffic originated
   from a Leaf Attachement Circuit.

6.1.  E-Tree Composite Tunnels

   A composite tunnel is tunnel type used by the Root PE to
   simultaneously indicate a P2MP tunnel in the transmit direction and
   an Ingress Replication tunnel in the receive direction for BUM
   traffic.  For this reason, an Inclusive Multicast Ethernet Tag route
   for a composite tunnel comprises both, a downstream allocated EVPN
   label for Ingress replication, and a P2MP tunnel identifier.  The
   EVPN label is extracted by the Border Router and programmed in the
   Forwarding Information Base, as described in Section 2.1.1 bullet
   "d".  Since the Ingress Replication procedures are followed, the
   Border Router generates a valid new label that is encoded in the
   (composite type) PMSI Tunnel Attribute of the route readvertised to
   the adjacent domain.  Also, as described in Section 2.1.1, the
   segmentation procedures in [I-D.ietf-bess-evpn-bum-procedure-updates]
   are followed for the encoded P2MP tunnel in the same PMSI Tunnel
   Attribute.

6.2.  Egress Filtering of BUM Traffic Originated from a Leaf Attachment
      Circuit

   E-Tree in EVPN networks requires the filtering of traffic originated
   from a Leaf Attachment Circuit.  While the ingress PE can determine
   if known unicast leaf traffic can be forwarded, based on whether the
   destination MAC address belongs to a leaf Attachment Circuit,
   filtering of the BUM traffic must be done at the egress PE.  For such
   filtering, the egress PE advertises a Leaf Label along with an

Rabadan, et al.         Expires 11 September 2023              [Page 21]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   Ethernet A-D per ES route (with ESI of zero), and the egress PE
   relies on the ingress PE to push that Leaf Label when sending Leaf
   BUM traffic to it [RFC8317].  If ingress and egress PEs are located
   in different domains of an Inter-Domain Option-B network, the ingress
   PE cannot correlate the received Inclusive Multicast Ethernet Tag
   route and A-D per ES route (comprising the Leaf Label) from the same
   egress PE.  Due to this issue when identifying the egress PE's Leaf
   Label, the ingress PE cannot push the Leaf Label below the EVPN
   multicast label for a given egress PE.  The issue is illustrated in
   Figure 4.

                             NHSelf      NHSelf
IMET orig-ip E-PE1          L55<-L33    L33<-L11        IMET orig-ip E-PE1
RD1 Label 55 NHop BR-1 <-------+ <---------+ <--------+ RD1 Label 11 NHop PE1

A-D per ES(Leaf-Lbl 1)<-------+ <---------+ <--------+ A-D per ES(Leaf-Lbl 1)
RDx/ESI0 NHop BR-1           NHSelf      NHSelf        RDx/ESI0 NHop PE1
                                                         E-PE1
                                                       +-------+
                                                       |+-----+|
                                                   +---|| VRF ||---CE1(Root)
                  I-PE3        BR-1        BR-2    |   |+-----+|
                +-------+   +-------+   +-------+  |   +-------+
                |+-----+|   |       |   |       |--+
    CE1(Leaf)---|| VRF ||---|       |---|       |         E-PE2
                |+-----+|   |       |   |       |--+   +-------+
                +-------+   +-------+   +-------+  |   |+-----+|---CE21(Leaf)
                                                   +---|| VRF ||
                                                       |+-----+|---CE22(Root)
                             NHSelf      NHSelf        +-------+
IMET orig-ip E-PE2          L66<-L44    L44<-L22        IMET orig-ip E-PE2
RD2 Label 66 NHop BR-1 <------+ <----------+ <--------+ RD2 Label 22 NHop PE2

A-D per ES(Leaf-Lbl 2)<------+ <----------+ <--------+ A-D per ES(Leaf-Lbl 2)
RDy/ESI0 NHop BR-1         NHSelf       NHSelf         RDy/ESI0 NHop E-PE2

     Figure 4: EVPN Inter-Domain Option-B and Leaf BUM filtering

Rabadan, et al.         Expires 11 September 2023              [Page 22]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   Suppose the egress PEs and ingress PE are in a different domain
   Figure 4, and that I-PE3 needs to forward EVPN-encapsulated BUM
   traffic from Leaf CE1, using Ingress Replication.  I-PE3 receives
   Inclusive Multicast Ethernet Tag routes and A-D per ES routes from
   the two egress PEs, however, I-PE3 is unable to identify what Leaf
   Label needs to push when sending EVPN-encapsulated BUM traffic to
   E-PE1 or E-PE2.  This is due to the fact that the A-D per ES routes
   cannot longer be associated with their corresponding Inclusive
   Multicast routes based on the next hop, since the four routes in the
   example are received from the same next hop.  This section suggests
   different solutions, as follows.

6.2.1.  Identication of the PE of Origin

   A way to solve the issue with E-Tree and the egress filtering of Leaf
   BUM traffic is to identify and correlate the Inclusive Multicast
   Ethernet Tag routes and A-D per ES routes (with ESI of zero)
   originated from the same egress PE.  In order to do that, any of the
   three techniques in Section 3.1 are valid, only that the
   identification is now done so that Inclusive Multicast Ethernet Tag
   routes and A-D per ES routes can be correlated, instead of MAC/IP
   Advertisement routes and A-D per ES routes.

6.2.2.  Domain-wide Common Block Leaf Labels

   The use of Leaf Labels allocated from a Domain-wide Common Block
   (DCB) and the same Leaf label value used by all the PEs attached to
   the E-Tree EVPN service simplify the procedures.  If that is the
   case, all the egress PEs advertise the same Leaf label in their A-D
   per ES routes for ESI of zero, and that Label value matches the local
   Leaf label on the ingress PE.  The ingress PE can then program the
   allocated Leaf label for all the destination egress PEs, without
   correlating the received Inclusive Multicast and A-D per ES routes.
   This assumes all the PEs in the Broadcast Domain allocate the same
   Leaf label.  If the ingress PE detects any inconsistency in the
   signaled Leaf label, that is, if at least one PE of the Broadcast
   Domain advertises a different label than the local Leaf label, then
   the ingress PE SHOULD NOT program the Leaf label when sending traffic
   to the egress PEs.

6.2.3.  Source MAC-based Egress Filtering

   Another potential solution is the use of source MAC-based egress
   filtering, as opposed to Leaf label-based egress filtering for EVPN-
   encapsulated BUM traffic.  If the ingress PE receives two or more A-D
   per ES routes (with ESI of zero) with the same next hop, then it does
   not program any of the received Leaf labels and forwards EVPN-
   encapsulated BUM packets with the EVPN label and without any Leaf

Rabadan, et al.         Expires 11 September 2023              [Page 23]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   label.  If we assume that the ingress PE has previously advertised
   the local Leaf MAC addresses, when the BUM packets get to the egress
   PE, a source MAC lookup in the MAC-VRF will determine if the BUM
   packet is coming from a Leaf or a Root Attachment Circuit.

   Taking the example of Figure 4, I-PE3 advertises CE1's MAC as a Leaf
   MAC in a route type 2, and hence CE1's MAC is programmed in E-PE1 and
   E-PE2 as Leaf.  Since I-PE3 receives two A-D per ES routes (with ESI
   of zero) from the same next hop, I-PE3 determines that it cannot
   program the received Leaf labels, and therefore I-PE3 forwards BUM
   packets from CE1 to E-PE1 and E-PE2 with their corresponding
   Inclusive Multicast labels and without any Leaf label.  When the
   packets get to the egress PEs, E-PE1 and E-PE2 perform a source MAC
   lookup in the MAC-VRF.  Since CE1's MAC appear as a Leaf MAC, E-PE1
   and E-PE2 can filter appropriately.  That is, e.g., E-PE2 forwards to
   CE22 (root) only and not to CE21 (leaf).

7.  Inter-Domain Option-B and PBB-EVPN

   Provider Backbone Bridging EVPN [RFC7623] is also supported in Inter-
   Domain Option-B.  The following considerations apply:

   *  PBB-EVPN does not have any of the issues described in Section 3.
      This is due to the fact that PBB-EVPN multi-homing procedures do
      not rely on Ethernet A-D per ES or per EVI routes at all.

   *  PBB-EVPN does not have any of the issues described in Section 6
      either, for the same reason.  For E-Tree egress filtering of the
      EVPN-encapsulated BUM packets (so that they are only forwarded to
      local Root Attachement Circuits and not Leaf Attachment Circuits),
      PBB-EVPN relies on the source B-MAC identification at the egress
      PE.  The procedures are not impacted by the presence of a Border
      Router between ingress and egress PEs.

   *  Also, this document assumes that the [I-D.ietf-bess-rfc7432bis]
      procedures to signal Flow Label, Control Word or Layer-2 MTU, do
      not apply to PBB-EVPN networks, hence there are no issues derived
      from those components.

8.  Security Considerations

   This document is intended to be published as Informational and hence
   does not impose and procedures that introduce any new security risks.
   The described solutions are based on existing specifications and
   therefore this document inherits the security considerations
   described in each of the normative reference documents.

Rabadan, et al.         Expires 11 September 2023              [Page 24]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

9.  IANA Considerations

   No IANA actions.

10.  Contributors

11.  Acknowledgments

12.  References

12.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC9135]  Sajassi, A., Salam, S., Thoria, S., Drake, J., and J.
              Rabadan, "Integrated Routing and Bridging in Ethernet VPN
              (EVPN)", RFC 9135, DOI 10.17487/RFC9135, October 2021,
              <https://www.rfc-editor.org/info/rfc9135>.

   [RFC9136]  Rabadan, J., Ed., Henderickx, W., Drake, J., Lin, W., and
              A. Sajassi, "IP Prefix Advertisement in Ethernet VPN
              (EVPN)", RFC 9136, DOI 10.17487/RFC9136, October 2021,
              <https://www.rfc-editor.org/info/rfc9136>.

   [RFC8365]  Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
              Uttaro, J., and W. Henderickx, "A Network Virtualization
              Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
              DOI 10.17487/RFC8365, March 2018,
              <https://www.rfc-editor.org/info/rfc8365>.

   [RFC4364]  Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
              Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
              2006, <https://www.rfc-editor.org/info/rfc4364>.

Rabadan, et al.         Expires 11 September 2023              [Page 25]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   [I-D.ietf-bess-rfc7432bis]
              Sajassi, A., Burdet, L. A., Drake, J., and J. Rabadan,
              "BGP MPLS-Based Ethernet VPN", Work in Progress, Internet-
              Draft, draft-ietf-bess-rfc7432bis-06, 5 January 2023,
              <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
              rfc7432bis-06>.

   [RFC9014]  Rabadan, J., Ed., Sathappan, S., Henderickx, W., Sajassi,
              A., and J. Drake, "Interconnect Solution for Ethernet VPN
              (EVPN) Overlay Networks", RFC 9014, DOI 10.17487/RFC9014,
              May 2021, <https://www.rfc-editor.org/info/rfc9014>.

   [RFC8214]  Boutros, S., Sajassi, A., Salam, S., Drake, J., and J.
              Rabadan, "Virtual Private Wire Service Support in Ethernet
              VPN", RFC 8214, DOI 10.17487/RFC8214, August 2017,
              <https://www.rfc-editor.org/info/rfc8214>.

   [RFC8317]  Sajassi, A., Ed., Salam, S., Drake, J., Uttaro, J.,
              Boutros, S., and J. Rabadan, "Ethernet-Tree (E-Tree)
              Support in Ethernet VPN (EVPN) and Provider Backbone
              Bridging EVPN (PBB-EVPN)", RFC 8317, DOI 10.17487/RFC8317,
              January 2018, <https://www.rfc-editor.org/info/rfc8317>.

   [RFC7623]  Sajassi, A., Ed., Salam, S., Bitar, N., Isaac, A., and W.
              Henderickx, "Provider Backbone Bridging Combined with
              Ethernet VPN (PBB-EVPN)", RFC 7623, DOI 10.17487/RFC7623,
              September 2015, <https://www.rfc-editor.org/info/rfc7623>.

   [RFC8584]  Rabadan, J., Ed., Mohanty, S., Ed., Sajassi, A., Drake,
              J., Nagaraj, K., and S. Sathappan, "Framework for Ethernet
              VPN Designated Forwarder Election Extensibility",
              RFC 8584, DOI 10.17487/RFC8584, April 2019,
              <https://www.rfc-editor.org/info/rfc8584>.

   [I-D.ietf-bess-evpn-irb-mcast]
              Lin, W., Zhang, Z. J., Drake, J., Rosen, E. C., Rabadan,
              J., and A. Sajassi, "EVPN Optimized Inter-Subnet Multicast
              (OISM) Forwarding", Work in Progress, Internet-Draft,
              draft-ietf-bess-evpn-irb-mcast-09, 21 February 2023,
              <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
              evpn-irb-mcast-09>.

Rabadan, et al.         Expires 11 September 2023              [Page 26]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   [I-D.ietf-bess-evpn-ipvpn-interworking]
              Rabadan, J., Sajassi, A., Rosen, E. C., Drake, J., Lin,
              W., Uttaro, J., and A. Simpson, "EVPN Interworking with
              IPVPN", Work in Progress, Internet-Draft, draft-ietf-bess-
              evpn-ipvpn-interworking-07, 6 July 2022,
              <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
              evpn-ipvpn-interworking-07>.

12.2.  Informative References

   [RFC9161]  Rabadan, J., Ed., Sathappan, S., Nagaraj, K., Hankins, G.,
              and T. King, "Operational Aspects of Proxy ARP/ND in
              Ethernet Virtual Private Networks", RFC 9161,
              DOI 10.17487/RFC9161, January 2022,
              <https://www.rfc-editor.org/info/rfc9161>.

   [RFC7348]  Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
              L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
              eXtensible Local Area Network (VXLAN): A Framework for
              Overlaying Virtualized Layer 2 Networks over Layer 3
              Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
              <https://www.rfc-editor.org/info/rfc7348>.

   [I-D.ietf-bess-evpn-bum-procedure-updates]
              Zhang, Z. J., Lin, W., Rabadan, J., Patel, K., and A.
              Sajassi, "Updates on EVPN BUM Procedures", Work in
              Progress, Internet-Draft, draft-ietf-bess-evpn-bum-
              procedure-updates-14, 18 November 2021,
              <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
              evpn-bum-procedure-updates-14>.

   [I-D.heitz-bess-evpn-option-b]
              Heitz, J., Sajassi, A., Drake, J., and J. Rabadan, "Multi-
              homing and E-Tree in EVPN with Inter-AS Option B", Work in
              Progress, Internet-Draft, draft-heitz-bess-evpn-option-
              b-01, 13 November 2017,
              <https://datatracker.ietf.org/doc/html/draft-heitz-bess-
              evpn-option-b-01>.

   [I-D.sr-bess-evpn-vpws-gateway]
              Rabadan, J., Sathappan, S., Prabhu, V., Lin, W., and P.
              Brissette, "Ethernet VPN Virtual Private Wire Services
              Gateway Solution", Work in Progress, Internet-Draft,
              draft-sr-bess-evpn-vpws-gateway-01, 10 October 2022,
              <https://datatracker.ietf.org/doc/html/draft-sr-bess-evpn-
              vpws-gateway-01>.

Rabadan, et al.         Expires 11 September 2023              [Page 27]
Internet-Draft         EVPN Inter-Domain Option-B             March 2023

   [I-D.sajassi-bess-evpn-ip-aliasing]
              Sajassi, A., Badoni, G., Warade, P., Pasupula, S.,
              Krattiger, L., Drake, J., and J. Rabadan, "EVPN Support
              for L3 Fast Convergence and Aliasing/Backup Path", Work in
              Progress, Internet-Draft, draft-sajassi-bess-evpn-ip-
              aliasing-06, 10 January 2023,
              <https://datatracker.ietf.org/doc/html/draft-sajassi-bess-
              evpn-ip-aliasing-06>.

   [I-D.ietf-bess-evpn-mh-split-horizon]
              Rabadan, J., Nagaraj, K., Lin, W., and A. Sajassi, "EVPN
              Multi-Homing Extensions for Split Horizon Filtering", Work
              in Progress, Internet-Draft, draft-ietf-bess-evpn-mh-
              split-horizon-04, 10 January 2023,
              <https://datatracker.ietf.org/doc/html/draft-ietf-bess-
              evpn-mh-split-horizon-04>.

Authors' Addresses

   Jorge Rabadan (editor)
   Nokia
   520 Almanor Avenue
   Sunnyvale, CA 94085
   United States of America
   Email: jorge.rabadan@nokia.com

   Senthil Sathappan
   Nokia
   520 Almanor Avenue
   Sunnyvale, CA 94085
   United States of America
   Email: senthil.sathappan@nokia.com

   Ali Sajassi
   Cisco
   Email: sajassi@cisco.com

Rabadan, et al.         Expires 11 September 2023              [Page 28]