IPv6 Operations (v6ops) Working Group                       N. Buraglio
Internet Draft                                  Energy Sciences Network
Intended status: Informational                                 K. Frank
Expires: July 2024                                              P. Nero
                                                                Private
                                                             P. Volpato
                                                           E. Vasilenko
                                                    Huawei Technologies

                                                       January 29, 2024



                  IPv6 Site connection to many Carriers
                  draft-fbnvv-v6ops-site-multihoming-03


Abstract

   Carrier resilience is a typical business requirement. IPv4
   deployments have traditionally solved this challenge through private
   internal site addressing in combination with separate NAT engines
   attached to multiple redundant carriers. IPv6 brings support for
   true end-to-end connectivity on the Internet, and hence NAT is the
   least desirable option in such deployments. Native IPv6 solutions
   for carrier resiliency, however, have drawbacks. This document
   discusses all currently-available options to organize carrier
   resiliency for a site, their strengths and weaknesses, and provides
   a history of past IETF efforts approaching the issue. The views
   presented here are the summary of discussions on the v6ops mailing
   list and are not just the personal opinion of the authors.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other documents
   at any time.  It is inappropriate to use Internet-Drafts as
   reference material or to cite them other than as "work in progress."

   This Internet-Draft will expire on July 2024.




FBNVV                   Expires July 29, 2024                  [Page 1]


Internet-Draft             site multihoming                January 2024


Copyright Notice

   Copyright (c) 2024 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with
   respect to this document. Code Components extracted from this
   document must include Simplified BSD License text as described in
   Section 4.e of the Trust Legal Provisions and are provided without
   warranty as described in the Simplified BSD License.



Table of Contents


   1. Terminology and pre-requisite..................................3
   2. Problem statement..............................................4
      2.1. General issues with network translation...................7
   3. Problem history for the host-driven solution...................7
   4. Solution requirements.........................................10
   5. Available Solutions...........................................12
      5.1. PI-based.................................................12
      5.2. PA-based solution........................................15
         5.2.1. Bind() case on the application side.................16
         5.2.2. Getaddrinfo() case on the application side..........17
         5.2.3. PA-based solution conclusion........................18
      5.3. Shifting the problem to the centralized site.............20
      5.4. ULA with NPTv6...........................................23
      5.5. ULA with NAT66...........................................26
      5.6. Application proxy........................................28
   6. Conclusion....................................................32
   7. Security Considerations.......................................35
   8. IANA Considerations...........................................35
   9. References....................................................35
      9.1. Normative References.....................................35
      9.2. Informative References...................................37
   Appendix A.......................................................39
   Acknowledgments..................................................40






FBNVV                   Expires July 29, 2024                  [Page 2]


Internet-Draft             site multihoming                January 2024


1. Terminology and pre-requisite

   Terminology is inherited from [ND] and [SLAAC]. Additional terms:

   RIR (Regional Internet Registry):  an organization that manages
      Internet numbering resources (such as IP addresses and autonomous
      system (AS) numbers) within a geographical region of the world.

   LIR (Local Internet Registry):  an organization (usually a carrier
      or an Enterprise/Academic institution) that receives its
      allocation of IP addresses from the Regional Internet Registry
      (RIR), then assigns parts of that allocation to its customers.

   PA (provider-assigned or provider-aggregatable) address space: a
      block of IP addresses assigned to the end customer by a carrier
      or other owner that received it inside a bigger block from an RIR
      or LIR. The principal characteristic of the PA address block is
      that it is aggregated in the bigger block (not announced
      separately) in the Internet routing tables.

   PI (provider-independent) address space:  a block of IP addresses
      assigned by RIR or LIR to the end customer with the possibility
      for independent announcements in the Internet routing tables.

   DHCP-PD:  IPv6 Prefix delegation by DHCP from the carrier to the
      client to be used for numbering throughout the client site (As
      currently defined by [DHCPv6] Section 6.3).

   End-to-end connectivity: The possibility to initiate the connection
      from any direction, including the case of complex protocols with
      many logically related sessions.

   ALG (Application-level gateway or Application-layer gateway): a
      component that monitors the data payload of connections to
      discover and fix application-level referrals (embedded IP
      addresses).

   Subscriber-only services:  The resource that is filtered to the
      public Internet and available only for some portion of the
      Internet, typically only for subscribers of the particular
      carrier.

   Multi-homing solution: The set of architectural, configurational and
      operational network and application changes necessary to be
      applied to a single-homed site plan in order to let all hosts in
      the site gain access to at least two paths towards the Internet,



FBNVV                   Expires July 29, 2024                  [Page 3]


Internet-Draft             site multihoming                January 2024


      effectively solving the Problem Statement, satisfying as many
      Solution Requirements as possible, and adding as little
      complexity as possible compared to a single-homed site.

2. Problem statement

   Always-on connectivity is a key requirement for the vast majority of
   businesses. [IAB report] predicts that there "might be as many as 10
   million multihomed sites by 2050". Unfortunately, several issues may
   affect the connection of a business to its upstream service
   provider. For example, the carrier's network, the network gateway,
   or the first-mile infrastructure may experience issues. It is
   especially true now after many recent examples of massive carrier
   outages.

   A redundant connection to the carrier is then the norm for business.

   A simple topology that showcases this key requirement is shown in
   Figure 1. Note that the topology could be more complex as shown, for
   example, in Figure 3 [MHMP Enterprise].

                                  +------+      _________

                                  |      |     /         \

                              +---| CPE1 |----/  Carrier  \

         2001:db8:0:1001::xx  |   |      |\   \     1     /

                  +------+    |   +------+ \   \_________/

                  |      |    |             \

                  | Host |----+              \

                  |      |    |               \

                  +------+    |   +------+     \_________

         2001:db8:0:2001::yy  |   |      |     /         \

                              +---| CPE2 |----/  Carrier  \

                                  |      |    \     2     /

                                  +------+     \_________/



FBNVV                   Expires July 29, 2024                  [Page 4]


Internet-Draft             site multihoming                January 2024




      Figure 1: The basic Carrier Resiliency topology

   Without entering too much into details, resilience is generally
   achieved by employing redundant elements. Two Customer Premises
   Equipment (CPE) systems are usually employed. Very often, each CPE
   connects the business to a different carrier. In some cases, a CPE
   may even connect to two different carriers, to achieve a higher
   level of protection against network failures.

   It is very desirable to have paths with diversity. Unfortunately, it
   is not always possible for the enterprise to understand what exactly
   is shared between paths offered by different carriers. The duct the
   fibers run through may even be shared in many cases, with the risk
   of both strands being damaged at the same time. Let's assume for the
   discussion in this document that the paths available for the site to
   reach the Internet are sufficiently diverse.

   In literature, a topology such as that shown in Figure 1 based on
   IPv6 connectivity is often referred to as Multi-Home, Multi-Prefix
   (MHMP). The name implies that a network is multi-homed to different
   carriers, receiving from them different network prefixes.

   It is important to mention that the topology in Figure 1 or all
   Figures of [MHMP Enterprise] assume redundant Carriers' connection
   to the router upstream. Direct Carrier connection to the host (for
   example 3GPP modem) is not in the scope of this document that has
   "site connection" in the name.

   The problem of providing network protection with IPv6 was thoroughly
   discussed in [Local Protection]. This document is an overview of the
   most commonly used methods to facilitate the desired protection in
   modern IPv6 networks, along with a discussion of the advantages and
   disadvantages of each method.

   In IPv4 environments, such a scenario is implemented through
   independent NAT translation on every CPE to the carrier in
   combination with private address space on-site. [Local Protection]
   has a good list of benefits. The solution was initially adopted due
   to the shortage of globally-reachable address space. Later, security
   and carrier resiliency were identified as additional benefits. Due
   to the prevalence of a solution based around address translation in
   IPv4, demands are often voiced for such a mechanism in IPv6 as well,
   running the risk of being chosen by network designers without
   evaluating alternative options.



FBNVV                   Expires July 29, 2024                  [Page 5]


Internet-Draft             site multihoming                January 2024


   [Local Protection] has a list of IPv6 tools that replace all
   functionality of the NAT solution except address conservation, which
   is not a necessity in IPv6. Time has shown that all IPv6 tools have
   been accepted as valid replacements except IPv6 "multi-homing and
   renumbering". Discussions about this last problem are ongoing - see
   [Multi-Homing] [MHMP Enterprise] [MHMP] [flash renumbering].

   This document considers 6 solutions with current operational
   deployment (more may be added in future versions of the draft):

     1. Static PI address space for the site, routed by multiple
        carriers. This method allows the allocation of Provider
        Independent (PI) addresses on-site while routing announcements
        are propagated by carriers on behalf of the client. It will be
        discussed in section 5.1.
     2. Dynamic PA addresses distribution and withdrawal from carriers.
        An [IPv6] host gets different Provider Aggregatable (PA)
        addresses for its interfaces, possibly from different carriers.
        It is the host's responsibility to properly choose the
        combination of a source address and the relevant next hop. This
        solution is primarily based on the interaction of [ND] and
        [SASA] and will be discussed in section 5.2.
     3. Shifting Internet access resilience to a central site. A branch
        site is granted redundant connectivity to a central hub
        location where the aspects related to resilient Internet
        connectivity are handled. The methods is discussed in section
        5.3.
     4. Static ULA address space for the site with NPTv6 translation.
        [NPTv6] is adopted in combination with Unique Local Addresses
        (ULA). This method is discussed in section 5.4.
     5. Static ULA address space for the site with NAT66 translation.
        In this case NAT66 is combined with Unique Local Addresses
        (ULA). The solution is discussed in section 5.5.
     6. Relying on an application proxy to handle path decisions. In
        this case, internal hosts are only required to send traffic to
        a central device in the site, which then independently selects
        which carrier to use to reach the Internet. This method is
        discussed in section 5.6.

   The IETF consensus to preserve end-to-end connectivity does not
   favor the general acceptance of solutions 3 and 4, yet they are
   reported in this document because they offer a degree of support for
   the requirements listed in section 4.






FBNVV                   Expires July 29, 2024                  [Page 6]


Internet-Draft             site multihoming                January 2024


2.1. General issues with network translation

   Part of the architectures described in this document rely on
   different forms of translation of the source IP address of internal
   hosts to offer redundant carrier connectivity. It is important to
   point out that, while these options have been included due to having
   a degree of real-world deployment, they incur a large number of
   drawbacks unrelated to carrier redundancy which have been
   extensively documented.

   From standards as well as practical perspectives, the [IAB request]
   to avoid the use of address and prefix translation has been heavily
   based on the long list of [NAT implications]. Some or all of the
   problems caused by network translation apply to the solutions in
   sections 5.4 and 5.5 breaking of the end-to-end model, the
   impossibility of initiating sessions from the outside, breaking
   IPSec encryption, breaking of application-layer referrals to the
   addresses, single point of failure, redundancy challenges (state
   replication), and performance challenges (stateful processing).

   The current document and its authors make no endorsement for the use
   of either NPTv6 or NAT66; rather, the intent is to illustrate and
   summarize operational alternatives currently in use when true route-
   based multi-homing, as exemplified by the use of PI in section 5.1,
   is not available.

   Network administrators looking to implement multi-homing as
   described in sections 5.4 or 5.5 are encouraged to consider [IAB
   request] and [NAT implications] before making a final decision.

3. Problem history for the host-driven solution

   Client applications typically utilize getaddrinfo() to establish
   communication and to perform source address selection. It was
   initially assumed in [SASA] and implemented by getaddrinfo() that
   the next hop is chosen before the source address.

   Bind() permits overriding this order, but it is typically used only
   in server-side applications.

   This specific process is the reason why in the case of a network
   fault, when a redundant CPE/link is promoted to the primary role,
   some specific destinations may become unreachable, causing the
   solutions listed hereafter to leave some unresolved scenarios.





FBNVV                   Expires July 29, 2024                  [Page 7]


Internet-Draft             site multihoming                January 2024


   The initial discussion on carrier resiliency occurred in the
   [multi6] and [Shim6] working groups. [multi6] requirements were
   reused in this document. [shim6] proposed to separate the location
   and identity properties of addresses. [Shim6] did not gain market
   acceptance. It is not currently supported by available software and
   hardware options.

   The host-based solution is highly dependent on the availability of
   the information that the prefix is stale. [6renum] has a deep
   discussion on how to best react in cases where there is missing
   information, including the use of other mechanisms to automatically
   resolve the problem. The best way to provide warnings about stale
   prefixes is still in discussion [flash renumbering].

   The next notable step was the addition of rule 5.5 to [SASA] to
   prefer the selection of a source address covered by a prefix
   advertised by the chosen next hop router. This allows the packet to
   be on the path to the PA-owning carrier and avoid packet filtering
   due to the application of [BCP38].

   Concerning Figure 1, if connectivity to Carrier 1 were lost then the
   host would select a source address in the prefix belonging to
   Carrier 2 to communicate with a certain destination outside of its
   local segment, thus achieving resiliency.

   On the other hand, this method is not a solution when only a
   particular source address is permitted (i.e., not filtered) to
   access a particular outside resource (e.g., "subscriber-only
   services") or when any type of deterministic traffic distribution
   policy is desired, because the random next hop choice would in turn
   lead to a random choice of source address.

   [MHMP] then discussed the problem thoroughly, attempting to use only
   already available tools. It was suggested that the solution must
   push related policies to the host as 1) "Routing Information
   Options" of [Route Preferences] and 2) [Policy by DHCP] to modify
   policies in the host's [SASA] selection algorithm. This solution has
   not gained market acceptance due to complexity reasons. Moreover,
   DHCPv6 is not universally implemented, being notably absent from
   some of the most widely deployed client platforms.

   Additionally, [HNCP] prescribes deprecating delegated prefixes (by
   setting their preferred lifetime to zero) when the router has
   information about loss of reachability to the carrier that sourced
   the prefix. This is particularly important when renumbering occurs




FBNVV                   Expires July 29, 2024                  [Page 8]


Internet-Draft             site multihoming                January 2024


   (the PA prefix may change after disconnection and re-connection to
   the carrier).

   The next solution was proposed in [Multi-Homing]. The document
   incorrectly assumed (errata 7009 and 7010 are published) that the
   source address is chosen first in the typical scenario when a client
   initiates outbound communications. [Multi-Homing] section 3.2
   proposed to prefer next hops from those routers that advertise the
   prefix covering the already selected source address. Hence, [Multi-
   Homing] unblocked the possibility for an application to use bind()
   to select the source address first since section 3.2 contains vital
   instructions on how to choose the next hop in that condition. It is
   important to note that the more common scenario of choosing the next
   hop before the source address is not solved by [Multi-Homing].

   Further progress in the problem discussion was made in [MHMP
   Enterprise], which discusses potentially complex on-site topologies
   and the source routing that is needed in such a scenario; it may be
   considered a comprehensive guide covering the source routing aspect
   of the complex site with carrier resiliency. Unfortunately, at this
   time it is not yet possible to use it in practice due to the lack of
   any market-accepted solutions to split and distribute PA prefixes
   throughout the complex site. All other solutions (PI, ULA+NAT66,
   ULA+NPTv6) do not require source routing. [MHMP Enterprise] might
   become vital in the future if a solution were to be adopted for
   splitting and propagating PA prefixes through the complex site;
   however, such a solution is currently unavailable.

   Restrictions to the list of applicable source addresses for a
   specific next hop (rule 5 or 5.5 of [SASA]) may not have been
   implemented in certain host operating systems. [Conditional PIO] can
   in this case mitigate the problem through selective PIO
   announcements to a particular host. This represents a valuable
   transition mechanism until rule 5.5 of [SASA] will be universally
   applied.

   [DNS Options] allows the router to supply the addresses of many
   different DNS resolvers, including those of completely unrelated
   carriers. It is not possible, however, to provide information to
   clients regarding which DNS resolver is related to which particular
   prefix; such information might be crucial in scenarios where traffic
   steering policies are required for successful communication
   (including when accessing filtered resources such as "subscriber-
   only services").





FBNVV                   Expires July 29, 2024                  [Page 9]


Internet-Draft             site multihoming                January 2024


   Finally, [Provisioning domains] permit virtualization of the router
   on the link, representing one physical device as many logical ones
   with fully separate sets of link parameters. This solution is
   valuable in some scenarios to deliver more diverse information to
   the host but does little to assist it with choosing the proper
   combination of next hop and source address that is still restricted
   by [SASA]. The challenge remains the same independent of how many
   physical or logical routers are present on the link. Moreover,
   virtualization of a single router on a link having two uplinks to
   different carriers creates a problem, because the host could
   randomly choose the wrong combination of source address and next hop
   announced by one of the virtual routers.

   Yet, [Provisioning domains] remains valuable in scenarios where
   several different routers are behind a single router, as well as
   when multiple physical routers are present on the same link (i.e.,
   the problem of host choice already exists). This is because, in
   contrast to [DNS Options], [Provisioning domains] retains the
   information that associates prefixes with the DNS resolvers of their
   respective carrier.

4. Solution requirements

   Solution Requirements are reflecting section 3.1 of [multi6] except
   for "Transport-Layer Survivability" (preservation of the session
   during the carrier disruption) which is not considered mandatory.

     1. Site resiliency to an arbitrary number of carriers, with an
        arbitrary number of routers on the link.
     2. End-to-end connectivity wherever possible.
     3. Possibility for internal communication using any prefixes
        distributed by local routers, irrespective of the status of the
        connectivity to the carriers that distributed such prefixes.
     4. The speed of convergence for the prefix deprecation on the site
        after connectivity is lost to any particular carrier should be
        comparable to the speed of routing convergence on the site. If
        flash renumbering affects the convergence of a particular
        solution then such a solution should have mechanisms to
        guarantee the desired level of convergence performance despite
        flash renumbering.
     5. Support for sites with complex topologies, including multiple
        internal on-site hops requiring many routers and links.



FBNVV                   Expires July 29, 2024                 [Page 10]


Internet-Draft             site multihoming                January 2024


     6. Access to resources from the carrier's "subscriber-only
        services" is permitted only when using the address space
        distributed by the particular carrier. There may exist a need
        for the host to check DNS resolvers from all carriers before it
        can even discover the restricted resource. A given host may
        thus need to choose the correct source address that would be
        accepted by the particular carrier.
     7. Possibility for traffic steering between different paths
        (including both internal to the site, and the Internet) based
        on bandwidth, cost, load, latency, packet loss, hop count, etc.
        (e.g. application traffic/path engineering) for both outbound
        and inbound packets.
   It is out of scope for this document to evaluate issues related to
   perimeter security; every system should assume an insecure
   environment, which is already the case since the host is
   establishing frequent connections to the open Internet.

   Different environments have diverse security policies, needs, and
   obligations that may be shaped by internal policy, regulatory
   compliance, or national security requirements, and as such will not
   be discussed in detail in this document.

   Privacy is furthermore assumed to be protected by [Temporary
   addressing] and is also kept outside of carrier resiliency design
   considerations. Address-based privacy considerations are not
   affected by carrier resiliency mechanisms and techniques.

   All solutions to the problem statement in this document would have
   different cost advantages and disadvantages. The associated costs
   may greatly vary for different geographies, market segments, and
   organization sizes, but the choices should be ultimately driven by
   any needs and requirements set upon the organization by their
   respective governing bodies, with input from the appropriate subject
   matter experts.

   It is recognized that cost is often a determining factor in IT and
   networking decisions, but it is also considered out of the scope of
   this document.








FBNVV                   Expires July 29, 2024                 [Page 11]


Internet-Draft             site multihoming                January 2024


5. Available Solutions

5.1. PI-based

   While building or expanding an IPv6 addressing plan, it may become
   necessary for an organization to procure or acquire additional
   network resources; it is reasonably straightforward to obtain
   additional Provider-Independent GUA address space and ASN from an
   RIR or LIR. However, not all organizations have the in-house
   expertise, desire, or capability to execute such a plan.

   PI prefixes allow the creation of an address plan that would never
   need renumbering, due to the non-dynamic and non-conditional nature
   of the prefix allocation. Hence, the address plan may be stable
   enough to be manually provisioned over all routers and links even in
   a complex topology.

   The long-term supportability and network convergence time of this
   solution are excellent because there is no need for renumbering;
   losing one of the Internet connectivities simply implies normal
   routing updates with default routing status withdrawal by the
   affected router.

   It may, however, be a more involved process to get the PI prefix
   routed by the carrier because such type of customer attachment is
   typically charged more by the carrier due to the more complicated
   nature of the connection and configuration (i.e. dynamic protocol
   configuration, configuration of appropriate prefix filtering, and
   equipment to support the necessary protocols). It generally requires
   a manual procedure on the carrier's side, and hence it is not
   possible to set this up without their cooperation.

   For faster convergence, routers should announce a "default" on-site
   and then add some more specific prefixes if needed. Additional
   destination prefixes facilitate the distribution of traffic between
   the carriers.

   In cases where internal routing hops are present between the edge
   devices, such announcements likely require other protocols in
   addition to Neighbor Discovery.

   As for the network interface of the internal host (the last hop),
   such an announcement comes in the form of default router status in
   [ND], and [Route Preferences] for the more specific routes.





FBNVV                   Expires July 29, 2024                 [Page 12]


Internet-Draft             site multihoming                January 2024


   The crucial requirement in the PI-based solution is that routers
   have to continuously track connectivity to carriers to be able to
   deprecate the "default" as well as any more specific announcements
   when such connectivity is lost.

   When multi-homing with PI space there are multiple entry points into
   the local network for traffic destined to every single address in
   the site, and this opens up discussion on how to load balance
   incoming traffic to such a site. An example can split the prefix
   into many smaller prefixes and announce them separately to different
   carriers. This requires that the initial prefix is of adequate size,
   to avoid ending up with announcements smaller than the longest
   acceptable prefix length within the default-free zone, which is
   conventionally /48. This advanced incoming traffic engineering is
   possible only with PI address space.

   Where traffic symmetry over the WAN is important, however, such as
   in environments performing application-based traffic steering, PI
   can fail to guarantee the necessary control over returning traffic.
   Traffic engineering using routing advertisements provides benefits
   for simple active/passive or active/active connectivity needs, but
   it can also be insufficient; routing advertisements are not granular
   or fast enough to make things work in scenarios where different
   application traffic must be steered towards different uplinks based
   on upper-layer information. As an example, an administrator might
   need FTP traffic generated by an internal host to always exit
   through the secondary Internet link, and for the return traffic to
   also arrive back on that same link; allowing the downstream
   component of FTP connections to occupy bandwidth on the primary link
   might be undesirable, and impact the performance of more sensitive
   protocols running over the primary link. Even in the extreme case
   where /128 prefixes for the internal host were advertised,
   engineering different return paths based on upper-layer information
   would not be feasible in a deterministic manner for the same source
   address.

   In addition, environments, where the edge devices (commonly,
   perimeter firewalls) are stateful might experience packet drops due
   to TCP session flows being split across Internet links. This would
   be especially true in the case where multiple stateful devices were
   deployed, one for each Internet link; the devices would thus only
   see a portion of the TCP sessions and never consider them fully
   established, or miss enough segments that the next arriving sequence
   number might be considered out of the window from their standpoint.
   Both conditions could cause the devices to drop segments. Note that
   these symptoms would be exaggerated for TCP traffic but also



FBNVV                   Expires July 29, 2024                 [Page 13]


Internet-Draft             site multihoming                January 2024


   possible with other protocols such as UDP, if the stateful devices
   had a way to track the sessions and allowed inbound UDP traffic only
   when a matching tuple already existed in their session tables.

   On the other hand, a crucial advantage of using a prefix that is
   globally reachable through multiple Internet links is the seamless
   failover of transport-layer sessions across the links without having
   to re-establish them. Conventionally, in IPv4 environments, such a
   failover would have been guaranteed to break existing communication
   due to the ubiquity of PA addressing combined with NAT44, and the
   consequent change in source address as seen by the remote endpoint.
   This issue would also be felt when leveraging IPv6 multi-homing in a
   way that caused source address changes upon failover, either due to
   NAT (see sections 5.4 and 5.5) or the prefix of the existing
   connection becoming unusable (see section 5.2).

   The biggest problem with PI addresses today remains the fact that
   the widespread practice of using PI space would bloat the Internet
   routing table by at least 10 times (reaching up to 10M entries in
   the default-free zone), which would greatly impact the cost and
   scalability of Internet routers. Admittedly, however, this problem
   would not be an immediate concern for any enterprise company
   implementing a PI-based solution.

   The implementation of RPKI (for prefix ownership protection) is also
   among the additional challenges that enterprises making use of PI
   may face in the future, requiring skilled personnel to the setup and
   operation of BGP.

   Advantages:

   - Preserves end-to-end communication,
   - Does not require any special functionality from the host,
   - Straightforward design,
   - Allows aggregation of multiple uplinks for increased throughput,
   - Supports sites with complex topology,
   - Seamless link failover without transport session re-establishment,
   - Supports outbound traffic steering.
   Disadvantages:
   - Hard to implement especially for smaller entities, as it requires
     knowledge of advanced routing protocols like BGP and availability
     of advanced networking hardware (it is not as simple as plugging
     two independent CPEs into the same switch as the client and having
     it work); moreover, skills related to RPKI may be needed in the
     future,




FBNVV                   Expires July 29, 2024                 [Page 14]


Internet-Draft             site multihoming                January 2024


   - In case of provider failure a prefix may not be correctly
     deprecated from global routing, and this could lead to complex
     error scenarios and downtime,
   - The need to pay and liaise with a RIR or LIR for the PI address
     space,
   - Needs carriers to accept PI prefix advertisements,
   - Carriers typically charge significantly more for such type of
     attachment; often SD-WAN contracts are required instead of
     business DSL contracts,
   - The return traffic path is not guaranteed to mirror the outbound
     path,
   - Bloats the Internet routing table with a potential 10 million new
     routes.

5.2. PA-based solution

   Let us assume that a site is connected to the Internet through many
   carriers, with every carrier delegating a PA prefix. How the prefix
   is delivered to the site is unimportant for the current example. It
   may be dynamic (most likely using DHCP-PD) or static (offline
   agreement with carrier).

   We will also assume for this example that the local CPE performs no
   proprietary functions and that hosts are provisioned with an address
   from each PA allocation.

   As soon as a router becomes aware that the path to the carrier that
   delegated a prefix is down, all more specific prefixes under that
   prefix must be deprecated (preferred lifetime set to zero) in all
   subnets. This is specified as requirement L-13 of [HNCP] but is not
   currently requested in other standards. Support for the [HNCP]
   protocol itself is not mandatory.

   The PA-based solution is the most affected by renumbering, a highly
   probable occurrence since 1/3 of carriers dynamically rotate
   prefixes leased by DHCP-PD after any re-connection. Measures
   specified in section 4.1 of [6renum] (i.e. usage of FQDN, DHCP+DNS,
   ULA, DHCP-PD, Parameterized Address Configuration, transition
   period, etc) alleviate the negative effects of renumbering.

   [Node Requirements] does not make DHCP mandatory for address
   assignment. Some popular client operating systems do not support
   DHCP, even though they "should", according to [Node Requirements].
   Hence, for the general case, we should assume that address
   assignment and configuration are done with [SLAAC]. Note that the
   logic below would not change if DHCP-IA [DHCPv6] were to be used for



FBNVV                   Expires July 29, 2024                 [Page 15]


Internet-Draft             site multihoming                January 2024


   address assignment. A similar statement applies to the address
   acquisition by DHCP-PD [DHCPv6], [PDperDevice], the size of the
   address pool received by hosts does not help to resolve the problem
   of source address and next hop choice, [SASA] restrictions remain
   the same.

   As a result, we have a host with many IP addresses that needs to
   choose the source address and the next hop before any communication
   attempts it may wish to make. This is because, if the source address
   of one carrier were to be used to send a packet to a different
   carrier, then the other carrier would filter it out in suspicion of
   an address spoofing attack.

5.2.1. Bind() case on the application side

   Potentially, the application on the host may use bind() to choose
   the source address first using any logic pre-programmed into the
   application. This application-driven solution is the only current
   way to access the "subscriber-only services" of the particular
   carrier and for steering traffic based on cost, latency, packet
   loss, hop count, etc.

   After the application selects the proper source address with bind(),
   it must choose the next hop to serve it.

   There are two options to accomplish this:

     1. [Multi-Homing] section 3.2 has an augmentation to [ND] section
        6.3.6 asking to prefer default routers that advertised the
        particular prefix already used for the source address.
     2. [Route Preferences] to install the external prefix into the
        host's routing table.

   It may be valuable to implement [Provisioning domains] to supply
   along with DNS resolvers the relationship of such resolvers to each
   prefix provided by the different carriers, as unrelated resolvers
   may respond with unusable or missing information when queried for
   "subscriber-only services". In any case, IP stacks of host OSes are,
   in practice, not capable of accepting and using this additional
   information - so it would not play any role in the decision-making
   process.

   The implementer should check that the functionality mentioned in
   this paragraph is supported on routers and applications before
   relying on it in designs. Unfortunately, [Provisioning domains] and
   [Multi-Homing] have low acceptance on the market, and hence the



FBNVV                   Expires July 29, 2024                 [Page 16]


Internet-Draft             site multihoming                January 2024


   application-driven solution for carrier resiliency has a low
   probability of having been implemented, and at best would be
   inconsistent.

5.2.2. Getaddrinfo() case on the application side

   Let us now consider the more typical case in which the application
   is simplified in its networking aspects. In this case, the
   application would use getaddrinfo(), which is typically compliant
   with [SASA] on all OSes. This solution is referred to as "host-
   driven" within this section.

   For historical reasons, getaddrinfo() selects the next hop first. By
   default, the next hop is chosen in a random round-robin manner
   between all available routers on the particular link (under [ND]
   section 6.3.6). If the routers and host support the "Default Router
   Preference" or "Route Preference" fields of [Route Preferences] in
   the RA header or the "Routing Information Option", priority can be
   assigned to certain default routers over others using a simple High,
   Medium, Low value; this functionality, however, is not universally
   implemented.

   Rule 5.5 of [SASA] section 5 will then prefer, in the list of
   available source addresses, those inside prefixes that are already
   advertised by the chosen next hop. Hence, the random next hop leads
   to a random source address choice among those available.

   Such behavior may block the possibility of accessing the
   "subscriber-only services" of a particular carrier (traffic would be
   filtered due to having a source address belonging to a different
   carrier) as well as prevent traffic steering by any sort of policy.

   Moreover, even if [Provisioning domains] were to be implemented, the
   recommendation for random next hop choice would prevent effective
   use of it.

   Simple Internet connectivity with carrier resiliency could be
   achieved in this manner, as carrier resiliency would work on a basic
   level albeit with unpredictable traffic distribution between the
   carriers. All that is needed for this is to support rule 5.5 of
   [SASA] section 5.

   Importantly, in addition to supporting rule 5.5 of [SASA], the
   router should also support the L-13 requirement of [HNCP] to
   deprecate prefixes individually and not the default router status




FBNVV                   Expires July 29, 2024                 [Page 17]


Internet-Draft             site multihoming                January 2024


   itself. This is the only way for convergence to take place
   effectively in the case of connectivity loss to the PA carrier.

   [MHMP] discusses that it is possible to achieve traffic steering
   supplying policies by two mechanisms:

     1. "Routing Information Options" of [Route Preferences] to
        influence next hop choice
     2. [Policy by DHCP] to modify policies in the [SASA] source
        address selection algorithm.

   The latter method is considered impractical because of its
   complexity and general lack of DHCPv6 support in many commonly
   deployed operating systems.

5.2.3. PA-based solution conclusion

   One of the drawbacks of the PA-based solution is that it can fail to
   meet the requirements for real-time traffic steering based on
   measured link quality and upper-layer information. While PA-based
   traffic steering using the methods described above can be sufficient
   in some environments, it must be kept in mind that the decision of
   which link to exit from is ultimately left to the host in the form
   of source address selection. In environments performing application-
   based traffic steering, it is instead crucial that the steering
   decision be made by the edge device, due to it having a policy table
   based on network, transport, and application-layer aspects of
   packets as well as real-time link quality metrics. This policy
   represents the intent of the administrator and, ultimately, the
   business requirements of the organization that owns the network; it
   may for various reasons not discussed in this document have been
   deemed unfeasible to replicate and apply this policy table directly
   at the host level inside the clients. In the PA-based solution, the
   host has access to prefixes from all available Internet links and
   can assign itself routable addresses from them, and is thus free to
   ignore any policy configured on the edge device. The gateway is
   forced to forward the packet in a way that honors the source address
   chosen by the host, lest the packet be dropped upstream due to
   implementations of [BCP38]. Once the traffic leaves the network
   through the correct uplink, however, the return path is guaranteed
   to be symmetric due to the address selected by the host being routed
   only to one of the Internet links; this could be considered to
   satisfy requirement 7 of section 4 with regards to the inbound
   portion of traffic steering.





FBNVV                   Expires July 29, 2024                 [Page 18]


Internet-Draft             site multihoming                January 2024


   The host stack (in [MPTCP] case) or the host application (in
   [MPQUIC] case) may resolve the MHMP problem in the PA environment by
   choosing the source address first (using bind()) as discussed in
   section 5.2.1. [MPTCP] socket interface extensions explain in
   sections 5.3.1 or 5.3.3 how to bind many local addresses to the
   future or current sessions. [MPQUIC] does not have a clear
   discussion on binding to local addresses, but the [MPQUIC] generally
   transfers the majority of tasks to the application layer where
   possible.

   Finally, in the case where a site has many links and routers
   (complex topology) then source routing or other connection tracking
   mechanisms in the internal network are mandatory to deliver the
   traffic to the carrier owning the source address of the packet. This
   aspect is properly discussed in [MHMP Enterprise], albeit it is not
   yet of any use in the PA-based scenario because the delegated PA
   prefix should since the beginning be dynamically split into smaller
   prefixes and propagated to all links throughout the site, and there
   is currently no accepted and available method to do this. Such a
   method should also assist with prefix deprecation in the case where
   connectivity to the carrier is lost. This is not supported at the
   moment because neither [HNCP] nor DHCP-PD have gained acceptance for
   this purpose. There is thus a chance that if this problem is
   addressed then [MHMP Enterprise] might become very important for
   sites with complex topologies.

   Advantages:

   - No need to own and operate a registered, Provider-Independent
     address space,
   - Preserves end-to-end communication,
   - In simple networks, it may be as easy as plugging all CPE routers
     and client devices into the same L2 domain (i.e. Switch).

   Disadvantages:

   - Scalability issues, easy for simple networks, but exponentially
     more difficult in complex networks,
   - Prefixes may not get deprecated when the CPE itself fails, as
     opposed to just the link,
   - Not all issues are resolved yet, only the simplest scenario is
     possible (simple topology and unpredictable traffic distribution),
   - Carriers may frequently change the prefix (flash renumbering), and
     this could disrupt communication,
   - Sites with complex topologies are not well supported yet,




FBNVV                   Expires July 29, 2024                 [Page 19]


Internet-Draft             site multihoming                January 2024


   - Traffic steering by any policies (including the capability to
     access "subscriber-only services") is not supported yet.

5.3. Shifting the problem to the centralized site

   Another possible approach is shifting the Internet access resiliency
   problem to a central site when the branch has a redundant, private
   WAN connectivity provisioned through any or multiple available
   methods (SDH/PDH/OTN links, MPLS VPNs, customer-managed overlays).
   In this scenario, the backhaul towards the central site (which then
   performs the ultimate handoff to the Internet) happens over
   physically or virtually dedicated links, and the actual addressing
   solution offered by the carrier serving the branch becomes
   irrelevant; in the extreme case the branch carrier does not provide
   an IP service at all (as in the case of optical transport or a
   layer-2 VPN) or the provided addressing is only used to establish
   tunnels (in the case of overlays). It is not uncommon for the branch
   edge device to not even have a default route configured towards any
   of the carrier next hops, instead configuring a default route
   through the overlay or having as next hop a loopback interface
   address reachable through the overlay itself.

   The reasoning behind this choice is based on several factors and
   commonly involves one or more of the following assumptions:

     1. Branch site Multi-Homing is mainly a matter of first-mile
        redundancy due to the increased difficulty of providing stable
        connectivity to remote sites compared to a large central site.
        Suppose the traffic can be made to traverse the first mile in
        an optimal environment (because the entire path is under the
        network administrator's control, at least at the network
        level). In that case, the relatively high-quality Internet
        circuits found at the central site can be managed using more
        traditional and resource-intensive techniques (for example, by
        significantly increasing capacity and carrier diversity, tuning
        routing advertisements, and using ECMP).
     2. Obtaining enterprise-class connectivity, where the customer has
        the option to announce their own address space dynamically to
        the carriers, can be complex and cannot thus be done for every
        site.
     3. Managing a geographically-distributed Internet breakout may
        pose greater operational, financial, and security-related
        challenges when the proper orchestration tools are not
        employed.





FBNVV                   Expires July 29, 2024                 [Page 20]


Internet-Draft             site multihoming                January 2024


   It is typically much easier to arrange the resiliency over internal
   WAN links. This is primarily because the addressing structure and
   the path selection are under the control of a single entity
   throughout the LAN and the private WAN. This can allow, for example,
   to number the entire internetwork using a single type of addressing,
   recreating the benefits of the solution in section 5.1 while
   avoiding some of its disadvantages, namely:

   - "In case of provider failure, a prefix may not be correctly
     deprecated": in the tunnel-based solution, all network devices are
     under the control of the same entity experiencing the hypothetical
     issue, simplifying troubleshooting and speeding up resolution.
   - "Need carriers to accept PI prefix advertisements": in the tunnel-
     based solution, this needs to be done only once, at the central
     site, instead of at every site implementing the solution.
   - "Carriers typically charge significantly more for such type of
     attachment": see the previous observation.
   - "Return traffic path not guaranteed to mirror the outbound path":
     path symmetry in the more critical first mile can be guaranteed by
     the network administrator through configuring the network devices
     at the far end of the WAN to honor the original link choice of the
     incoming sessions in a stateful manner, or by applying the same
     deterministic forwarding algorithms on devices at both ends.
     Several vendors provide this functionality out of the box in
     certain products. Symmetry is thus guaranteed where it matters,
     i.e., where traffic must traverse links having vastly different
     characteristics and quality (it is not uncommon for remote sites
     to be served by a primary DSL/fiber link and a secondary, much
     more limited cellular link).

   The tunnel-based solution described in this section may also be
   implemented as a scheme in which the central site is not owned by
   the organization at all and is instead part of a service offered by
   a tunnel broker somewhere on the Internet. Such a choice can be
   appealing due to factors such as outsourcing of operational burden
   and the possibility of superior performance due to the broker having
   a globally distributed and fine-tuned network of "hubs," to the
   closest of which each site can then connect to.

   Another advantage of having control over the path crossing the first
   mile of the branch site lies in the possibility of applying error-
   correcting algorithms to the traffic; several vendors offer this
   functionality which, although proprietary, can be made to work by
   placing compatible devices at the branch edge and the central site,
   usually terminating the tunnels comprising the overlay. Such
   techniques typically include forward error correction, compression,



FBNVV                   Expires July 29, 2024                 [Page 21]


Internet-Draft             site multihoming                January 2024


   packet duplication, and deduplication across multiple low-quality
   links, to prevent or lessen packet loss across the overlay. These
   techniques, however, cannot improve other metrics such as latency.

   The principal downside of the tunnel-based solution is not making
   use of the "local Internet breakout": users at the branch site are
   almost guaranteed to experience worse performance towards Internet
   destinations compared to solutions listed in sections 5.1-5.2, 5.4-
   5.5 due to the traffic having to be backhauled to the central site
   first.

   However, it should be noted that as long as the centralized site
   uses a solution similar to 5.1 or 5.2, it'll also enable
   communication that otherwise would have failed or required complex
   use-case-specific workarounds.

   In fact, despite the alleviating factors discussed above, shifting
   the problem to a different area of the network might not be
   considered a technical solution at all because the central site
   would face the same fundamental challenges, and it would ultimately
   have the same options for multi-homing as discussed in sections 5.1-
   5.2, 5.4-5.5. As such, what is described in this section could be
   considered a non-technical solution for a small site.

   Enterprise WAN design in itself remains outside the scope of this
   document.

   Advantages:

   - Shifting the problem to a different location may help solve it
     more efficiently,
   - The simplest solution for the small site,
   - Supported CE products (Wireguard),
   - No need to renumber, hence no issues with prefix deprecation,
   - No need for special support on hosts,
   - Traffic steering is easy to implement, including traffic symmetry
     requirements or active/passive failover,
   - A centralized Internet gateway simplifies perimeter security,
   - Possibility of applying WAN optimization techniques to the first
     section of the path toward the Internet,
   - Multiple free or paid tunnel brokers exist with different SLAs,
   - Avoid unnecessarily polluting the global routing table, and may
     also get better AS paths because of the aggregations compared to
     using a PI.
   Disadvantages:




FBNVV                   Expires July 29, 2024                 [Page 22]


Internet-Draft             site multihoming                January 2024


   - Looping the Internet traffic through the centralized site might
     increase latency, and additional links on the traffic path may
     contribute to jitter and packet loss,
   - More bandwidth is needed for rented WAN links,
   - Side effects related to tunnelings, such as encapsulation
     processing and overhead,
   - Convergence time in case of underlay network failures may be
     affected by the need to re-establish the tunnels and routing
     neighborships of the overlay,
   - The central site becomes a single point of failure for the
     Internet access of the entire organization,
   - Some or all of the disadvantages listed in sections 5.1-5.2, 5.4-
     5.5 apply, depending on the specific solution selected to solve
     the multi-homing issue at the central site. These may include end-
     to-end connectivity and traffic steering issues toward Internet
     destinations.

5.4. ULA with NPTv6

   [ULA] allows for the creation and use of local, non-globally
   reachable, and not centrally assigned address space that is
   sufficiently random to be treated like globally unique addressing
   within a given organization or environment. However, [ULA] has
   notable limits concerning the number of sites that one prefix may
   span, and well-documented usability limitations when considering
   address selection details across large, diverse network
   environments.

   Organizations requiring larger than a /48 prefix are often better
   served applying for and receiving a global PI allocation that is
   right-sized for their needs. There may exist creative tweaks for
   expanding ULA beyond its normal size, but that is outside the scope
   of this document.

   There is also one significant detail of [ULA] address space that is
   important to note in the presence of a dual-stacked environment, as
   [ULA] is prioritized below GUA and IPv4 address space on the hosts
   according to [SASA] section 2.1. Hence, in a dual-stack environment,
   it is necessary to modify the [SASA] policy table to insert the /48
   prefix with higher precedence, as recommended in section 10.6 of
   [SASA]. Automation for such configuration is OS-specific, and in
   some cases may not be possible.

   It is not mandatory to have [ULA] for the solution described in this
   section, registered GUA (PI addresses in particular) may be used
   too, but this has a low chance of happening as PI address space is



FBNVV                   Expires July 29, 2024                 [Page 23]


Internet-Draft             site multihoming                January 2024


   much more widely deployed as a routed policy directly to carriers or
   other upstream service providers.

   Network Prefix translation [NPTv6] is the unique IPv6 technology
   that enables a lightweight version of NAT with a 1:1 stateless
   relationship between addresses on the "inside" and "outside". It is
   formally an experimental protocol despite having a long list of
   production deployments. A stateless algorithmic relationship permits
   to have asymmetric routing and easy redundancy if multiple gateways
   are implemented facing a single carrier.

   Like any NAT it may create a challenge for protocols that embed IP
   addresses at the application level. It may require the usage of an
   external [STUN] server for address translation, or monitoring of the
   session by an ALG.

   Additionally, when crossing NAT environments, protocols such as
   IPSec require or fall back to "NAT traversal" schemes, which
   typically work by encapsulating the original session into UDP.
   Application support for this with IPv6 is often poor because a main
   talking point for the adoption of IPv6 is that "NAT traversal" is no
   longer required and that this simplifies the application logic.
   However, in practice, this is not always the case, and legacy
   applications may still require these techniques to operate.

   Contrary to other forms of NAT, with NPTv6 there is no need to
   generate or retain translation logs because translation is stateless
   and deterministic.

   Two-thirds of carriers lease a permanent prefix to subscribers, and
   such prefix would thus remain the same after the uplink is
   disconnected and re-connected. For this reason, it is possible to
   initiate an inbound session from the "outside" stably and
   continuously; the NPTv6 engine can apply the required translation on
   all outbound and inbound packets regardless of whether they are part
   of a new or existing connection. For other carriers which do not
   lease prefixes permanently, some additional efforts are needed to
   dynamically update DNS records, and use those to establish inbound
   connections. The possibility of connection initiation in any
   direction (when firewall rules allow it) is considered valuable by
   some engineers, while others value the one-way connectivity typical
   of NAT that is lost with NPTv6.

   Similarly, other NAT-related problems are not present with NPTv6
   because it does not require manipulation of the transport layer.




FBNVV                   Expires July 29, 2024                 [Page 24]


Internet-Draft             site multihoming                January 2024


   [NPTv6] is partially acting against the [IAB request] to preserve
   the end-to-end transparency of the Internet which is important for
   the Internet's future flexibility.

   Together, [ULA] and [NPTv6] may effectively mimic the typical IPv4
   carrier resilience practices: the organization might only have [ULA]
   inside its network (no GUA), and every site could have many
   redundant connections through separate [NPTv6] engines on every
   border gateway, making use of the actual PA space provided by each
   carrier on the external side of the translation to enable global
   reachability towards Internet destinations.

   There is, however, a principal difference from the typical IPv4 NAT
   solution in that [NPTv6] needs an equally-sized prefix on the
   "inside" and "outside". While it is typically possible to get a /56
   or /60 from the fixed broadband carrier, it is significantly less
   common to be delegated more than a /64 by the mobile carrier; hence,
   if the carrier is mobile then only a simplified site with one
   internal /64 subnet is feasible.

   For faster convergence, routers should announce a "default" on-site
   and then add some more specific prefixes if needed. Additional
   destination prefixes facilitate the distribution of traffic between
   the carriers.

   In cases where internal routing hops are present between the edge
   devices, such announcements likely require other protocols in
   addition to Neighbor Discovery.

   As for the network interface of the internal host (the last hop),
   such an announcement comes in the form of default router status in
   [ND], and [Route Preferences] for the more specific routes. The last
   one, in particular, is essential to be able to access "subscriber-
   only services".

   The crucial requirement remains that routers must continuously track
   connectivity to carriers to be able to deprecate the "default" as
   well as any more specific announcements when such connectivity is
   lost.

   Advantages:

   - No need for official address space, the ULA prefix is pseudo-
     randomly self-generated,
   - Easy to implement, similar in practice to current IPv4 carrier
     resiliency techniques,



FBNVV                   Expires July 29, 2024                 [Page 25]


Internet-Draft             site multihoming                January 2024


   - Potential for traffic distribution policy between different
     carriers.
   Disadvantages:
   - It is challenging to automate ULA prioritization above IPv4 on
     hosts,
   - NPTv6 breaks some applications with address referrals at the
     application level, some additional solutions are needed (STUN,
     ALG),
   - Custom distribution policies are needed for access to filtered
     resources ("subscriber-only services"),
   - Session initiation from the outside is practical only in cases
     where the carrier prefix is stable or DNS records are dynamically
     updated,
   - Currently limited to one subnet per site in mobile environments,
   - May hinder overall IPv6 adoption as IPv6 with NPTv6 loses the end-
     to-end connectivity advantage,
   - For applications, the drawbacks are similar to ULA with NAT66
     (section 5.4).

5.5. ULA with NAT66

   The observations regarding the use of ULA from the first paragraph
   of the previous section apply to this section as well. It is
   important to be mindful of the requirement and effort necessary to
   prioritize ULA above IPv4 in the [SASA] policy table of hosts.

   It is not mandatory to have [ULA] for the solution described in this
   section, alternatively, registered GUA (PI addresses in particular)
   may be used, but this brings its own set of requirements and effort,
   as PI address space is much more useful when routed directly to
   carriers. However, not all carriers may support this, or they might
   charge significantly more to do so.

   The [IAB request] to avoid the use of NAT has been heavily based on
   the long list of [NAT implications]. In the case of stateful NAT66,
   the full list of problems caused by NAT is applicable: breaking of
   the end-to-end model, the impossibility of initiating sessions from
   the outside, breaking IPSec encryption, breaking of application-
   layer referrals to the addresses, single point of failure,
   redundancy challenges (state replication), and performance
   challenges (stateful processing).

   Hence, it is easy to understand the IETF consensus for not having a
   stateful NAT standard for IPv6. There is no RFC for NAT66. A
   proprietary implementation may furthermore create interoperability
   challenges.



FBNVV                   Expires July 29, 2024                 [Page 26]


Internet-Draft             site multihoming                January 2024


   Those details aside, NAT has operational advantages: it avoids
   renumbering in case of PA address space change, and is historically
   the most common solution for carrier resiliency, especially in small
   to medium-sized environments that may not have the resources,
   availability, or expertise to leverage a PI-based solution with
   upstream carriers. Hence, it is supported by many products, both
   commercial and open source; one example is [nftables NAT66].

   Another issue with NAT is the lack of UPnPv6 standardization and
   implementation. With IPv4, applications behind a NAT44 can
   dynamically request to know their public IP as well as new port
   forwardings, thanks to UPnP. With IPv6 this is not available, and
   the end-user experience with NAT66 is thus likely to be worse than
   with NAT44.

   For faster convergence, routers should announce a "default" on-site
   and then additionally some more specific prefixes if needed.
   Additional destination prefixes facilitate the distribution of
   traffic between the carriers.

   In cases where internal routing hops are present between the edge
   devices, such announcements likely require other protocols in
   addition to Neighbor Discovery.

   As for the network interface of the internal host (the last hop),
   such an announcement comes in the form of default router status in
   [ND], and [Route Preferences] for the more specific routes. The last
   one, in particular, is essential to be able to access "subscriber-
   only services".

   The crucial requirement remains that routers must continuously track
   connectivity to carriers to be able to deprecate the "default" as
   well as any more specific announcements when such connectivity is
   lost.

   Advantages:

   - No need for official address space, as the ULA prefix is pseudo-
     randomly self-generated,
   - Easy to implement,
   - Equivalent in practice to current IPv4 carrier resiliency
     techniques,
   - NAT may be a normative requirement in itself (this is highly
     questionable, but brought forward in many discussions),
   - Support for sites with complex topologies,




FBNVV                   Expires July 29, 2024                 [Page 27]


Internet-Draft             site multihoming                January 2024


   - Potential for traffic distribution policy between different
     carriers.
   Disadvantages:
   - It is challenging to automate ULA prioritization above IPv4 on
     hosts,
   - NAT breaks some applications with address referrals at the
     application level,
   - Custom distribution policies needed for access to filtered
     resources ("subscriber-only services"),
   - Some additional solutions are needed (STUN, ALG),
   - Session initiation from the outside is blocked in practice (needs
     complex configuration),
   - NAT implies the requirement to keep logs for compliance and
     troubleshooting,
   - Expensive, due to the higher costs of stateful processing,
   - May hinder overall IPv6 adoption,
   - For applications, the drawbacks are similar to ULA with NPTv6
     (section 5.3).

5.6. Application proxy

   Another possible approach is shifting the Internet access resiliency
   problem to an explicit application proxy similar to the one
   described in [IP Proxy], that can be placed anywhere in the network.
   Traditionally, enterprise networks have often relied on proxies to
   enable Internet access for clients. The primary motivation for this
   has not been to provide network functionality but to enforce policy,
   authenticate users, and perform traffic filtering. The caveats for
   this mechanism to work are essentially two, namely that both the
   clients and the applications they are accessing support the use of
   an application proxy. "Transparent" proxies which sit in the traffic
   path, similar to those described in Section 4 of [IP Proxy], have
   also been employed to negate the requirement for clients to be
   configured to use the application proxy. With regards to the Site
   Multi-homing problem, however, this configuration is functionally
   equivalent to the scenario in 5.4 while having no advantages
   compared to it; for this reason, this section deals exclusively with
   explicitly configured proxies accepting traffic destined to an
   address owned by them.

   By proxying connections at the transport layer and above, thus
   splitting the connection into 2 parts, there is an inherent
   possibility to use a different source address for the second,
   upstream half of the connection; in an IPv4 environment, this can in
   some cases replace address translation in environments where private
   address space is used by the clients. A similar possibility exists



FBNVV                   Expires July 29, 2024                 [Page 28]


Internet-Draft             site multihoming                January 2024


   in an IPv6 environment, allowing end hosts to communicate with the
   Internet even if they only possess ULA or otherwise non-globally
   reachable addresses as long as they can reach the application proxy.
   The use of a proxy in an IPv6 site with multi-homing requirements
   thus allows for the following configuration:

   - Two or more Internet gateways from diverse carriers, receiving and
     advertising PA GUA address space on the network,
   - An application proxy provisioned with a ULA address as well as PA
     addresses from each Internet uplink,
   - Clients provisioned with ULA addresses to communicate with
     internal destinations and the proxy, and only optionally PA
     addresses.

   In such a configuration clients support, and are configured for, the
   proxying of outbound connections through the explicit application
   proxy. The connection to the proxy then uses ULA addresses
   exclusively, and the second part of the connection from the
   application proxy out to the Internet can use a viable PA source
   address selected from the available prefixes. In this manner,
   clients are relieved of the duty to select a GUA source as well as
   tracking of the health and validity of related prefixes. This leaves
   the source GUA selection, and in turn the link selection, to the
   proxy itself. In this case, tracking of the health of each Internet
   link needs to be performed in only one host, i.e. the proxy, and
   such tracking is facilitated further in the case where the proxy and
   the Internet gateways are the same devices.

   Available technologies for signaling and establishing proxied
   connections include [HTTP], [SOCKS], and, more recently, [MASQUE],
   which allow the application proxy host to interject at various
   layers of the stack up to the application itself, supporting TCP,
   UDP, and even the forwarding of raw IP datagrams. Discussion of the
   low-level workings of such technologies is out of the scope of this
   document.

   Owing to the extensive past usage of Web proxies and the dominance
   of HTTP and HTTPS in the outbound component of enterprise network
   traffic, a client in the enterprise can typically perform nearly all
   of its interactions with the Internet through a proxy, but several
   direct connections are often required for some applications; in the
   scenario described in this section, the client is free to reach
   Internet destinations in such a manner if it is provisioned with one
   or more of the actual PA prefixes coming from the Internet links,
   and if such traffic is not restricted by other on-path devices such
   as firewalls. Using its ULA addresses may even work for direct



FBNVV                   Expires July 29, 2024                 [Page 29]


Internet-Draft             site multihoming                January 2024


   communication, with the caveats outlined in sections 5.4 and 5.5.
   For any traffic generated by clients which bypasses the proxy, all
   considerations listed in sections 5.2, 5.4, and 5.5 apply.

   The device hosting the application proxy itself is in a situation
   comparable to the one described in section 5.2, but the process or
   daemon which performs the proxy functions can be placed arbitrarily
   close or even inside the gateways. This mitigates a number of the
   disadvantages listed in 5.2, namely:

   - "Prefixes may not get deprecated when the CPE itself fails": the
     application proxy can track the health of each prefix using active
     polling and other proprietary methods.
   - "Carriers may frequently change the prefix (flash renumbering)":
     renumbering is simplified as it only needs to occur on one host,
     and any patches and upgrades of the network stack required to
     support newer and more robust renumbering mechanisms are
     simplified. Additionally, if the application proxy is the gateway
     itself, it can become aware of any new prefixes without having to
     employ any kind of network signaling.
   - "Sites with complex topologies are not well supported": the
     application proxy can be positioned in the subnet and link which
     is shared by the internal interface of all the site's Internet
     gateways, recreating the "simple topology" condition. Clients
     reach the proxy using ULA and conventional routing is sufficient
     in the more complex, internal part of the network.
   - "Traffic steering by any policies is not supported yet": link
     selection is centralized in the application proxy, which can
     enforce steering policies of various kinds.

   Communication inbound from the Internet toward clients is impossible
   in a purely proxy-based scenario and requires that clients be
   provisioned with GUAs in addition to the ULA they employ to reach
   the proxy. All considerations in 5.2 are applicable in this case.
   Additionally, if an application supports and the clients opt to use
   the proxy for it, outbound usage of such an application results in a
   completely different mechanism (an application proxy) and a
   different GUA address than the one used for inbound sessions. It is
   however likely that the host inside the site is acting purely as a
   client for such an application, and any peer-to-peer or server
   applications running on the host are elected for bypassing the
   proxy.

   Modern client operating systems offer several standardized ways of
   configuring the use of an application proxy, including manual and
   automatic techniques in which the proxy, as well as the list of



FBNVV                   Expires July 29, 2024                 [Page 30]


Internet-Draft             site multihoming                January 2024


   destinations to be proxied, is auto-discovered by the client upon
   joining the network. Discussion of these techniques is out of the
   scope of this document, but they constitute a hard limit on the
   applicability of proxies as described in this section. If the number
   of hosts that do not support automatic configuration is large,
   provisioning of the proxy may become impractical or impossible. This
   is further compounded by the fact that an ever-increasing number of
   hosts in enterprise networks is headless and/or unmanaged, such as
   IoT devices, and may not even support proxies at all or not support
   a way of configuring them. Any such device would need to resort to
   one of the other options described in this document to gain Internet
   access and benefit from multi-homing. Network designers may find
   that, in such cases, the need to mix and match solutions has
   operational drawbacks that outweigh the advantages of using a proxy.

   It is also crucial to keep in mind that the use of an application
   proxy does not technically constitute a solution to the issue of
   IPv6 multi-homing, which is a network problem. This is obvious for
   two reasons:

   - The solution operates at the transport layer and above,
   - The device hosting the application proxy faces the same
     fundamental challenges and ultimately has the same options for
     multi-homing as discussed in sections 5.1-5.2, 5.4-5.5. As such,
     what is described in this section could be considered an upper-
     layer workaround to a network problem.

   Advantages:

   - No NAT, communication is terminated and re-established at a higher
     layer using a different source address, and the client is aware of
     this,
   - No need to own and operate a registered, Provider-Independent
     address space,
   - ULA usage is ULA to ULA, avoiding the need to engineer ULA
     prioritization above IPv4 in the [SASA] policy table,
   - Organizations may already be using an application proxy,
   - Supports sites with complex topology,
   - Supports outbound traffic steering,
   - Potential for traffic distribution policy between different
     carriers.

   Disadvantages:

   - No E2E unless the client has GUA as well, and even then outbound
     and inbound use different mechanisms,



FBNVV                   Expires July 29, 2024                 [Page 31]


Internet-Draft             site multihoming                January 2024


   - Proxy provisioning and discovery,
   - The application proxy is an additional point of failure in the
     network, and ensuring its redundancy could bring increased costs
     and complexity,
   - Requires explicit application and/or protocol support,
   - Requires explicit client support,
   - While not NAT66, sessions must still be tracked in a stateful
     manner by the application proxy,
   - Splitting the flow of data at the application layer is
     computationally expensive and can incur performance penalties far
     over a traditional network hop.

6. Conclusion

   Not all requirements can be satisfied by every solution:

   +--+--------------------------------+----+-----+---------+---------+

   |  |     Requirement                | PI | PA  |ULA+NPTv6| ULA+NAT |

   +--+--------------------------------+----+-----+---------+---------+

   | 1| Carriers Resiliency            |  + |  +  |    +    |    +    |

   | 2| End-to-End Connectivity        |  + |  +  |  +/-    |    -    |

   | 3| Internal Connectivity          |  + |  +  |  +/-    |  +/-    |

   | 4| Convergence speed              |  + | +/- |    +    |    +    |

   | 5| Complex Topology support       |  + |  -  |  +/-    |    +    |

   | 6| Subscriber-only services       |  - |  -  |  +/-    |  +/-    |

   | 7| Traffic Steering on Router     | +/-|  -  |    +    |    +    |

   | 7| Traffic Steering on Host OS    |  - |  -  |    -    |    -    |

   | 7| Traffic Steering on Application|  - |  -  |    -    |    -    |

   +--+--------------------------------+----+-----+---------+---------+

   The table above shows partial support for end-to-end connectivity
   for the ULA+NPTv6 solution because, while it does allow initiating
   connectivity in any direction, employing address references at the




FBNVV                   Expires July 29, 2024                 [Page 32]


Internet-Draft             site multihoming                January 2024


   application layer requires extra steps, for example in the form of
   an ALG or [STUN].

   Internal connectivity is marked as partial support in the ULA
   solutions due to the complexity involved in promoting the ULA
   address space above IPv4 in the [SASA] policy table of hosts.

   Convergence speed is partially satisfied by the PA-based solution
   because [HNCP] or DHCP-PD have not been adopted by the market, and
   they would be needed for prefix deprecation propagation over a
   complex site.

   Complex topology support is marked as partially satisfied by the
   ULA+NPTv6 solution because it is not possible in practice to get an
   external prefix larger than /64 from mobile carriers.

   Support for "subscriber-only services" is marked as partial in the
   ULA solutions because it needs a routing announcement as a "Routing
   Information Option" of [Route Preferences], which is not widely
   supported.

   Traffic steering on routers is marked as partially supported for the
   PI-based solution because of the high complexity involved in
   organizing the steering of incoming traffic. NAT/NPTv6-based
   solutions connect ingress traffic steering to egress which makes
   them simpler in this regard.

   Some of the functionality reflected in the table above may be
   improved in the future, but a roadmap (active IETF draft) is not
   available at the time of writing.

   Theoretically, from a purely technical point of view, the solutions
   in section 5 are ordered by the number of requirements satisfied,
   from most to least: PI is the best, PA-based is more complex, NPTv6
   breaks the end-to-end Internet model, and so on. In the real world,
   though, the company could have non-technical requirements that
   override the technical ones. For example, an organization might find
   that the tunnel broker solution described in 5.5 fits their use-case
   best, even though it doesn't solve the issue so much as outsource it
   to a different organization (the tunnel-based solution in 5.5 cannot
   be properly evaluated in the requirements table due to it
   technically not being a solution, and anything added to the table
   would merely reflect the choice of multi-homing techniques at the
   central site). The table above is, in this sense, not complete - it
   should be enriched with non-technical requirements as perceived by
   the network owner.



FBNVV                   Expires July 29, 2024                 [Page 33]


Internet-Draft             site multihoming                January 2024


   For many network owners, the main deciding factor may be the desire
   to have end-to-end connectivity, as it is the most notable advantage
   of deploying IPv6 compared to IPv4. This may be especially important
   when managing resources that need to be exposed to the Internet.

   If this is needed then PI or PA-based solutions are applicable, and
   network owners must undertake some additional steps to implement
   them:

   - obtain a PI prefix from a RIR or LIR,
   - pay a premium for the advertisement of the PI prefixes through the
     carriers (as these often impose different tariffs for such an
     attachment circuit).

   If both of these challenges are deemed acceptable then the PI-based
   solution is preferable: simple, reliable, and scalable. The
   universal adoption of PI by companies of all sizes would, however,
   create a burden for Internet routing. Routing tables in the default-
   free zone could reach up to 10M entries due to small allocations
   being advertised directly by end sites, without any ties to any
   particular carrier, and thus no summarization being possible at the
   carrier level. To support this, routers would need much larger
   amounts of specialized hardware memory, and this would not only
   disqualify all current hardware but it would also multiply expenses
   related to Internet routers in the future. If many organizations
   were to implement this option, the Internet would break for
   everyone. In any case, this issue is unlikely to be considered a
   priority by an enterprise organization whose decision-making process
   is already constrained by many other factors.

   If at least one of the aforementioned challenges is not acceptable
   then PA may be the solution of choice, even with all the
   restrictions and complexities discussed in section 5.2. A notable
   exception to this is if complex topology support is a requirement
   and the site is served by a mobile carrier (due to the
   unavailability of prefixes larger than /64 on such connections).

   It may also be the case that end-to-end connectivity is not a
   necessity, and may even be undesired. Here, the ULA+NPTv6 solution
   (discussed in section 5.4) satisfies a greater amount of
   requirements in the majority of situations, apart from cases where
   NAT66 is a strict (non-technical) requirement or the site has a
   combination of complex topology and mobile connectivity (problematic
   due to the small assignments on the WAN side and the 1:1 mechanism
   of NPTv6).




FBNVV                   Expires July 29, 2024                 [Page 34]


Internet-Draft             site multihoming                January 2024


   For sites having a complex topology (many links and routers), a PA-
   based solution is not an option yet, because it would need automatic
   PA address distribution over the site and neither [HNCP] nor DHCP-PD
   have gained market acceptance for this task.

   The logical steps in the design process would then be like the ones
   above, but after having evaluated the PI-based solution the next
   option would be ULA+NPTv6.

   Previous availability of PI space or perceived NAT66 regulatory
   requirements might also be primary factors, and then the logic may
   yet again be different.

   For further use cases that have not been discussed in this document,
   it is possible to get a general expectation of compatibility using
   the table above.

   It is however recommended to read section 5 to fine-tune the custom
   requirements matrix and grade each solution accordingly.

7. Security Considerations

   This document is informational.
   Hence, it may not create additional security challenges.

8. IANA Considerations

   This document has no request to IANA.

9. References

9.1. Normative References

   [BCP38] P. Ferguson, D. Senie, "Network Ingress Filtering: Defeating
             Denial of Service Attacks which employ IP Source Address
             Spoofing", BCP 38, RFC 2827, DOI 10.17487/RFC2827, May
             2000, <https://www.rfc-editor.org/info/rfc2827>.

   [IPv6] S. Deering, R. Hinden, "Internet Protocol, Version 6 (IPv6)
             Specification", RFC 8200, DOI 10.17487/RFC8200, July 2017,
             <https://www.rfc-editor.org/info/rfc8200>.

   [ND] T. Narten, E. Nordmark, W. Simpson, H. Soliman, "Neighbor
             Discovery for IP version 6 (IPv6)", RFC 4861, DOI
             10.17487/RFC4861, September 2007, <https://www.rfc-
             editor.org/info/rfc4861>.



FBNVV                   Expires July 29, 2024                 [Page 35]


Internet-Draft             site multihoming                January 2024


   [SLAAC] S. Thomson, T. Narten, T. Jinmei, "IPv6 Stateless Address
             Autoconfiguration", RFC 4862, DOI 10.17487/RFC4862,
             September 2007, <https://www.rfc-editor.org/info/rfc4862>.

   [NAT Implications] T. Hain, "Architectural Implications of NAT", RFC
             2993, DOI 10.17487/RFC2993, November 2000,
             <https://www.rfc-editor.org/info/rfc2993>.

   [Local Protection] G. Van de Velde, T. Hain, R. Droms, B. Carpenter,
             E. Klein, "Local Network Protection for IPv6", RFC 4864,
             DOI 10.17487/RFC4864, May 2007, <https://www.rfc-
             editor.org/info/rfc4864>.

   [IAB request] D. Thaler, L. Zhang, G. Lebovitz, "IAB Thoughts on
             IPv6 Network Address Translation", RFC 5902, DOI
             10.17487/RFC5902, July 2010, <https://www.rfc-
             editor.org/info/rfc5902>.

   [NPTv6] M. Wasserman, F. Baker, "IPv6-to-IPv6 Network Prefix
             Translation", RFC 6296, DOI 10.17487/RFC6296, June 2011,
             <https://www.rfc-editor.org/info/rfc6296>.

   [ULA] R. Hinden, B. Haberman, "Unique Local IPv6 Unicast Addresses",
             RFC 4193, DOI 10.17487/RFC4193, October 2005,
             <https://www.rfc-editor.org/info/rfc4193>.

   [SASA] D. Thaler, R. Draves, A. Matsumoto, T. Chown, "Default
             Address Selection for Internet Protocol Version 6 (IPv6)",
             RFC 6724, DOI 10.17487/RFC6724, September 2012,
             <https://www.rfc-editor.org/info/rfc6724>.

   [HNCP] M. Stenberg, S. Barth, P. Pfister, "Home Networking Control
             Protocol", RFC 7788, DOI 10.17487/RFC7788, April 2016,
             <https://www.rfc-editor.org/info/rfc7788>.

   [Node Requirements]  T. Chown, J. Loughney, T. Winters, "IPv6 Node
             Requirements", BCP 220, RFC 8504, DOI 10.17487/RFC8504,
             January 2019, <https://www.rfc-editor.org/info/rfc8504>.

   [MHMP Enterprise] F. Baker, C. Bowers, J. Linkova, "Enterprise
             Multihoming Using Provider-Assigned IPv6 Addresses without
             Network Prefix Translation: Requirements and Solutions",
             RFC 8678 DOI 10.17487/RFC8678, December 2019,
             <https://www.rfc-editor.org/info/rfc8678>.





FBNVV                   Expires July 29, 2024                 [Page 36]


Internet-Draft             site multihoming                January 2024


   [Multi-Homing] F. Baker, B. Carpenter, "First-Hop Router Selection
             by Hosts in a Multi-Prefix Network", RFC 8028, DOI
             10.17487/RFC8028, November 2016, <https://www.rfc-
             editor.org/info/rfc8028>.

9.2. Informative References

   [IAB report] D. Meyer, L. Zhang, K. Fall, "Report from the IAB
             Workshop on Routing and Addressing", RFC 4984, DOI
             10.17487/RFC4984, September 2007, <https://www.rfc-
             editor.org/info/rfc4984>.

   [Shim6] E. Nordmark, M. Bagnulo, "Shim6: Level 3 Multihoming Shim
             Protocol for IPv6", RFC 5533, DOI 10.17487/RFC5533, June
             2009, <https://www.rfc-editor.org/info/rfc5533>.

   [STUN] M. Petit-Huguenin, G. Salgueiro, J. Rosenberg, D. Wing, R.
             Mahy, P. Matthews, "Session Traversal Utilities for NAT
             (STUN)", RFC 8489, DOI 10.17487/RFC8489, February 2020,
             <https://www.rfc-editor.org/info/rfc8489>.

   [MHMP] O. Troan, D. Miles, S. Matsushima, T. Okimoto, D. Wing, "IPv6
             Multihoming without Network Address Translation", RFC
             7157, DOI 10.17487/RFC7157, March 2014, <https://www.rfc-
             editor.org/info/rfc7157>.

   [Route Preferences] R. Draves, D. Thaler, "Default Router
             Preferences and More-Specific Routes", RFC 4191, DOI
             10.17487/RFC4191, November 2005, <https://www.rfc-
             editor.org/info/rfc4191>.

   [Policy by DHCP] A. Matsumoto, T. Fujisaki, T. Chown, "Distributing
             Address Selection Policy Using DHCPv6", RFC 7078 DOI
             10.17487/RFC7078, January 2014, <https://www.rfc-
             editor.org/info/rfc7078>.

   [DNS Options] J. Jeong, S. Park, L. Beloeil, S. Madanapalli, "IPv6
             Router Advertisement Options for DNS Configuration", RFC
             8106 DOI 10.17487/RFC8106, March 2017, <https://www.rfc-
             editor.org/info/rfc8106>.

   [Conditional PIO] J. Linkova, M. Stucchi, "Using Conditional Router
             Advertisements for Enterprise Multihoming", RFC 8475 DOI
             10.17487/RFC8475, October 2018, <https://www.rfc-
             editor.org/info/rfc8475>.




FBNVV                   Expires July 29, 2024                 [Page 37]


Internet-Draft             site multihoming                January 2024


   [Provisioning domains] P. Pfister, E. Vyncke, T. Pauly, D. Schinazi,
             W. Shao, "Discovering Provisioning Domain Names and Data",
             RFC 8801 DOI 10.17487/RFC8801, July 2020,
             <https://www.rfc-editor.org/info/rfc8801>.

   [DHCPv6] T. Mrugalski, M. Siodelski, B. Volz, A. Yourtchenko, M.
             Richardson, S. Jiang, T. Lemon, T. Winters, " Dynamic Host
             Configuration Protocol for IPv6 (DHCPv6)", RFC 8415 DOI
             10.17487/RFC8415, November 2018, <https://www.rfc-
             editor.org/info/rfc8415>.

   [Temporary addressing] F. Gont, S. Krishnan, T. Narten, R. Draves,
             "Temporary Address Extensions for Stateless Address
             Autoconfiguration in IPv6", RFC 8981 DOI 10.17487/RFC8981,
             February 2021, <https://www.rfc-editor.org/info/rfc8981>.

   [6renum] S. Jiang, B. Liu, B. Carpenter, "IPv6 Enterprise Network
             Renumbering Scenarios, Considerations, and Methods", RFC
             6879 DOI 10.17487/RFC6879, February 2013,
             <https://www.rfc-editor.org/info/rfc6879>.

   [multi6] J. Abley, B. Black, V. Gill, "Goals for IPv6 Site-
             Multihoming Architectures", RFC 3582 DOI 10.17487/RFC3582,
             August 2003, <https://www.rfc-editor.org/info/rfc3582>.

   [flash renumbering] F. Gont, J. Zorz, R. Patterson, "Improving the
             Robustness of Stateless Address Autoconfiguration (SLAAC)
             to Flash Renumbering Events", draft-ietf-6man-slaac-renum-
             04 (work in progress), February 2023.

   [nftables NAT66] Marco Cilloni, "NAT66: The good, the bad, the
             ugly", <https://blog.apnic.net/2018/02/02/nat66-good-bad-
             ugly>.

   [IP Proxy] Chatel, M., "Classical versus Transparent IP Proxies",
             RFC 1919, DOI 10.17487/RFC1919, March 1996,
             <https://www.rfc-editor.org/info/rfc1919>.

   [HTTP] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, Ed.,
             "HTTP Semantics", STD 97, RFC 9110, DOI 10.17487/RFC9110,
             June 2022, <https://www.rfc-editor.org/info/rfc9110>.

   [SOCKS] Leech, M., Ganis, M., Lee, Y., Kuris, R., Koblas, D., and L.
             Jones, "SOCKS Protocol Version 5", RFC 1928, DOI
             10.17487/RFC1928, March 1996, <https://www.rfc-
             editor.org/info/rfc1928>.



FBNVV                   Expires July 29, 2024                 [Page 38]


Internet-Draft             site multihoming                January 2024


   [MASQUE] Pauly, T., Ed., Schinazi, D., Chernyakhovsky, A.,
             Kuhlewind, M., and M. Westerlund, "Proxying IP in HTTP",
             RFC 9484, DOI 10.17487/RFC9484, October 2023,
             <https://www.rfc-editor.org/info/rfc9484>.

   [MPTCP] A. Ford, C. Raiciu, M. Handley, O. Bonaventure, C. Paasch,
             "TCP Extensions for Multipath Operation with Multiple
             Addresses", RFC 8684 DOI 10.17487/RFC8684, March 2020,
             <https://www.rfc-editor.org/info/rfc8684>.

   [MPQUIC] Y. Liu, Y. Ma, Q. De Coninck, O. Bonaventure, C. Huitema,
             M. Kuehlewind, "Multipath Extension for QUIC", draft-ietf-
             quic-multipath-06 (work in progress), July 2023.

   [PDperDevice] L. Colitti, J. Linkova, X. Ma, "Using DHCPv6-PD to
             Allocate Unique IPv6 Prefix per Client in Large Broadcast
             Networks", draft-ietf-v6ops-dhcp-pd-per-device-06 (work in
             progress), November 2023.

Appendix A

   A survey was proposed to collect information on multi-homing
   solutions adopted in field deployment.

   As of IETF 118 (Prague, November 4-10 2023), 17 respondents provided
   answers to the following questions.

   Question 1: How many CPEs are connected to ISPs on the average site?

   15 responses:

   1.1. One CPE: 6.7%
   1.2. Two CPEs: 80.0%
   1.3. More than two: 13.3%

   Question 2: How many uplinks are configured per CPEs/CEs in your
   average site?

   15 responses:

   2.1. One (every CPE has just 1 uplink toward a certain ISP): 53.3%
   2.2. Two or more (a CPE is connected at least with 2 different
     ISPs): 46.7%

   Question 3: Which configuration do you support?




FBNVV                   Expires July 29, 2024                 [Page 39]


Internet-Draft             site multihoming                January 2024


   14 responses:

   3.1. Active/active: 78.6%
   3.2. Active/standby: 21.4%

   Question 4: Do your CPEs/CEs implement VRRP or any other dynamic
   exchange of redundancy control information?

   15 responses:
   4.1. Yes: 53.3%
   4.2. No: 46.7%

   Question 5: If the answer to the previous is "No", which method do
   you employ for supporting IPv6 multi-homing?

   10 responses:

   5.1. PI addressing: 40.0%
   5.2. PA addressing (GUA), each ISP assigns a prefix to every CPE/CE:
     30.0%
   5.3. ULA/GUA intra-site with ALG at the border: 0.0%
   5.4. ULA/GUA intra-site with NPTv6 at the border: 10.0%
   5.5. ULA/GUA intra-site with NAT66 at the border: 10.0%
   5.6. Other: 10.0%
   5.6.1 PA addressing (GUA) with address from CPE. ULA intra-site.
     NAT66 at border for mismatches: 10.0%

   Question 6: Which method do you employ for supporting IPv4 multi-
   homing (if any)?

   15 responses:

   6.1. Public Addressing: 53.3%
   6.2. NAT44: 46.7%

Acknowledgments

   Thanks to v6ops working group for problem discussion.











FBNVV                   Expires July 29, 2024                 [Page 40]


Internet-Draft             site multihoming                January 2024


Authors' Addresses

   Klaus Frank
   Email: klaus.frank@posteo.de

   Nick Buraglio
   Energy Sciences Network
   Email: buraglio@forwardingplane.net

   Paolo Nero
   Email: oselists@gmail.com

   Paolo Volpato
   Huawei Technologies
   Email: paolo.volpato@huawei.com

   Eduard Vasilenko
   Huawei Technologies
   Email: vasilenko.eduard@huawei.com






























FBNVV                   Expires July 29, 2024                 [Page 41]