Dynamic Networks to Hybrid Cloud DCs Problem Statement

Document Type Active Internet-Draft (rtgwg WG)
Last updated 2020-03-16
Replaces draft-dm-net2cloud-problem-statement
Stream IETF
Intended RFC status (None)
Formats plain text pdf htmlized bibtex
Stream WG state WG Document
Document shepherd No shepherd assigned
IESG IESG state I-D Exists
Consensus Boilerplate Unknown
Telechat date
Responsible AD (None)
Send notices to (None)
Network Working Group                                         L. Dunbar 
     Internet Draft                                                Futurewei  
     Intended status: Informational                               Andy Malis 
     Expires: September 16, 2020                                 Independent 
                                                                C. Jacquenet 
                                                                      M. Toy    
                                                              March 16, 2020 
                Dynamic Networks to Hybrid Cloud DCs Problem Statement 


        This document describes the problems that enterprises face today 
        when interconnecting their branch offices with dynamic workloads in 
        third party data centers (a.k.a. Cloud DCs). There can be many 
        problems associated with network connecting to or among Clouds, many 
        of which probably are out of the IETF scope. The objective of this 
        document is to identify some of the problems that need additional 
        work in IETF Routing area. Other problems are out of the scope of 
        this document. 

        It examines some of the approaches interconnecting cloud DCs with 
        enterprises  on-premises DCs & branch offices. This document also 
        describes some of the network problems that many enterprises face 
        when they have workloads & applications & data split among different 
        data centers, especially for those enterprises with multiple sites 
        that are already interconnected by VPNs (e.g., MPLS L2VPN/L3VPN).   

        Current operational problems are examined to determine whether there 
        is a need to improve existing protocols or whether a new protocol is 
        necessary to solve them.  

     Status of this Memo 

        This Internet-Draft is submitted in full conformance with the 
        provisions of BCP 78 and BCP 79.  

     xxx, et al.           Expires September 16, 2020               [Page 1] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

        Internet-Drafts are working documents of the Internet Engineering 
        Task Force (IETF), its areas, and its working groups.  Note that 
        other groups may also distribute working documents as Internet-

        Internet-Drafts are draft documents valid for a maximum of six 
        months and may be updated, replaced, or obsoleted by other documents 
        at any time.  It is inappropriate to use Internet-Drafts as 
        reference material or to cite them other than as "work in progress." 

        The list of current Internet-Drafts can be accessed at 

        The list of Internet-Draft Shadow Directories can be accessed at 

        This Internet-Draft will expire on August 16, 2020. 

     Copyright Notice 

        Copyright (c) 2020 IETF Trust and the persons identified as the 
        document authors. All rights reserved. 

        This document is subject to BCP 78 and the IETF Trust s Legal 
        Provisions Relating to IETF Documents 
        (http://trustee.ietf.org/license-info) in effect on the date of 
        publication of this document. Please review these documents 
        carefully, as they describe your rights and restrictions with 
        respect to this document. Code Components extracted from this 
        document must include Simplified BSD License text as described in 
        Section 4.e of the Trust Legal Provisions and are provided without 
        warranty as described in the Simplified BSD License. 

     Table of Contents 
        1. Introduction...................................................3 
           1.1. Key Characteristics of Cloud Services:....................3 
           1.2. Connecting to Cloud Services..............................3 
           1.3. The role of SD-WAN in connecting to Cloud Services........4 
        2. Definition of terms............................................5 
        3. High Level Issues of Connecting to Multi-Cloud.................6 
           3.1. Security Issues...........................................6 
     Dunbar, et al.                                                 [Page 2] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

           3.2. Authorization and Identity Management.....................6 
           3.3. API abstraction...........................................7 
           3.4. DNS for Cloud Resources...................................8 
           3.5. NAT for Cloud Services....................................9 
           3.6. Cloud Discovery...........................................9 
        4. Interconnecting Enterprise Sites with Cloud DCs...............10 
           4.1. Sites to Cloud DC........................................10 
           4.2. Inter-Cloud Interconnection..............................12 
        5. Problems with MPLS-based VPNs extending to Hybrid Cloud DCs...14 
        6. Problem with using IPsec tunnels to Cloud DCs.................15 
           6.1. Scaling Issues with IPsec Tunnels........................15 
           6.2. Poor performance over long distance......................16 
        7. Problems of Using SD-WAN to connect to Cloud DCs..............16 
           7.1. More Complexity to Edge Nodes............................17 
           7.2. Edge WAN Port Management.................................17 
           7.3. Forwarding based on Application..........................18 
        8. End-to-End Security Concerns for Data Flows...................18 
        9. Requirements for Dynamic Cloud Data Center VPNs...............18 
        10. Security Considerations......................................19 
        11. IANA Considerations..........................................19 
        12. References...................................................19 
           12.1. Normative References....................................19 
           12.2. Informative References..................................19 
        13. Acknowledgments..............................................20 

     1. Introduction 

     1.1. Key Characteristics of Cloud Services: 

        Key characteristics of Cloud Services are on-demand, scalable, 
        highly available, and usage-based billing. Cloud Services, such as, 
        compute, storage, network functions (most likely virtual), third 
        party managed applications, etc. are usually hosted and managed by 
        third parties Cloud Operators. Here are some examples of Cloud 
        network functions: Virtual Firewall services, Virtual private 
        network services, Virtual PBX services including voice and video 
        conferencing systems, etc. Cloud Data Center (DC) is shared 
        infrastructure that hosts the Cloud Services to many customers.  


     1.2. Connecting to Cloud Services 

        With the advent of widely available third-party cloud DCs and 
        services in diverse geographic locations and the advancement of 

     Dunbar, et al.                                                 [Page 3] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

        tools for monitoring and predicting application behaviors, it is 
        very attractive for enterprises to instantiate applications and 
        workloads in locations that are geographically closest to their end-
        users. Such proximity can improve end-to-end latency and overall 
        user experience. Conversely, an enterprise can easily shutdown 
        applications and workloads whenever end-users are in motion (thereby 
        modifying the networking connection of subsequently relocated 
        applications and workloads). In addition, enterprises may wish to 
        take advantage of more and more business applications offered by 
        cloud operators.  

        The networks that interconnect hybrid cloud DCs must address the 
        following requirements:  
          - High availability to access all workloads in the desired cloud 
             Many enterprises include cloud in their disaster recovery 
             strategy, such as enforcing periodic backup policies within the 
             cloud, or running backup applications in the Cloud. 

          - Global reachability from different geographical zones, thereby 
             facilitating the proximity of applications as a function of the 
             end users  location, to improve latency.  
          - Elasticity: prompt connection to newly instantiated 
             applications at Cloud DCs when usages increase and prompt 
             release of connection after applications at locations being 
             removed when demands change.  
          - Scalable security management.  

     1.3. The role of SD-WAN in connecting to Cloud Services  

        Some of the characteristics of SD-WAN [SDWAN-BGP-USAGE], such as 
        network augmentation and forwarding based on application IDs instead 
        of based on destination IP addresses, are very essential for 
        connecting to on-demand Cloud services.  

        Issues associated with using SD-WAN for connecting to Cloud services 
        are also discussed in this document.   

     Dunbar, et al.                                                 [Page 4] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

     2. Definition of terms 

        Cloud DC:   Third party Data Centers that usually host applications 
                    and workload owned by different organizations or 

        Controller: Used interchangeably with SD-WAN controller to manage 
                    SD-WAN overlay path creation/deletion and monitoring the 
                    path conditions between two or more sites.  

        DSVPN:      Dynamic Smart Virtual Private Network. DSVPN is a secure 
                    network that exchanges data between sites without 
                    needing to pass traffic through an organization's 
                    headquarter virtual private network (VPN) server or 

        Heterogeneous Cloud: applications and workloads split among Cloud 
                    DCs owned or managed by different operators.  

        Hybrid Clouds: Hybrid Clouds refers to an enterprise using its own 
                    on-premises DCs in addition to Cloud services provided 
                    by one or more cloud operators. (e.g. AWS, Azure, 
                    Google, Salesforces, SAP, etc).   

        SD-WAN:     Software Defined Wide Area Network. In this document, 
                     SD-WAN  refers to the solutions of pooling WAN 
                    bandwidth from multiple underlay networks to get better 
                    WAN bandwidth management, visibility & control. When the 
                    underlay networks are private networks, traffic can 
                    traverse without additional encryption; when the 
                    underlay networks are public, such as Internet, some 
                    traffic needs to be encrypted when traversing through 
                    (depending on user provided policies). 

        VPC:        Virtual Private Cloud is a virtual network dedicated to 
                    one client account. It is logically isolated from other 
                    virtual networks in a Cloud DC. Each client can launch 
                    his/her desired resources, such as compute, storage, or 
                    network functions into his/her VPC. Most Cloud 
                    operators  VPCs only support private addresses, some 
                    support IPv4 only, others support IPv4/IPv6 dual stack.  
     Dunbar, et al.                                                 [Page 5] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 


     3. High Level Issues of Connecting to Multi-Cloud 

        There are many problems associated with connecting to hybrid Cloud 
        Services, many of which are out of the IETF scope. This section is 
        to identify some of the high level problems that can be addressed by 
        IETF, especially by Routing area. Other problems are out of the 
        scope of this document. By no means has this section covered all 
        problems for connecting to Hybrid Cloud Services, e.g. difficulty in 
        managing cloud spending is not discussed here.  

     3.1. Security Issues 

        Cloud Services is built upon shared infrastructure, therefore not 
        secure by nature. Security has been a primary, and valid, concern 
        from the start of cloud computing: you are unable to see the exact 
        location where your data is stored or being processed. Headlines 
        highlighting data breaches, compromised credentials, and broken 
        authentication, hacked interfaces and APIs, account hijacking 
        haven t helped alleviate concerns.  

        Secure user identity management, authentication, and access control 
        mechanisms are important. Developing appropriate security 
        measurements can enhance the confidence needed by enterprises to 
        fully take advantage of Cloud Services.  


     3.2. Authorization and Identity Management 

        One of the more prominent challenges for Cloud Services is Identity 
        Management and Authorization. The Authorization not only includes 
        user authorization, but also the authorization of API calls by 
        applications from different Cloud DCs managed by different Cloud 
        Operators. In addition, there are authorization for Workload 
        Migration, Data Migration, and Workload Management.   

        There are many types of users in cloud environments, e.g. end users 
        for accessing applications hosted in Cloud DCs, Cloud-resource users 
        who are responsible for setting permissions for the resources based 
        on roles, access lists, IP addresses, domains, etc.  

     Dunbar, et al.                                                 [Page 6] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

        There are many types of Cloud authorizations: including MAC 
        (Mandatory Access Control)   where each app owns individual access 
        permissions, DAC (Discretionary Access Control)   where each app 
        requests permissions from an external permissions app, RBAC (Role-
        based Access Control)   where the authorization service owns roles 
        with different privileges on the cloud service, and ABAC (Attribute-
        based Access Control)   where access is based on request attributes 
        and policies. 

        IETF hasn t yet developed comprehensive specification for Identity 
        management and data models for Cloud Authorizations.  

     3.3. API abstraction 

        Different Cloud Operators have different APIs to access their Cloud 
        resources, security functions, the NAT, etc. 

        It is difficult to move applications built by one Cloud operator s 
        APIs to another. However, it is highly desirable to have a single 
        and consistent way to manage the networks and respective security 
        policies for interconnecting applications hosted in different Cloud 

        The desired property would be having a single network fabric to 
        which different Cloud DCs and enterprise s multiple sites can be 
        attached or detached, with a common interface for setting desired 

        The difficulty of connecting applications in different Clouds might 
        be stemmed from the fact that they are direct competitors. Usually 
        traffic flow out of Cloud DCs incur charges. Therefore, direct 
        communications between applications in different Cloud DCs can be 
        more expensive than intra Cloud communications. 

        It is desirable to have a common API shim layer or abstraction for 
        different Cloud providers to make it easier to move applications 
        from one Cloud DC to another.  


     Dunbar, et al.                                                 [Page 7] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

     3.4. DNS for Cloud Resources 

        DNS name resolution is essential for on-premises and cloud-based 
        resources. For customers with hybrid workloads, which include on-
        premises and cloud-based resources, extra steps are necessary to 
        configure DNS to work seamlessly across both environments. 

        Cloud operators have their own DNS to resolve resources within their 
        Cloud DCs and to well-known public domains. Cloud s DNS can be 
        configured to forward queries to customer managed authoritative DNS 
        servers hosted on-premises, and to respond to DNS queries forwarded 
        by on-premises DNS servers.    

        For enterprises utilizing Cloud services by different cloud 
        operators, it is necessary to establish policies and rules on 
        how/where to forward DNS queries to. When applications in one Cloud 
        need to communication with applications hosted in another Cloud, 
        there could be DNS queries from one Cloud DC being forwarded to the 
        enterprise s on premise DNS, which in turn be forwarded to the DNS 
        service in another Cloud. Needless to say, configuration can be 
        complex depending on the application communication patterns.  

        However, even with carefully managed policies and configurations, 
        collisions can still occur. If you use an internal name like .cloud 
        and then want your services to be available via or within some other 
        cloud provider which also uses .cloud, then it can't work. 
        Therefore, it is better to use the global domain name even when an 
        organization does not make all its namespace globally resolvable. An 
        organization's globally unique DNS can include subdomains that 
        cannot be resolved at all outside certain restricted paths, zones 
        that resolve differently based on the origin of the query, and zones 
        that resolve the same globally for all queries from any source.  

        Globally unique names do not equate to globally resolvable names or 
        even global names that resolve the same way from every perspective. 
        Globally unique names do prevent any possibility of collision at the 
        present or in the future and they make DNSSEC trust manageable. 
        Consider using a registered and fully qualified domain name (FQDN) 
        from global DNS as the root for enterprise and other internal 


     Dunbar, et al.                                                 [Page 8] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

     3.5. NAT for Cloud Services 

        Cloud resources, such as VM instances, are usually assigned with 
        private IP addresses. By configuration, some private subnets can 
        have the NAT function to reach out to external network and some 
        private subnets are internal to Cloud only.   

        Different Cloud operators support different levels of NAT functions. 
        For example, AWS NAT Gateway does not currently support connections 
        towards, or from VPC Endpoints, VPN, AWS Direct Connect, or VPC 
        Peering. https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-
        nat-gateway.html#nat-gateway-other-services. AWS Direct 
        Connect/VPN/VPC Peering does not currently support any NAT 

        Google s Cloud NAT allows Google Cloud virtual machine (VM) 
        instances without external IP addresses and private Google 
        Kubernetes Engine (GKE) clusters to connect to the Internet. Cloud 
        NAT implements outbound NAT in conjunction with a default route to 
        allow instances to reach the Internet. It does not implement inbound 
        NAT. Hosts outside of VPC network can only respond to established 
        connections initiated by instances inside the Google Cloud; they 
        cannot initiate their own, new connections to Cloud instances via 

        For enterprises with applications running in different Cloud DCs, 
        proper configuration of NAT have to be performed in Cloud DC and in 
        their own on-premise DC.  


     3.6. Cloud Discovery 

        One of the concerns of using Cloud services is not aware where the 
        resource is actually located, especially Cloud operators can move 
        application instances from one place to another. When applications 
        in Cloud communicate with on-premise applications, it may not be 
        clear where the Cloud applications are located or to which VPCs they 

        It is highly desirable to have tools to discover cloud services in 
        much the same way as you would discover your on-premises 
        infrastructure. A significant difference is that cloud discovery 
        uses the cloud vendor's API to extract data on your cloud services, 
        rather than the direct access used in scanning your on-premises 
     Dunbar, et al.                                                 [Page 9] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

        Standard data models, APIs or tools can alleviate concerns of 
        enterprise utilizing Cloud Resources, e.g. having a Cloud service 
        scan that connects to the API of the cloud provider and collects 
        information directly. 


     4. Interconnecting Enterprise Sites with Cloud DCs 

        Considering that many enterprises already have existing VPNs (e.g. 
        MPLS based L2VPN or L3VPN) interconnecting branch offices & on-
        premises data centers, connecting to Cloud services will be mixed of 
        different types of networks. When an enterprise s existing VPN 
        service providers do not have direct connections to the 
        corresponding cloud DCs that the enterprise prefers to use, the 
        enterprise has to face additional infrastructure and operational 
        costs to utilize Cloud services. 

     4.1. Sites to Cloud DC  

        Most Cloud operators offer some type of network gateway through 
        which an enterprise can reach their workloads hosted in the Cloud 
        DCs. AWS (Amazon Web Services) offers the following options to reach 
        workloads in AWS Cloud DCs: 

          - AWS Internet gateway allows communication between instances in 
             AWS VPC and the internet. 
          - AWS Virtual gateway (vGW) where IPsec tunnels [RFC6071] are 
             established between an enterprise s own gateway and AWS vGW, so 
             that the communications between those gateways can be secured 
             from the underlay (which might be the public Internet).  
          - AWS Direct Connect, which allows enterprises to purchase direct 
             connect from network service providers to get a private leased 
             line interconnecting the enterprises gateway(s) and the AWS 
             Direct Connect routers. In addition, an AWS Transit Gateway can 
             be used to interconnect multiple VPCs in different Availability 
             Zones. AWS Transit Gateway acts as a hub that controls how 
             traffic is forwarded among all the connected networks which act 
             like spokes. 

     Dunbar, et al.                                                [Page 10] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

        Microsoft s ExpressRoute allows extension of a private network to 
        any of the Microsoft cloud services, including Azure and Office365. 
        ExpressRoute is configured using Layer 3 routing. Customers can opt 
        for redundancy by provisioning dual links from their location to two 
        Microsoft Enterprise edge routers (MSEEs) located within a third-
        party ExpressRoute peering location. The BGP routing protocol is 
        then setup over WAN links to provide redundancy to the cloud. This 
        redundancy is maintained from the peering data center into 
        Microsoft's cloud network. 

        Google s Cloud Dedicated Interconnect offers similar network 
        connectivity options as AWS and Microsoft. One distinct difference, 
        however, is that Google s service allows customers access to the 
        entire global cloud network by default. It does this by connecting 
        your on-premises network with the Google Cloud using BGP and Google 
        Cloud Routers to provide optimal paths to the different regions of 
        the global cloud infrastructure. 

        Figure below shows an example of some of a tenant s workloads are 
        accessible via a virtual router connected by AWS Internet Gateway; 
        some are accessible via AWS vGW, and others are accessible via AWS 
        Direct Connect.  

        Different types of access require different level of security 
        functions. Sometimes it is not visible to end customers which type 
        of network access is used for a specific application instance.  To 
        get better visibility, separate virtual routers (e.g. vR1 & vR2) can 
        be deployed to differentiate traffic to/from different cloud GWs. It 
        is important for some enterprises to be able to observe the specific 
        behaviors when connected by different connections. 

        Customer Gateway can be customer owned router or ports physically 
        connected to AWS Direct Connect GW. 

     Dunbar, et al.                                                [Page 11] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

          |    ,---.         ,---. |        
          |   (TN-1 )       ( TN-2)|   
          |    `-+-'  +---+  `-+-' |          
          |      +----|vR1|----+   | 
          |           ++--+        | 
          |            |         +-+----+     
          |            |        /Internet\ For External  
          |            +-------+ Gateway  +----------------------  
          |                     \        / to reach via Internet 
          |                      +-+----+   
          |                        |   
          |    ,---.         ,---. |        
          |   (TN-1 )       ( TN-2)|   
          |    `-+-'  +---+  `-+-' |          
          |      +----|vR2|----+   | 
          |           ++--+        | 
          |            |         +-+----+     
          |            |        / virtual\ For IPsec Tunnel  
          |            +-------+ Gateway  +----------------------  
          |            |        \        /  termination 
          |            |         +-+----+   
          |            |           |   
          |            |         +-+----+              +------+ 
          |            |        /        \ For Direct /customer\ 
          |            +-------+ Gateway  +----------+ gateway  | 
          |                     \        /  Connect   \        / 
          |                      +-+----+              +------+  
          |                        |   
          Figure 1: Examples of Multiple Cloud DC connections. 

     4.2. Inter-Cloud Interconnection 

        The connectivity options to Cloud DCs described in the previous 
        section are for reaching Cloud providers  DCs, but not between cloud 
        DCs. When applications in AWS Cloud need to communicate with 
        applications in Azure, today s practice requires a third-party 
        gateway (physical or virtual) to interconnect the AWS s Layer 2 
        DirectConnect path with Azure s Layer 3 ExpressRoute.  

        Enterprises can also instantiate their own virtual routers in 
        different Cloud DCs and administer IPsec tunnels among them, which 
        by itself is not a trivial task. Or by leveraging open source VPN 
        software such as strongSwan, you create an IPSec connection to the 
        Azure gateway using a shared key. The StrongSwan instance within AWS 
     Dunbar, et al.                                                [Page 12] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

        not only can connect to Azure but can also be used to facilitate 
        traffic to other nodes within the AWS VPC by configuring forwarding 
        and using appropriate routing rules for the VPC.  

        Most Cloud operators, such as AWS VPC or Azure VNET, use non-
        globally routable CIDR from private IPv4 address ranges as specified 
        by RFC1918. To establish IPsec tunnel between two Cloud DCs, it is 
        necessary to exchange Public routable addresses for applications in 
        different Cloud DCs. [BGP-SDWAN] describes one method. Other methods 
        are worth exploring.   

        In summary, here are some approaches, available now (which might 
        change in the future), to interconnect workloads among different 
        Cloud DCs: 

            Utilize Cloud DC provided inter/intra-cloud connectivity 
             services (e.g., AWS Transit Gateway) to connect workloads 
             instantiated in multiple VPCs. Such services are provided with 
             the cloud gateway to connect to external networks (e.g., AWS 
             DirectConnect Gateway).  
            Hairpin all traffic through the customer gateway, meaning all 
             workloads are directly connected to the customer gateway, so 
             that communications among workloads within one Cloud DC must 
             traverse through the customer gateway. 
            Establish direct tunnels among different VPCs (AWS  Virtual 
             Private Clouds) and VNET (Azure s Virtual Networks) via 
             client s own virtual routers instantiated within Cloud DCs. 
             DMVPN (Dynamic Multipoint Virtual Private Network) or DSVPN 
             (Dynamic Smart VPN) techniques can be used to establish direct 
             Multi-point-to-Point or multi-point-to multi-point tunnels 
             among those client s own virtual routers. 
        Approach a) usually does not work if Cloud DCs are owned and managed 
        by different Cloud providers.  

        Approach b) creates additional transmission delay plus incurring 
        cost when exiting Cloud DCs. 

        For the Approach c), DMVPN or DSVPN use NHRP (Next Hop Resolution 
        Protocol) [RFC2735] so that spoke nodes can register their IP 
        addresses & WAN ports with the hub node. The IETF ION 
        (Internetworking over NBMA (non-broadcast multiple access) WG 
        standardized NHRP for connection-oriented NBMA network (such as ATM) 
        network address resolution more than two decades ago.  
     Dunbar, et al.                                                [Page 13] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

        There are many differences between virtual routers in Public Cloud 
        DCs and the nodes in an NBMA network. NHRP cannot be used for 
        registering virtual routers in Cloud DCs unless an extension of such 
        protocols is developed for that purpose, e.g. taking NAT or dynamic 
        addresses into consideration. Therefore, DMVPN and/or DSVPN cannot 
        be used directly for connecting workloads in hybrid Cloud DCs. 

        Other protocols such as BGP can be used, as described in [BGP-


     5. Problems with MPLS-based VPNs extending to Hybrid Cloud DCs 

        Traditional MPLS-based VPNs have been widely deployed as an 
        effective way to support businesses and organizations that require 
        network performance and reliability. MPLS shifted the burden of 
        managing a VPN service from enterprises to service providers. The 
        CPEs attached to MPLS VPNs are also simpler and less expensive, 
        because they do not need to manage routes to remote sites; they 
        simply pass all outbound traffic to the MPLS VPN PEs to which the 
        CPEs are attached (albeit multi-homing scenarios require more 
        processing logic on CPEs).  MPLS has addressed the problems of 
        scale, availability, and fast recovery from network faults, and 
        incorporated traffic-engineering capabilities. 

        However, traditional MPLS-based VPN solutions are sub-optimized for 
        connecting end-users to dynamic workloads/applications in cloud DCs 

          - The Provider Edge (PE) nodes of the enterprise s VPNs might not 
             have direct connections to third party cloud DCs that are used 
             for hosting workloads with the goal of providing an easy access 
             to enterprises  end-users.   

          - It takes some time to deploy provider edge (PE) routers at new 
             locations. When enterprise s workloads are changed from one 
             cloud DC to another (i.e., removed from one DC and re-
             instantiated to another location when demand changes), the 
             enterprise branch offices need to be connected to the new cloud 
             DC, but the network service provider might not have PEs located 
             at the new location.    

     Dunbar, et al.                                                [Page 14] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

             One of the main drivers for moving workloads into the cloud is 
             the widely available cloud DCs at geographically diverse 
             locations, where apps can be instantiated so that they can be 
             as close to their end-users as possible. When the user base 
             changes, the applications may be migrated to a new cloud DC 
             location closest to the new user base.  

          - Most of the cloud DCs do not expose their internal networks. An 
             enterprise with a hybrid cloud deployment can use an MPLS-VPN 
             to connect to a Cloud provider at multiple locations.  The 
             connection locations often correspond to gateways of different 
             Cloud DC locations from the Cloud provider.  The different 
             Cloud DCs are interconnected by the Cloud provider's own 
             internal network.  At each connection location (gateway), the 
             Cloud provider uses BGP to advertise all of the prefixes in the 
             enterprise's VPC, regardless of which Cloud DC a given prefix 
             is actually in. This can result in inefficient routing for the 
             end-to-end data path. 

        Another roadblock is the lack of a standard way to express and 
        enforce consistent security policies for workloads that not only use 
        virtual addresses, but in which are also very likely hosted in 
        different locations within the Cloud DC [RFC8192]. The current VPN 
        path computation and bandwidth allocation schemes may not be 
        flexible enough to address the need for enterprises to rapidly 
        connect to dynamically instantiated (or removed) workloads and 
        applications regardless of their location/nature (i.e., third party 
        cloud DCs).   

     6. Problem with using IPsec tunnels to Cloud DCs 
        As described in the previous section, many Cloud operators expose 
        their gateways for external entities (which can be enterprises 
        themselves) to directly establish IPsec tunnels. Enterprises can 
        also instantiate virtual routers within Cloud DCs to connect to 
        their on-premises devices via IPsec tunnels.   

     6.1. Scaling Issues with IPsec Tunnels 

        If there is only one enterprise location that needs to reach the 
        Cloud DC, an IPsec tunnel is a very convenient solution.  
     Dunbar, et al.                                                [Page 15] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

        However, many medium-to-large enterprises have multiple sites and 
        multiple data centers. For multiple sites to communicate with 
        workloads and apps hosted in cloud DCs, Cloud DC gateways have to 
        maintain many IPsec tunnels to all those locations. In addition, 
        each of those IPsec Tunnels requires pair-wise periodic key 
        refreshment. For a company with hundreds or thousands of locations, 
        there could be hundreds (or even thousands) of IPsec tunnels 
        terminating at the cloud DC gateway, which is very processing 
        intensive. That is why many cloud operators only allow a limited 
        number of (IPsec) tunnels & bandwidth to each customer.   

        Alternatively, you could use a solution like group encryption where 
        a single IPsec SA is necessary at the GW but the drawback is key 
        distribution and maintenance of a key server, etc. 

     6.2. Poor performance over long distance 

        When enterprise CPEs or gateways are far away from cloud DC gateways 
        or across country/continent boundaries, performance of IPsec tunnels 
        over the public Internet can be problematic and unpredictable. Even 
        though there are many monitoring tools available to measure delay 
        and various performance characteristics of the network, the 
        measurement for paths over the Internet is passive and past 
        measurements may not represent future performance.   

        Many cloud providers can replicate workloads in different available 
        zones. An App instantiated in a cloud DC closest to clients may have 
        to cooperate with another App (or its mirror image) in another 
        region or database server(s) in the on-premises DC. This kind of 
        coordination requires predicable networking behavior/performance 
        among those locations.  


     7. Problems of Using SD-WAN to connect to Cloud DCs 
        SD-WAN lets enterprises augment their current VPN network with cost-
        effective, readily available Broadband Internet connectivity, 
        enabling some traffic offloading to paths over the Internet 
        according to differentiated, possibly application-based traffic 
        forwarding policies, or when the MPLS VPN connection between the two 
        locations is congested, or otherwise undesirable or unavailable. 

     Dunbar, et al.                                                [Page 16] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

     7.1. More Complexity to Edge Nodes 

        Augmenting transport path is not as simple as it appears. For an 
        enterprise with multiple sites, CPE managed overlay paths among 
        sites requires each CPE to manage all the addresses that local hosts 
        have potential to reach, i.e., map internal VPN addresses to 
        appropriate Overlay paths. This is similar to the complexity of 
        Frame Relay based VPNs, where each CPE needed to maintain mesh 
        routing for all destinations if they were to avoid an extra hop 
        through a hub router. Even with the  assistance from a central 
        controller (instead of running a routing protocol) to resolve the 
        mapping between destinations and SD-WAN paths, SD-WAN CPEs are still 
        responsible for routing table maintenance as remote destinations 
        change their attachments, e.g., the dynamic workload in other DCs 
        are de-commissioned or added. 

        In addition, overlay path for interconnecting branch offices are 
        different from connecting to Cloud DCs:  

          - Overlay path interconnecting branch offices usually have two 
             end-points (e.g. CPEs) controlled by one entity (e.g. 
             controllers or management systems operated by the enterprise).  
          - Connecting to Cloud DC may consists of CPEs owned or managed by 
             the enterprise, and the remote end-points being managed or 
             controlled by Cloud DCs.  

     7.2. Edge WAN Port Management 

             An SDWAN edge node can have WAN ports connected to different 
             networks or public internet managed by different operators. 
             There is therefore a need to propagate WAN port property to 
             remote authorized peers in third party network domains in 
             addition to route propagation. Such an exchange cannot happen 
             before communication between peers is properly secured. 

     Dunbar, et al.                                                [Page 17] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

     7.3. Forwarding based on Application 
          Forwarding based on application IDs instead of based on 
          destination IP addresses is often referred to as Application based 
          Segmentation. If the Applications have unique IP addresses, then 
          the Application Based Segmentation can be achieved by propagating 
          different BGP UPDATE messages to different nodes, as described in 
          [BGP-SDWAN-USAGE]. If the Application cannot be uniquely 
          identified by the IP addresses, more work is needed.  
     8. End-to-End Security Concerns for Data Flows 

          When IPsec tunnels established from enterprise on-premises CPEs 
          are terminated at the Cloud DC gateway where the workloads or 
          applications are hosted, some enterprises have concerns regarding 
          traffic to/from their workload being exposed to others behind the 
          data center gateway (e.g., exposed to other organizations that 
          have workloads in the same data center).  
          To ensure that traffic to/from workloads is not exposed to 
          unwanted entities, IPsec tunnels may go all the way to the 
          workload (servers, or VMs) within the DC.  

     9. Requirements for Dynamic Cloud Data Center VPNs 

        In order to address the aforementioned issues, any solution for 
        enterprise VPNs that includes connectivity to dynamic workloads or 
        applications in cloud data centers should satisfy a set of 

          - The solution should allow enterprises to take advantage of the 
             current state-of-the-art in VPN technology, in both traditional 
             MPLS-based VPNs and IPsec-based VPNs (or any combination 
             thereof) that run over the public Internet. 
          - The solution should not require an enterprise to upgrade all 
             their existing CPEs.  
          - The solution should support scalable IPsec key management among 
             all nodes involved in DC interconnect schemes. 
          - The solution needs to support easy and fast, on-the-fly, VPN 
             connections to dynamic workloads and applications in third 

     Dunbar, et al.                                                [Page 18] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

             party data centers, and easily allow these workloads to migrate 
             both within a data center and between data centers. 
          - Allow VPNs to provide bandwidth and other performance 
          - Be a cost-effective solution for enterprises to incorporate 
             dynamic cloud-based applications and workloads into their 
             existing VPN environment. 


     10. Security Considerations 

        The draft discusses security requirements as a part of the problem 
        space, particularly in sections 4, 5, and 8. 

        Solution drafts resulting from this work will address security 
        concerns inherent to the solution(s), including both protocol 
        aspects and the importance (for example) of securing workloads in 
        cloud DCs and the use of secure interconnection mechanisms. 

     11. IANA Considerations 

        This document requires no IANA actions. RFC Editor: Please remove 
        this section before publication. 

     12. References 

     12.1. Normative References 

     12.2. Informative References 

        [RFC2735]   B. Fox, et al  NHRP Support for Virtual Private 
        networks . Dec. 1999. 

        [RFC8192] S. Hares, et al  Interface to Network Security Functions 
                  (I2NSF) Problem Statement and Use Cases , July 2017 

     Dunbar, et al.                                                [Page 19] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

         [ITU-T-X1036] ITU-T Recommendation X.1036,  Framework for creation, 
                  storage, distribution and enforcement of policies for 
                  network security , Nov 2007.  

         [RFC6071] S. Frankel and S. Krishnan,  IP Security (IPsec) and 
                  Internet Key Exchange (IKE) Document Roadmap , Feb 2011.  

        [RFC4364] E. Rosen and Y. Rekhter,  BGP/MPLS IP Virtual Private 
                  Networks (VPNs) , Feb 2006 

        [RFC4664] L. Andersson and E. Rosen,  Framework for Layer 2 Virtual 
                  Private Networks (L2VPNs) , Sept 2006.  

        [BGP-SDWAN] L. Dunbar, et al.  BGP Extension for SDWAN Overlay 
                  Networks , draft-dunbar-idr-bgp-sdwan-overlay-ext-03, 
                  work-in-progress, Nov 2018.  

     13. Acknowledgments 

        Many thanks to Alia Atlas, Chris Bowers, Paul Vixie, Paul Ebersman, 
        Timothy Morizot, Ignas Bagdonas, Michael Huang, Liu Yuan Jiao, 
        Katherine Zhao, and Jim Guichard for the discussion and 

     Dunbar, et al.                                                [Page 20] 

     Internet-Draft        Net2Cloud Problem Statement            March 2020 

     Authors  Addresses 
        Linda Dunbar 
        Email: Linda.Dunbar@futurewei.com 
        Andrew G. Malis 
        Email: agmalis@gmail.com 
        Christian Jacquenet 
        Rennes, 35000 
        Email: Christian.jacquenet@orange.com 
        Mehmet Toy 
        One Verizon Way 
        Basking Ridge, NJ 07920 
        Email: mehmet.toy@verizon.com  

     Dunbar, et al.                                                [Page 21]