Scaling the Address Resolution Protocol for Large Data Centers (SARP)
draft-nachum-sarp-02
The information below is for an old version of the document.
| Document | Type |
This is an older version of an Internet-Draft that was ultimately published as RFC 7586.
|
|
|---|---|---|---|
| Authors | Youval Nachum , Linda Dunbar , Tal Mizrahi , Ilan Yerushalmi | ||
| Last updated | 2012-06-03 | ||
| RFC stream | (None) | ||
| Formats | |||
| IETF conflict review | conflict-review-nachum-sarp, conflict-review-nachum-sarp, conflict-review-nachum-sarp, conflict-review-nachum-sarp, conflict-review-nachum-sarp, conflict-review-nachum-sarp, conflict-review-nachum-sarp, conflict-review-nachum-sarp | ||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | Became RFC 7586 (Experimental) | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-nachum-sarp-02
INTAREA Working Group Youval Nachum
Internet Draft Marvell
Intended status: Proposed Standard Linda Dunbar
Expires: December 2012 Huawei
Tal Mizrahi
Ilan Yerushalmi
Marvell
June 4, 2012
Scaling the Address Resolution Protocol for Large Data Centers
(SARP)
draft-nachum-sarp-02.txt
Abstract
This document provides a recommended architecture and network
operation named SARP. SARP is based on fast proxies that
significantly reduce broadcast domains and ARP/ND broadcast
transmissions. SARP supports smooth and fast virtual machine (VM)
mobility without any modification to the VM, while keeping the
connection up and running efficiently. SARP is targeted for massive
scaling data centers with a significant number of VMs using ARP and
ND protocols.
Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on December 4, 2012.
Nachum, et al. Expires December 4, 2012 [Page 1]
Internet-Draft SARP June 2012
Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction ................................................. 3
1.1. SARP Motivation.......................................... 3
1.2. SARP Overview ........................................... 3
1.3. SARP Deployment Options ................................. 4
2. Terms and Abbreviations Used in this Document ................ 5
3. SARP Description ............................................. 6
3.1. Control Plane: ARP/ND ................................... 6
3.1.1. ARP/ND Request for a Local VM ...................... 6
3.1.2. ARP/ND Request for a Remote VM ..................... 6
3.2. Data Plane: Packet Transmission ......................... 7
3.2.1. Local Packet Transmission .......................... 7
3.2.2. Packet Transmission Between Sites .................. 7
3.3. VM Migration ............................................ 7
3.3.1. VM Local Migration ................................. 8
3.3.2. VM Migration from One Site to Another .............. 8
3.3.2.1. Impact to ARP/ND Table of VMs being moved ..... 9
3.4. Multicast and Broadcast ................................ 10
3.5. Non IP packet .......................................... 10
3.6. ARP caching ............................................ 10
3.7. High availability and load balancing ................... 11
3.8. SARP Interaction with Overlay networks ................. 12
4. Conclusions ................................................. 12
5. Security Considerations ..................................... 13
6. IANA Considerations ......................................... 13
7. References .................................................. 13
7.1. Normative References ................................... 13
7.2. Informative References ................................. 13
8. Acknowledgments ............................................. 14
Nachum, et al. Expires December 4, 2012 [Page 2]
Internet-Draft SARP June 2012
1. Introduction
1.1. SARP Motivation
SARP provides operational recommendations for network in data
center(s) with a large number of virtual Machines which can migrate
from one location to another without changing their IP/MAC addresses.
[ARMD] has documented various impacts and scaling issues to data
center networks, especially to L2/L3 boundary routers, when large
number of VMs can move without changing their MAC/IP addresses.
1.2. SARP Overview
SARP is a type of proxies that constrain the ARP/ND
broadcast/multicast messages to small segments regardless how wide
their corresponding Layer 2 domain spread.
In order to enable VMs to be moved across greater number of servers
while maintaining their MAC/IP addresses unchanged, the layer-2
network (e.g. VLAN) which interconnect those VMs may spread across
different server racks, different rows of server racks, or even
different data centers.
For ease of description, let's break the entire network which
interconnects all those VMs into two segments: interconnecting
segment and "access" segments. While the "Access" network is mostly
likely Layer 2, the "interconnecting" segment might be not.
The SARP proxies are located at the boundaries where the "Access"
segment connects to its "Interconnecting" segment. The boundary node
could be a Hypervisor virtual switch, a Top of Rack switch, an
Aggregation switch (or end of row switch), or a data center core
switch. Figure 1 depicts an example of two remote data centers that
are managed as a single flat Layer 2 domain. SARP proxies are
implemented at the edge devices connecting the data center to the
transport network. SARP significantly reduces the ARP/ND
transmissions over the "interconnection" network. The ARP/ND
broadcast/multicast messages are bounded by the SARP proxies.
Nachum, et al. Expires December 4, 2012 [Page 3]
Internet-Draft SARP June 2012
*-------------------*
| |
+-------| Interconnect |-------+
| | | |
| *-------------------* |
| |
*-----------------* *----------------*
| SARP Proxies | | SARP Proxies |
*-----------------* *----------------*
| | | |
*-------* *-------* *-------* *-------*
| ACC | | ACC | | ACC | | ACC |
*-------* *-------* *-------* *-------*
|
*----------*
|Hypervisor|
*----------*
|
*--------*
|Virtual |
|Machine |
*--------*
(West Site) (East Site)
Figure 1 SARP Networking Architecture Example.
1.3. SARP Deployment Options
SARP deployment is tightly coupled with the data center architecture.
SARP proxies are located at the point where the Layer 2
infrastructure connects to its Layer 2 cloud using overlay networks.
SARP proxies can be located at the data center edge (as Figure 1
depicts), data center core, or data center aggregation. SARP can also
be implemented by the hypervisor (as Figure 2 depicts).
To simplify the description, we will focus on data centers that are
managed as a single flat Layer 2 network, where SARP proxies are
located at the boundary where the data center connects to the
transport network (as Figure 1 depicts).
Nachum, et al. Expires December 4, 2012 [Page 4]
Internet-Draft SARP June 2012
*-------------------*
| |
+-------| TRANSPORT |-------+
| | | |
| *-------------------* |
| |
*-----------------* *----------------*
| Edge Device | | Edge Device |
*-----------------* *----------------*
| |
*-----------------* *----------------*
| Core | | Core |
*-----------------* *----------------*
| | | |
*-------* *-------* *-------* *-------*
| Agg | | Agg | | Agg | | Agg |
*-------* *-------* *-------* *-------*
|
*----------*
|Hypervisor|
*----------*
(West Site) (East Site)
Figure 2 SARP deployment options.
2. Terms and Abbreviations Used in this Document
ARP: Address Resolution Protocol
FIB: Forwarding Information Base
IP-D: IP address of the destination virtual machine
IP-S: IP address of the source virtual machine
MAC-D: MAC address of the destination virtual machine
MAC-E: MAC address of the East Proxy SARP Device
MAC-S: MAC address of the source virtual machine
ND: Neighbor Discovery
SARP Proxy: The components that participates in the SARP protocol.
Nachum, et al. Expires December 4, 2012 [Page 5]
Internet-Draft SARP June 2012
VM: Virtual Machine
3. SARP Description
3.1. Control Plane: ARP/ND
This section describes the ARP/ND procedure scenarios. In the first
scenario, VMs share the same Access Segment. In the second scenario,
the source VM is local Access Segment and the destination VM is
located at the remote Access Segment.
In all scenarios, the VMs (source and destination) share the same L2
broadcast domain.
3.1.1. ARP/ND Request for a Local VM
When source and destination VMs are located at the same Access
Segment, the Address Resolution process is as described in [ARP] and
[ND]. When the VM sends an ARP request to learn the IP to MAC mapping
of another local VM, it receives a reply from the other local VM with
the IP-D to MAC-D mapping.
3.1.2. ARP/ND Request for a Remote VM
When the source and destination VMs are located at different Access
Segments, the Address Resolution process is as follows.
In our example, the source VM is located at the west Access Segment
and the destination VM is located at the east Access Segment.
When the source VM sends an ARP/ND request to find out the IP to MAC
mapping of a remote VM, if the local SARP proxy doesn't have the ARP
cache for the target IP address or the cache entry has expired, the
ARP request is propagated to all Access Segments which might have VMs
in the same virtual network as the originating VM, including the east
Access Segment.
The destination VM responds to the ARP/ND request and transmits an
ARP/ND reply having the IP-D to MAC-D mapping.
The east SARP proxy functions as the proxy ARP of its Local VMs. The
east SARP proxy modifies the ARP reply message to be MAC-D to MAC-E
and forwards the modified ARP reply message to all the SARP proxies.
Nachum, et al. Expires December 4, 2012 [Page 6]
Internet-Draft SARP June 2012
The West SARP Proxy forwards the modified ARP reply message to the
source VM.
The west SARP proxy can also functions as an ARP cache of the Remote
VMs. By doing so, it significantly reduces the volume of the ARP/ND
transmission over the network.
3.2. Data Plane: Packet Transmission
3.2.1. Local Packet Transmission
When a VM transmits packets to a destination VM that is located at
the same site, there is no change in the data plane. The packets are
sent from (IP-S, MAC-S) to (IP-D, MAC-D).
3.2.2. Packet Transmission Between Sites
Packets that are sent between sites traverse the SARP proxy of both
sites. In our example, all packets sent from the VM located at the
west site to the destination VM located at the east site traverse the
west SARP proxy and the east SARP proxy.
The source VM follows its ARP table and sends packets to (IP-D, MAC-
E) destination addresses and with (IP-s, MAC-S) as the source
addresses.
The west SARP proxy replaces the packet source address to its own
source address (MAC-W), keeps the destination address to be (MAC-E),
and forwards the packet to the east proxy SARP.
When the east proxy SARP receives the packet, it replaces the
destination MAC address to be (MAC-D) based on the packet destination
IP (i.e., IP-D), but it does not change the source MAC addresses.
When the destination VM receives the packet, the Source Address field
would be the MAC address of the west side SARP proxy,
Noted: it is common for data center network to have security policies
to enforce some VMs can communicate with each other, and some VMs
can't. Most likely, those policies are configured by VM's IP
addresses. Even though the originating VM's MAC address is lost when
the packet arrives at the destination VM, the originating VM's IP
address is still present in the data packets for security policy to
be enforced.
3.3. VM Migration
Nachum, et al. Expires December 4, 2012 [Page 7]
Internet-Draft SARP June 2012
3.3.1. VM Local Migration
When a VM migrates locally within its Access segment, the SARP
protocol is not required to perform any action. VM migration is
resolved entirely by the Layer 2 mechanisms.
3.3.2. VM Migration from One Site to Another
In our example, the VM migrates from the west site to the east site
while maintaining its MAC and IP addresses.
VM migration might affect networking elements based on their
respective location:
- Origin site (west site)
- Destination site (east site)
- Other sites
Origin site:
The Origin site is the site where the VM is before migration. It is
the west site in our example.
Before the VM (IP=IP-D, MAC=MAC-D) is moved, all VMs at the west site
that have an ARP entry of IP-D in their ARP table have the (IP-D to
MAC-D) mapping. VMs on any other "Access Segments" will have ARP
entry of (IP-D to MAC-W) mapping where MAC-W is the MAC address of
the SARP proxy on the West Access Segment.
After the VM (IP-D) in the West Site moves to East Site, if there is
gratuitous ARP sent out by the destination hypervisor for the VM (IP-
D), then the ARP mapping on VMs on all Access Segments will be
updated by (IP-D to MAC-E) where MAC-E is the MAC address of the SARP
proxy on the East Site . If there isn't any gratuitous ARP sent out
by the destination hypervisor, the ARP mapping on the VMs in west
site (and other sites) will eventually aged out.
Until ARP tables are updated, the source VMs from the west site
continue sending packets to MAC-D. Switches at the west site are
still configured with the old location of MAC-D. This can be resolved
by VM manager sending out a fake gratuitous ARP on behalf of
destination Hypervisor, MAC table aging or by redirecting the packets
to the proxy SARP of the west site.
Destination Site:
Nachum, et al. Expires December 4, 2012 [Page 8]
Internet-Draft SARP June 2012
The destination site is the site to which the VM migrated, the east
site in our example.
Before any gratuitous ARP messages are sent out by the destination
hypervisor, all VMs at the east site (and all other sites) that have
an ARP entry of IP-D in their ARP table have the (IP-D to MAC-W)
mapping. ARP mapping is updated by aging or by a gratuitous ARP
message sent by the destination hypervisor. Until ARP tables are
updated, the source VMs from the east site continue to send packets
to MAC-W. This can be resolved by VM manager sending out a fake
gratuitous ARP immediately after the VM migration, or redirecting the
packets from the SARP proxy of the east site to the migrated VM by
updating the destination MAC of the packets to MAC-D.
Other Sites:
All VMs at the other sites that have an ARP entry of IP-D in their
ARP table have the (IP-D to MAC-W) mapping. ARP mapping is updated by
aging or by a gratuitous ARP message sent by the destination
hypervisor of the migrated VM and modified by the SARP proxy of the
east site (IP-D to MAC-E) mapping. Until ARP tables are updated, the
source VMs from the west site continue sending packets to MAC-W. This
can be resolved by redirecting the packets from the SARP proxy of the
west site to the SARP proxy of the east site by updating the
destination MAC of the packets to MAC-E.
3.3.2.1. Impact to ARP/ND Table of VMs being moved
When a VM (IP-D) is moved from one site to another site, its ARP
entries for VMs located at the other sites (i.e. neither east site
nor west site) are unaffected by the migration.
The VM (IP-D)'s ARP entries (i.e. IP to MAC mapping) for VMs located
at east site can be kept with no change until the ARP aging time
since they are mapped to MAC-E. All traffic originated from the VM
(IP-D) in its new location to VMs located at the east site traverses
the SARP proxy of the east Site. This can be mitigated by ARP
advertisement sent by the SARP proxy of the east site or by the
hypervisor.
The VM (IP-D)'s ARP entries (i.e. IP to MAC mapping) for VMs located
at west sites will not be changed either until the ARP entries age
out or new data frames are received from the remote sites. Since all
MAC addresses of the VMs located at the west site are unknown at the
east site. All unknown traffic from the VM is intercepted by the SARP
proxy of the east site and forwarded to the SARP proxy of the west
site (just for ARP aging time). This can be resolved by the east SARP
Nachum, et al. Expires December 4, 2012 [Page 9]
Internet-Draft SARP June 2012
proxy having mapping entries for VMs in the west side. Upon receiving
unknown packets, it can update the migrating VM with the new IP to
MAC mapping by sending a modified gratuitous ARP with (IP-D to MAC-W)
mapping.
Note that overlay networks providing the Layer 2 network
virtualization services configure their Edge Device MAC aging timers
to be greater than the ARP request interval.
3.4. Multicast and Broadcast
To be added in a future version of this document
3.5. Non IP packet
To be added in a future version of this document
3.6. ARP caching
ARP/ND Requests for a VM located at a remote site require flooding
messages over the interconnecting network to all sites which have
enabled the virtual network on which the VM belongs to. This
scenario is described in details at 3.1.2. In such cases, SARP
caching can reduce the number of ARP/ND transmissions over
interconnecting networks.
In the example presented at section 3.1.2. the source VM is located
at the west site and the destination VM is located at the east site.
When the source VM sends an ARP/ND request to discover the IP to MAC
mapping of the remote VM, the ARP/ND request can be intercepted by
the west SARP proxy.
The west SARP proxy learns or refreshes the source IP to source MAC
mapping and looks up the IP to MAC translation of the destination IP.
If the destination IP entry is found and is valid, the west SARP
proxy replies with an ARP/ND reply without propagating the packet to
other sites. Otherwise, the packet is propagated to all sites which
have the virtual network enabled including the east site.
The propagated ARP/ND request is intercepted again by the east SARP
proxy. It learns or refreshes the source IP to source MAC mapping and
looks up the destination IP to MAC translation. If the destination IP
entry is found and is valid the SARP proxy replies with an ARP/ND
reply without propagating the ARP request to the east site.
Otherwise, the ARP/ND request is broadcasted within the east site.
Nachum, et al. Expires December 4, 2012 [Page 10]
Internet-Draft SARP June 2012
The destination VM responds to the ARP/ND request and transmits an
ARP/ND reply having the IP-D to MAC-D mapping.
The east side SARP proxy intercepts the ARP/ND reply and learns or
refreshes the Destination IP to Destination MAC mapping and
propagates the ARP/ND reply to the west SARP proxy.
The West SARP Proxy intercepts the ARP/ND reply and learns or
refreshes the Destination IP to Destination MAC mapping and
propagates the ARP reply to the source VM.
The SARP proxies maintain an ARP caching table of IP to MAC mapping
for all recent ARP/ND requests and replies. This table allows the
SARP proxy to respond with low latency to the ARP/ND requests sent
locally and avoid the broadcast transmissions of such requests over
the transport network and all over the broadcast domains at the
remote sites.
3.7. High availability and load balancing
The SARP proxy is located at the boundary where the local Layer 2
infrastructure connects to the interconnecting network. All traffic
from the local site to the remote sites traverses the SARP proxy. The
SARP proxy is subject to high availability and bandwidth
requirements.
The SARP architecture supports multiple SARP proxies connecting a
single site to the transport network. In SARP architecture all
proxies can be active and can backup one another. The SARP
architecture is robust and allows the network administrator to
allocate proxies according to the bandwidth and high availability
requirements.
Traffic is segregated between SARP proxies by using VLANs. An SARP
proxy is the Master-SARP proxy of a set of VLANs and the Backup-SARP
proxy of another set of VLANs.
For example the SARP proxies of the west site (as Figure 1 depicts)
are SARP proxy-1 and SARP proxy-2. The west site supports VLAN-1 and
VLAN-2 while SARP proxy-1 is the Master SARP proxy of VLAN-1 and the
Backup proxy of VLAN-2 and SARP proxy-2 is the Master SARP proxy of
VLAN-2 and the Backup SARP proxy of VLAN-1. Both proxies are members
of VLAN-1 and VLAN-2.
Nachum, et al. Expires December 4, 2012 [Page 11]
Internet-Draft SARP June 2012
The Master SARP proxy updates its Backup proxy with all the ARP reply
messages. The Backup SARP proxy maintains a backup database to all
the VLANs that it is the Backup SARP proxy.
The Master and the Backup SARP proxies maintain a keepalive
mechanism. In case of a failure the Backup proxy becomes the Master
SARP proxy. The failure decision is per VLAN. When the Master and
the Backup proxies switchover, the backup SARP proxy can use the MAC
address of the Master SARP proxy. The backup SARP proxy sends locally
a gratuitous ARP message with the MAC address of the Master SARP
proxy to update the forwarding tables on the local switches. The
backup SARP proxy also updates the remote SARP proxies on the change.
3.8. SARP Interaction with Overlay networks
SARP interaction with overlay networks providing L2 network
virtualization (such as IP, VPLS, Trill, OTV, NVGRE and VxLAN) is
efficient and scalable.
The mapping of SARP to overlay networks is straightforward. The VM
does the destination IP to SARP proxy MAC mapping. The mapping of the
proxy MAC to its correct tunnel is done by the overlay networks. SARP
significantly scales down the complexity of the overlay networks and
transport networks by reducing the mapping tables to the number of
SARP proxies.
4. Conclusions
SARP distributes the Layer 2 Forwarding Information Base (FIB) from
the edge devices (functioning as SARP proxies) to the VMs. By doing
so, it significantly reduces table sizes on the edge devices. The
source VM maintains the mapping of its destination VMs to the
destination site/cloud in the ARP table. The destination VM IP is
translated to the destination MAC address of the SARP proxy at the
destination site. The SARP proxies only maintain Layer 2 FIB of local
VMs and remote edge devices.
SARP proxies can support FAST VM migration and provide minimum
transition phase. When SARP proxy indicates or is informed of VM
migration, it can update all its peers and trigger a fast update.
SARP seamlessly supports Layer 2 network virtualization services over
the overlay network and significantly reduces their complexity in
terms of table size and performance. The overlay networks are only
required to map MAC addresses of the SARP proxies to the correct
tunnel.
Nachum, et al. Expires December 4, 2012 [Page 12]
Internet-Draft SARP June 2012
5. Security Considerations
The SARP proxies are located at the boundaries where the local Layer
2 infrastructure connects to its Layer 2 cloud. The SARP proxies
interoperate with overlay network protocols that extend the Layer-2
subnet across data centers or between different systems within a
datacenter.
SARP control plane and data plane are traversed by the overlay
network hence SARP does not expose the network to additional security
threats.
SARP proxies may be exposed to Denial of Service (DoS) attacks by
means of ARP/ND message flooding. Thus, the SARP proxies must have
sufficient resources to support the SARP control plane without making
the network more vulnerable to DoS than without SARP proxies.
SARP adds security to the data plane by hiding all the local layer 2
MAC addresses from potential attacker located at the remote clouds.
The only MAC addresses that are exposed at remote sites are the MAC
addresses of the SARP proxies.
6. IANA Considerations
There are no IANA actions required by this document.
RFC Editor: please delete this section before publication.
7. References
7.1. Normative References
[ARP] Plummer, D., "An Ethernet Address Resolution Protocol",
RFC 826, November 1982.
[ND] Narten, T., Nordmark, E., Simpson, W., and H. Soliman,
"Neighbor Discovery for IP version 6 (IPv6)", RFC
4861, September 2007.
7.2. Informative References
[ARMD] Narten, T., Karir, M., Foo, I., " Problem Statement for
ARMD", draft-ietf-armd-problem-statement, February
2012.
Nachum, et al. Expires December 4, 2012 [Page 13]
Internet-Draft SARP June 2012
8. Acknowledgments
This document was prepared using 2-Word-v2.0.template.dot.
Nachum, et al. Expires December 4, 2012 [Page 14]
Internet-Draft SARP June 2012
Authors' Addresses
Youval Nachum
Marvell
6 Hamada St.
Yokneam, 20692 Israel
Email: youvaln@marvell.com
Linda Dunbar
Huawei Technologies
5430 Legacy Drive, Suite #175
Plano, TX 75024, USA
Phone: (469) 277 5840
Email: ldunbar@huawei.com
Tal Mizrahi
Marvell
6 Hamada St.
Yokneam, 20692 Israel
Email: talmi@marvell.com
Ilan Yerushalmi
Marvell
6 Hamada St.
Yokneam, 20692 Israel
Email: yilan@marvell.com
Nachum, et al. Expires December 4, 2012 [Page 15]