Network Working Group Y. Gu
Internet-Draft Huawei
Intended status: Standards Track March 7, 2011
Expires: September 8, 2011
Policies and dynamic data migration in DC
draft-gu-dc-management-problem-statement-00
Abstract
Virtualization provides Data Center with feasibility and improves the
utilization of limited physical resource, e.g. switches/routers,
servers and links. Virtual machines (VM) are allowed to migrate to
any place in the Data Center. A variety of policies (e.g. ACL,
firewalls, load balancers, IPS and QoS) are deployed in Data Center
to guarantee the SLA the provider signed with their clients. Dynamic
information, such as TCP Connection Table, dynamic ACLs and cumulated
data, is generated on network devices. In order to keep running
services uninterrupted while VM migrating, relevant policies and
dynamic information, also need to migrate with VM.
This document describes some examples of the policies and dynamic
information that need to migrate with VM, the influence if they are
not migrated with VM, the problems that need to consider when migrate
polices and dynamic information. It also describes some existing
network management protocols standardized by IETF and the advantages
and disadvantages of them for operating policies and dynamic
information migration respectively. The goal is to justify that it
is necessary for IETF to make effort on policy and dynamic
information migration for large virtualized Data Center.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 8, 2011.
Gu Expires September 8, 2011 [Page 1]
Internet-Draft Policies and dynamic data migration in DC March 2011
Copyright Notice
Copyright (c) 2011 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Terminologies and concepts . . . . . . . . . . . . . . . . . . 4
3. Policies on several network devices . . . . . . . . . . . . . 4
3.1. Policies and configurations . . . . . . . . . . . . . . . 5
4. Dynamic Information and the influence if lack or unaccurate . 6
4.1. TCP connection tables . . . . . . . . . . . . . . . . . . 7
4.1.1. If TCP Connection Table isn't migrated . . . . . . . . 7
4.1.2. If TCP Connection Table is not accurately migrated . . 8
4.2. Cumulated data . . . . . . . . . . . . . . . . . . . . . . 9
4.3. Dynamic ACLs . . . . . . . . . . . . . . . . . . . . . . . 9
4.4. DHCP Snooping . . . . . . . . . . . . . . . . . . . . . . 10
4.5. Multicast Membership . . . . . . . . . . . . . . . . . . . 10
5. Existing network management protocol and the limitations . . . 10
5.1. Limitations . . . . . . . . . . . . . . . . . . . . . . . 10
5.1.1. For Policies migration . . . . . . . . . . . . . . . . 10
5.1.2. For Dynamic Information Migration . . . . . . . . . . 11
6. Security Considerations . . . . . . . . . . . . . . . . . . . 11
7. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 11
8. Normative Reference . . . . . . . . . . . . . . . . . . . . . 11
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 11
Gu Expires September 8, 2011 [Page 2]
Internet-Draft Policies and dynamic data migration in DC March 2011
1. Introduction
Data centers can host tens or even thousands of different
applications. Some are simple applications such as web servers
providing static content, while some may be very complex, e.g.
e-commerce, that requiring all around privacy protection and data
security. Clients of Data Center, unlike server hosting clients,
raise more strict QoS and Security requirements. Clients may sign
Service Level Agreement (SLA) with Data Center Provider to make sure
their requirement can be guaranteed. To satisfy different level of
security requirements and to manage and improve the performance of
these applications, data centers typically deploy a large variety of
middleboxes, including firewalls, load balancers, SSL offloaders, web
caches, and intrusion prevention boxes.
To satisfy QoS requirements, Data Center also implement QoS mechanism
as ISP network. For example, to deploy polices on Switches to
execute traffic classification and marking. IEEE 802.1 DCB working
group defines a series of standards to guarantee quality of service.
802.1Qau - Congestion Notification
802.1Qaz - Enhanced Transmission Selection
802.1Qbb - Priority-based Flow Control
Without regard to mobile network, the existing DC network management
has a pre-assumption that the end hosts will not move. If an end
host moves, because the physical link has to break down and the
service also has to break down, the network can treat it as two
separated parts: one host leave the network and another host join the
network.
Server Virtualization and Virtual Machine Migration changes the
situation and break the preassumption. Server Virtualization is not
a new technology. But, because Cloud services become popular, which
requires flexible resource assignment and effectively resource
integration, server virtualization revitalizes again. Using server
virtualization technologies, network adminitrator can reduce
networking cost. To support the same volume of services, fewer
network devices, servers and links are required than before.
Multiple Virtual Machines (VMs) are established within a single
physical server and the VMs are allowed to relocate to a different
servers within the same subnet of Data Center, or even among
different sites of a Data Center. This is so called VM Migration.
VM Migration brings flexibility to Data Center, meanwhile it makes
network management more complex and challenging.
Gu Expires September 8, 2011 [Page 3]
Internet-Draft Policies and dynamic data migration in DC March 2011
While VM migrates, a very important requirement is that running
services on the VM mustn't been interrupted. Though a 'zero delay'
on running services is not realistic, but the services should be able
to continue after a very short delay.
In order to avoid service interruption and minimize delay on running
services, polices and dynamic information on network devices must be
migrated timely and accurately. The policies and dynamic information
includes those on switches, routers and middleboxes.
In the following section, we describe the policies and dynamic
information migration on several example network devices. The
influence to running services if they are not migrated accurately and
timely. Then we will introduce the limitations of existing network
management protocol.
2. Terminologies and concepts
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
Source Network Device, Source switch, or Source device: the network
device/switch/device from where the VM migrates. I.E. VM is
originally located under the source network device/switch/device.
Destination Network Device, Destination switch, or Destination
device: the network device/switch/device to where the VM migrates.
I.E. VM is relocated to the destination network device/switch/device.
TCP connection table: A table containing TCP connection-specific
information.
3. Policies on several network devices
In this draft, our discussion using the following figure as an
example networking. The links between AGG1/AGG2 and Gateway2, AGG3/
AGG4 and Gateway1 are omitted for simplicity. the new VM1 under
Server4 represents the VM1 after migration. VM1 and new VM1 don't
exist at the same time. In the real world, the networking of DC
could be different.
Gu Expires September 8, 2011 [Page 4]
Internet-Draft Policies and dynamic data migration in DC March 2011
------------------ ------------------
| Gateway1 |-------------| Gateway2 |
------------------ ------------------
/ \ / \
/ \ / \
------------ -------- -------- -------- -------- ------------
|Firewall-A|------| AGG1 |----| AGG2 | | AGG3 |----| AGG4 |----- |Firewall-B|
------------ -------- -------- -------- -------- ------------
| | | | | | | |
| \ / | | \ / |
| \ / | | \ / |
| \/ | | \/ |
| /\ | | /\ |
| / \ | | / \ |
| / \ | | / \ |
| | | | | | | |
---------- ---------- ---------- ----------
|Switch1 | |Switch2 | |Switch3 | |Switch4 |
---------- ---------- ---------- ----------
| \ / | | \ / |
| \/ | | \/ |
| /\ | | /\ |
| / \ | | / \ |
---------- ---------- ---------- ----------
|Server1 | |Server2 | |Server3 | |Server4 |
---------- ---------- ---------- ----------
| | | |
VM1 VM2 VM3 new VM1
Fig1. Basic networking for discussion in this draft.
3.1. Policies and configurations
SLA is parsed into a collection of polices, which can be described by
natural languages or mathematics fomula. Then policies are
represented by specific configurations on different network elements,
e.g. physical ports on routers and switches. In this draft, we
discuss the migration of policies, but the policies migration also
implies the migration of configurations on network devices, because
configurations are embodiment of policies.
Policies that need to migrate with VM are those can influence VM's
running services. The policies could be different on different
network devices. For example, it can be static Access Control Lists
on Access switches, QoS on switches and routers, security rules on
Firewalls, etc.
Take Access Control List as an example. Figure 1 shows the influence
Gu Expires September 8, 2011 [Page 5]
Internet-Draft Policies and dynamic data migration in DC March 2011
of lack of ACLs on destination switch. There is an ACL 100 on source
switch (Switch1) deny all packets from IP subnet 10.138.3.0 to
Internet. And another ACL 101 allows IP Address 10.138.3.1, VM1's IP
Address, to send packets to Internet. VM1 has a running service on
it. During service provisioning, VM1 is migrated to Server4 under
Switch4, Where there is no ACL 100 and ACL 101. VM1's IP Address
falls into a default ACL which deny all unmatching packets. As a
result, packets belonging to the running services are dropped, hence
the running service is interrupted.
------------------ ------------------
| Gateway1 |-------------| Gateway2 |
------------------ ------------------
/ \ / \
/ \ / \
------------ -------- -------- -------- -------- ------------
|Firewall-A|------| AGG1 |----| AGG2 | | AGG3 |----| AGG4 |----- |Firewall-B|
------------ -------- -------- -------- -------- ------------
| | | | | | | |
| \ / | | \ / |
| \ / | | \ / |
| \/ | | \/ |
| /\ | | /\ |
| / \ | | / \ |
| / \ | | / \ |
| | | | | | | |
ACL 101 ---------- ---------- ---------- ----------
ACL 100 |Switch1 | |Switch2 | |Switch3 | |Switch4 | Default ACL:
---------- ---------- ---------- ---------- deny all
| \ / | | \ / |
| \/ | | \/ |
| /\ | | /\ |
| / \ | | / \ |
---------- ---------- ---------- ----------
|Server1 | |Server2 | |Server3 | |Server4 |
---------- ---------- ---------- ----------
| | | |
VM1 VM2 VM3 new VM1
Fig.2 VM migration without ACL migration
4. Dynamic Information and the influence if lack or unaccurate
Network Manager (NM) can configure static configuration on network
devices. Except for the static configuration, some dynamic
information could also be recorded and processed by network devices.
TCP connection table is an obvious example. Normally, TCP Connection
Table is not configured by NM, but is generated by network devices,
e.g. Firewalls, by looking into the packets passing them. TCP
Gu Expires September 8, 2011 [Page 6]
Internet-Draft Policies and dynamic data migration in DC March 2011
Connection Table can be used for forwarding and security reasons.
Another example is cumulated data, e.g. how many packets/TCP
connecition requests an end host has sent. This information can only
be generated by network devices according to real traffic.
Configurations could be generated dynamically by network devices
themselves according to the dynamic informaiton, e.g. Dynamic ACLs.
4.1. TCP connection tables
A typical TCP Connection Table includes the following data:
tcpConnState: The state of this TCP connection.
tcpConnLocalAddress: The local IP address for this TCP connection.
tcpConnLocalPort: The local port number for this TCP connection.
tcpConnRemAddress: The remote IP address for this TCP connection.
tcpConnRemPort: The remote port number for this TCP connection.
A TCP Connectin Table could also includes the following information:
Sequence Number: the sequence number in the packet header the
sender is going to send.
Acknowledgement Number: the sequence number in the packet header
the receiver is hoping to receive.
Idle time: the time that the tcp connection table hasn't been
updated.
4.1.1. If TCP Connection Table isn't migrated
Assuming TCP Connection Table item is generated for VM1 on
Firewall-A, the information is as follows:
tcpConnState == Established
tcpConnLocalAddress == 10.138.3.1
tcpConnLocalPort == 1234
tcpConnRemAddress: == 192.167.22.3
tcpConnRemPort == 4321
Assuming VM1 is migrated to Server4 under Switch4, without TCP
Gu Expires September 8, 2011 [Page 7]
Internet-Draft Policies and dynamic data migration in DC March 2011
Connection Table migration. In order to keep the running service
uninterrupted, the IP Address of VM1 will keep unchanged. The
packets belonging to this TCP Connection will continue coming, which
will pass Firewall-B, instead of Firewall-A. Because there is no TCP
Connection Table for VM1 on Firewall-B, the following packets
belonging to the TCP Connection will be dropped, hence the running
service is broken down.
4.1.2. If TCP Connection Table is not accurately migrated
VM migration needs a period to finish memory and register copy.
Fig.3 shows the VMware VMotion process. There are three points we
should pay attention to.
Pre-copy period: VM begins to prepare for migration. In this
period, VM pre-copy memory state to the new VM on destination
device. The original VM is still power on and service is still
running, which means the memory and register could keep changing.
The new VM is power-on.
VM not running period: The end phase of memory copy. In this
period, original VM stop running service, the memory will not
change. Original VM finish copying the rest changed memory and
register to new VM. New VM is still power-on.
VM power-off point: After original VM receives the OK message from
new VM, it turns off the power, and meanwhile the new memory
starts to run.
We can see that it's unrealistic to make a 'zero delay' VM migration,
because there is at least about 1 second period (VM not running
period) when neither VM is running.
Assuming there is a NM can GET and SET dynamic information. The NM
GET dynamic information at Time A, and finish SET at Time B. At Time
A, the Sequence Number of VM1's TCP Connection is 99. After NM GET
dynamic information, VM1's TCP connection keeps transferring packets
and Sequence Number increase to 110, until VM not running period
begins. During VM not running period, no TCP packet is acknowledged
by VM1, so the Sequence Number is 110. At Time B, the destination
Firewall is SET by Sequence Number 99. When new VM1 starts, the
packets belonging to VM1's TCP connection comes to Firewall-B with
Sequence Number 111. Since the receiving Sequence Number doesn't
equal to the Acknowleadge Sequence Number of Firewall-B, this packet
will be dropped and the running service is broken down.
Gu Expires September 8, 2011 [Page 8]
Internet-Draft Policies and dynamic data migration in DC March 2011
----------------------------------------------------------------------------
^ || | Power-on destination VM ^
| || | |
| ||--------->| --------------------------- VMotion
VM || | Pre-copy memory state abortable
running ||--------->| |------------>Time A
on source|| | | ^
| ||--------->| | Dynamic Information
V || | | V keep changing
---------------------------------------------------------------------------|----
^ |--------->| Checkpoint-save src VM and transfer state |
| | |------------------------------------------- |
VM not |--------->| transfer remaining changed memory and |------------->Time B
running | | checkpoint-restore dst VM V
| | |-------------------------------------------------------
~1sec |<---------| send restore OK
V | |
----------------------------------------------------------------
| || Power-off source VM
| || VM running on destination
Fig.3 VMware VMotion process
4.2. Cumulated data
One example for cumulated data is unfinished TCP Connection
established by a specific VM. In order to avoid TCP SYN flood, a
network device may control the unfinished TCP Connection established
by a single end host by setting a threshold. For example, the NM set
the threshold to 5, and VM1 has established 3 unfinished TCP
connections. If the cumulated TCP connection number isn't migrated
to destination devices, the destination device will allow VM1 to
establish up to 5 unfinished TCP connections. For the single
destination device, the unfinished TCP connections established by VM1
is under control, but for the whole DC, VM1 has established 8
unfinished TCP connections. So VM1 has consume more resoureces than
allowed.
4.3. Dynamic ACLs
Assuming all traffic is denied unless the end host is authorized and
authenticated. VM1 has been authenticated on source device and a
dynamic ACL has been generated to allow VM1's traffic to pass. If
VM1 migrates to destination device without the dynamic ACL, the
destination device will drop VM1's traffic, because VM1 is an
unathenticated end host for it. So in this case, the dynamic ACL
Gu Expires September 8, 2011 [Page 9]
Internet-Draft Policies and dynamic data migration in DC March 2011
needs to migrate with VM.
4.4. DHCP Snooping
Assuming source device is DHCP Snooping Enabled and a DHCP Snooping
mapping item is created for VM1: (IP-VM1: MAC-VM1). This mapping is
created dynamically by listening to DHCP Response message. If VM1
migrate to destination device, since the IP Address of VM1 doesn't
change, there is no DHCP Request sent by VM1. So on destination
device, there is (IP-VM1: MAC-VM1) mapping, all traffic from VM1 will
be dropped. So DHCP Snooping mapping item need to migrate to
destination device.
4.5. Multicast Membership
Multicast membership is similar to DHCP Snooping. Multicast
membership is created on ports by listening to IGMP membership report
messages. If VM1 migrates to destination, VM1 will not send IGMP
membership report until next IGMP General Query. Before that, VM1
may not be able to recevie Multicast packets since network devices on
and above destination devices don't knwo VM1's Multicast membership
and don't forwarding the Multicast packets to VM1.
5. Existing network management protocol and the limitations
RFC3535 introduces many Network Management architectures and
protocols. Basically, there are two kinds of architectures: network
element oriented (SNMP and NETCONF) and Policy based (COPS-PR). In
this section, we will introduce why these NMPs can not resolve the
problem described in this draft.
5.1. Limitations
We analyze the problem described above into two aspects. One is
Policies migration and the other is dynamic information migration.
5.1.1. For Policies migration
Existing NMP could be used to migrate Policies from source device to
destination device. But we still need to face some questions:
Is device-oriented NMP suitable for policies migration?
Is C/S based NMP suitable for DC management?
How does NM know the source and destination device?
Gu Expires September 8, 2011 [Page 10]
Internet-Draft Policies and dynamic data migration in DC March 2011
Do we need an automatic policies migration mechanism?
5.1.2. For Dynamic Information Migration
Currently, NMP is not used to configure dynamic information.
And, as Fig.3 shows, if we fail to begin migrating dynamic
information at appropriate time (the time during VM not running
period), the running services will be interrupted. For example, if
dynamic information is migrated before 'VM not running period',
dynamic information is inaccurate and the running services may be
broken down when new VM restarts. In order to make accurate
migration and keep running service uninterrupted, we need to know the
exact timing for migration.
6. Security Considerations
The policies and dynamic information described above are all about
security.
7. Acknowledgments
I would like to thank the following people for contributing to this
draft: Ning Zong, David harrington, Linda dunbar, Susan Hares, Serge
manning, Barry Leiba, Jiang xingfeng, Song Wei, Robert Sultan, and
many others.
8. Normative Reference
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", March 1997.
Author's Address
Gu Yingjie
Huawei
No. 101 Software Avenue
Nanjing, Jiangsu Province 210001
P.R.China
Phone: +86-25-56624760
Fax: +86-25-56624702
Email: guyingjie@huawei.com
Gu Expires September 8, 2011 [Page 11]