Network Working Group Y. Gu
Internet-Draft Huawei
Intended status: Standards Track Y. Fan
Expires: December 10, 2011 China Telecom
June 12, 2011
Policies and dynamic information migration in DC
draft-gu-opsa-policies-migration-00
Abstract
Virtualization and Virtual Machine (VM) migration provide Data Center
with feasibility and improves the utilization of limited physical
resource, e.g. switches/routers, servers and links. Meanwhile, a
variety of policies (e.g. ACL, firewalls, load balancers, IPS and
QoS) are deployed in Data Center to improve system security and
gurantee SLA. Those polices are executed by rules configured or
generated on network devices. E.g. packet filtering policies are
executed by Access Control List on switches or firewalls. Another
example is Load balancer (LB) who extablishes TCP/HTTP connections
with external clients and balances connections among server farm.
During this process, TCP connection tables are dynamically generated
on LB. When VM migrates, the network devices that processing and
forward VM's packets may change. In order to keep VM's running
serives and guanrantee security on new place, VM-relevant policies,
including static policies as well as the dynamically generated
information, need to migrate with VM.
This draft describes some examples of the policies that need to
migrate with VM, the problems that need to consider when migrate
polices in Data Center. The goal is to justify that it is necessary
for IETF to make new effort on management of virtualized Data Center.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
Gu & Fan Expires December 10, 2011 [Page 1]
Internet-Draft policy and dynamic information migration June 2011
This Internet-Draft will expire on December 10, 2011.
Copyright Notice
Copyright (c) 2011 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Gu & Fan Expires December 10, 2011 [Page 2]
Internet-Draft policy and dynamic information migration June 2011
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Terminologies and concepts . . . . . . . . . . . . . . . . . . 5
3. Policies Classification . . . . . . . . . . . . . . . . . . . 6
3.1. Static Policies . . . . . . . . . . . . . . . . . . . . . 7
3.1.1. Use Cases . . . . . . . . . . . . . . . . . . . . . . 7
3.2. Dynamic Information . . . . . . . . . . . . . . . . . . . 9
3.2.1. Use Cases . . . . . . . . . . . . . . . . . . . . . . 10
4. Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.1. Policy migration within a single site . . . . . . . . . . 22
4.2. Polices migration in asymmetric architecture . . . . . . . 23
4.3. Policies migration between DC sites . . . . . . . . . . . 24
5. General Considerations . . . . . . . . . . . . . . . . . . . . 25
5.1. Time-sensitive . . . . . . . . . . . . . . . . . . . . . . 26
5.2. Notification . . . . . . . . . . . . . . . . . . . . . . . 27
5.3. Functionality . . . . . . . . . . . . . . . . . . . . . . 27
5.4. Resource Capability . . . . . . . . . . . . . . . . . . . 29
5.5. Migration Feedback . . . . . . . . . . . . . . . . . . . . 29
6. Security Considerations . . . . . . . . . . . . . . . . . . . 29
7. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 29
8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 29
8.1. Normative Reference . . . . . . . . . . . . . . . . . . . 29
8.2. Informative Reference . . . . . . . . . . . . . . . . . . 30
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 30
Gu & Fan Expires December 10, 2011 [Page 3]
Internet-Draft policy and dynamic information migration June 2011
1. Introduction
Data centers can host tens or even thousands of different
applications. Some are simple applications such as web servers
providing static content, while some may be very complex, e.g.
e-commerce, that requiring all around privacy protection and data
security. Clients of Data Center, unlike server hosting clients,
raise more strict QoS and Security requirements.
To satisfy different level of security requirements and to manage and
improve the performance of these applications, data centers typically
deploy a large variety of policies on network devices, e.g. interface
security functions on switches and routers, and middleboxes,
including firewalls, load balancers, SSL offloaders, web caches and
intrusion prevention boxes.
To satisfy QoS requirements, Data Center also implement QoS mechanism
as ISP network. IEEE 802.1 DCB working group defines a series of
standards to guarantee quality of service.
802.1Qau - Congestion Notification
802.1Qaz - Enhanced Transmission Selection
802.1Qbb - Priority-based Flow Control
Without regard to mobile network, the existing DC network management
has a pre-assumption that end hosts will not move. If an end host
moves, because the physical link has to break down and the service
also has to break down, the network can treat it as two separated
parts: one host leave the network and another host join the network.
Server Virtualization and Virtual Machine Migration changes the
situation and disable the preassumption. With server virtualization,
providers can reduce networking cost. To support the same volume of
services, fewer network devices, servers and links are required than
before. Multiple Virtual Machines (VMs) are established within a
single physical server and the VMs are allowed to relocate to a
different servers within the same subnet of Data Center, or even
among different sites of a Data Center. This is so called VM
Migration. VM Migration brings flexibility to Data Center, meanwhile
it makes network management more complex and challenging.
Unlike servers power off and power on again, in which case, servers
know and can bear services disruption, VM Migration requires that
VM's IP Address shouldn't change and running service mustn't be
disrupted. Meawhile, security level should be guaranteed. In order
to satisfy the above requirements, the VM relevant polices need to
Gu & Fan Expires December 10, 2011 [Page 4]
Internet-Draft policy and dynamic information migration June 2011
migrate with VM.
IEEE 802.1Qbg has developed VSI Discovery and Configuration Protocol
(VDP) to enable ajacent bridge to discover VM and configure
corresponding static polices.
The author has presented the problem to IEEE 802.1 DCB task group.
DCB has recognized this problem and make extension to VDP to notify
VM status, which will be introduced in Section 5.2. IEEE 802.1 DCB
would like to know furture progress of this problem in IETF and ask
the author to let know the furthur progress.
In the following sections, detail examples and senarios are given to
explain why and which polices need to migrate with VM.
2. Terminologies and concepts
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
Source Network Device, Source switch, or Source device: the network
device/switch/device from where the VM migrates. I.E. VM is
originally located under the source network device/switch/device.
Destination Network Device, Destination switch, or Destination
device: the network device/switch/device to where the VM migrates.
I.E. VM is relocated to the destination network device/switch/device.
TCP connection table: A table containing TCP connection-specific
information.
VM: Virtual Machine; A completely isolated operation system which is
installed by software on a normal operation system. An normal
operation system can be virtualized into several VM.
ACL: Access Control List; A list of rules that specifies which
packets can be permited, which should be denied.
QoS: Quality of Service;
FW: Firewall; a policy based security function, typically used for
restricting access to/from specific devices and applications.
LB: Load Balancer; A methodology to distribute workload across
multiple computers or a computer cluster, network links, central
processing units, disk drives, or other resources, to achieve optimal
Gu & Fan Expires December 10, 2011 [Page 5]
Internet-Draft policy and dynamic information migration June 2011
resource utilization, maximize throughput, minimize response time,
and avoid overload.
NAT: Network Address Translation. Refer to RFC3303.
3. Policies Classification
In this section, we use the following figure <Fig1> as an example
network architecture. Some redundant links are omitted for
simplicity. the new VM1 under Server4 represents the VM1 after
migration. VM1 and new VM1 don't exist at the same time. In the
real world, the networking of DC could be different.
-------------------------------------------------
| Gateway |
-------------------------------------------------
/ \ / \
/ \ / \
----- -------- -------- -------- -------- ------
|FW-A|--| AGG1 |----| AGG2 | | AGG3 |----| AGG4 |---|FW-B|
----- -------- -------- -------- -------- ------
| | | | | | | |
| \ / | | \ / |
| \ / | | \ / |
| \/ | | \/ |
| /\ | | /\ |
| / \ | | / \ |
| / \ | | / \ |
| | | | | | | |
---------- ---------- ---------- ----------
|Switch1 | |Switch2 | |Switch3 | |Switch4 |
---------- ---------- ---------- ----------
| \ / | | \ / |
| \/ | | \/ |
| /\ | | /\ |
| / \ | | / \ |
---------- ---------- ---------- ----------
|Server1 | |Server2 | |Server3 | |Server4 |
---------- ---------- ---------- ----------
| | | |
VM1 VM2 VM3 new VM1
Figure 1: Basic networking for discussion in this draft.
One key feature of VM migration is that VM's IP Address shouldn't
change, otherwise the running service could be disrupted. In order
to achieve this, additional mechanism need to be developed. However,
these mechanism are out of the scope of this draft. In policy
Gu & Fan Expires December 10, 2011 [Page 6]
Internet-Draft policy and dynamic information migration June 2011
migration problem statement, we assume that there already exist an
effective mechanism to guarantee VM's IP Address unchanged.
Two kinds of polices are introduced in this draft, i.e. static
polices and dynamic polices (or dynamic information).
3.1. Static Policies
Administrators of DC establish VM profile, which may include static
policies for the VM, e.g. VLAN Name, bandwidth indication, and Qos
parameters, etc. And they are suppposed to be consistent during the
lifetime of the VM. No matter where VM migrates, the static policies
need to move with VM for the sake of security, privacy and QoS.
Static polices can be described by natural languages or mathematics
fomula. They are implementated by specific configurations on
different network elements, e.g. a ACL item on physical ports of
switches enables a packet filtering policies. In this draft, we
discuss the migration of policies, but the policies migration also
implies the migration of configurations on network devices, because
configurations are embodiment of policies.
Policies that need to migrate with VM are those can influence VM's
running services or are related to security and billing&accounting.
Different network devices will contain different kind of policies.
For example, it can be static Access Control Lists on Access
switches, QoS on switches and routers, security rules on Firewalls,
and load balancing mechanism on Load Balancer, etc.
3.1.1. Use Cases
It's hard to enumerate all the VM-related static polices. In this
section, we just give two examples of static policies to show the
necessity of static polices migration.
3.1.1.1. Access Control List
<Fig2> shows the influence of lack of ACLs on destination switch.
There is an Default ACL on source switch (Switch1) deny all packets
from IP subnet 10.138.3.0 to Internet. And another ACL 101 allows IP
Address 10.138.3.1, VM1's IP Address, to send packets to Internet.
VM1 has a running service on it. During service provisioning, VM1 is
migrated to Server4 under Switch4, Where there is default ACL to deny
all packets. Since ACL 101 on Switch1 is not migrated to Switch4,
VM1's IP Address falls into a default ACL which deny all unmatching
packets. As a result, packets belonging to the running services are
dropped, hence the running service is disrupted.
Gu & Fan Expires December 10, 2011 [Page 7]
Internet-Draft policy and dynamic information migration June 2011
-------------------------------------------------
| Gateway |
-------------------------------------------------
//\ \ / \
/ * \ / \
------ -------- -------- -------- -------- ------
|FW-A|----| AGG1 |----| AGG2 | | AGG3 |----| AGG4 |---|FW-B|
------ -------- -------- -------- -------- ------
|* | | | | | | |
|* \ / | | \ / |
|* \ / | | \ / |
|* \/ | | \/ |
ACL 101: |* /\ | | /\ |
permit VM1 |* / \ | | / \ |Default ACL:
Default ACL:|* / \ | | / \ |deny all
deny all |* | | | | | | |Packets dropped
---------- ---------- ---------- ----------
|Switch1 | |Switch2 | |Switch3 | |Switch4 |
---------- ---------- ---------- ----------
|* \ / | | \ / |/\ \/
|* \/ | | \/ |* /\
|* /\ | | /\ |*
|* / \ | | / \ |*
---------- ---------- ---------- ----------
|Server1 | |Server2 | |Server3 | |Server4 |
---------- ---------- ---------- ----------
|* | | |*
|* | | |*
VM1 VM2 VM3 new VM1
Figure 2: VM migration without ACL migration
3.1.1.2. VLAN
VMs with similar properties could be grouped into a VLAN, e.g. end
hosts providing Web service are grouped into Web-VLAN. Each VLAN has
a unique VLAN ID. Only when a VM is configured into a specific VLAN,
can it receives broadcast packets from other VMs in the same VLAN.
In <Fig3>, VM1 belongs to VLAN1. Assuming VLAN1 is not configured on
the port on AGG4 that attaching Switch4, and VM1 migrates to new VM1,
Broadcast pakcets from another VM belonging to VLAN1 are forwarded to
AGG4, however they can not be forwarded to Switch4, since broadcast
packets are sending only to VLAN1-enabled port. The *** line
represents VLAN1 broadcast, which can not access Server4. New VM1
can not receive VLAN1 broadcast.
Gu & Fan Expires December 10, 2011 [Page 8]
Internet-Draft policy and dynamic information migration June 2011
-----------------------------------
| Gateway |
-----------------------------------
|
|
------------------------------------------
| Core Switch |
------------------------------------------
/* \ / *\
/* \ / \/\
------ ------- ------- -------- ------- ------
|FW-A|----|AGG1 |----| AGG2| | AGG3 |--| AGG4|---|Fw-B|
------ ------- ------- -------- ------- ------
|* *| | | | | | | \/ No VLAN1
|* *\ / | | \ / | /\
|* *\ / | | \ / |
|* *\/ | | \/ |
|* /*\ | | /\ |
|* / * \ | | / \ |
|* / * \ | | / \ |
|* | * | | | | | |
--------- --------- --------- ----------
|Switch1| |Switch2| |Switch3| |Switch4 |
--------- --------- --------- ----------
VLAN1 |* \ / * |VLAN1 | \ / |
|* \/ * | | \/ |
|* /\ * | | /\ |
|* / \ \/ | | / \ |
--------- --------- --------- ----------
|Server1| |Server2| |Server3| |Server4 |
--------- --------- --------- ----------
|* | | |
|\/ | | |
VM1 VM2 VM3 new VM1
Figure 3: VM migration without VLAN migration
3.2. Dynamic Information
Except for the static policies, some dynamic information could also
be generated by network devices to assist packet processing. TCP
connection table on proxy is an obvious example. Proxy intervenes
the direct connection between external client and internal server,
which can protect internal server from attacking. Proxy establishes
TCP connection with external client for internal server, a TCP
connection Table being generated on proxy. Then proxy establishes
TCP connection with internal server for external client, another TCP
connection table being generated. These TCP Connection table can not
Gu & Fan Expires December 10, 2011 [Page 9]
Internet-Draft policy and dynamic information migration June 2011
be configured by network manager, but is generated by network
devices, e.g. Firewalls and proxies, according to real connections.
Another example is cumulative data, e.g. how many packets/TCP
connecition requests an end host has sent. This information can only
be generated by network devices according to real traffic.
Configurations could be generated dynamically by network devices
themselves according to the dynamic informaiton, e.g. Dynamic ACLs.
In following section, we enumerate several kinds of dynamic
information. A dynamic information could be generated on different
network devices, meanwhile a specific network devices could hold more
than one kind of dynamic information. In following section, we
introduce possible dynamic informaiton on specific network devices.
3.2.1. Use Cases
3.2.1.1. Switches
We use switch to represent Top of Racks, Bridges and switches. These
devices have slightly different definition and functions, and may be
used or combinedly used in differenct scenarios.
3.2.1.1.1. Dynamic ACL
In <Fig4>, assuming a default ACL is configured on Switch1 that all
traffic is denied unless the end host is authorized and
authenticated. VM1 has been authenticated on source device and a
dynamic ACL has been generated which allows VM1's traffic to pass
through.
A 'Deny all' default ACL is configured on Switch4 too. VM1 migrates
to destination device which attaches to Swtich4. Because Switch4
regards new VM1 as an unathenticated end host, according to default
ACL, all data packets from VM1 will be dropped. In order to continue
service, VM1 has to authenticate again. In this case, running
service is disrupted. If authentication and dynamic ACL, generated
on Switch1, can be migrated to Switch4, data packets from VM1 are
permitted, hence, without regard to influence caused by features,
running service can continue.
Gu & Fan Expires December 10, 2011 [Page 10]
Internet-Draft policy and dynamic information migration June 2011
-------------------------------------------------
| Gateway |
-------------------------------------------------
//\ \ / \
/ * \ / \
------ ------- ------- ------- ------ ------
|FW-A|-----| AGG1 |---| AGG2 | | AGG3|----| AGG4|---|FW-B|
------ ------- ------- ------- ------ ------
|* | | | | | | |
|* \ / | | \ / |
|* \ / | | \ / |
|* \/ | | \/ |
|* /\ | | /\ |
|* / \ | | / \ |
|* / \ | | / \ |
|* | | | | | | |
Default ACL: --------- --------- --------- --------- Default ACL:
deny all |Switch1| |Switch2| |Switch3| |Switch4| deny all
--------- --------- --------- ---------
|* \ / | | \ / /\| \/ VM1's pakcets
Dynamic ACL: |* \/ | | \/ *| /\ are dropped.
Permit VM! |* /\ | | /\ *|
|* / \ | | / \ *|
--------- --------- --------- ---------
|Server1| |Server2| |Server3| |Server4|
--------- --------- --------- ---------
|* | | *|
|* | | *|
VM1 VM2 VM3 new VM1
Figure 4: VM migration without dynamic ACL migration
3.2.1.1.2. DHCP snooping
Assuming Switch1 is DHCP Snooping Enabled and a DHCP Snooping mapping
item is created for VM1: (IP-VM1: MAC-VM1). This mapping is created
dynamically by listening to DHCP Response message. If VM1 migrate to
destination device, since the IP Address of VM1 doesn't change, there
is no DHCP Request sent by VM1 before lease time expires. So on
Switch4, no mapping information can be generated by listening to DHCP
response, and all traffic from VM1 will be dropped.
If DHCP snooping table on Switch1 can be migrated to Switch4, Switch4
learns from the migrated DHCP snooping talbe that packet with IP-VM1
and MAC-VM1 can be regularly forwarded.
Gu & Fan Expires December 10, 2011 [Page 11]
Internet-Draft policy and dynamic information migration June 2011
-------------------------------------------------
| Gateway |
-------------------------------------------------
//\ \ / \
/ * \ / \
------ ------- ------ -------- ------- ------
|FW-A|---|AGG1 |----| AGG2| | AGG3 |---|AGG4 |---|FW-B|
------ -------- ------ -------- ------- ------
|* | | | | | | |
DHCP |* \ / | | \ / |
Response |* \ / | | \ / |
\ |* \/ | | \/ |
\ |* /\ | | /\ |
\ |* / \ | | / \ |DHCP snooping enabled
\ |* / \ | | / \ |No DHCP request
DHCP >|* | | | | | | |from new VM1
snooping --------- --------- --------- ----------
enabled |Switch1| |Switch2| |Switch3| |Switch4 |
--------- --------- --------- ----------
---------- |* \ / | | \ / /\| \/ VM1's packets
|Snooping| |* \/ | | \/ *| /\ are dropped
|Table | |* /\ | | /\ *|
---------- |* / \ | | / \ *|
-------- --------- --------- ----------
|Server1| |Server2| |Server3| |Server4 |
-------- --------- --------- ----------
|* | | *|
|* | | *|
VM1 VM2 VM3 new VM1
Figure 5: VM migration without DHCP snooping table migration
3.2.1.1.3. IGMP snooping
IGMP snooping is similar to DHCP Snooping. Multicast membership is
created on ports by listening to IGMP JOIN or membership report
messages. When a switch receives a multicast packet, it forwards the
packet to the port which has joined the multicast group. In <Fig6>,
assuming VM1 sends an IGMP JOIN Group1 message to Multicast server,
port1 and port-uplink on Switch1 check the JOIN message and tag on
port1 and port-uplink that they should forward Multicast packets from
Group1. If VM1 migrates to destination, since migration is
transparent to VM, VM1 will not send IGMP membership report until
next IGMP General Query is received. If port2 on Swtich4 didn't join
Group1, multicast packet from Group1 won't be forward to port2, then
VM1 won't be able to receive Multicast packets.
Gu & Fan Expires December 10, 2011 [Page 12]
Internet-Draft policy and dynamic information migration June 2011
-------------------------------------------------
| Gateway |
-------------------------------------------------
/* \ / *\
/* \ / *\
------ -------- -------- -------- -------- ------
|FW-A|-----| AGG1 |----| AGG2 | | AGG3 |---| AGG4 |--- |FW-B|
------ -------- -------- -------- -------- ------
|* | | | | | | *|
IGMP |* \ / | | \ / *|
JOIN Snooping|* \ / | | \ / *|
\ |* \/ | | \/ *|
\ |* /\ | | /\ *|
\ |* / \ | | / \ *|IGMP enabled
\ |* / \ | | / \ *|No IGMP JOIN
>|* | | | | | | \/|from new VM1
IGMP ---------- ---------- ---------- ---------- \/
enabled |Switch1 | |Switch2 | |Switch3 | |Switch4 | /\
---------- ---------- ---------- ----------
|* \ / | | \ / |
---------- |* \/ | | \/ |
|Group1: | |* /\ | | /\ |
|Port1 | |* / \ | | / \ |
---------- ---------- --------- --------- ----------
|Server1 | |Server2| |Server3| |Server4 |
---------- --------- --------- ----------
|* | | |
|\/ | | |
VM1 VM2 VM3 new VM1
Figure 6: VM migration without IGMP snooping table migration
3.2.1.1.4. Cumulative Data
To ensure VMs consume no more than assigned bandwidth and to enable
QoS control, billing and accounting, bridges need to cumulate packets
number and calculate packet rate. Lack of these cumulative data and
statistic data on destination bridge decrease the accuracy.
3.2.1.2. Firewall
Fireall (FW) is a filtering device that separates LAN segments,
giving each segment a different security level and establishing a
security perimeter that controls the traffic flow between segments.
There are different types of firewalls based on their packet-
processing capabilities and their awareness of application-level
information:
Gu & Fan Expires December 10, 2011 [Page 13]
Internet-Draft policy and dynamic information migration June 2011
Packet-filetering firewalls
Proxy firewalls
Stateful firewalls
Hybrid firewalls
3.2.1.2.1. Dynamic ACL
Packet filtering Firewall filters packet according to ACL. Dynamic
ACL could be generated to protect internal network/servers from
attacks, similar to dynamic ACL on Switches.
3.2.1.2.2. TCP Connection Table
Proxy Firewall proxies TCP connections between internal servers and
external clients. It establishes TCP connection with external client
and then it establishes TCP connection with internal server. TCP
connection tables are cached on Proxy Firewall to indicate how to
transform following packets belonging to this TCP connection.
External Client Firewall Internal server
|----- TCP SYN ----->|
|<-----SYN/ACK ------|
|-------ACK ----->|
|--------TCP SYN ------>|
|<-------SYN/ACK -------|
|--------ACK ------>|
|-------DATA ------>|--------DATA ------>|
|<------DATA -------|<-------DATA -------|
Figure 7: Proxy Firewall
Gu & Fan Expires December 10, 2011 [Page 14]
Internet-Draft policy and dynamic information migration June 2011
-------------------------------------------------
| Gateway |
-------------------------------------------------
---------------- /* \ / *\
|TCP Conn Table| /* \ / *\ No corresponding
---------------- /* \ / *\ TCP Conn Table
------ **** -------- -------- -------- -------- ****>------
|FW-A|-----| AGG1 |----| AGG2 | | AGG3 |----| AGG4 |----- |FW-B|
------ **** -------- -------- -------- -------- \/ ------
|* | | | | | | | /\
|* \ / | | \ / |
|* \ / | | \ / |
|* \/ | | \/ |
|* /\ | | /\ |
|* / \ | | / \ |
|* / \ | | / \ |
|* | | | | | | |
---------- ---------- ---------- ----------
|Switch1 | |Switch2 | |Switch3 | |Switch4 |
---------- ---------- ---------- ----------
|* \ / | | \ / |
|* \/ | | \/ |
|* /\ | | /\ |
|* / \ | | / \ |
---------- ---------- ---------- ----------
|Server1 | |Server2 | |Server3 | |Server4 |
---------- ---------- ---------- ----------
|* | | |
|\/ | | |
VM1 VM2 VM3 new VM1
Figure 8: VM migration without TCP connection table migration
A typical TCP Connection Table includes the following data:
tcpConnState: The state of this TCP connection.
tcpConnLocalAddress: The local IP address for this TCP connection.
tcpConnLocalPort: The local port number for this TCP connection.
tcpConnRemAddress: The remote IP address for this TCP connection.
tcpConnRemPort: The remote port number for this TCP connection.
A TCP Connectin Table could also includes the following information:
Gu & Fan Expires December 10, 2011 [Page 15]
Internet-Draft policy and dynamic information migration June 2011
Sequence Number: the sequence number in the packet header the
sender is going to send.
Acknowledgement Number: the sequence number in the packet header
the receiver is hoping to receive.
Idle time: the time that the tcp connection table hasn't been
updated.
Assuming TCP Connection Table item is generated for VM1 on Fw-A, the
information is as follows:
tcpConnState == Established
tcpConnLocalAddress == 10.138.3.1
tcpConnLocalPort == 1234
tcpConnRemAddress: == 192.167.22.3
tcpConnRemPort == 4321
VM1 migrates to Server4 under Switch4, without TCP Connection Table
items migration. The packets belonging to this TCP Connection will
continue coming, which will pass FW-B, instead of FW-A. Because
there is no TCP Connection Table for VM1 on FW-B, the following
packets belonging to the TCP Connection will be dropped, hence the
running service is broken down.
3.2.1.2.3. Cumulative Data
One example for cumulative data is unfinished TCP Connection
established by a specific VM. In order to avoid TCP SYN flood,
Firewall may control the unfinished TCP Connection established by a
single end host by setting a threshold. For example, the NM set the
threshold to 50, and VM1 has established 30 unfinished TCP
connections. If the cumulated TCP connection number isn't migrated
to destination devices, the destination device will allow VM1 to
establish up to 50 unfinished TCP connections. For the single
destination device, the unfinished TCP connections established by VM1
is under control, but for the whole DC, VM1 has established 80
unfinished TCP connections. So VM1 has consume more resoureces than
allowed.
3.2.1.2.4. NAT Address Mapping
Data center may implement network address translation. Sometimes NAT
function is enabled on Firewall. Private IP Add. is assigned to
Gu & Fan Expires December 10, 2011 [Page 16]
Internet-Draft policy and dynamic information migration June 2011
internal VM. When VM communicate with external client, NAT function
assigns a public IP Add. to the communicating internal VM. A mapping
between private IP Add. and public IP Add. is recorded on NAT
function.
Assuming a mapping from Pri-IP-VM1 to Pub-IP-VM1 is generated on
FW-A. When VM1 migrates to Server4 without the address mapping,
following packets from external client can not be translated to
correct Pri-IP-VM1, and vice versa.
-------------------------------------------------
| Gateway |
-------------------------------------------------
//\ \ / \* Dst: Pub-IP-VM1
Src: Pub-IP-VM1 /* \ / \* No Address mapping
----------------- /* \ / \* from Pub-IP-VM1
|Address Mapping| /* \ / \* to Private-IP-VM1
----------------- /* \ / \*Packets are dropped
------- ****>-------- -------- ------- ------- ****>------
|NAT-A|------| AGG1 |---| AGG2 | |AGG3 |----| AGG4 |-----|NAT-B|
------- <****-------- -------- ------- -------\/ -------
|* | | | | | | | /\
|* \ / | | \ / |
|* \ / | | \ / |
|* \/ | | \/ |
|* /\ | | /\ |
|* / \ | | / \ |
|* / \ | | / \ |
|* | | | | | | |
--------- --------- --------- ----------
|Switch1| |Switch2| |Switch3| |Switch4 |
--------- --------- --------- ----------
|* \ / | | \ / |
Src: |* \/ | | \/ |
Pri-IP-VM1 |* /\ | | /\ |
|* / \ | | / \ |
--------- --------- --------- ----------
|Server1| |Server2| |Server3| |Server4 |
--------- --------- --------- ----------
|* | | |
|\/ | | |
VM1 VM2 VM3 new VM1
Figure 9: VM migration without NAT Address Mapping migration
Gu & Fan Expires December 10, 2011 [Page 17]
Internet-Draft policy and dynamic information migration June 2011
3.2.1.3. Load Balancer
When a cluster of servers are organized to provide same services,
Load Balancer (LB) is used to balance service load between servers.
All external requests are sent to LB, before they are redirected to a
specific server. First, external clients extablish TCP connection
with LB. Then LB decides which specific server should take care of
the requests. Two ways to redirect the requests, i.e. transparent LB
and proxy LB. The following dynamic information applies for both
ways.
3.2.1.3.1. Connection Table
Layer 4 LB identifies and makes load-balancing decesion by Layer 4
Protocol, e.g. TCP. Layer 4 LB generates TCP connection table which
is similar to TCP connection table on Firewall. <Fig10>
External Client Load Balancer Internal server
| ------- TCP SYN ------->|-------TCP SYN ----->|
|<--------SYN/ACK --------|<------SYN/ACK ------|
|---------ACK ------->|-------ACK ----->|
|---------HTTP REQ ------->|-------HTTP REQ----->|
|<--------HTTP ACK --------|<------HTTP ACK------|
|---------DATA ------->|-------DATA ----->|
|<--------DATA --------|<------DATA ------|
Figure 10: Layer 4 Load Balancing
Layer 5 LB identifies and makes load-balancing decesion by Layer 5
protocol, e.g. HTTP, SMTP and FTP, that is found in packet payload.
The Layer 5 information could be HTTP URL, HTTP cookie, or HTTP
header field user agent. Layer 5 LB generates connection table for
HTTP, SMTP, and FTP, etc.<Fig11>
External Client Load Balancer Internal server
| ------- TCP SYN ----->|
|<--------SYN/ACK ------|
|---------ACK ----->|
|---------HTTP Req ----->|
|-------TCP SYN ----->|
|<------SYN/ACK ------|
|-------ACK ----->|
|-------HTTP Req----->|
|---------HTTP ACK ----->|-------HTTP ACK----->|
|---------DATA ----->|-------DATA ----->|
|<--------DATA ------|<------DATA ------|
Figure 11: Layer 5 Load Balancing
<Fig12> shows the result of lack of Connection table on destination
Gu & Fan Expires December 10, 2011 [Page 18]
Internet-Draft policy and dynamic information migration June 2011
LB.
-------------------------------------------------
| Gateway |
-------------------------------------------------
------------- /* \ / *\ No VM1's Conn Table
|Conn Table | /* \ / *\Packets could be dropped
------------- /* \ / *\or mis-redirected
------ <**** ------- -------- ------- ------- ****>-------
|LB-A|------| AGG1 |---| AGG2 | | AGG3|---|AGG4 |----- |LB-B|
------ ****> ------- -------- ------- ------- <**** ------
|* | | | | | | |*
|* \ / | | \ / |*
|* \ / | | \ / |*
|* \/ | | \/ |*
|* /\ | | /\ |*
|* / \ | | / \ |*
|* / \ | | / \ |*
|* | | | | | | |*
---------- ---------- --------- ---------
|Switch1 | |Switch2 | |Switch3| |Switch4|
---------- ---------- --------- ---------
|* \ / | | \ / |*
|* \/ | | \/ |*
|* /\ | | /\ |*
|* / \ | | / \ |*
---------- ---------- ---------- ----------
|Server1 | |Server2 | |Server3 | |Server4 |
---------- ---------- ---------- ----------
|* | |* |
|\/ | |\/ |
VM1 VM2 VM3 new VM1
Figure 12: VM migration without Connection table migration
3.2.1.3.2. Sticky Table
Session persistence refers to the capability of the Load alancer to
logically group multiple connections from the same client transaction
to the service. Session persistence is also known as stickiness or
sticky connections because the goal is to stick two or more
connections together as part of a single session. This grouping is
done to ensure the connections are handled and forwarded to the
groups of servers that are aware of and expect to see the remaining
connections. This ensures that the client has a successful
interaction with the application.
LB usually keeps a sticky table for session persistence, which is
Gu & Fan Expires December 10, 2011 [Page 19]
Internet-Draft policy and dynamic information migration June 2011
used to match incoming requests to existing connections so that they
can be grouped together. A sticky table could include the following
information.
Sticky groups
Group name (sticky group identifier)
Type (sticky group type, according to stick methods)
Sticky server farm
Backup server farm
Aggregate state (to indicate whether the state of the backup
server is tied to the virtual server state)
Sticky enabled on backup server farm (to indicate whether the
backup server farm is sticky)
Replicate (to indicate whether to replicate sticky entries on
the standby device)
Timeout (a mechanism to age out sticky table entries)
Timeout active connections (to specify whether to time out
sticky table entries if active connections exists after the
sticky timer expires)
Sticky methods
Sticky connections
Real servers
In <Fig13>, two connections, represented by * line and $ line, are
redirected to VM1, and a sticky table is generated on LB-A. VM1
migrates to new VM1, without Sticky table migrates with VM1. A new
connection, represented by & line, arriving at LB-B, which can not
know that this connection should be redirected to new VM1, because of
lack of sticky table. As a result, LB-B redirect the new connection
to other VM than new VM1. This mis-redirection could cause key
information lost. For example, in e-business, refer to
Gu & Fan Expires December 10, 2011 [Page 20]
Internet-Draft policy and dynamic information migration June 2011
External Client-X
* $ &
* $ &
* $ &
--------------------------------------------
| Gateway |
--------------------------------------------
/*$ \ /\*$&
/*$ \ / \*$& No Sticky table for client-X
------------- /*$ \ / \*$& Connections from client-X
|Sticky Table| /*$ \ / \*$& may be redirected to
------------- /*$ \ / \*$& different servers
----- <*$*$*------- ------ ------- -------- *$&*> ------
|LB-A|------| AGG1 |----| AGG2| | AGG3|----| AGG4 |----- |LB-B|
----- *$*$> ------- ------ ------- --------<*$&* ------
|*$ | | | | | | |*$&
|*$ \ / | | \ / |*$&
|*$ \ / | | \ / |*$&
|*$ \/ | | \/ |*$&
|*$ /\ | | /\ |*$&
|*$ / \ | | / \ |*$&
|*$ / \ | | / \ |*$&
|*$ | | | | | | |*$&
--------- -------- --------- ----------
|Switch1| |Switch2| |Switch3| |Switch4 |
--------- -------- --------- ----------
|*$ \ / | | \ / |*$&
|*$ \/ | | \/ |*$&
|*$ /\ | | / \ |*$&
|*$ / \ | | / \ |*$&
---------- ---------- --------- ----------
|Server1 | |Server2 | |Server3| |Server4 |
---------- ---------- --------- ----------
|$% | |& |*$
|\/ | |\/ |\/
VM1 VM2 VM3 new VM1
******
$$$$$$ Three connectons from the same external clients.
&&&&&&
Figure 13: VM migration without Sticky table migration
While server or server farm move to a new location under a new LB, in
order to keep the existing connections and sessions, as well as
session persistence, both connection table and sticky table need to
move with servers.
Gu & Fan Expires December 10, 2011 [Page 21]
Internet-Draft policy and dynamic information migration June 2011
4. Scenarios
In previous section, we introduce two classes of policies, i.e.
static policies and dynamic information. The introduction is based
on a simplized networking architecture and intentionally omit other
factors. The polices in the example network implies <1:1> partner,
that is polices need to move from one device to another. However, in
real world, network architectures are much complicate than the
simplized example networking. We introduce several scenarios in this
section. Different architecture raise different requirements and
migration partner style.
4.1. Policy migration within a single site
A DC site means a geographic location, which could include one
subnet, or several subnets. A VM migrates to a new location which is
in the same subnet as original location. In this case, there might
not be policies on Middleboxes that need to be migrated, since new
VM1 is under the same Middleboxes as VM1. But dynamic information on
switches need to migrate as well.
Gu & Fan Expires December 10, 2011 [Page 22]
Internet-Draft policy and dynamic information migration June 2011
-------------------------------------------------
| Gateway |
-------------------------------------------------
/ \ / \
/ \ / \
------ -------- -------- -------- -------- ------
|FW-A|---| AGG1 |----| AGG2 | | AGG3 |----| AGG4 |----- |FW-B|
------ -------- -------- -------- -------- ------
| | | | | | | |
| \ / | | \ / |
| \ / | | \ / |
| \/ | | \/ |
| /\ | | /\ |
| / \ | | / \ |
| / \ | | / \ |
| | | | | | | |
---------- ---------- ---------- ----------
|Switch1 | |Switch2 | |Switch3 | |Switch4 |
---------- ---------- ---------- ----------
| \ / | | \ / |
| \/ | | \/ |
| /\ | | /\ |
| / \ | | / \ |
---------- ---------- ---------- ----------
|Server1 | |Server2 | |Server3 | |Server4 |
---------- ---------- ---------- ----------
| |
VM1 new VM1
Figure 14: Policy migration within a single DataCenter site
4.2. Polices migration in asymmetric architecture
The network architecture could be asymmetric. Refer to Fig16. In
this case, there is possible to encounter <1:n> or <n:1> partnership.
The former partnership implies policies on one orginal device might
need to migrate to n destination devices. The latter partnership
implies policies on multiple original devices might need to migrate
to one destination devices. For example, when VM1 migrate from
Server 4 to Server1, the dynamic information on both Switch4 and ToR
need to migrate to Switch1, which is 2:1 partnership。
Gu & Fan Expires December 10, 2011 [Page 23]
Internet-Draft policy and dynamic information migration June 2011
-------------------------------------------------
| Gateway |
-------------------------------------------------
/ \ / \
/ \ / \
------ ------- ------- -------- -------- ------
|FW-A|---| AGG1|----| AGG2 | | AGG3 |----| AGG4 |---|FW-B|
------ ------- ------- -------- -------- ------
| | | | | \ / |
| \ / | | \/ |
| \ / | | /\ |
| \/ | ---------- ----------
| /\ | |Switch3 | |Switch4 |
| / \ | ---------- ----------
| / \ | | \ / |
| | | | | \/ |
--------- ---------- | /\ |
|Switch1| |Switch2 | ---------- ----------
--------- ---------- | ToR | | ToR |
| \ / | ---------- ----------
| \/ | | \/ |
| /\ | | /\ |
| / \ | | / \ |
---------- --------- ---------- ----------
|Server1 | |Server2| |Server3 | |Server4 |
---------- --------- ---------- ----------
| |
new VM1 VM1
Figure 15: Polices migration in asymmetric architecture
4.3. Policies migration between DC sites
Currently, most VM hot migration is limited in the same data center
site. But in the furture there might be requirements for VM
migration between data center sites, as shown in <Fig17>. A use case
for this scenario is resource coordination between data center sites.
A DC provier has two data center sites in different countries, and
both of them serve external clients with similar applications. Some
of the applications are high priority applications, others are normal
priority ones. At first, requests from external clients are
redirected to the data center site which are physically close to the
external clients. However, at some time, one data center is
approaching its capability limits, which could be servers capability
limits (i.e. few server capability are available for new requests) or
network capability limits (i.e. network can not support so much
packet processing requirement, and packet loss happens) and
performance is degraded. In order to guarantee high priority
Gu & Fan Expires December 10, 2011 [Page 24]
Internet-Draft policy and dynamic information migration June 2011
applications, and try best to make fewest damage to normal priority
applications, some VMs, who provide normal priority applications, can
be migrated to the other data center sites.
In this case, policies on switches, middleboxes and gateways need to
migrate to new devices on the other data center site.
Policies migration between data centers also need to consider the
asymmetric structure.
------------------ ---------------------
| Gateway | | Gateway2 |
------------------ ---------------------
/ \ / \
/ \ / \
------ -------- -------- -------- -------- ------
|FW-A|---| AGG1 |----| AGG2 | | AGG3 |----| AGG4 |-----|FW-B|
------ -------- -------- -------- -------- ------
| | | | | | | |
| \ / | | \ / |
| \ / | | \ / |
| \/ | | \/ |
| /\ | | /\ |
| / \ | | / \ |
| / \ | | / \ |
| | | | | | | |
---------- ---------- ---------- ----------
|Switch1 | |Switch2 | |Switch3 | |Switch4 |
---------- ---------- ---------- ----------
| \ / | | \ / |
| \/ | | \/ |
| /\ | | /\ |
| / \ | | / \ |
---------- ---------- ---------- ----------
|Server1 | |Server2 | |Server3 | |Server4 |
---------- ---------- ---------- ----------
| |
VM1 new VM1
Figure 16: Polices migration between DC sites
5. General Considerations
In this section, we enumerate some general considerations on Policy
migration. Only with deep consideration on these factors can we
achieve a successful VM migration and policy migration.
Gu & Fan Expires December 10, 2011 [Page 25]
Internet-Draft policy and dynamic information migration June 2011
5.1. Time-sensitive
VM migration needs a period to finish memory and register copy.
Fig.3 shows the VMware VMotion process. There are three points we
should pay attention to.
Pre-copy period: VM begins to prepare for migration. In this
period, VM pre-copy memory state to the new VM on destination
device. The original VM is still power on and service is still
running, which means the memory and register could keep changing.
The new VM is power-on.
VM not running period: The end phase of memory copy. In this
period, original VM stop running service, the memory will not
change. Original VM finish copying the rest changed memory and
register to new VM. New VM is still power-on.
VM power-off point: After original VM receives the OK message from
new VM, it turns off the power, and meanwhile the new memory
starts to run.
We can see that it's unrealistic to make a 'zero delay' VM migration,
because there is at least about 1 second period (VM not running
period) when neither VM is running.
Assuming there is a 3rd-party device can GET and SET dynamic
information.(This is only an possible way to migrate polices.
Polices could also be migrated distributedly. The author has no
preferrence to any approaches in this draft.) The 3rd-party device
GET dynamic information at Time A, and finish SET at Time B. At Time
A, the Sequence Number of VM1's TCP Connection is 99. After 3rd-
party device GET dynamic information, VM1's TCP connection keeps
transferring packets and Sequence Number increase to 110, until VM
not running period begins. During VM not running period, no TCP
packet is acknowledged by VM1, so the Sequence Number is 110. At
Time B, the destination Firewall is SET by Sequence Number 99. When
new VM1 starts, the packets belonging to VM1's TCP connection comes
to Firewall-B with Sequence Number 111. Since the receiving Sequence
Number doesn't equal to the Acknowleadge Sequence Number of
Firewall-B, this packet will be dropped and the running service is
broken down.
Another example: After Time A, VM1 may establish new TCP connections,
which are not migrated to new Firewall, then following packets
belonging to new TCP connections will be dropped and running services
are disrupted.
Gu & Fan Expires December 10, 2011 [Page 26]
Internet-Draft policy and dynamic information migration June 2011
-------------------------------------------------------
^ || | Power-on destination VM ^
| || | |
| ||--->| --------------------------- VMotion
VM || | Pre-copy memory state abortable
running ||--->| |-------->Time A
on source|| | | /\
| ||--->| |Dynamic Information
| ||--->| |keep changing
V || | | \/
---------------------------------------------------|-----------
^ |---->| Checkpoint-save src VM and |
| |---->| transfer state |
| | |-----------------------------------|
VM not |---->| transfer remaining changed memory |-------->Time B
running | | and checkpoint-restore dst VM V
| | |--------------------------------------
~1sec |<----| send restore OK
V | |
------------------------------------------------------
| || Power-off source VM
| || VM running on destination
Figure 17: VMware VMotion process
5.2. Notification
As Section 5.1shows, policy migration is very sensitive to migration
time. Either later or earlier may cause service disruption. The
best migration time is after the original VM stops running and before
the destination VM begins running. Only Hypervisor knows the acurate
VM status. So we need a notification from server side, Hypervisor in
specific, to tell network side when to migrate polices.
In fact, the author has presented the problem to IEEE 802.1 DCB task
group, which defines VSI Discovery and Configuration protocols (VDP).
DCB has recognized this problem and make effort within its scope.
DCB has extended VDP, in 802.1 Qbg, to enable notification from
Hypervisor to adjacent bridge. DCB expects corresponding IETF WG to
make furthur effort to distribute the notification to devices from
Layer 3 to Layer 7. And IEEE 802.1 DCB working group would like to
see further progress at IETF on this problem.
5.3. Functionality
Data center architecture could be heterogeneous, both in physical
structure and functionality. Section 4.2 and <Fig16> shows the
asymmetric physical structure. The following figure shows the case
Gu & Fan Expires December 10, 2011 [Page 27]
Internet-Draft policy and dynamic information migration June 2011
where functionality on devices is heterogenous. While VM migrates
from one location to a heterogeneous location, it's necessary to
negotiate and make sure that essential functions are supported at
destination location. Otherwise, VM migration maybe fail. Even VM
successfully migrates, there might be security risk and/or service
failure. VM Managers who want to migrate VM between heterogeneous
architecutre must be aware of the risk.
<Fig19> shows two examples of functionality heterogeneous. FW-A is
the source Firewall with packet filtering function, and a dynamic ACL
is generated on FW-A for VM1. FW-B is the destination Firewall
without packet filtering funtion. In this case, dynamic ACL on FW-A
can not be implemented on FW-B. The similar case happens on Switch1
and Switch4. Switch1 has DHCP snooping function enabled, while
Switch has not. DHCP snooping table from Switch1 can not be
implemented on Switch4.
-------------------------------------------------
| Gateway |
-------------------------------------------------
/ \ / \
/ \ / \
------ -------- -------- ------ ------- ------
|FW-A|----| AGG1 |----| AGG2 | | AGG3|----| AGG4|---|FW-B|
------ -------- -------- ------ ------- ------
with | | | | | | | | without
Packet | \ / | | \ / | Packet filtering
filtering | \ / | | \ / |
| \/ | | \/ |
| /\ | | /\ |
| / \ | | / \ |
| / \ | | / \ |
| | | | | | | |
---------- ---------- --------- ---------
with |Switch1 | |Switch2 | |Switch3| |Switch4| withou
DHCP ---------- ---------- --------- --------- DHCP
Snooping | \ / | | \ / | Snooping
| \/ | | \/ |
| /\ | | /\ |
| / \ | | / \ |
---------- ---------- ---------- ----------
|Server1 | |Server2 | |Server3 | |Server4 |
---------- ---------- ---------- ----------
| |
VM1 new VM1
Figure 18: Heterogeneous functions on devices
Gu & Fan Expires December 10, 2011 [Page 28]
Internet-Draft policy and dynamic information migration June 2011
5.4. Resource Capability
Even if destination devices can support specific function that is
required by dynamic information migration, there might be not enough
resources to implement migrated polices. For example, there are 5
connection state items related to VM1 on source LB. However, the
destination LB has only 4 table items available. That means the
destination LB doesn't have enough resource to support this VM
migration, and policies migration on network devices may fail.
5.5. Migration Feedback
Sometimes, VM manager has ensure, by some way, that proper
functionality and plenty of resources are available on destionation
devices, by which we mean all the devices that need to handle VM's
packets, but dynamic information migration still fails. In this
case, VM manager or Hypervisor needs to know the results of policy
migration. The feedback from network can help VM Manager to decide
whether to rollback the migration or continue migration with risk.
6. Security Considerations
The policies and dynamic information described above are all about
security. Besides, we need to be careful to avoid poisoned polices
from untrusted source. That means no mather how the polices are
migrated, be it distributed or centralized way, authentication and
verification are required.
A reliable notifation from server side to network side is also
necessary, so that the destination and original devices can be sure
that a real VM migration happens.
7. Acknowledgments
I would like to thank the following people for contributing to this
draft: Ning Zong, David harrington, Linda dunbar, Susan Hares, Serge
manning, Barry Leiba, Jiang xingfeng, Song Wei, Robert Sultan, and
many others.
8. References
8.1. Normative Reference
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", March 1997.
Gu & Fan Expires December 10, 2011 [Page 29]
Internet-Draft policy and dynamic information migration June 2011
[RFC3303] Srisuresh, P., Kuthan, J., Rosenberg, J., Molitor, A., and
A. Rayhan, "Middlebox communication architecture and
framework", August 2002.
8.2. Informative Reference
[Data_Center_Fundamentals]
"Data Center Fundamentals", 2003.
Authors' Addresses
Gu Yingjie
Huawei
No. 101 Software Avenue
Nanjing, Jiangsu Province 210001
P.R.China
Phone: +86-25-56624760
Fax: +86-25-56624702
Email: guyingjie@huawei.com
Fan Yongbing
China Telecom
No. 109 Zhongshan Road West
Guangzhou, Guangdong Province
P.R.China
Phone: 86-20-38639121
Fax: 86-20-38639487
Email: fanyb@gsta.com
Gu & Fan Expires December 10, 2011 [Page 30]