MIF Working Group T. Zheng
Internet Draft X. Deng
Intended status: Informational France Telecom
Expires: January 11, 2012 D. Wing
Cisco
L. Wang
France Telecom
Y. Lee
Comcast
July 10, 2011
Applications Test Results in MIF environment
draft-zheng-mif-apps-test-02.txt
Abstract
With the development of IPv6 and deployment of transition solutions,
more and more hosts can access Internet with dual stack by multi-
interface and applications behaviors are worth to be concerned under
this environment. This memo describes the test results of some well-
known applications in different scenarios under MIF environment and
provides an analysis and suggestions to develop and deploy under MIF
environment.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 11, 2012.
Copyright Notice
Copyright (c) 2011 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Zheng, et al. Expires January 11, 2012 [Page 1]
Internet-Draft MIF Applications Test Results July 11, 2011
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this
Section 4.e of the Trust Legal Provisions and are provided without
warranty as described in the Simplified BSD License.
Table of Contents
1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Scenario 1: One network adapter with dual stack . . . . . . . 4
2.1. Test environment. . . . . . . . . . . . . . . . . . . . . 4
2.2. Test results on Windows . . . . . . . . . . . . . . . . . 5
2.2.1. Behaviors of DNS lookups . . . . . . . . . . . . . . 5
2.2.2. Applications results . . . . . . . . . . . . . . . . 5
2.2.3. Problems and suggestions . . . . . . . . . . . . . . 6
2.3. Test results on Fedora. . . . . . . . . . . . . . . . . . 6
2.3.1. Behaviors of DNS lookups . . . . . . . . . . . . . . 6
2.3.2. Applications results . . . . . . . . . . . . . . . . 6
2.3.3. Problems and suggestions . . . . . . . . . . . . . . 7
3. Scenario 2: Two network adapters respectively with IPv6 and IPv4
connections. . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1. Test environment. . . . . . . . . . . . . . . . . . . . . 7
3.2. Test results on Windows . . . . . . . . . . . . . . . . . 8
3.2.1. Behaviors of DNS lookups . . . . . . . . . . . . . . 8
3.2.2. Applications results . . . . . . . . . . . . . . . . 9
3.2.3. Problems and suggestions . . . . . . . . . . . . . . 9
3.3. Test results on Fedora. . . . . . . . . . . . . . . . . . 9
4. Transition solutions in MIF . . . . . . . . . . . . . . . . . 9
4.1. Case 1: AplusP and IPv6 provisioned Host. . . . . . . . . 10
4.1.1. Test Environment and DNS configuration . . . . . . . 10
4.1.2. Test Results and suggestions . . . . . . . . . . . . 10
4.2. Case 2: DS-Lite and IPv6 provisioned Host . . . . . . . . 11
4.2.1. Test Environment and DNS configuration . . . . . . . 11
4.2.2. Test Results and suggestions . . . . . . . . . . . . 11
4.3. Case 3: NAT64 and DS-Lite provisioned Host. . . . . . . . 11
4.3.1. Use Case . . . . . . . . . . . . . . . . . . . . . . 12
4.3.2. Test Environment and DNS configuration . . . . . . . 12
4.3.3. NAT64 impacts on applications and suggestions. . . . 13
4.3.4. Test in a both NAT64 and DS-Lite accessible network and
suggestions . . . . . . . . . . . . . . . . . . . . . . . . 14
5. Security Considerations . . . . . . . . . . . . . . . . . . . 15
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15
7. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 15
Zheng, et al. Expires January 11, 2012 [Page 2]
Internet-Draft MIF Applications Test Results July 11, 2011
8. References. . . . . . . . . . . . . . . . . . . . . . . . . . 15
8.1. Normative References . . . . . . . . . . . . . . . . . . 15
8.2. Informative References. . . . . . . . . . . . . . . . . . 15
Zheng, et al. Expires January 11, 2012 [Page 3]
Internet-Draft MIF Applications Test Results July 11, 2011
1. Introduction
The IPv6 introduction and deployment of IPv6 to IPv4 transition
solutions increase the chances of multi-interface hosts, which face
some challenges described in the MIF problem statement, [I-D.ietf-
mif-problem-statement]. The applications on these hosts will be
impacted not impacted on single-interface hosts. This document
provides some test results of well-known applications in MIF and
transition solutions environment.
The tested applications include:
o Windows 7: ping, IE 9, Live Messenger 2009, Skype 5.1, Firefox 3.6,
Google Earth 6
o Fedora 13: ping and ping6, Skype 2.2, Firefox 3.6, Google Earth 6.
The following scenarios are tested:
o Host has one network adapter with dual stack
o Host has two network adapters respectively with IPv6 and IPv4
Connections
The transition solutions in the test include AplusP [I-D.ymbk-
aplusp], DS-Lite [I-D.ietf-softwire-dual-stack-lite], NAT64
[RFC6145]/DNS64 [RFC 6147] and the combination of them.
The test behinds two DNS Servers that one in IPv4 Internet is not on
Google's IPv6 whitelist and another in IPv6 Internet is on it.
2. Scenario 1: One network adapter with dual stack
2.1. Test environment
In this scenario as showed in the Figure 1, the host accesses
Internet by one network adapter with IPv6 and IPv4 addresses.
Applications run on the host equipped with Windows 7 or Fedora 13
system. DNS Servers are in IPv6 Internet and IPv4 Internet
respectively.
Zheng, et al. Expires January 11, 2012 [Page 4]
Internet-Draft MIF Applications Test Results July 11, 2011
IPv4 DNS Server IPv6 DNS Server
| |
-+--IPv4 Internet--- ---IPv6 Internet--+-
| |
| +--------+ |
+--------| CPE |--------+
+--------+
| |
IPv4 | | IPv6
Logical | | Logical
Link | | Link
+--------+
| Host |
+--------+
Figure 1 Test environment of Scenario 1
2.2. Test results on Windows
2.2.1. Behaviors of DNS lookups
When application needs to lookup a host name, which one DNS Server is
selected and which type (A or AAAA) query is sent are decided by
Windows system. In this scenario, the addresses of DNS Servers are
provided through DHCP messenger to system. The IPv6 DNS Server has
high priority. The system sends an A type query to IPv6 DNS Server at
first. If the non-NXDOMAIN error response of the A type query is
received, the system sends an AAAA query to IPv6 DNS Server, then
provides DNS records coming from IPv6 DNS Server to applications and
the DNS lookup finishes. Otherwise, the system sends an A type query
to IPv4 DNS Server, then sends an AAAA query if the non-NXDOMAIN
error response of the A type query is received, then DNS records
coming from IPv4 DNS Server are provided to applications.
2.2.2. Applications results
In this scenario, if the tested application is IPv6-enabled and the
AAAA resource record is available, IPv6 has higher priority to be
selected to communicate.
Ping, IE and Firefox: IPv6 has high priority if AAAA resource record
is available;
Google Earth: IPv6;
Skype and Live Messenger: IPv4 (because AAAA resource record is
unavailable)
Zheng, et al. Expires January 11, 2012 [Page 5]
Internet-Draft MIF Applications Test Results July 11, 2011
2.2.3. Problems and suggestions
A problem was observed with Windows 7 when querying a host name that
has only AAAA records (and no A or CNAME records), and receives
NXDOMAIN from a faulty DNS server ([RFC4074], [CERT-714121]). Windows
7 always first queries a host name's A record, and if it receives an
NXDOMAIN it will not query for an AAAA record. To accommodate this
behavior, Windows 7 should ask for both type records in each DNS
lookup or a host name that has only an AAAA record should have a
CNAME which points to the AAAA.
2.3. Test results on Fedora
2.3.1. Behaviors of DNS lookups
In this scenario, the addresses of DNS Servers are configured in
/etc/resolv.conf. The configured order decides which DNS Server has
high priority.
When an application can be decided which type queries it needs, such
as ping (A type) or ping6 (AAAA type), the system will send this type
query to the DNS Servers. Otherwise, the system will send A and AAAA
queries to the DNS Server at the same time. The system sends DNS
query to DNS Servers according to the configured order in
resolv.conf. Once the non-NXDOMAIN response of the queries is
received, the query process will stop and the query results will be
returned to the application. Otherwise, the system will query next
DNS Server in the order.
2.3.2. Applications results
In this scenario, which type IPv6 or IPv4 is used to communicate by
the applications is impacted by the DNS Server configured order,
records in queried zone and the applications' selection.
o IPv6 DNS Server has high priority
Ping: IPv4 (just sending A query);
Ping6: IPv6 (just sending AAAA query);
Firefox: IPv6 has high priority if AAAA resource record available;
Google Earth: IPv6;
Skype: IPv4 (because AAAA resource record is unavailable)
o IPv4 DNS Server has high priority
Zheng, et al. Expires January 11, 2012 [Page 6]
Internet-Draft MIF Applications Test Results July 11, 2011
Ping: IPv4 (just sending A query);
Ping6: IPv6 (just sending AAAA query and if the queried host name
does not have an AAAA resource record through querying IPv4 DNS
Server, system will ask IPv6 DNS Server for AAAA record);
Firefox: IPv4, IPv6 only in the case: the AAAA record of the
queried host name can be returned in the additional records section
in the A query response, such as www.kame.net or be gotten through
AAAA query response;
Google Earth: IPv4;
Skype: IPv4 (because IPv6 server side is unavailable)
2.3.3. Problems and suggestions
Because the DNS Servers order configured in resolv.conf impacts the
protocol type used to communicate by applications, the IPv6 DNS
Server should be configured with high priority or the AAAA record of
the queried host name can be returned in the additional records
section in the A query response if IPv6 is wished for.
Note: analysis of how Fedora orders its DNS servers when it is DHCP-
configured is for future study.
3. Scenario 2: Two network adapters respectively with IPv6 and IPv4
connections
3.1. Test environment
In this scenario as showed in the Figure 2, the host has IPv6 and
IPv4 two connections to accesses Internet by two network adapters.
Applications run on the host equipped with Windows 7 or Fedora 13
system. DNS Servers are in IPv6 Internet and IPv4 Internet
respectively.
Zheng, et al. Expires January 11, 2012 [Page 7]
Internet-Draft MIF Applications Test Results July 11, 2011
IPv4 DNS Server IPv6 DNS Server
| |
-+--IPv4 Internet---- ----IPv6 Internet--+-
| |
| |
+-------+ +-------+
|WLAN AP| | CPE |
+-------+ +-------+
| |
IPv4 | | IPv6
Link | +--------+ | Link
+--------| Host |--------+
+--------+
Figure 2 Test environment of Scenario 2
3.2. Test results on Windows
In this scenario, there are two cases:
o Case 1: The two interfaces of IPv6 and IPv4 all with IPv6 and IPv4
dual protocol stacks
o Case 2: IPv6 interface with IPv6-only protocol stack and IPv4
interface with IPv4-only protocol stack
3.2.1. Behaviors of DNS lookups
In this scenario, the addresses of DNS Servers are provided through
DHCP messenger to system. The order of two adapters in which they can
be accessed can be configured in Windows system. However, in our
test, the order of adapters had no impact on the priority of DNS
Servers. Compared with IPv4 DNS server, the IPv6 DNS Server always
has high priority in Case 1 and low priority in Case 2.
In Case 1, the IPv6 DNS Server has high priority. Behavior of DNS
lookups is same as in section 2.2.1.
In Case 2, the IPv4 DNS Server has high priority. The system sends an
A type query to IPv4 DNS Server at first. If the non-NXDOMAIN
response of the A type query is received, the system provides DNS
records coming from IPv4 DNS Server to applications and the DNS
lookup stops. Otherwise, the system sends an A type query to IPv6 DNS
Server, then sends an AAAA query if the non-NXDOMAIN response of the
A type query is received and DNS records coming from IPv6 DNS Server
are provided to applications.
Zheng, et al. Expires January 11, 2012 [Page 8]
Internet-Draft MIF Applications Test Results July 11, 2011
3.2.2. Applications results
In Case 1, the results are same as in section 2.2.2.
In Case 2:
Ping, IE and Firefox: IPv4 (most of websites), IPv6 only when the
AAAA record of the queried host name can be returned in the
additional records section in the A query response, such as
www.kame.net;
Google Earth: IPv4 (because only A record is got from IPv4 DNS
Server);
Skype and Live Messenger: IPv4 (because AAAA resource record is
unavailable)
3.2.3. Problems and suggestions
If IPv6 is wished for, the configuration of case 2 should be avoided.
In case 1, to avoid the problem same as in section 2.2.3, a host name
that has only an AAAA record should have a CNAME which points to the
AAAA.
3.3. Test results on Fedora
The configuration of DNS Server is same as in scenario 1 and the test
results are same as in section 2.3.2.
4. Transition solutions in MIF
To investigate MIF issues in the context of transition solutions,
three transition solutions Aplusp, DS-Lite and NAT64 are enabled in
the test bed, and three cases of combination are tested.
Zheng, et al. Expires January 11, 2012 [Page 9]
Internet-Draft MIF Applications Test Results July 11, 2011
4.1. Case 1: AplusP and IPv6 provisioned Host
4.1.1. Test Environment and DNS configuration
IPv4 DNS Server IPv6 DNS Server
| |
-+--IPv4 Internet--- ---IPv6 Internet--+-
| |
+----------+ |
| PRR | |
=== +----------+ |
IPv4 | | | |
| | | |
| +--------------+ |
| | PPPoE/DHCPv6 |-------------------+
over | | Server |
| +--------------+
| IPv6 | |
| over | |
IPv6 | PPPoE | |
=== | |
+----------+
| A+P |
| CPE |
+----------+
IPv4 | | IPv6
Logical | | Logical
Link | | Link
+----------+
| Host |
+----------+
Figure 3 : Test Environment
For A+P CPE and PRR implementation, please refer to [I-D. deng-
aplusp-experiment-results].
4.1.2. Test Results and suggestions
Please refer to case 2 in Section 2.2.
Zheng, et al. Expires January 11, 2012 [Page 10]
Internet-Draft MIF Applications Test Results July 11, 2011
4.2. Case 2: DS-Lite and IPv6 provisioned Host
4.2.1. Test Environment and DNS configuration
IPv4 DNS Server IPv6 DNS Server
| |
-+--IPv4 Internet--- ---IPv6 Internet--+-
| |
+----------+ |
| AFTR | |
=== +----------+ |
IPv4 | | | |
| | | |
| +--------------+ |
| | PPPoE/DHCPv6 |-------------------+
over | | Server |
| +--------------+
| IPv6 | |
| over | |
IPv6 | PPPoE | |
=== | |
+----------+
| B4 |
| CPE |
+----------+
IPv4 | | IPv6
Logical| | Logical
Link | | Link
+----------+
| Host |
+----------+
Figure 4 : Test Environment
Tested AFTR is provided by a Vendor.
4.2.2. Test Results and suggestions
Please refer to Section 2.2.
4.3. Case 3: NAT64 and DS-Lite provisioned Host
A concern that NAT64 may break existing popular applications has been
arising. Hence, applications' NAT64 compatibility has been tested and
tests results are listed in the section 4.5. In addition, since
solutions addressing different scenarios may co-exist in the same
subscribers' network, it is necessary to understand potential
Therefore, both Dual-stack Lite and NAT64 are enabled in the test bed
Zheng, et al. Expires January 11, 2012 [Page 11]
Internet-Draft MIF Applications Test Results July 11, 2011
network, to see whether apps work properly, to see whether apps
choose IPv6 prior to IPv4 or if they try both, and the corresponding
results are shown in the section 4.6.
Test Environments and DNS configuration details are described in
Section 4.3.2.
4.3.1. Use Case
Consider a host (e.g. Smartphone) which contains both wifi and
wireless interfaces. When it connects a dual-stack network provided
by a B4 element via wifi and also connects to an IPv6-only network
via 3G/4G provided by a wireless ISP. The wireless IPv6-only networks
has NAT64/DNS64 deployed.
4.3.2. Test Environment and DNS configuration
An open-source implementation of a NAT64/DNS64 gateway funded by the
NLnet Foundation and Viageie, Ecdysis has been used as NAT64/DNS64
gateway. The host runs Windows XP.
A DS-Lite AFTR is provided by a Vendor.
IPv4 DNS Server
|
--+-----IPv4 Internet---------+-
DNS | |
queries | |
| |
+-----------+ +----+
|NAT64/DNS64| |AFTR|
+-----------+ +----+------
| |
IPv6 | DNS | IPv4
Packet| queries | in
| | IPv6
| +--------+ | Tunnel
| | B4/ | |
+--------| Dnsmasq|------+--------
+--------+
IPv6 | | IPv4
Logical| |Logical
Link | | Link
+--------+
| Host |
+--------+
Figure 5 : Test Environment
Zheng, et al. Expires January 11, 2012 [Page 12]
Internet-Draft MIF Applications Test Results July 11, 2011
A Dnsmasq (a DNS forwarder), whose upstream DNS server is configured
to DNS64, was installed in the B4 element as shown in figure 5. DNS64
delegates A queries via IPv4 Internet and returns the A RRs to
Dnsmasq, and also responsible for generating AAAA RRs to AAAA queries
by forming a DNS64 prefix from A RRs and NAT64's prefix.
Both AAAA and A RRs are then delegated by Dnsmasq to the Host behind
a B4.
4.3.3. NAT64 impacts on applications and suggestions
Introducing NAT64 may bring impacts, for instance some applications
may break due to the IP protocol translation. First, AFTR
provisioning was switched off to see applications' compatibility with
NAT64. Results are shown in the figure 6.
+-------------------+--------------------------------------+
| Application | NAT64 |
+-------------------+--------------------------------------+
| Firefox v3.6.12 | works fine except watching video |
| | on line failed |
+-------------------+--------------------------------------+
| IE v6.0 | works fine except watching video |
| | on line failed |
+-------------------+--------------------------------------+
| Google Earth | works fine |
| v5.2.1 | |
+-------------------+--------------------------------------+
| Skype v5.0 | works fine |
| | |
+-------------------+--------------------------------------+
| Live Messenger | fails (see suggestions blow) |
| 2009 | |
+-------------------+--------------------------------------+
| uTorrent v2.2 | fails (see suggestions blow) |
| | |
+-------------------+--------------------------------------+
| BitComet v1.23 | fails (not IPv6 enabled) |
+-------------------+--------------------------------------+
Figure 6 : Applications' compatibility with NAT64
Live Messenger uses IPv6 to login to the server, but the IPv6 client
did not send the same application layer message as the IPv4 client,
so the IPv6 client (behind NAT64) failed to get reply from IPv4
server. Therefore, the application layer of Live Messenger should be
IP version independent.
There are two suggestions for P2P applications:
Zheng, et al. Expires January 11, 2012 [Page 13]
Internet-Draft MIF Applications Test Results July 11, 2011
o There is a need for P2P applications behind NAT64 to learn the
NAT64's prefix, so that IPv6 peer can talk with IPv4 peers via
NAT64.
o Otherwise, IPv4 connectivity is required to support remaining
access to IPv4 peer, according to test results suggested in section
4.3.4.
4.3.4. Test in a both NAT64 and DS-Lite accessible network and
suggestions
In this scenario, both AFTR and NAT64/DNS64 are provisioned to B4.
Applications works fine with a both NAT64 and DS-Lite accessible
network.
Apps' behaviors in terms of IP preference, more specifically, how
applications prioritize AAAA and A DNS records and which IP version
protocol (IPv4 or IPv6) is preferred to initiate communication, are
described in the figure 7.
+-------------------+--------------------------------------+
| Application | Both NAT64&DS-Lite accessible |
+-------------------+--------------------------------------+
| Firefox v3.6.12 | video content via DS-Lite; |
| | other content via NAT64 |
+-------------------+--------------------------------------+
| IE v6.0 | video content via DS-Lite; |
| | other content via NAT64 |
+-------------------+--------------------------------------+
| Google Earth | NAT64 |
| v5.2.1 | |
+-------------------+--------------------------------------+
| Skype v5.0 | NAT64 |
| | |
+-------------------+--------------------------------------+
| Live Messenger | DS-Lite |
| 2009 | (see Section 4.7 |
| | for more information) |
+-------------------+--------------------------------------+
| uTorrent v2.2 | DS-Lite |
+-------------------+--------------------------------------+
| BitComet v1.23 | DS-Lite |
+-------------------+--------------------------------------+
Figure 7 : Applications' transition solution preference
Except BitComet, which although issues both A and AAAA quires,
Zheng, et al. Expires January 11, 2012 [Page 14]
Internet-Draft MIF Applications Test Results July 11, 2011
ignores AAAA RR and only uses IPv4 to initiate communication, all the
other applications first try A RR to initiate IPv6 communications,
and if it fails A RR is used to initiate IPv4 communications.
Live messenger first fails IPv6 due to DNS64 translation, so it is
suggested to avoid that translation when the application uses DNS64,
as described in [I-D.wing-behave-dns64-config].
Yet, after failing IPv6 Live messenger automatically switches to IPv4
by which all the rest of communication were done. What we have
learned from this test case is that the application layer should be
IP version agnostic in order to decrease impacts introduced by IPv6
transition solutions.
5. Security Considerations
TBD
6. IANA Considerations
This document includes no request to IANA.
7. Conclusions
Despite some mainstream operating systems have already support IPv6,
but in terms of our test results, there are still some issues
impacting on the IPv6 practice, especially under MIF and transition
solutions environment. It's worth to be considered during IPv6
introduction.
8. References
8.1. Normative References
[I-D.ietf-mif-problem-statement]
Blanchet, M. and P. Seite, "Multiple Interfaces and
Provisioning Domains Problem Statement", draft-ietf-mif-
problem-statement-15 (work in progress), May 2011.
8.2. Informative References
[I-D.ymbk-aplusp]
R. Bush, "The A+P Approach to the IPv4 Address Shortage",
draft-ymbk-aplusp-10 (work in progress), May 2011.
[I-D.ietf-softwire-dual-stack-lite]
Zheng, et al. Expires January 11, 2012 [Page 15]
Internet-Draft MIF Applications Test Results July 11, 2011
A. Durand, "Dual-Stack Lite Broadband Deployments Following
IPv4 Exhaustion", draft-durand-softwire-dual-stack-lite-11
[RFC 6145] X. Li, "IP/ICMP Translation Algorithm", RFC 6145, April
2011.
[RFC 6147] M. Bagnulo, "DNS64: DNS Extensions for Network Address
Translation from IPv6 Clients to IPv4 Servers", RFC 6147,
April, 2011.
[RFC 4074] Y. Morishita, "Common Misbehavior Against DNS Queries for
IPv6 Addresses", RFC 4074, May 2005.
[CERT-714121] https://www.kb.cert.org/vuls/id/714121.
[I-D.deng-aplusp-experiment-results]
X.Deng, "Implementing A+P in the provider's IPv6-only
network", draft-deng-aplusp-experiment-results-00 (work on
progress), March, 2011.
[I-D.wing-behave-dns64-config]
D.Wing, "IPv6-only and Dual Stack Hosts on the Same Network
with DNS64", draft-wing-behave-dns64-config-03 (work on
progress), February, 2011
Zheng, et al. Expires January 11, 2012 [Page 16]
Internet-Draft MIF Applications Test Results July 11, 2011
Authors' Addresses
Tao Zheng
France Telecom
Hai dian district, 100190, Beijing, China
Email: tao.zheng@orange-ftgroup.com
Xiaohong Deng
France Telecom
Hai dian district, 100190, Beijing, China
Email: xiaohong.deng@orange-ftgroup.com
Lan Wang
France Telecom
Hai dian district, 100190, Beijing, China
Email: lan.wang@orange-ftgroup.com
Dan Wing
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134
USA
Email: dwing@cisco.com
Yiu L. Lee
Comcast
One Comcast Center
Philadelphia, PA 19103
USA
Email: yiu_lee@cable.comcast.com
Zheng, et al. Expires January 11, 2012 [Page 17]