CORE WG F. Zheng
INTERNET-DRAFT B. Fu
Intended Status: Informational Z. Cao (Ed.)
Expires: Feb 3 , 2016 Huawei Tech
July 4, 2016
CoAP Latency Evaluation
draft-zheng-core-coap-lantency-evaluation-00
Abstract
This document presents the evaluation results of CoAP in terms of
various latency metrics over UDP/TCP under different network
environments. We conduct experiments via both GPRS and WiFi. We
also evaluate how the latency metrics are impacted by the background
traffic.
Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as
Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/1id-abstracts.html
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
Copyright and License Notice
Copyright (c) 2016 IETF Trust and the persons identified as the
document authors. All rights reserved.
Expires <Expiry Date> [Page 1]
INTERNET DRAFT draft-zheng-core-coap-latency-evaluation <Issue Date>
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1 General topology . . . . . . . . . . . . . . . . . . . . . . 3
2.2 CoAP Client . . . . . . . . . . . . . . . . . . . . . . . . 3
2.3 CoAP Server . . . . . . . . . . . . . . . . . . . . . . . . 3
2.4 Bare Latency . . . . . . . . . . . . . . . . . . . . . . . . 4
2.5 Packet loss rate . . . . . . . . . . . . . . . . . . . . . . 4
2.6 Tested flows . . . . . . . . . . . . . . . . . . . . . . . . 4
3 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
4.1 How much CoCoA helps? . . . . . . . . . . . . . . . . . . . 5
4.2 How much impact of the competing flows under GPRS and WiFi . 6
4.4 How much difference of CoAP-TCP and CoAP-over-UDP . . . . . 7
5. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
6 Security Considerations . . . . . . . . . . . . . . . . . . . . 8
7 IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 8
8 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . 8
9 References . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 9
Expires <Expiry Date> [Page 2]
INTERNET DRAFT draft-zheng-core-coap-latency-evaluation <Issue Date>
1 Introduction
CoAP [RFC7252] is a light-weight application protocol for constrained
networks. CoAP has been widely used in many Internet of Things
scenarios and deployed by many products as well.
In order to know how CoAP performs under various network environment,
we conduct the experimental study and present the results in this
document. This work is a complementary work of the previous
evaluation work presented in [COAPCCCOMP] and [COAPCC]. In
particular, (a) we intentionally measure the latency related metrics;
(b) we introduce competing flows while testing, which reveals the
difference of Bufferbloat impact under GPRS and WiFi; (c) we conduct
study in both cellular networks (GPRS) and WiFi network.
This document presents the evaluation setup and results, and intents
to provide insights for further protocol design and implementation.
2 Experiment Setup
This section describes the evaluation setup.
2.1 General topology
The topology is depicted in Fig.1
---GPRS--+ ________
| | ( )
CoAP Client-{ +- + Internet +----+CoAP Server
| | (________)
- Wi-Fi--+
Figure 1: Topology
2.2 CoAP Client
We use an Android phone as the CoAP client, which has both cellular and
Wi-Fi interfaces. This device is physically located at our lab.
libcoap is ported to this device with the extension defined in CoCoA
algorithm [CoCoA] and CoAP over TCP [COT].
2.3 CoAP Server
The CoAP server is a rented Linux server from some public cloud company
in China. We also use libcoap and our extensions on the server side.
Expires <Expiry Date> [Page 3]
INTERNET DRAFT draft-zheng-core-coap-latency-evaluation <Issue Date>
2.4 Bare Latency
The ICMP RTT between the client and server is as below.
| | GPRS | Wi-Fi |
+------+-----------+-----------+
| RTT | ~572ms | ~19ms |
+------+-----------+-----------+
2.5 Packet loss rate
We conduct these tests with the native packet loss rate, as well as
increased packet loss rate by using Linux TC command at the server (+2%
and +5% separately). We use 0% to denote the native packet loss rate.
2.6 Tested flows
The client sends 100 CoAP requests to the server, with interval of 100
ms. The size of the CoAP request is 9 Bytes, and the response payload
size is 23 Bytes. We believe this reflects a certain IoT scenario,
e.g., the reporting of data or status from device to server. For TCP
flows, long-lived connection has been established before the test, so
that TCP three-way handshake overhead has not been taken into account.
We also introduce background traffic as the competing flow while
measuring the latency. The background traffic is a FTP downloading
flow.
3 Metrics
We intend to evaluate the latency of the CoAP flow with the following
metrics:
1. Flow completing time (FCT): defined as the time elapsed from the
sending of first request until the receiving of the last response.
2. CoAP RTT (C-RTT): defined as the average elapsed time between
the sending of the original CoAP request and receiving of the CoAP
response. Note: if the original request has been lost and then
retransmitted when RTO is triggered, it will certainly increase the
CoAP-RTT.
3. Retransmission ratio (RTR): defined as the percentage of CoAP-
layer retransmission. The maximum number of retransmission is
Expires <Expiry Date> [Page 4]
INTERNET DRAFT draft-zheng-core-coap-latency-evaluation <Issue Date>
defined as FOUR in [RFC7252] for UDP.
4 Results
4.1 How much CoCoA helps?
Table.1 summarizes the CoAP latency performance with and without
CoCoA. The results clearly indicate that CoCoA effectively reduces
the latency, especially when the packet loss rate is high, due to
its dynamic calculation and update of RTO. The trade-off of CoCoA
is its payment of more retransmission, alike other congestion
control algorithm.
GPRS | CoAP-RAW | CoAP-CoCoA |
+------+---------------+----------------+
| FCT |+0%* | 10173 |+0% | 10175 |
|(ms) +---------------+----------------+
| |+2% | 10868 |+2% | 10170 |
| +---------------+----------------+
| |+5% | 10697 |+5% | 10176 |
+------+---------------+----------------+
|C-RTT |+0% | 178 |+0% | 172 |
|(ms) +---------------+----------------+
| |+2% | 243 |+2% | 182 |
| +---------------+----------------+
| |+5% | 332 |+5% | 203 |
+------+---------------+----------------+
| RTR |+0% | 0 |+0% | 7 |
| (% ) +---------------+----------------+
| |+2% | 2.7 |+2% | 18 |
| +---------------+----------------+
| |+5% | 5.3 |+5% | 9 |
+++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++
WiFi | CoAP-RAW | CoAP-CoCoA |
+------+---------------+----------------+
| FCT |+0% | 9988 |+0% | 10175 |
|(ms) +---------------+----------------+
| |+2% | 10818 |+2% | 10170 |
| +---------------+----------------+
| |+5% | 10889 |+5% | 10176 |
+------+---------------+----------------+
|C-RTT |+0% | 18 |+0% | 18 |
|(ms) +---------------+----------------+
| |+2% | 105 |+2% | 18 |
| +---------------+----------------+
Expires <Expiry Date> [Page 5]
INTERNET DRAFT draft-zheng-core-coap-latency-evaluation <Issue Date>
| |+5% | 180 |+5% | 18 |
+------+---------------+----------------+
| RTR |+0% | 0 |+0% | 3.5 |
| (% ) +---------------+----------------+
| |+2% | 3.3 |+2% | 5.3 |
| +---------------+----------------+
| |+5% | 6.3 |+5% | 6.8 |
+------+---------------+----------------+
* 0% indicates the native packet loss rate, without
any manually introduced packet loss rate.
Table.1 CoAP latency performance with and without CoCoA
4.2 How much impact of the competing flows under GPRS and WiFi
Table.2 summarizes the results. With competing flows under GRPS, both
FCT and C-RTT deteriorate heavily, and especially C-RTT has been
increased by around seven times. Comparatively, in Wi-Fi case,
existence of the competing flows does not deteriorates the two latency
factors much.
The latency bloat given the existence of competing flows most likely
stems from the famous buffer-bloat problem. The GPRS and WiFi differ
from each other because of their queue management deployed in the
middle. This result we present here about GPRS is consistent with what
have been reported in [BufbloatLTE]. In addition, we find the WiFi
suffers less from Bufferbloat than GPRS.
GPRS | CoAP-RAW | CoAP-CoCoA |
+------+---------------+----------------+
| FCT |wo | 10173 |wo | 10175 |
|(ms) +---------------+----------------+
| |w | 11270 |w | 11288 |
+------+---------------+----------------+
|C-RTT |wo | 178 |wo | 172 |
|(ms) +---------------+----------------+
| |w | 1344 |w | 1315 |
+------+---------------+----------------+
WiFi | CoAP-RAW | CoAP-CoCoA |
+------+---------------+----------------+
| FCT |wo | 9988 |wo | 9985 |
|(ms) +---------------+----------------+
| |w | 9969 |w | 9969 |
+------+---------------+----------------+
|C-RTT |wo | 18 |wo | 18.3 |
|(ms) +---------------+----------------+
| |w | 30.9 |w | 25.4 |
Expires <Expiry Date> [Page 6]
INTERNET DRAFT draft-zheng-core-coap-latency-evaluation <Issue Date>
+------+---------------+----------------+
Table.2 CoAP latency performance under competing flows
(wo=without competing flow, w=with competing flow)
4.4 How much difference of CoAP-TCP and CoAP-over-UDP
We implement the CoAP-TCP per [COAP-T], which is still working-in-
progress. The C-RTT comparison of the two algorithm is shown in
Table.3. In the CoAP-TCP implementation, we used the default Linux
congestion control algorithm -Cubic. In the test, we first establish
the TCP connection and then run the CoAP flows, which means that the
handshake overhead has not been included into the results. CoAP-TCP
bears a longer latency possibly because of (a) TCP header overhead (20
bytes as composed to 8 bytes, and TCP/UDP header constitutes a larger
portion of the packet due to the small CoAP payload; (b) GPRS low data
rate, which makes this effect more obvious in GPRS than in WiFi. (d)
under WiFi, retransmission contributes more to latency than packet
length.
GPRS | CoAP-CoCoA | CoAP-TCP |
+------+---------------+----------------+
|C-RTT |+0% | 172 |+0% | 450 |
|(ms) +---------------+----------------+
| |+2% | 182 |+2% | 421 |
| +---------------+----------------+
| |+5% | 203 |+5% | 454 |
+------+---------------+----------------+
WiFi | CoAP-CoCoA | CoAP-TCP |
+------+---------------+----------------+
|C-RTT |+0% | 18 |+0% | 18 |
|(ms) +---------------+----------------+
| |+2% | 18 |+2% | 31 |
| +---------------+----------------+
| |+5% | 18 |+5% | 40 |
+------+---------------+----------------+
Table.3 CoAP latency performance of UDP-CoCoA and CoAP-TCP
5. Summary
As a summary, we find:
1. CoCoA algorithm helps shorten the CoAP message layer latency
effectively for our tested cases.
Expires <Expiry Date> [Page 7]
INTERNET DRAFT draft-zheng-core-coap-latency-evaluation <Issue Date>
2. With competing flows under GRPS, both FCT and C-RTT deteriorate
heavily. Comparatively, WiFi suffers less from competing flows
than GPRS, because of deployment of active queue management to
fight with Bufferbloat.
3. COAP-TCP (with Cubic) in general bears a longer latency than
that of CoCoA in UDP. Many factors contribute to this fact as
analyzed in Section.4.4.
This document intents to provide insights for further protocol
design and implementation. We will be glad to generate more
results if the IETF audience finds this document helpful.
6 Security Considerations
We have not evaluate the CoAP with DTLS/TLS yet. Security tunnels
definitely add to the overall latency.
7 IANA Considerations
Not applicable to this document.
8 Acknowledgement
This document was inspired by related work presented in [COAPCC]
[COAPCCCOMP]. We also appreciate all contributors to CoAP
Congestion Control. We also thanks Dr. Fanzhao Wang for discussion
and reasoning about the results.
9 References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, DOI
10.17487/RFC2119, March 1997.
[RFC7252] Shelby, Z., Hartke, K., and C. Bormann, "Constrained
Application Protocol (CoAP)", RFC 7252, June 2014.
[CoCoA] C. Bormann, A. Betzler, C. Gomez, I. Demirkol, "CoAP Simple
Congestion Control/Advanced", draft-bormann-core-cocoa-03,
Oct., 2015.
Expires <Expiry Date> [Page 8]
INTERNET DRAFT draft-zheng-core-coap-latency-evaluation <Issue Date>
[COAP-T] C. Bormann, S. Lemay, and H. Tschofenig, "A TCP and TLS
Transport for the Constrained Application Protocol
(CoAP)", draft-ietf-core-coap-tcp-tls-03, May 2016.
[COAPCCCOMP] Ilpo Jaervinen, Laila Daniel, Markku Kojo, "Experimental
Evaluation of Alternative Congestion Control Algorithms
for Constrained Application Protocol (CoAP)", Internet of
Things (WF-IoT), 2015.
[COAPCC] August Betzler, Carles Gomez, Ilker Demirkol, Josep
Paradells, "CoAP Congestion Control for the Internet of
Things", IEEE Networks Magazine, 2016.
[BufbloatLTE] Stefan Alfredsson, Giacomo Del Giudice, Johan Garcia,
Anna Brunstrom, Luca De Cicco, Saverio Mascolo, "Impact of
TCP congestion control on bufferbloat in cellular
networks", IEEE WOWMOM 2013.
Authors' Addresses
Fei Zheng
Huawei Tech,
Shenzhen, China
EMail: zhengfei10@huawei.com
Baicheng Fu
Huawei Tech,
Shenzhen, China
EMail: fubaicheng@huawei.com
Zhen Cao (Ed.)
Huawei Tech,
Beijing, China
EMail: zhencao.ietf@gmail.com
Expires <Expiry Date> [Page 9]