Resource Orchestration for Multi-Domain Data Analytics
draft-xiang-alto-exascale-network-optimization-03

Document Type Expired Internet-Draft (individual)
Last updated 2018-01-04 (latest revision 2017-07-03)
Stream (None)
Intended RFC status (None)
Formats
Expired & archived
plain text pdf html bibtex
Stream Stream state (No stream defined)
Consensus Boilerplate Unknown
RFC Editor Note (None)
IESG IESG state Expired
Telechat date
Responsible AD (None)
Send notices to (None)

This Internet-Draft is no longer active. A copy of the expired Internet-Draft can be found at
https://www.ietf.org/archive/id/draft-xiang-alto-exascale-network-optimization-03.txt

Abstract

Data-intensive analytics is entering the era of multi-domain, geographically-distributed, collaborative computing, where different organizations contribute various resources to collaboratively collect, share and analyze extremely large amounts of data. Examples of this paradigm include the Compact Muon Solenoid (CMS) and A Toroidal LHC ApparatuS (ATLAS) experiments of the Large Hadron Collider (LHC) program. Massive datasets continue to be acquired, simulated, processed and analyzed by globally distributed science networks in these collaborations. Applications that manage and analyze such massive data volumes can benefit substantially from the information about networking, computing and storage resources from each member's site, and more directly from network-resident services that optimize and load balance resource usage among multiple data transfers and analytics requests, and achieve a better utilization of multiple resources in clusters. The Application-Layer Traffic Optimization (ALTO) protocol can provide via extensions the network information about different clusters/sites, to both users and proactive network management services where applicable, with the goal of improving both application performance and network resource utilization. In this document, we propose that it is feasible to use existing ALTO services to provides not only network information, but also information about computation and storage resources in data analytics networks. We introduce a uniform resource orchestration framework (Unicorn), which achieves an efficient multi-resource allocation to support low-latency dataset transfer and data intensive analytics for collaborative computing. It collects cluster information from multiple ALTO services utilizing topology extensions and leverages emerging SDN control capabilities to orchestrate the resource allocation for dataset transfers and analytics tasks, leading to improved transfer and analytics latency as well as more efficient utilization of multi-resources in sites.

Authors

Qiao Xiang (qiao.xiang@cs.yale.edu)
Harvey Newman (newman@hep.caltech.edu)
Greg Bernstein (gregb@grotto-networking.com)
Haizhou Du (duhaizhou@gmail.com)
Kai Gao (gaok12@mails.tsinghua.edu.cn)
Azher Mughal (azher@hep.caltech.edu)
Justas Balcas (justas.balcas@cern.ch)
J. Zhang (jingxuan.n.zhang@gmail.com)
Yang Yang (yry@cs.yale.edu)

(Note: The e-mail addresses provided for the authors of this Internet-Draft may no longer be valid.)