TY - GEN
T1 - Multi-job Merging Framework and Scheduling Optimization for Apache Flink
AU - Ji, Hangxu
AU - Wu, Gang
AU - Zhao, Yuhai
AU - Yuan, Ye
AU - Wang, Guoren
N1 - Publisher Copyright:
© 2021, Springer Nature Switzerland AG.
PY - 2021
Y1 - 2021
N2 - With the popularization of big data technology, distributed computing systems are constantly evolving and maturing, making substantial contributions to the query and analysis of massive data. However, the insufficient utilization of system resources is an inherent problem of distributed computing engines. Particularly, when more jobs lead to execution blocking, the system schedules multiple jobs on a first-come-first-executed (FCFE) basis, even if there are still many remaining resources in the cluster. Therefore, the optimization of resource utilization is key to improving the efficiency of multi-job execution. We investigated the field of multi-job execution optimization, designed a multi-job merging framework and scheduling optimization algorithm, and implemented them in the latest generation of a distributed computing system, Apache Flink. In summary, the advantages of our work are highlighted as follows: (1) the framework enables Flink to support multi-job collection, merging and dynamic tuning of the execution sequence, and the selection of these functions are customizable. (2) with the multi-job merging and optimization, the total running time can be reduced by 31% compared with traditional sequential execution. (3) the multi-job scheduling optimization algorithm can bring 28% performance improvement, and in the average case can reduce the cluster idle resources by 61%.
AB - With the popularization of big data technology, distributed computing systems are constantly evolving and maturing, making substantial contributions to the query and analysis of massive data. However, the insufficient utilization of system resources is an inherent problem of distributed computing engines. Particularly, when more jobs lead to execution blocking, the system schedules multiple jobs on a first-come-first-executed (FCFE) basis, even if there are still many remaining resources in the cluster. Therefore, the optimization of resource utilization is key to improving the efficiency of multi-job execution. We investigated the field of multi-job execution optimization, designed a multi-job merging framework and scheduling optimization algorithm, and implemented them in the latest generation of a distributed computing system, Apache Flink. In summary, the advantages of our work are highlighted as follows: (1) the framework enables Flink to support multi-job collection, merging and dynamic tuning of the execution sequence, and the selection of these functions are customizable. (2) with the multi-job merging and optimization, the total running time can be reduced by 31% compared with traditional sequential execution. (3) the multi-job scheduling optimization algorithm can bring 28% performance improvement, and in the average case can reduce the cluster idle resources by 61%.
KW - Distributed computing
KW - Flink
KW - Multi-job merging
KW - Scheduling optimization
UR - http://www.scopus.com/inward/record.url?scp=85104810813&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-73194-6_2
DO - 10.1007/978-3-030-73194-6_2
M3 - Conference contribution
AN - SCOPUS:85104810813
SN - 9783030731939
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 20
EP - 36
BT - Database Systems for Advanced Applications - 26th International Conference, DASFAA 2021, Proceedings
A2 - Jensen, Christian S.
A2 - Lim, Ee-Peng
A2 - Yang, De-Nian
A2 - Lee, Wang-Chien
A2 - Tseng, Vincent S.
A2 - Kalogeraki, Vana
A2 - Huang, Jen-Wei
A2 - Shen, Chih-Ya
PB - Springer Science and Business Media Deutschland GmbH
T2 - 26th International Conference on Database Systems for Advanced Applications, DASFAA 2021
Y2 - 11 April 2021 through 14 April 2021
ER -