TY - GEN
T1 - Scheduling Containerized Workflow in Multi-cluster Kubernetes
AU - Liu, Danyang
AU - Xia, Yuanqing
AU - Shan, Chenggang
AU - Wang, Guan
AU - Wang, Yongkang
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd 2023.
PY - 2023
Y1 - 2023
N2 - Docker and Kubernetes have revolutionized the cloud-native technology ecosystem by offering robust solutions for containerization and orchestration workflows. This combination provides unprecedented speed, scalability, and efficiency in deploying and managing applications in distributed environments. However, when scheduling complex workflows across multi-cluster Kubernetes environments, existing workflow scheduling systems often fail to provide the necessary support. Integrating workflow scheduling algorithms with multi-cluster scheduling algorithms poses a complex and challenging problem. In this paper, we present a comprehensive framework known as the Containerized Workflow Engine (CWE), specifically designed for multi-cluster Kubernetes deployments. The CWE framework employs a two-level scheduling scheme, which combines the benefits of workflow containerization and establishes seamless connections between multi-cluster scheduling algorithms and multi-cluster Kubernetes environments. By integrating workflow scheduling algorithms with Kubernetes schedulers across Kubernetes environments, the CWE framework enables efficient utilization of resources and improved overall workflow performance. Compared to the state-of-the-art Argo workflows, CWE performs better in average task pod execution time and resource utilization.
AB - Docker and Kubernetes have revolutionized the cloud-native technology ecosystem by offering robust solutions for containerization and orchestration workflows. This combination provides unprecedented speed, scalability, and efficiency in deploying and managing applications in distributed environments. However, when scheduling complex workflows across multi-cluster Kubernetes environments, existing workflow scheduling systems often fail to provide the necessary support. Integrating workflow scheduling algorithms with multi-cluster scheduling algorithms poses a complex and challenging problem. In this paper, we present a comprehensive framework known as the Containerized Workflow Engine (CWE), specifically designed for multi-cluster Kubernetes deployments. The CWE framework employs a two-level scheduling scheme, which combines the benefits of workflow containerization and establishes seamless connections between multi-cluster scheduling algorithms and multi-cluster Kubernetes environments. By integrating workflow scheduling algorithms with Kubernetes schedulers across Kubernetes environments, the CWE framework enables efficient utilization of resources and improved overall workflow performance. Compared to the state-of-the-art Argo workflows, CWE performs better in average task pod execution time and resource utilization.
KW - Containerized
KW - Scheduling
KW - Workflow
UR - http://www.scopus.com/inward/record.url?scp=85180632249&partnerID=8YFLogxK
U2 - 10.1007/978-981-99-8979-9_12
DO - 10.1007/978-981-99-8979-9_12
M3 - Conference contribution
AN - SCOPUS:85180632249
SN - 9789819989782
T3 - Communications in Computer and Information Science
SP - 149
EP - 163
BT - Big Data - 11th CCF Conference, BigData 2023, Proceedings
A2 - Chen, Enhong
A2 - Gao, Yang
A2 - Gu, Rong
A2 - Cao, Longbing
A2 - Xiao, Fu
A2 - Cui, Yiping
A2 - Yang, Wanqi
A2 - Wang, Li
A2 - Cui, Laizhong
PB - Springer Science and Business Media Deutschland GmbH
T2 - 11th CCF Big Data Conference, BigData 2023
Y2 - 8 September 2023 through 10 September 2023
ER -