Sparkaibench: a benchmark to generate ai workloads on spark

Zifeng Liu; Xiaojiang Zuo; Zeqing Li; Rui Han

doi:10.1007/978-3-030-49556-5_21

Sparkaibench: a benchmark to generate ai workloads on spark

Zifeng Liu, Xiaojiang Zuo, Zeqing Li, Rui Han^*

^*Corresponding author for this work

School of Computer Science and Technology

Beijing Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Citations (Scopus)

Abstract

With the rapid development of artificial intelligence and cloud computing technologies, more and more workloads embed AI algorithms are deploying on cloud systems. For lack of sufficient achievements on generating AI workloads in recent years, designing and developing an efficient benchmark for AI workloads will be significant helpful for optimization of job execution time and cluster resource utilization. SparkAIBench proposed in this paper is a user customized benchmark and is able to automatically generate a variety of AI workloads by transforming user requirements into JSON objects. Besides, cooperated with a DRL-based job scheduling optimizer, a real scenario is introduced in this paper to demonstrate how SparkAIBench works.

Original language	English
Title of host publication	Benchmarking, Measuring, and Optimizing - 2nd BenchCouncil International Symposium, Bench 2019, Revised Selected Papers
Editors	Wanling Gao, Jianfeng Zhan, Geoffrey Fox, Xiaoyi Lu, Dan Stanzione
Publisher	Springer
Pages	215-221
Number of pages	7
ISBN (Print)	9783030495558
DOIs	https://doi.org/10.1007/978-3-030-49556-5_21
Publication status	Published - 2020
Event	2nd International Symposium on Benchmarking, Measuring, and Optimization, Bench 2019 - Denver, United States Duration: 14 Nov 2019 → 16 Nov 2019

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12093 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	2nd International Symposium on Benchmarking, Measuring, and Optimization, Bench 2019
Country/Territory	United States
City	Denver
Period	14/11/19 → 16/11/19

Keywords

AI workloads
Benchmark
Cloud computing

Access to Document

10.1007/978-3-030-49556-5_21

Cite this

Liu, Z., Zuo, X., Li, Z., & Han, R. (2020). Sparkaibench: a benchmark to generate ai workloads on spark. In W. Gao, J. Zhan, G. Fox, X. Lu, & D. Stanzione (Eds.), Benchmarking, Measuring, and Optimizing - 2nd BenchCouncil International Symposium, Bench 2019, Revised Selected Papers (pp. 215-221). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12093 LNCS). Springer. https://doi.org/10.1007/978-3-030-49556-5_21

Liu, Zifeng ; Zuo, Xiaojiang ; Li, Zeqing et al. / Sparkaibench : a benchmark to generate ai workloads on spark. Benchmarking, Measuring, and Optimizing - 2nd BenchCouncil International Symposium, Bench 2019, Revised Selected Papers. editor / Wanling Gao ; Jianfeng Zhan ; Geoffrey Fox ; Xiaoyi Lu ; Dan Stanzione. Springer, 2020. pp. 215-221 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{2aa1524ca387467a9e0db3bac3c657c4,

title = "Sparkaibench: a benchmark to generate ai workloads on spark",

abstract = "With the rapid development of artificial intelligence and cloud computing technologies, more and more workloads embed AI algorithms are deploying on cloud systems. For lack of sufficient achievements on generating AI workloads in recent years, designing and developing an efficient benchmark for AI workloads will be significant helpful for optimization of job execution time and cluster resource utilization. SparkAIBench proposed in this paper is a user customized benchmark and is able to automatically generate a variety of AI workloads by transforming user requirements into JSON objects. Besides, cooperated with a DRL-based job scheduling optimizer, a real scenario is introduced in this paper to demonstrate how SparkAIBench works.",

keywords = "AI workloads, Benchmark, Cloud computing",

author = "Zifeng Liu and Xiaojiang Zuo and Zeqing Li and Rui Han",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2020.; 2nd International Symposium on Benchmarking, Measuring, and Optimization, Bench 2019 ; Conference date: 14-11-2019 Through 16-11-2019",

year = "2020",

doi = "10.1007/978-3-030-49556-5_21",

language = "English",

isbn = "9783030495558",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "215--221",

editor = "Wanling Gao and Jianfeng Zhan and Geoffrey Fox and Xiaoyi Lu and Dan Stanzione",

booktitle = "Benchmarking, Measuring, and Optimizing - 2nd BenchCouncil International Symposium, Bench 2019, Revised Selected Papers",

address = "Germany",

}

Liu, Z, Zuo, X, Li, Z & Han, R 2020, Sparkaibench: a benchmark to generate ai workloads on spark. in W Gao, J Zhan, G Fox, X Lu & D Stanzione (eds), Benchmarking, Measuring, and Optimizing - 2nd BenchCouncil International Symposium, Bench 2019, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12093 LNCS, Springer, pp. 215-221, 2nd International Symposium on Benchmarking, Measuring, and Optimization, Bench 2019, Denver, United States, 14/11/19. https://doi.org/10.1007/978-3-030-49556-5_21

Sparkaibench: a benchmark to generate ai workloads on spark. / Liu, Zifeng; Zuo, Xiaojiang; Li, Zeqing et al.
Benchmarking, Measuring, and Optimizing - 2nd BenchCouncil International Symposium, Bench 2019, Revised Selected Papers. ed. / Wanling Gao; Jianfeng Zhan; Geoffrey Fox; Xiaoyi Lu; Dan Stanzione. Springer, 2020. p. 215-221 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12093 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Sparkaibench

T2 - 2nd International Symposium on Benchmarking, Measuring, and Optimization, Bench 2019

AU - Liu, Zifeng

AU - Zuo, Xiaojiang

AU - Li, Zeqing

AU - Han, Rui

N1 - Publisher Copyright: © Springer Nature Switzerland AG 2020.

PY - 2020

Y1 - 2020

N2 - With the rapid development of artificial intelligence and cloud computing technologies, more and more workloads embed AI algorithms are deploying on cloud systems. For lack of sufficient achievements on generating AI workloads in recent years, designing and developing an efficient benchmark for AI workloads will be significant helpful for optimization of job execution time and cluster resource utilization. SparkAIBench proposed in this paper is a user customized benchmark and is able to automatically generate a variety of AI workloads by transforming user requirements into JSON objects. Besides, cooperated with a DRL-based job scheduling optimizer, a real scenario is introduced in this paper to demonstrate how SparkAIBench works.

AB - With the rapid development of artificial intelligence and cloud computing technologies, more and more workloads embed AI algorithms are deploying on cloud systems. For lack of sufficient achievements on generating AI workloads in recent years, designing and developing an efficient benchmark for AI workloads will be significant helpful for optimization of job execution time and cluster resource utilization. SparkAIBench proposed in this paper is a user customized benchmark and is able to automatically generate a variety of AI workloads by transforming user requirements into JSON objects. Besides, cooperated with a DRL-based job scheduling optimizer, a real scenario is introduced in this paper to demonstrate how SparkAIBench works.

KW - AI workloads

KW - Benchmark

KW - Cloud computing

UR - http://www.scopus.com/inward/record.url?scp=85087002964&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-49556-5_21

DO - 10.1007/978-3-030-49556-5_21

M3 - Conference contribution

AN - SCOPUS:85087002964

SN - 9783030495558

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 215

EP - 221

BT - Benchmarking, Measuring, and Optimizing - 2nd BenchCouncil International Symposium, Bench 2019, Revised Selected Papers

A2 - Gao, Wanling

A2 - Zhan, Jianfeng

A2 - Fox, Geoffrey

A2 - Lu, Xiaoyi

A2 - Stanzione, Dan

PB - Springer

Y2 - 14 November 2019 through 16 November 2019

ER -

Liu Z, Zuo X, Li Z, Han R. Sparkaibench: a benchmark to generate ai workloads on spark. In Gao W, Zhan J, Fox G, Lu X, Stanzione D, editors, Benchmarking, Measuring, and Optimizing - 2nd BenchCouncil International Symposium, Bench 2019, Revised Selected Papers. Springer. 2020. p. 215-221. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-49556-5_21

Sparkaibench: a benchmark to generate ai workloads on spark

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this