Cutting the Unnecessary Long Tail: Cost-Effective Big Data Clustering in the Cloud

Dongwei Li, Shuliang Wang, Nan Gao, Qiang He*, Yun Yang*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

4 Citations (Scopus)

Abstract

Clustering big data often requires tremendous computational resources where cloud computing is undoubtedly one of the promising solutions. However, the computation cost in the cloud can be unexpectedly high if it cannot be managed properly. The long tail phenomenon has been observed widely in the big data clustering area, which indicates that the majority of time is often consumed in the middle to late stages in the clustering process. In this research, we try to cut the unnecessary long tail in the clustering process to achieve a sufficiently satisfactory accuracy at the lowest possible computation cost. A novel approach is proposed to achieve cost-effective big data clustering in the cloud. By training the regression model with the sampling data, we can make widely used k-means and EM (Expectation-Maximization) algorithms stop automatically at an early point when the desired accuracy is obtained. Experiments are conducted on four popular data sets and the results demonstrate that both k-means and EM algorithms can achieve high cost-effectiveness in the cloud with our proposed approach. For example, in the case studies with the much more efficient k-means algorithm, we find that achieving a 99 percent accuracy needs only 47.71-71.14 percent of the computation cost required for achieving a 100 percent accuracy while the less efficient EM algorithm needs 16.69-32.04 percent of the computation cost. To put that into perspective, in the United States land use classification example, our approach can save up to $94,687.49 for the government in each use.

Original languageEnglish
Pages (from-to)292-303
Number of pages12
JournalIEEE Transactions on Cloud Computing
Volume10
Issue number1
DOIs
Publication statusPublished - 2022

Keywords

  • Cloud computing
  • big data
  • clustering algorithms
  • cost-effectiveness
  • data mining

Fingerprint

Dive into the research topics of 'Cutting the Unnecessary Long Tail: Cost-Effective Big Data Clustering in the Cloud'. Together they form a unique fingerprint.

Cite this