Deep Learning and Its Parallelization

In recent years, deep learning has been extensively studied as a new way to train multilayer neural networks. Deep learning is a set of algorithms in machine learning, which attempts to model high-level abstractions in input data by using multiple nonlinear transformations. Many great achievements of deep learning have been made in speech recognition, computer vision, and natural language processing. Considering that data volume increases rapidly, deep learning becomes more and more important in predictive analytics of big data. We need tens of millions of parameters and billions of samples to train a high quality and practical deep learning model. As the number of parameters and training data are still growing rapidly in the Big Data era, the speed to train a practical model is limited by sequential algorithms and intensive data computation. Therefore, deep learning has been accelerated in parallel with GPUs and clusters in recent years. This chapter introduces several mainstream deep learning approaches developed over the past decades, as well as optimization methods for deep learning in parallel.