Layer-wise Principal Gradient Compression for Distributed Machine Learning

Distributed Machine Learning (DML) can accelerate model training by using multiple training nodes and large datasets. As model and data sizes grow, DML often uses more training nodes, and frequent model information exchange creates high communication overhead. To reduce this overhead, we propose PrincipalCatcher, a layer-wise gradient compression method based on Principal Component Analysis (PCA) and error feedback. PrincipalCatcher uses PCA to capture the main gradient components in each layer and transmits only the compressed gradient representation during iterative training. To limit the accuracy loss caused by lossy compression, each worker accumulates compression errors and adds them to gradients in the next epoch. We evaluate PrincipalCatcher on image classification tasks under distributed training settings. The results show that PrincipalCatcher can reduce gradient transmission by up to 96.4% while keeping accuracy close to conventional stochastic gradient descent in most tested settings.