A PSO-based Method to Test Deep Learning Library at API Level

In recent years, deep learning (DL) is widely used in various fields. DL library bugs could result in security issues and even some losses like data loss and model stealing. As a result, testing DL libraries is the focus of an increasing number of studies. However, there are still issues with these works, such as poor test sample selection and overly general test oracle, which result in ineffective and insufficient testing. In this paper, we present a LEAPI-PSO method based on particle swarm optimization (PSO) algorithm for testing DL libraries at API level, which tackles the inadequacies of existing testing techniques. LEAPI-PSO initially chooses high-quality seed samples by using input coverage and the seed progeny tree. Then, by using PSO, LEAPI-PSO generates test samples that are more likely to reveal API bugs. Based on eight mutation strategies, LEAPI-PSO can produce rich and varied test samples and input them into the API to check for bugs using test oracle. The two most popular DL libraries, PyTorch and Tensorflow, have been used in this study to validate and evaluate LEAPI-PSO. The result shows that LEAPI-PSO is capable to successfully find bugs including crashes, logical errors, and documentation errors. We report 115 bugs to the DL library developers, 95 of which are confirmed, and 63 of which are fixed.