Empirical study on tangent loss function for classification with deep neural networks

Deep neural networks have been widely applied in natural language processing and computer vision tasks, which have achieved great successes due to their powerful ability for capturing sophisticated deep features. Currently, most of the neural networks are trained with cross-entropy (CE). However, the traditional CE loss function is sensitive to randomness induced from training samples. In this paper, we propose a novel loss function, namely tangent loss (TG), aiming to make classification models more stable while achieving comparable performance. The TG loss function trains the neural network in a way that emphasizes samples whose predictions deviate greatly from the targets at each training step. We make a systematical empirical study on TG loss, which is compared with CE loss on various classification tasks. Extensive experimental results on the real-world datasets demonstrate that the TG loss function can be readily applied with the existing neural networks and improves the stability of classification models, which can obtain better or comparable classification performance than the CE loss function.