MMReLU: A Simple and Smooth Activation Function with High Convergence Speed

Activation functions have a major effect on deep networks' performance. In past few years, there is an increasing interest in the construction of novel activation functions. In this paper, we introduced a novel non-monotonic activation function, named Moreau Mish Rectified Linear Unit (MMReLU). It's simple, efficient, and robust, comparing with Mish, ReLU, and etc. The experimental results on several classical datasets demonstrate that MMReLU outperforms its counterparts in both convergence speed and accuracy. We show that the capacity of neural networks could be enhanced by MMReLU without changing the network structure, especially the convergence speed.