Probabilistic neural network equalizer for nonlinear mitigation in OAM mode division multiplexed optical fiber communication

Orbital angular momentum (OAM) mode-division multiplexing (MDM) is a key technique to achieve ultra-high-capacity optical fiber communications. However, the high nonlinear impairment from optoelectronic devices, such as spatial light modulators, modulators, and photodiodes, is a long-standing challenge for OAM-MDM. In this paper, an equalizer based on a probabilistic neural network (PNN) is presented to mitigate the nonlinear impairment for an OAM-MDM fiber communication system with 32 GBaud Nyquist pulse amplitude modulation-8 (PAM8) intensity-modulation direct-detection (IM-DD) signals. PNN equalizer can calculate the distribution of the nonlinearity using Bayesian decision theory and thus mitigate the stochastic nonlinear impairment of the received signal. Experimental results show that compared with the convolutional neural network (CNN) equalizer, the PNN equalizer improves the receiver sensitivity by 0.6dB and 2dB for two OAM modes with l = + 3 and l = + 4 at the 20% FEC limit, respectively. Moreover, compared with Volterra or CNN equalizers, the PNN equalizer can reduce the computation complexity significantly, which has great potential to mitigate the nonlinear signal distortions in high-speed IM-DD OAM-MDM fiber communication systems.