Adaptive turbulence compensation with a hybrid input–output algorithm in orbital angular momentum-based free-space optical communication

Atmospheric turbulence is a major challenge for practical orbital angular momentum (OAM)-based free-space optical (FSO) communication systems that causes intermodal crosstalk and degrades the performance of the system. Herein, we propose a hybrid input–output algorithm (HIOA)-based adaptive optics (AO) system to compensate for distorted OAM beams. The principle and parameters of the HIOA-based AO system in an OAM-based FSO system are analyzed, and the performance is discussed. The simulation results indicate that the HIOA-based AO system can effectively correct distorted OAM beams and that the HIOA improves the compensation performance and convergence speed compared to the traditional Gerchberg–Saxton algorithm. Moreover, we analyze the compensation performance based on different probe beams. Using an OAM beam with state l 1 as a probe beam can yield better correction effects than a Gaussian beam. This work verifies the feasibility of using an HIOA for adaptive turbulence compensation and provides new insights into OAM communication systems.