4×4 RLS-CMA Equalization for IQ Imbalance Compensation in Coherent Optical Receivers

Receiver-side IQ imbalance introduces I/Q amplitude mismatch, quadrature phase error, and timing skew in coherent optical communication systems, thereby significantly degrading system transmission performance. Conventional 2×2 complex-valued MIMO equalizers are mainly used for polarization demultiplexing and linear channel equalization. However, conventional 2×2 complex-valued equalizers process the I and Q components as coupled complex signals, which reduces the available degrees of freedom and limits their ability to separately compensate I/Q imbalance, skew, and arbitrary real-domain coupling. To address this issue, this paper proposes a 4×4 real-valued RLS-CMA equalization scheme for dual-polarization coherent optical transmission systems. By reconstructing the dual-polarization complex-valued signal into a four-dimensional real-valued vector, the proposed scheme increases the degrees of freedom for adaptive compensation of receiver-side IQ impairments while preserving the fast convergence property of RLS-CMA. Simulation results for a 32 GBaud DP-16QAM coherent optical transmission system over 20 km fiber show that, in the absence of receiver-side IQ imbalance, the proposed 4×4 RLS-CMA achieves BER performance comparable to that of the conventional 2×2 RLS-CMA. In the presence of IQ skew, phase imbalance, and amplitude imbalance, the 4×4 RLS-CMA exhibits stronger robustness. When the IQ phase imbalance is 15°, the EVM is reduced from 17.80% to 11.28%; when the IQ amplitude imbalance is 3 dB, the EVM is reduced from 22.80% to 18.12%. These results demonstrate that the proposed 4×4 RLS-CMA can effectively improve the tolerance of coherent optical receivers to IQ imbalance.