Hybrid Nonlinear Transceiver Optimization for the RIS-Aided MIMO Downlink

The hybrid nonlinear transceiver optimization problem of reconfigurable intelligent surface (RIS)-aided multi-user multiple-input multiple-output (MU-MIMO) downlink is investigated. Specifically, the Tomlinson-Harashima precoder (THP) and the hybrid transmit precoder (TPC) of the base station are jointly optimized with the linear digital receivers of mobile users. The triangular feedback matrix of the THP is optimized and the optimal solution is derived in closed form based on a matrix inequality. Moreover, in order to tackle the nonconvexity of the constant-modulus constraints imposed on the analog TPC, the Majorization-Minimization (MM) based reconfigurable optimization framework is proposed, which strikes a trade-off between the implementation complexity and system performance in a reconfigurable manner. Explicitly, our MM-based reconfigurable optimization framework is capable of optimizing the analog TPC in a dynamically reconfigurable manner on an element-by-element, column-by-column, row-by-row or block-by-block basis. Moreover, an MM-based reconfigurable algorithm is proposed for the optimization of the phase shifting matrix at RIS, which also suffers from constant-modulus constraints. In the proposed MM-based reconfigurable algorithm, the RIS can be partitioned into a series of subarrays for striking different performance vs. complexity tradeoffs. Finally, our numerical results demonstrate the performance advantages of the proposed nonlinear hybrid transceiver optimization techniques.