Joint Calibration and Angle Estimation for MIMO Sensing Systems in the Presence of Multipath

Multiple-input multiple-output (MIMO) sensing systems, including integrated sensing and communication (ISAC) platforms, often suffer from angle-dependent array distortions. These distortions degrade performance significantly in multipath environments where the direction of departure (DOD) and direction of arrival (DOA) are mismatched. Conventional calibration, which typically applies a single compensation matrix derived under a line-of-sight assumption, fails to correct such complex distortions. This paper proposes a joint calibration and angle-estimation framework leveraging a decoupled calibration dictionary that independently models transmit- and receive-side distortions across angular sectors. We formulate the estimation task as a joint optimization problem and solve it via an alternating optimization (AO) algorithm that iteratively refines path parameters and selects the optimal per-path calibration matrix. Validated with high-fidelity full-wave electromagnetic simulation data of an automotive MIMO array, the proposed method demonstrates superior robustness and accuracy, effectively eliminating the error floor observed in conventional methods under challenging multipath conditions.