Iterative Interference Cancellation Refinement for OMP Channel Estimation in SEFDM-based OTFS Systems

Spectrally efficient frequency division multiplexing (SEFDM)-based orthogonal time frequency space (OTFS) maps signals to the delay-Doppler (DD) domain to cope with high-mobility scenarios, and improves spectral efficiency via subcarrier compression. However, subcarrier compression introduces inter-carrier interference (ICI), which further couples into the DD domain and degrades channel estimation performance. In this paper, we first derive explicit input-output relations in both the time-frequency (TF) domain and the DD domain for SEFDM-based OTFS with practical rectangular pulses, revealing the coupling effects induced by subcarrier compression and fractional Doppler shifts. Then, to improve channel estimation performance under SEFDM-induced coupling interference, we propose an iterative interference cancellation refinement (IICR) framework to refine fractional Doppler based on an initial orthogonal matching pursuit (OMP) estimation. When OMP with fractional refinement (OMPFR) is adopted as the coarse estimator, the resulting scheme is termed IICR-OMPFR. To further reduce computational complexity, we develop a low-complexity OMP (LCOMP) algorithm comprising noise-free offline calibration and online refinement. In the online stage, a piecewise cubic Hermite interpolating polynomial (PCHIP) is used for Doppler estimation. Meanwhile, LCOMP can also serve as the coarse estimator within the proposed IICR framework, yielding IICR-LCOMP.Simulation results demonstrate that the proposed IICR-OMPFR and IICR-LCOMP improve the robustness and accuracy of channel estimation in the presence of SEFDM-induced coupling interference, and that the proposed LCOMP achieves estimation performance comparable to traditional OMPFR algorithm while maintaining a complexity close to that of on-grid OMP.