Hybrid Iterative Detection for OTFS: Interplay between Local L-MMSE and Global Message Passing

Abstract

Orthogonal time frequency space (OTFS) modulation has emerged as a robust solution for high-mobility wireless communications. However, conventional detection algorithms, such as linear equalizers and message passing (MP) methods, either suffer from noise enhancement or fail under complex doubly-selective channels, especially in the presence of fractional delay and Doppler shifts. In this paper, we propose a hybrid low-complexity iterative detection framework that combines linear minimum mean square error (L-MMSE) estimation with MP-based probabilistic inference. The key idea is to apply a new delay-Doppler (DD) commutation precoder (DDCP) to the DD domain signal vector, such that the resulting effective channel matrix exhibits a structured form with several locally dense blocks that are sparsely inter-connected. This precoding structure enables a hybrid iterative detection strategy, where a low-dimensional L-MMSE estimation is applied to the dense blocks, while MP is utilized to exploit the sparse inter-block connections. Furthermore, we provide a detailed complexity analysis, which shows that the proposed scheme incurs lower computational cost compared to the full-size L-MMSE detection. The simulation results of convergence performance confirm that the proposed hybrid MP detection achieves fast and reliable convergence with controlled complexity. In terms of error performance, simulation results demonstrate that our scheme achieves significantly better bit error rate (BER) under various channel conditions. Particularly in multipath scenarios, the BER performance of the proposed method closely approaches the matched filter bound (MFB), indicating its near-optimal error performance.