MambaST: Hexagonal State Space Modeling for Spatial Domain Identification

Spatial transcriptomics has transformed tissue analysis by preserving spatial context in gene expression data, enabling deeper insights into tissue microenvironments. However, current spatial domain identification methods largely focus on adjacent cellular similarities, limiting their ability to capture long-range spatial dependencies and identify identical cell types distributed across distant and non-contiguous areas. To address these challenges, we introduce MambaST, a hybrid deep-learning framework that integrates selective state space modeling (Mamba) and self-supervised learning for Spatial Transcriptomics data analysis. Specifically, MambaST incorporates a Six-Directional Selective Scan (SS6D) algorithm to convert graph-structured spatial data into topology-preserving pseudo-sequences, effectively bridging sequential modeling with spatial topology. Additionally, we propose HexMambaBlock (HMB), which applies Mamba to simultaneously denoise gene expression data and capture global spatial dependencies. Furthermore, contrastive learning enhanced with a Dynamic Context-aware Readout (DCR) module improves the biological specificity of local representations. Comprehensive evaluations across three spatial transcriptomic datasets demonstrate MambaST’s superior performance in spatial domain identification, achieving a 0.58 average Adjusted Rand Index (ARI) on the DLPFC dataset, which surpasses state-of-the-art methods by 2.7%.