Sequential Differentiable Alignment-Based 2D-3D Optimal Gaussian Distribution Registration for Monocular Endoscopic Video AR Guidance

Registration between preoperative 3D data and intraoperative 2D endoscopic videos is crucial for advancing Augmented Reality (AR) guidance in surgery. However, due to the complexity of intraoperative scene, it is difficult to achieve accurate real-time 2D–3D registration without prior camera poses or anatomical landmarks. To overcome these challenges, a novel 2D–3D gaussian mixture splatting registration architecture is proposed, which integrates optimal gaussian distribution registration with sequential differentiable alignment. First, to achieve consistent feature representations across 2D and 3D data, the gaussian mixture model feature distributions are constructed through the gaussian splatting pipeline from the preoperative 3D data and the intraoperative 2D endoscopic images, respectively. Secondly, freeing from prior pose or landmarks, a novel optimal gaussian distribution registration module is designed to establish the global coarse registration by minimizing the feature divergence of gaussian mixture models. Finally, a sequential differentiable fine alignment module is employed to achieve real-time local alignment between the rasterized 3D data and the intraoperative endoscopic video. The evaluation is carried out on the public Scared endoscopic dataset, the Clinical Endoscopic Kidney and Liver dataset from the local hospital. The experimental results show that the proposed method outperforms state-of-the-art methods in quantitative and qualitative comparisons.