MedMM: A Multimodal Fusion Framework for 3D Medical Image Classification with Multigranular Text Guidance

Deep learning approaches are widely used in medical image analysis and have shown impressive results on many analytical tasks. However, textual information related to medical images are often underutilized in existing methods, despite the great semantic value and potential multigranular guidance in medical image analysis. Meanwhile, many medical images, like magnetic resonance (MR) images are usually in 3D format consisting of multiple slices which contain more complex and redundant information, making them especially hard to be represented. In this paper, we propose a multimodal funsion framework for 3D medical image classification, which utilizes the medical text paired with the 3D medical image to guide the generation and aggregation of image features. Results show that our method significantly outperforms uni-modal and multimodal baseline methods. Ablation studies validate the effectiveness of each component, and visualization results also reveal the strong ability of our model on capturing fine-grained and coarse-grained information.