Fine-Grained Feature-Driven Incomplete Multi-modal Hashing

Traditional multi-modal hashing methods map instances into hash codes for multi-modal retrieval tasks, achieving low storage costs and fast retrieval speed. In reality, it is common to encounter missing modality scenarios, i.e., instances that should originally contain all modalities may lack certain modal data points. Faced with this issue, existing methods typically employ an instance-level completion strategy. This strategy selects similar integrated instances based on coarse-grained label similarity and then fuses the corresponding modal data points of the selected instances to complete the missing data. However, this strategy typically involves noisy feature representation due to irrelevant label information. For example, completing a ”sunset” labeled instance with an integrated instance labeled by ”sunset” and ”structure” will include information about ”structure”. To address this issue, we propose a novel Fine-Grained Feature-Driven Incomplete Multi-modal Hashing (FDIMH) to directly model the relationship between labels and features rather than adopting instances as a bridge to complete the missing data without injecting noisy information. Specifically, FDIMH initializes a high-dimensional memory bank where each unit denotes a fine-grained feature. During the training process, all the fine-grained features in this memory bank are first pre-optimized to fit the complete instances through an adaptive weight allocation mechanism. Subsequently, the learned memory bank and the adaptive weight allocation mechanism are optimized together with the hashing network through our proposed intra-modality and inter-modality loss functions to bridge the gap between the completed data and real data. Extensive experiments on three widely used datasets demonstrate the superiority of the proposed method compared to the state-of-the-art base-lines in multi-modal retrieval tasks.