An information-Enhanced memory library for multi-modal continuous clustering

Multi-modal continuous clustering is oriented to the dynamic information flow of multiple modalities, which mines and propagates relevant information within incremental multi-modal data to improve the clustering performance. Popular methods construct a memory library to preserve historical knowledge while updating this memory library to accumulate new knowledge. However, most of them use a simple first-in-first-out strategy to update the memory library, which results in the loss of existing knowledge. In this paper, we propose an Information-enhanced Memory Library (InfoML) for multi-modal continuous clustering, which theoretically guarantees the continuous accumulation of relevant information in the memory library. InfoML follows an information-theoretic-based update strategy, maximizing the mutual information between the memory library and multi-modal data to determine the optimal update location. At each update, we try all candidate update locations and compute the mutual information between the current memory library and each modality. Then, we consider the location that maximizes the sum of mutual information as the optimal update location to ensure the accumulation of relevant information. Finally, we integrate InfoML in a contrastive multi-modal clustering framework to achieve multi-modal continuous clustering. We conducted extensive experiments across five multimodal datasets. The results demonstrate that the proposed method achieves average improvements of 1.5 %, 2 %, and 1.7 % in terms of ACC, NMI, and ARI over the optimal baseline, thereby validating the effectiveness of InfoML.