Selective Pseudo Word Inversion with MLLM Reasoning for Zero-Shot Composed Image Retrieval

Composed Image Retrieval (CIR) enables users to search for target images via a multimodal query including a reference image and a modification text, aiming to retain key visual features while integrating textual modifications. Since supervised CIR requires costly annotated triplets, researchers have explored Zero-Shot CIR (ZS-CIR). Current ZS-CIR primarily includes two kinds of methods. Textual inversion methods map reference images to pseudo-word tokens, which often fail to understand manipulation intentions and retain excessive visual noise from the reference image. Training-free methods leverage Multimodal Large Language Models (MLLMs), which lose fine-grained visual details by expressing visual content via only text. To overcome these limitations, we propose a novel Selective Pseudo Word Inversion with MLLM Reasoning (SPIR), which leverages textual inversion to supplement visual details and enhances reasoning using MLLM-generated descriptions. To align the inference and training schema, we leverage MLLMs to process image-text pairs, automatically generating modification texts and corresponding target descriptions that serve as supervision information. Extensive experiments conducted on three benchmark datasets demonstrate the superiority of our proposed method. Our code is released at https://github.com/fancySummer19/SPIR.