“Car or Bus?" CLearSeg: CLIP-Enhanced Discrimination Among Resembling Classes for Few-Shot Semantic Segmentation

Few-shot semantic segmentation aims at learning to segment query images of unseen classes with the guidance of limited segmented support examples. However, existing models tend to confuse the resembling classes (e.g., ‘car’ and ‘bus’) thus generating erroneous predictions. To address this, we propose the CLIP-enhanced discrimination among resembling classes for few-shot semantic Segmentation (CLearSeg), which leverages information beyond support images, including the class name, through Contrastive Language-Image Pretraining (CLIP), to discriminate between resembling classes. Firstly, we modify the CLIP structure and design the Sliding Attention Pooling (SAP) to construct the Text-Driven Activation (TDA) module, learning the Class-Specific Activation (CSA) maps with class names. Since the semantic information is explicitly involved by the class name, the CSA maps exhibit clear distinctions among resembling classes. Meanwhile, to enrich fine-grained features ensuring distinguishability, the Multi-Level Correlation (MLC) module is designed to extract multi-level features of support and query images and generate various correlation maps. We further applied a decoder to fuse the CSA map and correlation maps with encoded features and obtain the final prediction. Experiments on the Pascal-5ⁱ and COCO-20ⁱ datasets have shown that CLearSeg outperforms previous methods, achieving the mIoU of 69.2 % and 48.9 % for 1-shot segmentation, particularly in distinguishing objects from resembling classes.