Cross-Image Pixel Contrasting for Semantic Segmentation

Tianfei Zhou, Wenguan Wang

科研成果: 期刊稿件文章同行评审

16 引用 (Scopus)

摘要

This work studies the problem of image semantic segmentation. Current approaches focus mainly on mining &#x201C;local&#x201D; context, <italic>i.e.</italic>, dependencies between pixels within individual images, by specifically-designed, context aggregation modules (<italic>e.g.</italic>, dilated convolution, neural attention) or structure-aware optimization objectives (<italic>e.g.</italic>, IoU-like loss). However, they ignore &#x201C;global&#x201D; context of the training data, <italic>i.e.</italic>, rich semantic relations between pixels across different images. Inspired by recent advance in unsupervised contrastive representation learning, we propose a pixel-wise contrastive algorithm, dubbed as PiCo, for semantic segmentation in the fully supervised learning setting. The core idea is to enforce pixel embeddings belonging to a same semantic class to be more similar than embeddings from different classes. It raises a pixel-wise metric learning paradigm for semantic segmentation, by explicitly exploring the structures of labeled pixels, which were rarely studied before. Our training algorithm is compatible with modern segmentation solutions without extra overhead during testing. We experimentally show that, with famous segmentation models (<italic>i.e.</italic>, DeepLabV3, HRNet, OCRNet, SegFormer, Segmenter, MaskFormer) and backbones (<italic>i.e.</italic>, MobileNet, ResNet, HRNet, MiT, ViT), our algorithm brings consistent performance improvements across diverse datasets (<italic>i.e.</italic>, Cityscapes, ADE20K, PASCAL-Context, COCO-Stuff, CamVid). We expect that this work will encourage our community to rethink the current de facto training paradigm in semantic segmentation. Our code is available at <uri>https://github.com/tfzhou/ContrastiveSeg</uri>.

源语言英语
页(从-至)1-15
页数15
期刊IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI
出版状态已接受/待刊 - 2024

指纹

探究 'Cross-Image Pixel Contrasting for Semantic Segmentation' 的科研主题。它们共同构成独一无二的指纹。

引用此