Audio-Visual Segmentation by Leveraging Multi-scaled Features Learning

Audio-visual segmentation with semantics (AVSS) is an advanced approach that enriches Audio-visual segmentation (AVS) by incorporating object classification, making it a more challenging task to accurately delineate sound-producing objects based on audio-visual cues. To achieve successful audio-visual learning, a model must accurately extract pixel-wise semantic information from images and effectively connect audio-visual data. However, the robustness of such models is often hindered by the limited availability of comprehensive examples in publicly accessible datasets. In this paper, we introduce an audio-visual attentive feature fusion module designed to guide the visual segmentation process by injecting audio semantics. This module is seamlessly integrated into a widely adopted U-Net-like model. Meanwhile, to enhance the model’s ability to capture both high and low-level features, we implement double-skip connections. Besides, in the exploration of intra- and inter-frame correspondences, we also propose an ensemble model proficient in learning two distinct tasks: frame-level and video-level segmentation. To address the task’s diverse demands, we introduce two model variants, one based on the ResNet architecture and the other based on the Swin Transformer model. Our approach leverages transfer learning and employs data augmentation techniques. Additionally, we introduce a custom regularization function aimed at enhancing the model’s robustness against unseen data while simultaneously improving segmentation boundary confidence through self-supervision. Extensive experiments demonstrate the effectiveness of our method as well as the significance of cross-modal perception and dependency modeling for this task.