ReBac: Current-to-Past Residual Background Correction for Class-Incremental Semantic Segmentation

Class-Incremental Semantic Segmentation (CISS) extends incremental learning to dense prediction, where catastrophic forgetting is intensified by background shift: as new classes appear, regions once labeled as background are reassigned, destabilizing both background and foreground predictions. Unlike incremental classification, CISS methods have to balance precision and recall for old and new classes under the stringent mean Intersection-over-Union (mIoU) metric. Existing methods address background shift mostly implicitly, e.g., by anticipating future-class interference from a current-to-future view. In contrast, we propose a principled residual modeling framework, termed ReBac (Residual Background Correction), which explicitly models background evolution from a current-to-past perspective. ReBac introduces residual corrections by treating newly introduced foreground regions as past-step background, enabling incremental and consistent refinement of background logits across time. This residual accumulation preserves stability for old class predictions and enhances adaptability to new classes. We present two instantiations of ReBac: ReBac-B, using a basic shared residual channel for background refinement, and ReBac-C, incorporating class-aware residual selection for finer granularity and reduced interference. To optimize residual modeling, we design a synergistic objective combining a Pseudo-Background Binary Cross-Entropy Loss with Background Adaptation Losses, jointly supervising and stabilizing background refinement across steps. Furthermore, Group Knowledge Distillation preserves old-class decision boundaries, while Background Feature Distillation enforces consistent background representations, yielding a robust plasticity–stability trade-off. On Pascal VOC 2012 and ADE20K, ReBac achieves state-of-the-art exemplar-free performance, improving new-class segmentation by 4.5% mIoU on VOC 10-1 and 5.2% on ADE 100-5. These results highlight the value of explicitly modeling background dynamics in continual semantic segmentation. Code is available in https://github.com/ANDYZAQ/ReBac.