Adaptive Causal Counterfactual Network for Weakly Supervised Group Activity Recognition

Weakly supervised group activity recognition (WSGAR) aims to recognize collective activities from video clips using only clip-level labels, without actor-level annotations or explicit person detectors. Existing detector-free methods often compress noisy spatial responses through fixed or uniform aggregation, which can weaken local interaction cues. To address this limitation, we propose an Adaptive Causal Counterfactual Network (ACCN), a unified detector-free framework that integrates spatial refinement, structured relation–interaction reasoning, and token-space intervention. Specifically, Adaptive Spatial Resampling (ASR) refines frame-wise feature maps before tokenization through learnable local resampling, preserving discriminative responses before compact tokens are formed. On top of the clip-level tokens, the Relation–Interaction Compositional Head (RICH) separately models latent person-oriented relation cues and interaction-target-oriented cues, and composes them into structured group-context representations with auxiliary branch supervision. Furthermore, Saliency-guided Prototype Counterfactual Learning (SPCL) identifies decision-sensitive token partitions in hard samples and replaces them with class-prototype tokens. SPCL serves as a counterfactual regularizer during training and can also be enabled for counterfactual evidence analysis during evaluation, encouraging a more robust decision boundary and more reliable evidence attribution. Extensive experiments on the Volleyball, NBA, and Collective Activity datasets demonstrate that ACCN achieves strong and consistent performance, while providing faithful visual explanations.