Facial AU Recognition With Feature-Based AU Localization and Confidence-Based Relation Mining

Facial Action Unit (AU) recognition involves identifying subtle muscle movements corresponding to different AUs. Recent approaches have focused on localizing AUs using predefined Regions of Interest (RoIs) or learnable modules. However, these methods either overly depend on the precision of predefined RoIs or inaccurately localize background regions instead of the actual AU positions. To address this challenge, we propose a novel method, which automatically localizes each AU without relying on predefined RoIs or introducing learnable modules during the inference phase. Specifically, our approach decomposes the task into two subtasks: AU localization and AU state verification. We first align the direction between spatial features and the corresponding AU class weights to guide the model in localizing AUs. Next, we incorporate spatial and temporal aspects for precise AU state detection. From the perspective of spatial information learning, we propose a confidence-based AU relationship mining module that directs the model to focus on uncertain AUs. From the aspect of temporal information learning, we introduce a temporal sampling strategy that implicitly captures time-dependent features. Experimental results on the BP4D and DISFA datasets demonstrate the effectiveness of our method, showing that it outperforms existing approaches and achieves state-of-the-art performance in AU recognition.