Deep Detector and Optical Flow-based Tracking Approach of Facial Markers for Animation Capture

Marker-based facial motion capture methods are still commonly used in the film and game industry. However, these methods can lose track of target markers under certain conditions such as occlusion and blur, requiring extensive manual revisions. Therefore, there is a need for a more robust marker tracking method that can accurately and stably track markers over a longer period of time, thereby simplifying manual operations. In this paper, we present a new facial marker tracking system that focuses on improving the accuracy and stability of performance capture. Our system integrates a robust optical flow tracking method with the proposed Marker-DETR detector to achieve synthetic analysis of the marker tracking process. To evaluate the performance of our system, we collected a real-world dataset that records the performance of voluntary actors, with ground truth labels provided by artists. Our experimental results demonstrate that our approach outperforms state-of-the-art trackers such as SiamDW and ECO in terms of RMSE (root mean square error) and AUC (area under the curve). These results confirm the improved accuracy and stability of our approach.