Hybrid neural network for event-based object tracking

Event stream has been used in various vision tasks due to the low latency and high dynamic range of event camera. However, because of the temporal dynamic of event stream, convolution neural networks (CNNs) are difficult to effectively extract features from event streams to achieve object tracking tasks. Besides, SNNs is suitable for processing data with temporal information because of its spiking delivery mechanism and membrane potential accumulation over time. In this work, we propose a Hybrid Neural Network (HNNet) to achieve effective event-based single object tracking tasks by combining the advantages of SNNs and Swin-Transformer. For higher feature expression ability of SNNs, we adopt the Swin-Transformer to extract features from sparse event stream. Then we use these features to modulate the threshold of SNNs neurons. What’s more, for improving tracking performance for both special and temporal features, a cross-modality fusion module is designed to fuse the two features extracted by the Swin-Transformer and SNNs. We conduct extensive experiments on three public event-based datasets (FE240, FE108, and VisEvent) and our tracker outperforms other trackers maximum at 1.1% and 6.8% in terms of area under curve (AUC) scores and precision rate respectively.