Transformer-Based Band Regrouping With Feature Refinement for Hyperspectral Object Tracking

Hyperspectral videos (HSVs) offer not only spatial information but also diagnostic spectral features. Due to the fact that spectral features are only related to the material of the object, this advantage can address the issue of RGB video tracking failure when the object and background are visually similar. However, the effectiveness of deep learning models is limited due to insufficient HSV training data. Existing methods tend to divide a hyperspectral image (HSI) into several three-channel false-color images to leverage the existing RGB trackers for transfer learning. Nonetheless, these methods lack adequate exploration of band interrelations and overlook correlation among objects prior to similarity calculation. In this article, a transformer-based band regrouping and feature refinement network (TBR-Net) is introduced, which is specifically tailored for hyperspectral object tracking. To maximize the potential of the RGB tracker and enhance the use of available training data, we propose a transformer-based band regrouping (TBR) method. By modeling long-range spectral dependencies, the inherent context information among bands is captured, which is subsequently utilized to reorganize bands into several false-color images. Furthermore, to combine the relationship of the template and the search (T & S) frames into a correlation calculation, a feature refinement module (FRM) is designed. The cross-attention mechanism enables mutual relation modeling, allowing similar regions to be perceived and form discriminative feature representation. As a result, a hyperspectral tracker can be efficiently trained via transfer learning to address the data insufficiency challenge, while the mutual perception between objects further enhances the tracking performance. Its effectiveness is validated by extensive benchmark experiments, which demonstrate that the TBR-Net surpasses state-of-the-art methods.