Capturing geometric structure change through deformation aware correlation

Capturing the inner structure change of the target is crucial for a siamese tracker confronted with severe deformation or occlusion. The conventional siamese tracker utilizes a strict grid-to-grid correlation to pass the template information towards the current candidate, which may suffer severe mismatch compared to the initial frame caused by undesirable conditions such as rotation and deformation. Thus such mismatch would lead to insufficient information passed from the template and eventually end with a poor performance in classification and regression branches. The inherent CNN structure with the restricted grid-to-grid matching mechanism during correlation is not able to capture the corresponding change of the target structure. Motivated by DCNv2, we propose a novel deformation aware correlation to replace the widely-used depthwise-correlation. An offset-prediction module is proposed based on the similarity matrix to compensate for the mismatch of the grid assignment between template and candidate. Then we utilize an importance bilinear sampling algorithm to assess the similarity of the matching pair, alleviating the occlusion drift problem. Finally, a modulated deformable correlation is fed into the aforementioned input and realizes a better information modulation. We replace the correlation operator on SiamRPN++ with our deformation aware correlation operator and obtain our tracker, SiamDAC. Massive experiments on OTB100, UAV123, VOT2019, LaSOT and DTB70 validate the effectiveness of our proposed operator, especially confronted with severe deformation or occlusion.