TransCNN-MLDFN: A Multi-level Dynamic Fusion Framework for Incomplete Palmprint and Palm Vein Recognition

As one of the most popular biometric recognition methods in identity identification, palmprint and palm vein patterns have recently attracted much attention due to their uniqueness and stability. Most existing deep learning-based palmprint and palm vein fusion recognition methods are based on convolutional neural network (CNN) model. However, these methods struggle with preserving global features and the recognition performance will significantly decrease when dealing with incomplete images caused by injury, lighting, or hand posture. In addition, the feature maps output from the network are usually fused through element addition or cascading, without considering different weights of palmprint and palm vein modalities. To alleviate this problem, we present a novel two-stage multimodal dynamic fusion framework for palmprint and palm vein fusion recognition, called TransCNN-MLDFN. Specifically, we design a multi-level dynamic fusion network, which includes a global feature extraction block based on Transformer network and a local feature extraction block based on CNN. Following this, we propose a two-stage dynamic weighted fusion strategy combining shallow and deep features by setting the weights of palmprint and vein images as learnable parameters to fully integrate the feature information of two modalities. Finally, we design a new loss function to improve the accuracy and generalization ability of the model. Numerous experiments on four widely used and public available palmprint and palm vein datasets show that our proposed method outperforms other methods in terms of accuracy and equal error rate on both complete and incomplete image recognition scenarios.