Infrared and Passive Millimeter Wave image fusion based on multi-resolution deep learning method

Infrared & Passive Millimeter Wave (IR/PMMW) composite guidance is the development hotspot of multimode composite guidance technology. Considering the low penetrability of IR imaging system under nonideal visibility conditions, while the PMMW imaging technology has high atmospheric transmittance but low resolution, a fusion method of infrared/millimeter wave images based on multi-resolution deep learning is proposed. In this method, the infrared and millimeter wave images are first decomposed by NSCT transformation to separate the low-frequency and high-frequency components of the input image. For low-frequency components fusion, we design a specific generative adversarial convolution neural network for activity-level measurement and fusion rules to preserve information in the both scenes as much as possible. The high-frequency components are fused by the Pulse Couple Neural Network algorithm because of its similar processing mechanism with human visual nervous system; the fusion results of the low and high frequency components are subjected to inverse NSCT transformation, the final fused image is obtained. Data augment technology, such as image style transferring, is applied to extend IR/PMMW training set. Extensive results demonstrate that the proposed method can generate image with higher qualities with salient targets inside, deliver better performance than the state-of-the-art methods in both subjective and objective evaluation.