A perceptual framework for infrared–visible image fusion based on multiscale structure decomposition and biological vision

Infrared–visible image fusion is of great value in many applications due to their highly complementary information. However, it is hard to obtain high-quality fused image for current fusion algorithms. In this paper, we reveal an underlying deficiency in current fusion framework limiting the quality of fusion, i.e., the visual features used in the fusion can be easily affected by external physical conditions (e.g., the characteristics of different sensors and environmental illumination), indicating that those features from different sources have not been ensured to be fused on a consistent basis during the fusion. Inspired by biological vision, we derive a framework that transforms the image intensities into the visual response space of human visual system (HVS), within which all features are fused in the same perceptual state, eliminating the external physical factors that may influence the fusion process. The proposed framework incorporates some key characteristics of HVS that facilitate the simulation of human visual response in complex scenes, and is built on a new variant of multiscale decomposition, which can accurately localize image structures of different scales in visual-response simulation and feature fusion. A bidirectional saliency aggregation is proposed to fuse the perceived contrast features within the visual response space of HVS, along with an adaptive suppression of noise and intensity-saturation in this space prior to the fusion. The final fused image is obtained by transforming the fusion results in human visual response space back to the physical domain. Experiments demonstrate the significant improvement of fusion quality brought about by the proposed method.