OF-DFN: Optical flow prediction network for different perspective image fusion

Currently, non-decision-level image fusion algorithms require extremely high registration precision of the images to be fused. In the face of different perspective image fusion scenarios, traditional feature registration algorithms and learning-based methods have poor robustness and are unsuitable for large image differences because of the Registration-Fusion separation. In addition, the lack of relevant datasets also hinders the development of different perspective image fusion methods. Given the above problems, we collect 5000 sets of different perspective RGB-MONO datasets in multiple scenes for raw data support. We present an end-to-end learned system for fusing two different perspective photographs into a chosen target view. The cascaded feature extraction based on encoder–decoder structure enables learning optical flow at different feature levels systematically. Then the optical flow module enables the image to be continuously registered and optimized during the fusion process, thus avoiding the deviations introduced by non-end-to-end algorithms. Extensive quantitative and qualitative experiments demonstrate that our proposed system can effectively fuse images from different perspectives in our self-built dataset. Compared with non-end-to-end fusion, our method provides superior performance in several fusion evaluation indicators.