HSVCNN: CNN-Based Hyperspectral Reconstruction from RGB Videos

Hyperspectral video acquisition usually requires high complexity hardware and reconstruction algorithms. In this paper, we propose a low complexity CNN-based method for hyperspectral reconstruction from ubiquitous RGB videos, which effectively exploits the temporal redundancies within RGB videos and generates high-quality hyperspectral output. Specifically, given an RGB video, we first design an efficient motion compensation network to align the RGB frames and reduce the large motion. Then, we design a temporal-adaptive fusion network to exploit the inter-frame correlation. The fusion network has the ability to determine the optimum temporal dependency within successive frames, which further promotes the hyperspectral reconstruction fidelity. Preliminary experimental results validate the superior performance of the proposed method over previous learning-based methods. To the best of our knowledge, this is the first time that RGB videos are utilized for hyperspectral reconstruction through deep learning.