全像素双核成像技术及应用研究综述

Dual-pixel (DP) sensor is a kind of Canon-originated hardware technique for autofocusing in 2013. Conventional autofocus methods are divided into two major categories: phase-based and contrast-based methods. However, phase-based autofocusing has higher electronic complexity, and contrast-based autofocusing runs slower in practice. Therefore, current hybrid detection autofocus technique is more concerned, which yields some pixels to imaging and focusing. However, the resolution loss issue cannot be avoided. Hybrid DP-based autofocusing methods enable each pixel to be integrated for both imaging and focusing, which improves the cost-efficiency focusing accuracy. Therefore, it has been widely used in mobile phone cameras and digital single lens reflex (DSLR) cameras. In recent years, DP sensors have been offered by sensor manufacturers and occupied the vast majority of camera sensor market. To guarantee the performance of focusing and imaging, each pixel in a DP-based sensor is equipped with two photodiodes. A DP-based sensor can assign each pixel to two halves and two images can be obtained simultaneously. These two images (DP image pair) could be viewed as a perfectly rectified stereo image pair in relation to a tiny baseline and the same exposure time, or as a two viewing angled light field camera. Unlike stereo image pairs, DP image pairs only have disparity at the out-of-focus regions, while the in-focus regions have no disparity. Defocus disparity, which only exists in the out-of-focus regions, is directly connected with the depth of the scene being captured and is generated by the point spread function. The point spread functions of the left and right views of DP are approximately symmetric, and the point spread functions based focus are relatively symmetric before and after as well. This relationship and the special point spread function can provide extra information for various computer vision tasks. Therefore, the obtained DP image pair can also be used for depth estimation, defocus deblurring and reflection removal beyond automatic focusing applications of the DP sensors. In particular, the relationship between depth and blur size in DP sensor is effective to deal with the depth from defocus task and the defocus deblur task. We critically review the autofocus, imaging principle and current situation of the DP-based sensor. 1) To provide a basic understanding of the dual-pixel sensors, we introduce the dual-pixel imaging model and imaging principle. 2) To specify the breakthrough of them, we carry out comparative analysis related to dual-pixel research in recent years. 3) To develop a reference for researchers further, we trace the current open-source dual-pixel datasets and simulators to facilitate data acquisition. Specifically, we firstly describe dual-pixel from the point of view of enabling automatic focus, where three conventional autofocus methods: 1) phase detection autofocus (PDAF), 2) contrast detection autofocus (CDAF), and 3) hybrid autofocus. The principle and priority of dual-pixel autofocus are critically reviewed in Section I. In Section II, we review the relevant optical concepts and camera imaging model. The imaging principle and geometric features of dual-pixel are introduced on four aspects: 1) dual-pixel geometry, 2) dual-pixel affine ambiguity, 3) dual-pixel point spread function, and 4) the difference between dual-pixel image pair and stereo image pair. It shows that DP image pairs can aid downstream tasks and how to mine effective hidden information from the DP image pairs. DP-based defocus disparity is linked to the contexted depth in terms of the affine ambiguity of dual-pixel, which can be used as a cue of depth estimation, defocus deblur and other related tasks. In Section III, we summarize the applications of the DP image pairs in the context of three computer vision tasks: 1) depth estimation, 2) reflection removal, and 3) defocus deblur. As appropriate datasets are fundamental to designing deep learning based architecture of neural networks better in contrast to conventional methods, we briefly introduce the community-derived DP datasets and summarize the algorithm principles of the current DP simulators. Finally, the future challenges and opportunities of the DP sensor have been discussed further in Section V.