Hole-filling based on content loss indexed 3D partial convolution network for freehand ultrasound reconstruction

Background and objective: During the 3D reconstruction of ultrasound volume from 2D B-scan ultrasound images, holes are usually found in the reconstructed 3D volumes due to the fast scans. This condition will affect the positioning and judgment of the doctor to the lesion. Hence, in this study, we propose to fill the holes by using a novel content loss indexed 3D partial convolution network for 3D freehand ultrasound volume reconstruction. The network can synthesize novel ultrasound volume structures and reconstruct ultrasound volume with missing regions with variable sizes and at arbitrary locations. Methods: First, the 3D partial convolution is introduced into the convolutional layer, which is masked and renormalized to be conditioned on only valid voxels. Then, the mask in the next layer is automatically updated as a part of the forward pass. To better preserve texture and structure details of the reconstruction results, we couple the adversarial loss of the least squares generative adversarial network (LSGAN) with the innovative content loss, which consists of the context loss, the feature-matching loss and the total variation loss. Thereafter, we introduce a novel spectral-normalized LSGAN by adding spectral normalization (SN) to the generator and discriminator of the LSGAN. The proposed method is simple in formulation, and is stable in training. Results: Experiments on public and in-vivo ultrasound datasets and comparisons with popular algorithms demonstrate that the proposed approach can generate high-quality hole-filling results with preserved perceptual image details. Conclusions: Considering the high quality of the hole-filling results, the proposed method can effectively fill the missing regions in the reconstructed 3D ultrasound volume from 2D ultrasound image sequences.