Fine-tuning the Diffusion Model and Distilling Informative Priors for Sparse-view 3D Reconstruction

Jiadong Tang, Yu Gao, Tianji Jiang, Yi Yang*, Mengyin Fu

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

3D reconstruction methods such as Neural Radiance Fields (NeRFs) are capable of optimizing high-quality 3D representation from images. However, NeRF is limited by the requirement for a large number of multi-view images, making its application to real-world scenarios challenging. In this work, we propose a method that can reconstruct real-world scenes from a few input images and a simple text prompt. Specifically, we fine-tune a pretrained diffusion model to constrain its powerful priors to the visual inputs and generate 3D-aware images, leveraging the coarse renderings obtained from input images as the image condition, along with the text prompt as the text condition. Our fine-tuning method saves a significant amount of training time and GPU memory usage while also generating credible results. Moreover, to enable our method to have self-evaluation capabilities, we design a semantic switch to filter out generated images that do not match real scenes, ensuring that only informative priors from the fine-tuned diffusion model are distilled into the 3D model. The semantic switch we designed can be used as a plug-in and improve performance by 13%. We perform our approach on a real-world dataset and demonstrate competitive results compared to existing sparse-view 3D reconstruction methods. Please see our project page for more visualizations and code: https://bityia.github.io/FDfusion.

Original languageEnglish
Title of host publication2024 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2024
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages7437-7444
Number of pages8
ISBN (Electronic)9798350377705
DOIs
Publication statusPublished - 2024
Event2024 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2024 - Abu Dhabi, United Arab Emirates
Duration: 14 Oct 202418 Oct 2024

Publication series

NameIEEE International Conference on Intelligent Robots and Systems
ISSN (Print)2153-0858
ISSN (Electronic)2153-0866

Conference

Conference2024 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2024
Country/TerritoryUnited Arab Emirates
CityAbu Dhabi
Period14/10/2418/10/24

Fingerprint

Dive into the research topics of 'Fine-tuning the Diffusion Model and Distilling Informative Priors for Sparse-view 3D Reconstruction'. Together they form a unique fingerprint.

Cite this