Temporal Residual Guided Diffusion Framework for Event-Driven Video Reconstruction

Lin Zhu*, Yunlong Zheng, Yijun Zhang, Xiao Wang, Lizhi Wang, Hua Huang*

*此作品的通讯作者

科研成果: 书/报告/会议事项章节会议稿件同行评审

摘要

Event-based video reconstruction has garnered increasing attention due to its advantages, such as high dynamic range and rapid motion capture capabilities. However, current methods often prioritize the extraction of temporal information from continuous event flow, leading to an overemphasis on low-frequency texture features in the scene, resulting in over-smoothing and blurry artifacts. Addressing this challenge necessitates the integration of conditional information, encompassing temporal features, low-frequency texture, and high-frequency events, to guide the Denoising Diffusion Probabilistic Model (DDPM) in producing accurate and natural outputs. To tackle this issue, we introduce a novel approach, the Temporal Residual Guided Diffusion Framework, which effectively leverages both temporal and frequency-based event priors. Our framework incorporates three key conditioning modules: a pre-trained low-frequency intensity estimation module, a temporal recurrent encoder module, and an attention-based high-frequency prior enhancement module. In order to capture temporal scene variations from the events at the current moment, we employ a temporal-domain residual image as the target for the diffusion model. Through the combination of these three conditioning paths and the temporal residual framework, our framework excels in reconstructing high-quality videos from event flow, mitigating issues such as artifacts and over-smoothing commonly observed in previous approaches. Extensive experiments conducted on multiple benchmark datasets validate the superior performance of our framework compared to prior event-based reconstruction methods.

源语言英语
主期刊名Computer Vision – ECCV 2024 - 18th European Conference, Proceedings
编辑Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, Gül Varol
出版商Springer Science and Business Media Deutschland GmbH
411-427
页数17
ISBN(印刷版)9783031736605
DOI
出版状态已出版 - 2025
活动18th European Conference on Computer Vision, ECCV 2024 - Milan, 意大利
期限: 29 9月 20244 10月 2024

出版系列

姓名Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
15098 LNCS
ISSN(印刷版)0302-9743
ISSN(电子版)1611-3349

会议

会议18th European Conference on Computer Vision, ECCV 2024
国家/地区意大利
Milan
时期29/09/244/10/24

指纹

探究 'Temporal Residual Guided Diffusion Framework for Event-Driven Video Reconstruction' 的科研主题。它们共同构成独一无二的指纹。

引用此

Zhu, L., Zheng, Y., Zhang, Y., Wang, X., Wang, L., & Huang, H. (2025). Temporal Residual Guided Diffusion Framework for Event-Driven Video Reconstruction. 在 A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, & G. Varol (编辑), Computer Vision – ECCV 2024 - 18th European Conference, Proceedings (页码 411-427). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); 卷 15098 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-73661-2_23