TY - GEN
T1 - UniDriveDreamer
T2 - 19th European Conference on Computer Vision, ECCV 2026
AU - Zhao, Guosheng
AU - Wang, Yaozeng
AU - Wang, Xiaofeng
AU - Zhu, Zheng
AU - Tingdong, Yu
AU - Huang, Guan
AU - Zai, Yongchen
AU - Jiao, Ji
AU - Xue, Changliang
AU - Wang, Xiaole
AU - Yang, Zhen
AU - Zhu, Futang
AU - Wang, Xingang
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.
PY - 2026
Y1 - 2026
N2 - World models have demonstrated significant promise for data synthesis in autonomous driving. However, existing methods predominantly concentrate on single-modality generation, typically focusing on either multi-camera video or LiDAR sequence synthesis. In this paper, we propose UniDriveDreamer, a single-stage unified multimodal world model for autonomous driving, which directly generates multimodal future observations without relying on intermediate representations or cascaded modules. Our framework introduces a LiDAR-specific variational autoencoder (VAE) designed to encode input LiDAR sequences, alongside a video VAE for multi-camera images. To ensure cross-modal compatibility and training stability, we propose Unified Latent Anchoring (ULA), which explicitly aligns the latent distributions of the two modalities. The aligned features are fused and processed by a diffusion transformer that jointly models their geometric correspondence and temporal evolution. Additionally, structured scene layout information is projected per modality as a conditioning signal to guide the synthesis. Extensive experiments demonstrate that UniDriveDreamer outperforms previous state-of-the-art methods in both video and LiDAR generation, while also yielding measurable improvements in downstream driving tasks.
AB - World models have demonstrated significant promise for data synthesis in autonomous driving. However, existing methods predominantly concentrate on single-modality generation, typically focusing on either multi-camera video or LiDAR sequence synthesis. In this paper, we propose UniDriveDreamer, a single-stage unified multimodal world model for autonomous driving, which directly generates multimodal future observations without relying on intermediate representations or cascaded modules. Our framework introduces a LiDAR-specific variational autoencoder (VAE) designed to encode input LiDAR sequences, alongside a video VAE for multi-camera images. To ensure cross-modal compatibility and training stability, we propose Unified Latent Anchoring (ULA), which explicitly aligns the latent distributions of the two modalities. The aligned features are fused and processed by a diffusion transformer that jointly models their geometric correspondence and temporal evolution. Additionally, structured scene layout information is projected per modality as a conditioning signal to guide the synthesis. Extensive experiments demonstrate that UniDriveDreamer outperforms previous state-of-the-art methods in both video and LiDAR generation, while also yielding measurable improvements in downstream driving tasks.
KW - Autonomous driving
KW - Multimodal generation
UR - https://www.scopus.com/pages/publications/105052886310
U2 - 10.1007/978-3-032-37314-4_20
DO - 10.1007/978-3-032-37314-4_20
M3 - Conference contribution
AN - SCOPUS:105052886310
SN - 9783032373137
T3 - Lecture Notes in Computer Science
SP - 360
EP - 377
BT - Computer Vision – ECCV 2026 - 19th European Conference, 2026, Proceedings
A2 - Favaro, Paolo
A2 - Kukelova, Zuzana
A2 - Maki, Atsuto
A2 - Rohrbach, Anna
A2 - Schindler, Konrad
A2 - Tombari, Federico
PB - Springer Science and Business Media Deutschland GmbH
Y2 - 8 September 2026 through 12 September 2026
ER -