TY - GEN
T1 - Optimization-Guided Diffusion for Interactive Scene Generation
AU - Li, Shihao
AU - Ye, Naisheng
AU - Li, Tianyu
AU - Chitta, Kashyap
AU - An, Tuo
AU - Su, Peng
AU - Wang, Boyang
AU - Liu, Haiou
AU - Lv, Chen
AU - Li, Hongyang
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.
PY - 2026
Y1 - 2026
N2 - Realistic and diverse multi-agent driving scenes are crucial for evaluating autonomous vehicles, but safety-critical events which are essential for this task are rare and underrepresented in driving datasets. Data-driven scene generation offers a low-cost alternative by synthesizing complex traffic behaviors from existing driving logs. However, existing models often lack controllability or yield samples that violate physical or social constraints, limiting their usability. We present OMEGA, an optimization-guided, training-free framework that enforces structural consistency and interaction awareness during diffusion-based sampling from a scene generation model. OMEGA re-anchors each reverse diffusion step via constrained optimization, steering the generation towards physically plausible and behaviorally coherent trajectories. Building on this framework, we formulate ego–attacker interactions as a game-theoretic optimization in the distribution space, approximating Nash equilibria to generate realistic, safety-critical adversarial scenarios. Experiments on nuPlan and Waymo show that OMEGA improves generation realism, consistency, and controllability, increasing the ratio of physically and behaviorally valid scenes from 32.35% to 72.27% for free exploration capabilities, and from 11% to 80% for controllability-focused generation. Our approach can also generate 5× more near-collision frames with a time-to-collision under three seconds while maintaining the overall scene realism.
AB - Realistic and diverse multi-agent driving scenes are crucial for evaluating autonomous vehicles, but safety-critical events which are essential for this task are rare and underrepresented in driving datasets. Data-driven scene generation offers a low-cost alternative by synthesizing complex traffic behaviors from existing driving logs. However, existing models often lack controllability or yield samples that violate physical or social constraints, limiting their usability. We present OMEGA, an optimization-guided, training-free framework that enforces structural consistency and interaction awareness during diffusion-based sampling from a scene generation model. OMEGA re-anchors each reverse diffusion step via constrained optimization, steering the generation towards physically plausible and behaviorally coherent trajectories. Building on this framework, we formulate ego–attacker interactions as a game-theoretic optimization in the distribution space, approximating Nash equilibria to generate realistic, safety-critical adversarial scenarios. Experiments on nuPlan and Waymo show that OMEGA improves generation realism, consistency, and controllability, increasing the ratio of physically and behaviorally valid scenes from 32.35% to 72.27% for free exploration capabilities, and from 11% to 80% for controllability-focused generation. Our approach can also generate 5× more near-collision frames with a time-to-collision under three seconds while maintaining the overall scene realism.
KW - Autonomous Driving
KW - Constrained Optimization
KW - Diffusion Models
KW - Guided Sampling
KW - Interactive Scene Generation
UR - https://www.scopus.com/pages/publications/105052556188
U2 - 10.1007/978-3-032-37167-6_13
DO - 10.1007/978-3-032-37167-6_13
M3 - Conference contribution
AN - SCOPUS:105052556188
SN - 9783032371669
T3 - Lecture Notes in Computer Science
SP - 228
EP - 244
BT - Computer Vision – ECCV 2026 - 19th European Conference, 2026, Proceedings
A2 - Favaro, Paolo
A2 - Kukelova, Zuzana
A2 - Maki, Atsuto
A2 - Rohrbach, Anna
A2 - Schindler, Konrad
A2 - Tombari, Federico
PB - Springer Science and Business Media Deutschland GmbH
T2 - 19th European Conference on Computer Vision, ECCV 2026
Y2 - 8 September 2026 through 12 September 2026
ER -