S2R-Bench: A Sim-to-Real Evaluation Benchmark for Autonomous Driving

Safety is a long-standing and final pursuit in the development of autonomous driving systems, with many safety challenge arising from perception. How to effectively evaluate the safety and reliability of perception algorithms is becoming an emerging issue. Despite its critical importance, existing perception methods exhibit a limitation in their robustness, primarily due to benchmarks being entirely simulated, which fail to align predicted results with actual outcomes, particularly under extreme weather conditions and sensor anomalies that are prevalent in real-world scenarios. To fill this gap, in this study, we propose a Sim-to-Real Evaluation Benchmark for autonomous driving (S2R-Bench). We collect diverse sensor anomaly data across various road conditions to evaluate the robustness of perception methods comprehensive and realistic manner. This is the first corruption robustness dataset based on real-world scenarios, encompassing various road conditions, weather conditions, lighting intensities, and time periods. By comparing real-world data with simulated data, we demonstrate the reliability the collected data for real-world use and aim to foster research on more robust perception models for autonomous driving.