Abstract
The testing of Autonomous Vehicles (AVs) requires driving the AV billions of miles under varied scenarios in order to find bugs, accidents and otherwise inappropriate behavior. Because driving a real AV that many miles is too slow and costly, this motivates the use of sophisticated ‘world simulators’, which present the AV’s perception pipeline with realistic input scenes, and present the AV’s control stack with realistic traffic and physics to which to react. Thus the simulator is a crucial piece of any CAD toolchain for AV testing. In this work, we build a test harness for driving an arbitrary AV’s code in a simulated world. We demonstrate this harness by using the game Grand Theft Auto V (GTA) as world simulator for AV testing. Namely, our AV code, for both perception and control, interacts in real-time with the game engine to drive our AV in the GTA world, and we search for weather conditions and AV operating conditions that lead to dangerous situations. This goes beyond the current state-of-the-art where AVs are tested under ideal weather conditions, and lays the ground work for a more comprehensive testing effort. We also propose and demonstrate necessary analyses to validate the simulation results relative to the real world. The results of such analyses allow the designers and verification engineers to weigh the results of simulation-based testing.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Since a countable infinite set always has measure 0 in a continuous search space, uniform random sampling cannot offer such a guarantee. That said, currently known bounds on the convergence rate of Simulated Annealing are too loose in our context.
References
Godil, A.A., et al.: Performance metrics for evaluating object and human detection and tracking systems, nIST Interagency/Internal Report (NISTIR) - 7972, July 2014
Alexander Blade, A.S.D.: Script hook v.net, September 2017. https://github.com/crosire/scripthookvdotnet
Baidu: Apollo platform, September 2017. apollo.auto
Chen, C., Seff, A., Kornhauser, A., Xiao, J.: Deepdriving: learning affordance for direct perception in autonomous driving. In: International Conference on Computer Vision (2015)
Chib, S., Greenberg, E.: Understanding the Metropolis-Hastings algorithm. Am. Stat. 49(4), 327–335 (1995)
Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the kitti dataset. Int. J. Robot. Res. 32(11), 1231–1237 (2013)
Johnson-Roberson, M., Barto, C., Mehta, R., Sridhar, S.N., Rosaen, K., Vasudevan, R.: Driving in the matrix: can virtual worlds replace human-generated annotations for real world tasks? In: ICRA, May 2017
Kato, S.: Autoware, September 2017. https://github.com/CPFL/Autoware
Kundu, D.: Subjective and objective quality evaluation of synthetic and high dynamic range images, Ph.D. Dissertation, May 2016
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Conference on Computer Vision and Pattern Recognition (2016)
Redmon, J., Farhadi, A.: Yolo9000: better, faster, stronger. https://arxiv.org/abs/1612.08242
Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: ground truth from computer games. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 102–118. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_7
Udacity, A.R.: Udacity self-driving car dataset, p. 2 (2017). http://bit.ly/udacity-annotations-autti
Winkler, S.: Analysis of public image and video databases for quality assessment. IEEE J. Sel. Top. Signal Process. 6(6), 616–625 (2012)
Zhao, D., Peng, H.: From the lab to the street, m-City White Paper, May 2017
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Abbas, H., O’Kelly, M., Rodionova, A., Mangharam, R. (2019). Safe At Any Speed: A Simulation-Based Test Harness for Autonomous Vehicles. In: Chamberlain, R., Taha, W., Törngren, M. (eds) Cyber Physical Systems. Design, Modeling, and Evaluation. CyPhy 2017. Lecture Notes in Computer Science(), vol 11267. Springer, Cham. https://doi.org/10.1007/978-3-030-17910-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-17910-6_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-17909-0
Online ISBN: 978-3-030-17910-6
eBook Packages: Computer ScienceComputer Science (R0)