Abstract
Today’s automotive vehicles are often equipped with powerful data processing systems for driver assistance and/or autonomous driving. To meet the rigorous safety standard, ensuring extremely small failure probability over all possible operation conditions is one of the critical tasks for an autonomous driving system. In this chapter, we describe a novel methodology to validate vision-based autonomous driving systems over different circuit corners with consideration of temperature variation and circuit aging. Our approach seamlessly integrates the image data recorded under nominal conditions with comprehensive statistical circuit models to synthetically generate the critical corner cases for which an autonomous driving system is likely to fail. As such, a given automotive system can be robustly validated for these worst-case scenarios that cannot be easily captured by physical experiments. To efficiently estimate the rare failure rate of an autonomous system, we further propose a novel Subset Sampling (SUS) algorithm. In particular, a Markov Chain Monte Carlo algorithm based on graph mapping is developed to accurately estimate the rare failure rate with a minimal amount of test data, thereby minimizing the validation cost. Our numerical experiments show that SUS achieves 15.2× runtime speed-up over the conventional brute-force Monte Carlo method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
A. Lindgren, F. Chen, in State of the art analysis: an overview of advanced driver assistance systems (ADAS) and possible human factors issues. Human Factors and Economics Aspects on Safety (2006), pp. 38–50
M. Aeberhard et al., Experience, results and lessons learned from automated driving on Germany’s highways. IEEE Intell. Transp. Syst. Mag. 7(1), 42–57 (2015)
J. Ziegler et al., Making bertha drive—an autonomous journey on a historic route. IEEE Intell. Transp. Syst. Mag. 6(2), 8–20 (2014)
M. Fu, Y. Huang, in A survey of traffic sign recognition. ICWAPR (2010), pp. 119–124
N. Kalra, S.M. Paddock, Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability. Transp. Res. A Pol. Pract. 94, 182–193 (2016)
D. Hospach et al., in Simulation of falling rain for robustness testing of video-based surround sensing systems. DATE (2016), pp. 233–236
R. Gallen et al., Nighttime visibility analysis and estimation method in the presence of dense fog. IEEE Trans. Intell. Transp. Syst. 16(1), 310–320 (2015)
M. Nentwig, M. Stamminger, in Hardware-in-the-loop testing of computer vision based driver assistance systems. IEEE Intelligent Vehicles Symposium (2011), pp. 339–344
M. Nentwig et al., in Concerning the applicability of computer graphics for the evaluation of image processing algorithms. ICVES (2012), pp. 205–210
A. Broggi et al., Extensive tests of autonomous driving technologies. IEEE Trans. Intell. Transp. Syst. 14(3), 1403–1415 (2013)
E. Roth, T. Calapoglu, in Advanced driver assistance system testing using OptiX. NVIDIA GTC (2012)
L. Raffaëlli et al., in Facing ADAS validation complexity with usage oriented testing. ERTS (2016)
M. Lankamp et al., MGSim-simulation tools for multi-core processor architectures, arXiv (2013)
S. Au, J. Beck, Estimation of small failure probabilities in high dimensions by subset simulation. Probabilist. Eng. Mech. 16(4), 263–277 (2001)
S. Sun, X. Li, in Fast statistical analysis of rare circuit failure events via subset simulation in high-dimensional variation space. ICCAD (2014), pp. 324–331
T. Kuroda, Essential Principles of Image Sensors (CRC Press, Boca Raton, 2014)
G.C. Holst, CCD Arrays, Cameras, and Displays (JCD Publication, Winter Park, 1998)
R. Ramanath et al., Color image processing pipeline. IEEE Signal Process. Mag. 22(1), 34–43 (2005)
C.M. Kyung, Theory and Applications of Smart Cameras (Springer, Dordrecht, 2016)
C. Bishop, Pattern Recognition and Machine Learning (Prentice Hall, Upper Saddle River, 2007)
P. Viola, M. Jones, in Rapid object detection using a boosted cascade of simple features. CVPR (2001), pp. I511–I518
W. Shi et al., in Efficient statistical validation of machine learning systems for autonomous driving. ICCAD (2016)
X. Xi et al., in BSIM4.3.0 MOSFET Model—User’s Manual, vol. 2, no. 1 (University of California, Berkley, 2003), pp. 351–354
P. Seitz, A.J. Theuwissen, Single-Photon Imaging (Springer, Berlin, 2011)
G.G. Nampoothiri et al., in Ageing effects on image sensors due to terrestrial cosmic radiation. IS&T/SPIE Electronic Imaging, vol. 7875 (2011)
G.H. Chapman et al., in Empirical formula for rates of hot pixel defects based on pixel size, sensor area and ISO. IS&T/SPIE Electronic Imaging, vol. 8659, (2013)
H.T. Sencar, N. Memon, Digital Image Forensics (Springer, New York, 2013)
W. Shi et al., An FPGA-based hardware accelerator for traffic sign detection. IEEE Trans. Very Large Scale Integr. Syst. 25(4), 1362–1372 (2017)
P. Dollar, Piotr’s computer vision MATLAB toolbox (PMT), http://vision.ucsd.edu/~pdollar/toolbox/doc/index.html
R. Timof et al., in Traffic sign recognition—how far are we from the solution? IJCNN (2013)
J. Stallkamp et al., Man vs. computer: benchmarking machine learning algorithms for traffic sign recognition. Neural Netw. 32, 323–332 (2012)
S. Houben et al., in Detection of traffic signs in real-world images: The German traffic sign detection benchmark. IJCNN (2014)
S. Sahni, T. Gonzalez, P-complete approximation problems. J. ACM 23(3), 555–565 (1976)
D. Harel, Y. Koren, in Graph Drawing by High Dimensional Embedding. International Symposium on Graph Drawing (Springer, Berlin, 2002). pp. 207–219
S. Banerjee, A. Roy, Linear Algebra and Matrix Analysis for Statistics (Chapman and Hall/CRC, Hoboken, 2014)
I. Borg, Modern Multidimensional Scaling: Theory and Applications (Springer, New York, 2005)
P. Gopalakrishnan, X. Li, L. Pileggi, in Architecture-aware FPGA placement using metric embedding. DAC (2006), pp. 460–465
J. Bentley, Multidimensional binary search trees used for associative searching. Commun. ACM 18(9), 509–517 (1975)
Acknowledgments
This research work has been supported in part by Toyota InfoTechnology Center, Cadence Design Systems, National Key Research and Development Program of China 2016YFB0201304, and National Natural Science Foundation of China (NSFC) research project 61376040, 61574046 and 61674042.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Yu, H. et al. (2019). Efficient Statistical Validation of Autonomous Driving Systems. In: Yu, H., Li, X., Murray, R., Ramesh, S., Tomlin, C. (eds) Safe, Autonomous and Intelligent Vehicles. Unmanned System Technologies. Springer, Cham. https://doi.org/10.1007/978-3-319-97301-2_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-97301-2_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-97300-5
Online ISBN: 978-3-319-97301-2
eBook Packages: EngineeringEngineering (R0)