Abstract
Automated random testing is useful in finding faulty corner cases that are difficult to find by using manually-defined fixed test suites. However, random test inputs can be inefficient in finding faults, particularly in systems where test execution is time- and resource-consuming. Hence, filtering out less-effective test cases by applying domain knowledge constraints can contribute to test effectiveness and efficiency. In this paper, we provide a domain specific language (DSL) for formalising locality-based test selection constraints for autonomous agents. We use this DSL for filtering randomly generated test inputs. To evaluate our approach, we use a simple case study of autonomous agents and evaluate our approach using the QuickCheck tool. The results of our experiments show that using domain knowledge and applying test selection filters significantly reduce the required number of potentially expensive test executions to discover still existing faults. We have also identified the need for applying filters earlier during the test data generation. This observation shows the need to make a more formal connection between the data generation and the DSL-based filtering, which will be addressed in future work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
The experimental data and the code of statistical tests are available in “exp” sub-directory of [8].
References
Arts, T., Hughes, J., Johansson, J., Wiger, U.: Testing telecoms software with QUVIQ quickcheck. In: Proceedings of the 2006 ACM SIGPLAN Workshop on Erlang, pp. 2–10 (2006)
ASAM: ASAM openSCENARIO (2021). https://www.asam.net/standards/detail/openscenario/
Boyapati, C., Khurshid, S., Marinov, D.: Korat: Automated testing based on java predicates. ACM SIGSOFT Softw. Eng. Notes 27(4), 123–133 (2002)
Carsten, O., Merat, N., Janssen, W., Johansson, E., Fowkes, M., Brookhuis, K.: Human machine interaction and safety of traffic in europe. HASTE final Report 3 (2005)
Chen, T.Y., Kuo, F.C., Merkel, R.G., Tse, T.: Adaptive random testing: the art of test case diversity. J. Syst. Softw. 83(1), 60–66 (2010)
Daniel, B., Dig, D., Garcia, K., Marinov, D.: Automated testing of refactoring engines. In: Proceedings of the the 6th Joint Meeting of the European Software Engineering Conference and the ACM Sigsoft Symposium on the Foundations of Software Engineering, pp. 185–194 (2007)
Dreossi, T., et al.: Verifai: a toolkit for the formal design and analysis of artificial intelligence-based systems. In: Dillig, I., Tasiran, S. (eds.) Computer Aided Verification. CAV 2019. LNCS, vol. 11561, pp. 432–442. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25540-4_25
Entekhabi, S., Arts, T.: Safesmartturtle (2021). https://github.com/ThomasArts/SafeSmartTurtle
Fisher, R.A.: Xv.-the correlation between relatives on the supposition of mendelian inheritance. Trans. Roy. Soc. Edinburgh 52(2), 399–433 (1919). https://doi.org/10.1017/S0080456800012163
Foretellix Inc.: M-SDL (2021). https://www.foretellix.com/open-language/
Fremont, D.J., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Scenic: a language for scenario specification and scene generation. In: Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 63–78 (2019)
Fremont, D.J., et al.: Formal scenario-based testing of autonomous vehicles: from simulation to the real world. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1–8. IEEE (2020)
Gligoric, M., Gvero, T., Jagannath, V., Khurshid, S., Kuncak, V., Marinov, D.: Test generation through programming in Udita. In: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1, pp. 225–234 (2010)
Hamlet, D.: When only random testing will do. In: Proceedings of the 1st International Workshop on Random Testing, pp. 1–9 (2006)
Holmes, J., et al.: TSTL: the template scripting testing language. Int. J. Softw. Tools Technol. Transfer 20(1), 57–78 (2018)
Broy, M., Jonsson, B., Katoen, J.-P., Leucker, M., Pretschner, A. (eds.): Model-Based Testing of Reactive Systems. LNCS, vol. 3472. Springer, Heidelberg (2005). https://doi.org/10.1007/b137241
Khurshid, S., Marinov, D.: Testera: specification-based testing of java programs using sat. Autom. Softw. Eng. 11(4), 403–434 (2004)
Kruber, F., Wurst, J., Botsch, M.: An unsupervised random forest clustering technique for automatic traffic scenario categorization. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 2811–2818. IEEE (2018)
Kruskal, W.H., Wallis, W.A.: Use of ranks in one-criterion variance analysis. J. Am. Stat. Assoc. 47(260), 583–621 (1952)
LG Electronics Inc.: SVL Simulator (2021). https://www.svlsimulator.com/
Liu, H., Xie, X., Yang, J., Lu, Y., Chen, T.Y.: Adaptive random testing through test profiles. Softw.: Pract. Exper. 41(10), 1131–1154 (2011)
Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing, 3rd edn. Wiley Publishing (2011)
Najm, W.G., Toma, S., Brewer, J., et al.: Depiction of priority light-vehicle pre-crash scenarios for safety applications based on vehicle-to-vehicle communications. Technical report, United States. National Highway Traffic Safety Administration (2013)
Queiroz, R., Berger, T., Czarnecki, K.: Geoscenario: an open DSL for autonomous driving scenario representation. In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 287–294. IEEE (2019)
Roesener, C., Fahrenkrog, F., Uhlig, A., Eckstein, L.: A scenario-based assessment approach for automated driving by using time series classification of human-driving behaviour. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pp. 1360–1365. IEEE (2016)
Rothermel, G., Untch, R.H., Chu, C., Harrold, M.J.: Test case prioritization: An empirical study. In: Proceedings IEEE International Conference on Software Maintenance-1999 (ICSM 1999). ‘Software Maintenance for Business Change’(Cat. No. 99CB36360), pp. 179–188. IEEE (1999)
SHAPIRO, S.S., WILK, M.B.: An analysis of variance test for normality (complete samples). Biometrika 52(3–4), 591–611 (1965). https://doi.org/10.1093/biomet/52.3-4.591
Student: The probable error of a mean. Biometrika, pp. 1–25 (1908)
Thunberg, J., Sidorenko, G., Sjöberg, K., Vinel, A.: Efficiently bounding the probabilities of vehicle collision at intelligent intersections. IEEE Open J. Intell. Transp. Syst. 2, 47–59 (2021). https://doi.org/10.1109/OJITS.2021.3058449
Wilcoxon, F.: Individual comparisons by ranking methods. Biom. Bull. 1, 80–83 (1945)
Acknowledgements
We thank Jan Tretmans, Verónica Gaspes, and the anonymous reviewers of ICTSS for their valuable comments on this work. Our research has been partially funded by the Knowledge Foundation (KKS) in the framework of “Safety of Connected Intelligent Vehicles in Smart Cities – SafeSmart” project (2019–2023). Mohammad Reza Mousavi has been partially supported by the UKRI Trustworthy Autonomous Systems Node in Verifiability, Grant Award Reference EP/V026801/1.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 IFIP International Federation for Information Processing
About this paper
Cite this paper
Entekhabi, S., Mostowski, W., Mousavi, M.R., Arts, T. (2022). Locality-Based Test Selection for Autonomous Agents. In: Clark, D., Menendez, H., Cavalli, A.R. (eds) Testing Software and Systems. ICTSS 2021. Lecture Notes in Computer Science, vol 13045. Springer, Cham. https://doi.org/10.1007/978-3-031-04673-5_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-04673-5_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-04672-8
Online ISBN: 978-3-031-04673-5
eBook Packages: Computer ScienceComputer Science (R0)
