Automatically learning usage behavior and generating event sequences for black-box testing of reactive systems
- 85 Downloads
We propose a novel technique based on recurrent artificial neural networks to generate test cases for black-box testing of reactive systems. We combine functional testing inputs that are automatically generated from a model together with manually-applied test cases for robustness testing. We use this combination to train a long short-term memory (LSTM) network. As a result, the network learns an implicit representation of the usage behavior that is liable to failures. We use this network to generate new event sequences as test cases. We applied our approach in the context of an industrial case study for the black-box testing of a digital TV system. LSTM-generated test cases were able to reveal several faults, including critical ones, that were not detected with existing automated or manual testing activities. Our approach is complementary to model-based and exploratory testing, and the combined approach outperforms random testing in terms of both fault coverage and execution time.
KeywordsTest case generation Black-box testing Recurrent neural networks Long short-term memory networks Learning usage behavior
We would like to thank the software developers, test engineers, and technicians at Vestel Electronics for sharing their resources with us and supporting our case study. We also thank the anonymous reviewers for their comments on this paper.
- Agruss, C., & Johnson, B. (2000). Ad hoc software testing: a perspective on exploration and improvisation. In Florida institute of technology, pp. 68–69.Google Scholar
- Belli, F. (2001). Finite state testing and analysis of graphical user interfaces. In Proceedings of 12th international symposium on software reliability engineering, pp. 34–43.Google Scholar
- Berner, S., Weber, R., Keller, R. K. (2005). Observations and lessons learned from automated testing. In Proceedings of the 27th international conference on software engineering, pp. 571–579.Google Scholar
- Bottou, L. (2012). Stochastic gradient descent tricks. In Neural networks: tricks of the trade, pp. 421–436. Springer.Google Scholar
- Cho, K., van Merrienboer, B., Gülçehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 conference on empirical methods in natural language processing, pp. 1724–1734.Google Scholar
- Cotter, A., Shamir, O., Srebro, N., Sridharan, K. (2011). Better mini-batch algorithms via accelerated gradient methods. In Advances in neural information processing systems, pp. 1647–1655.Google Scholar
- Dalal, S. R., Jain, A., Karunanithi, N., Leaton, J. M., Lott, C. M., Patton, G. C., Horowitz, B. M. (1999). Model-based testing in practice. In Proceedings of the international conference on software engineering, pp. 285–294.Google Scholar
- Entin, V., Winder, M., Zhang, B., Christmann, S. (2011). Combining model-based and capture-replay testing techniques of graphical user interfaces: an industrial approach. In Proceedings of the 4th IEEE international conference on software testing, verification and validation workshops, pp. 572–577.Google Scholar
- Fard, A., Mirzaaghaei, M., Mesbah, A. (2014). Leveraging existing tests in automated test generation for web applications. In Proceedings of the 29th ACM/IEEE international conference on automated software engineering, pp. 67–78.Google Scholar
- Gebizli, C., & Sozer, H. (2016). Automated refinement of models for model-based testing using exploratory testing. Software Quality Journal. Published online. https://doi.org/10.1007/s11219-016-9338-2.
- Gebizli, C.S., & Sozer, H. (2014). Improving models for model-based testing based on exploratory testing. In Proceedings of the 6th IEEE workshop on software test automation, pp. 656–661. (COMPSAC Companion).Google Scholar
- Gers, F., & Schmidhuber, J. (2000). Recurrent nets that time and count. In Proceedings of the IEEE-INNS-ENNS international joint conference on neural networks, pp. 189–194.Google Scholar
- Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv:1308.0850.
- Guen, H. L., Marie, R., Thelin, T. (2004). Reliability estimation for statistical usage testing using Markov chains. In Proceedings of the 15th international symposium on software reliability engineering, pp. 54–65.Google Scholar
- Hagan, M., Demuth, H., Beale, M. (1995). Neural network design. New York: PWS Publishing.Google Scholar
- Itkonen, J. (2011). Empirical studies on exploratory software testing. Ph.D. thesis Aalto University.Google Scholar
- Itkonen, J., Mantyla, M. V., Lassenius, C. (2007). Defect detection efficiency: test case based vs. exploratory testing. In First international symposium on empirical software engineering and measurement, pp. 61–70. IEEE computer society.Google Scholar
- Štefanovič, J. (2000). A neural network algorithm for digital circuits test generation. In Proceedings of the European symposium on the state of the art in computational intelligence, pp. 56-60, Physica-Verlag HD, Heidelberg.Google Scholar
- Bach, J. (2003). Exploratory testing explained. Tech. rep., http://www.satisfice.com/articles/et-article.pdf.
- Kaner, C. (2006). Exploratory testing. In Quality assurance institute worldwide annual software testing conference.Google Scholar
- Karpathy, A. (2015). char-rnn https://github.com/karpathy/char-rnn.
- Kingma, D., & Ba, J. (2014). Adam: a method for stochastic optimization. arXiv:1412.6980.
- Lukac, Z., Zlokolica, V., Mlikota, B., Radonjic, M., Velikic, I. (2012). A testing methodology and system for functional verification of general HbbTV device. In Proceedings of the IEEE international conference on consumer electronics, pp. 325–326.Google Scholar
- Meinke, K., & Sindhu, M.A. (2013). LBTest: a learning-based testing tool for reactive systems. In Proceedings of the 6th IEEE international conference on software testing, verification and validation, pp. 447–454.Google Scholar
- Memon, A., Banerjee, I., Nguyen, B. N., Robbins, B. (2013). The first decade of GUI ripping: extensions, applications, and broader impacts. In Proceedings of the 20th working conference on reverse engineering, pp. 11–20.Google Scholar
- Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807–814.Google Scholar
- Neto, A. C. D., Subramanyan, R., Vieira, M., Travassos, G.H. (2007). A survey on model-based testing approaches: a systematic review. In Proceedings of the 1st ACM international workshop on empirical assessment of software engineering languages and technologies, pp. 31–36.Google Scholar
- Pacheco, C., Lahiri, S., Ernst, M., Ball, T. (2006). Feedback-directed random test generation. In Proceedings of the 29th international conference on software engineering, pp. 396–405.Google Scholar
- Peković, V., Teslić, N., Resetar, I., Tekcan, T. (2010). Test management and test execution system for automated verification of digital television systems. In IEEE International symposium on consumer electronics (ISCE 2010), pp. 1–6. https://doi.org/10.1109/ISCE.2010.5523721.
- Rafi, D., Moses, K., Petersen, K., Mäntylä, M. (2012). Benefits and limitations of automated software testing: systematic literature review and practitioner survey. In Proceedings of the 7th international workshop on automation of software test, pp. 36–42.Google Scholar
- Robinson, H. (1999). Finite state model-based testing on a shoestring. In Proceedings of the software testing and analysis and review west conference.Google Scholar
- Robinson, H. (2000). Intelligent test automation – a model-based method for generating tests from a description of an application’s behavior. Software Testing and Quality Engineering Magazine, pp. 24–32.Google Scholar
- Sak, H., Senior, A., Beaufays, F. (2014). Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In Proceedings of the 15th annual conference of the international speech communication association, pp. 338–342.Google Scholar
- Sivaraman, G., César, P., Vuorimaa, P. (2001). System software for digital television applications. In IEEE International conference on multimedia and expo, pp. 784–787.Google Scholar
- Sprenkle, A., Gibson, E., Sampath, S., Pollock, L. (2005). Automated replay and failure detection for web applications. In Proceedings of the 20th IEEE/ACM international conference on automated software engineering, pp. 253–262.Google Scholar
- Tinkham, A., & Kaner, C. (2003). Exploring exploratory testing. In Proceedings of the software testing and analysis and review east conference.Google Scholar
- Tretmans, J. (2011). Formal methods for eternal networked software systems, Springer, Berlin.Google Scholar
- Wu, Y., & et al. (2016). Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv:1609.08144.