Abstract
Software testing is a key component in software quality assurance; it typically involves generating test data that exercises all instructions and tested conditions in a program and, due to its complexity, can consume as much as 50% of overall software development budget. Some evolutionary computing techniques have been successfully applied to automate the process of test data generation but no existing techniques exploit variable interdependencies in the process of test data generation, even though several studies from the software testing literature suggest that the variables examined in the branching conditions of real life programs are often interdependent on each other, for example, if (x == y), etc.
We propose the Ariadne system which uses Grammatical Evolution (GE) and a simple Attribute Grammar to exploit the variable interdependencies in the process of test data generation. Our results show that Ariadne dramatically improves both effectiveness and efficiency when compared with existing techniques based upon well-established criteria, attaining coverage (the standard software testing success metric for these sorts of problems) of 100% on all benchmarks with far fewer program evaluations (often between a third and a tenth of other systems).
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Where possible we used the same functions, but where the actual source code isn’t available we have tried to reproduce the code as closely as possible and in order to facilitate future comparisons we have made available the source code at http://bds.ul.ie/?page_id=390/.
References
Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing. Wiley, Hoboken (2011)
Beizer, B.: Software Testing Techniques, 2nd edn. Van Nostrand Reinhold Inc., New York (1990). ISBN 0-442-20672-0
Sauder, R.L.: A general test data generator for COBOL. In: Proceedings of the 1–3 May 1962, Spring Joint Computer Conference, pp. 317–323. ACM (1962)
McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verif. Reliab. 14(2), 105–156 (2004)
Ali, S., Briand, L.C., Hemmati, H., Panesar-Walawege, R.K.: A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans. Softw. Eng. 36(6), 742–762 (2010)
Holland, J.H.: Genetic algorithms. Sci. Am. 267(1), 66–73 (1992)
Aleti, A., Buhnova, B., Grunske, L., Koziolek, A., Meedeniya, I.: Software architecture optimization methods: a systematic literature review. IEEE Trans. Softw. Eng. 39(5), 658–683 (2013)
Michael, C.C., McGraw, G., Schatz, M.A.: Generating software test data by evolution. IEEE Trans. Softw. Eng. 12, 1085–1110 (2001)
Miller, J., Reformat, M., Zhang, H.: Automatic test data generation using genetic algorithm and program dependence graphs. Inf. Softw. Technol. 48(7), 586–605 (2006)
Harman, M., McMinn, P.: A theoretical and empirical study of search-based testing: local, global, and hybrid search. IEEE Trans. Softw. Eng. 36(2), 226–247 (2010)
Cohen, E.I.: A finite domain-testing strategy for computer program testing. Ph.D. thesis, The Ohio State University (1978)
Elshoff, J.L.: An analysis of some commercial PL/I programs. IEEE Trans. Softw. Eng. 2, 113–120 (1976)
Ryan, C., Collins, J.J., Neill, M.O.: Grammatical evolution: evolving programs for an arbitrary language. In: Banzhaf, W., Poli, R., Schoenauer, M., Fogarty, T.C. (eds.) EuroGP 1998. LNCS, vol. 1391, pp. 83–96. Springer, Heidelberg (1998). https://doi.org/10.1007/BFb0055930
O’Neill, M., Ryan, C.: Grammatical evolution. IEEE Trans. Evol. Comput. 5(4), 349–358 (2001)
Harman, M., Jia, Y., Zhang, Y.: Achievements, open problems and challenges for search based software testing. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–12. IEEE (2015)
Boyer, R.S., Elspas, B., Levitt, K.N.: Selecta formal system for testing and debugging programs by symbolic execution. ACM SIGPLAN Not. 10(6), 234–245 (1975)
Miller, W., Spooner, D.L.: Automatic generation of floating-point test data. IEEE Trans. Softw. Eng. 3, 223–226 (1976)
Clarke, L.A.: A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Eng. 3, 215–222 (1976)
Ramamoorthy, C.V., Ho, S.B., Chen, W.: On the automated generation of program test data. IEEE Trans. Softw. Eng. 4, 293–300 (1976)
Offutt, A.J.: An integrated automatic test data generation system. In: Yeh, R.T. (ed.) Case Technology, pp. 129–147. Springer, Boston (1991). https://doi.org/10.1007/978-1-4615-3644-4_7
DeMilli, R., Offutt, A.J.: Constraint-based automatic test data generation. IEEE Trans. Softw. Eng. 17(9), 900–910 (1991)
Offutt, A.J., Jin, Z., Pan, J.: The dynamic domain reduction procedure for test data generation. Softw. Pract. Exp. 29(2), 167–193 (1999)
Korel, B.: Automated software test data generation. IEEE Trans. Softw. Eng. 16(8), 870–879 (1990)
Korel, B.: Dynamic method for software test data generation. Softw. Test. Verif. Reliab. 2(4), 203–213 (1992)
Korel, B.: Automated test data generation for programs with procedures. In: ACM SIGSOFT Software Engineering Notes, vol. 21, pp. 209–215. ACM (1996)
Ferguson, R., Korel, B.: The chaining approach for software test data generation. ACM Trans. Softw. Eng. Methodol. (TOSEM) 5(1), 63–86 (1996)
Tracey, N., Clark, J., Mander, K., McDermid, J.: An automated framework for structural test-data generation. In: ASE, p. 285. IEEE (1998)
Tracey, N., Clark, J.A., Mander, K.: The way forward for unifying dynamic test-case generation: the optimisation-based approach. In: Proceedings of the IFIP International Workshop on Dependable Computing and Its Applications (DCIA), York (1998)
Xanthakis, S., Ellis, C., Skourlas, C., Le Gall, A., Katsikas, S., Karapoulios, K.: Application of genetic algorithms to software testing. In: Proceedings of the 5th International Conference on Software Engineering and Applications, pp. 625–636 (1992)
Jones, B.F., Sthamer, H.H., Eyres, D.E.: Automatic structural testing using genetic algorithms. Softw. Eng. J. 11(5), 299–306 (1996)
Pargas, R.P., Harrold, M.J., Peck, R.R.: Test-data generation using genetic algorithms. Softw. Test. Verif. Reliab. 9(4), 263–282 (1999)
Wegener, J., Baresel, A., Sthamer, H.: Evolutionary test environment for automatic structural testing. Inf. Softw. Technol. 43(14), 841–854 (2001)
Arcuri, A., Yao, X.: Search based software testing of object-oriented containers. Inf. Sci. 178(15), 3075–3095 (2008)
Fraser, G., Arcuri, A., McMinn, P.: A memetic algorithm for whole test suite generation. J. Syst. Softw. 103, 311–327 (2015)
Fraser, G., Arcuri, A.: Whole test suite generation. IEEE Trans. Softw. Eng. 39(2), 276–291 (2013)
Panichella, A., Kifetew, F.M., Tonella, P.: Reformulating branch coverage as a many-objective optimization problem. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–10. IEEE (2015)
Karim, M.R., Ryan, C.: Sensitive ants are sensible ants. In: Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation, pp. 775–782. ACM (2012)
bibclean.c (1995). http://www.cs.bham.ac.uk/~wbl/biblio/tools/bibclean.c. Accessed 09 Nov 2018
Reeves, C.R., Rowe, J.E.: Genetic Algorithms Principles and Presentation: A Guide to GA Theory. Springer, New York (2002). https://doi.org/10.1007/b101880
Mitchell, M., Forrest, S., Holland, J.H.: The royal road for genetic algorithms: fitness landscapes and GA performance. In: Proceedings of the First European Conference on Artificial Life, pp. 245–254 (1992)
DeMillo, R.A., Offutt, A.J.: Experimental results from an automatic test case generator. ACM Trans. Softw. Eng. Methodol. (TOSEM) 2(2), 109–127 (1993)
Acknowledgments
The authors would like to thank Muhammad Hamad Khan for his help with the graphic designs. This work is supported by Lero, the Irish Software Research Centre, and the Science Foundation of Ireland.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Anjum, M.S., Ryan, C. (2019). Ariadne: Evolving Test Data Using Grammatical Evolution. In: Sekanina, L., Hu, T., Lourenço, N., Richter, H., GarcÃa-Sánchez, P. (eds) Genetic Programming. EuroGP 2019. Lecture Notes in Computer Science(), vol 11451. Springer, Cham. https://doi.org/10.1007/978-3-030-16670-0_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-16670-0_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-16669-4
Online ISBN: 978-3-030-16670-0
eBook Packages: Computer ScienceComputer Science (R0)