Abstract
Software-based optimization techniques have been increasingly used to automate code coverage analysis since the nineties. Although several studies suggest that interdependencies can exist between condition constructs in branching conditions of real life programs e.g. (\(i<=100\)) or (\(i==j\)), etc., to date, only the Ariadne system, a Grammatical Evolution (GE)-based Search Based Software Testing (SBST) technique, exploits interdependencies between variables to efficiently automate code coverage analysis.
Ariadne employs a simple attribute grammar to exploit these dependencies, which enables it to very efficiently evolve highly complex test cases, and has been compared favourably to other well-known techniques in the literature. However, Ariadne does not benefit from the interdependencies involving constants e.g. (\(i<=100\)), which are equally important constructs of condition predicates. Furthermore, constant creation in GE can be difficult, particularly with high precision.
We propose to seed the grammar with constants extracted from the source code of the program under test in order to enhance and extend Ariadne’s capability to exploit richer types of dependencies (involving all combinations of both variables and constant values). We compared our results with the original system of Ariadne against a large set of benchmark problems which include 10 numeric programs in addition to the ones originally used for Ariadne. Our results demonstrate that the seeding strategy not only dramatically improves the generality of the system, as it improves the code coverage (effectiveness) by impressive margins, but it also reduces the search budgets (efficiency) often up to an order of magnitude.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
In order to facilitate future comparisons we have made available the source code at http://bds.ul.ie/?page_id=390/.
- 2.
The source code of these benchmark functions was made available by [11] at http://bds.ul.ie/?page_id=390/.
References
Beizer, B.: Software Testing Techniques, 2nd edn. Van Nostrand Reinhold Inc., New York (1990). ISBN 0-442-20672-0
Myers, G.J., Badgett, T., Thomas, T.M., Sandler, C.: The Art of Software Testing, vol. 2. Wiley Online Library (2004)
McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verif. Reliab. 14(2), 105–156 (2004)
Afzal, W., Torkar, R., Feldt, R.: A systematic review of search-based testing for non-functional system properties. Inf. Softw. Technol. 51(6), 957–976 (2009)
Ali, S., Briand, L.C., Hemmati, H., Panesar-Walawege, R.K.: A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans. Software Eng. 36(6), 742–762 (2010)
Anand, S., et al.: An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw. 86(8), 1978–2001 (2013)
Holland, J.H.: Genetic algorithms. Sci. Am. 267(1), 66–73 (1992)
Aleti, A., Buhnova, B., Grunske, L., Koziolek, A., Meedeniya, I.: Software architecture optimization methods: a systematic literature review. IEEE Trans. Software Eng. 39(5), 658–683 (2013)
Elshoff, J.L.: An analysis of some commercial PL/I programs. IEEE Trans. Software Eng. 2, 113–120 (1976)
Cohen, E.I.: A finite domain-testing strategy for computer program testing. Ph.D. thesis, The Ohio State University (1978)
Anjum, M.S., Ryan, C.: Ariadne: evolving test data using grammatical evolution. In: Sekanina, L., Hu, T., Lourenço, N., Richter, H., García-Sánchez, P. (eds.) EuroGP 2019. LNCS, vol. 11451, pp. 3–18. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-16670-0_1
Ryan, C., Collins, J.J., Neill, M.O.: Grammatical evolution: evolving programs for an arbitrary language. In: Banzhaf, W., Poli, R., Schoenauer, M., Fogarty, T.C. (eds.) EuroGP 1998. LNCS, vol. 1391, pp. 83–96. Springer, Heidelberg (1998). https://doi.org/10.1007/BFb0055930
O’Neill, M., Ryan, C.: Grammatical evolution. IEEE Trans. Evol. Comput. 5(4), 349–358 (2001)
Dempsey, I., O’Neill, M., Brabazon, A.: Constant creation in grammatical evolution. Int. J. Innovative Comput. Appl. 1(1), 23–38 (2007)
Azad, R.M.A., Ryan, C.: The best things don’t always come in small packages: constant creation in grammatical evolution. In: Nicolau, M., et al. (eds.) EuroGP 2014. LNCS, vol. 8599, pp. 186–197. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-44303-3_16
Barros, R.C., Basgalupp, M.P., Cerri, R., da Silva, T.S., de Carvalho, A.C.: A grammatical evolution approach for software effort estimation. In: Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, pp. 1413–1420. ACM (2013)
Sparks, S., Embleton, S., Cunningham, R., Zou, C.: Automated vulnerability analysis: leveraging control flow for evolutionary input crafting. In: Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007), pp. 477–486. IEEE (2007)
Mariani, T., Guizzo, G., Vergilio, S.R., Pozo, A.T.: Grammatical evolution for the multi-objective integration and test order problem. In: Proceedings of the Genetic and Evolutionary Computation Conference 2016, pp. 1069–1076. ACM (2016)
Patten, J.V., Ryan, C.: Procedural content generation for games using grammatical evolution and attribute grammars (2014)
Kifetew, F.M., Jin, W., Tiella, R., Orso, A., Tonella, P.: Reproducing field failures for programs with complex grammar-based input. In: 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation, pp. 163–172. IEEE (2014)
de Andrade, J., Silva, L., Britto, A., Amaral, R.: Solving the software project scheduling problem with hyper-heuristics. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2019. LNCS (LNAI), vol. 11508, pp. 399–411. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20912-4_37
Lima, J.A.P., Vergilio, S.R., et al.: Automatic generation of search-based algorithms applied to the feature testing of software product lines. In: Proceedings of the 31st Brazilian Symposium on Software Engineering, pp. 114–123. ACM (2017)
Michael, C.C., McGraw, G., Schatz, M.A.: Generating software test data by evolution. IEEE Trans. Software Eng. 12, 1085–1110 (2001)
Harman, M., McMinn, P.: A theoretical and empirical study of search-based testing: local, global, and hybrid search. IEEE Trans. Software Eng. 36(2), 226–247 (2010)
Sauder, R.L.: A general test data generator for COBOL. In: Proceedings of the May 1–3, 1962, Spring Joint Computer Conference, pp. 317–323. ACM (1962)
Harman, M., Jia, Y., Zhang, Y.: Achievements, open problems and challenges for search based software testing. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–12. IEEE (2015)
Clarke, L.A.: A system to generate test data and symbolically execute programs. IEEE Trans. Software Eng. 3, 215–222 (1976)
DeMilli, R., Offutt, A.J.: Constraint-based automatic test data generation. IEEE Trans. Software Eng. 17(9), 900–910 (1991)
Offutt, A.J., Jin, Z., Pan, J.: The dynamic domain reduction procedure for test data generation. Softw.: Pract. Exp. 29(2), 167–193 (1999)
Miller, W., Spooner, D.L.: Automatic generation of floating-point test data. IEEE Trans. Software Eng. 3, 223–226 (1976)
Korel, B.: Automated software test data generation. IEEE Trans. Software Eng. 16(8), 870–879 (1990)
Ferguson, R., Korel, B.: The chaining approach for software test data generation. ACM Trans. Softw. Eng. Methodol. (TOSEM) 5(1), 63–86 (1996)
Jones, B.F., Sthamer, H.H., Eyres, D.E.: Automatic structural testing using genetic algorithms. Softw. Eng. J. 11(5), 299–306 (1996)
Pargas, R.P., Harrold, M.J., Peck, R.R.: Test-data generation using genetic algorithms. Softw. Test. Verif. Reliab. 9(4), 263–282 (1999)
Wegener, J., Baresel, A., Sthamer, H.: Evolutionary test environment for automatic structural testing. Inf. Softw. Technol. 43(14), 841–854 (2001)
Miller, J., Reformat, M., Zhang, H.: Automatic test data generation using genetic algorithm and program dependence graphs. Inf. Softw. Technol. 48(7), 586–605 (2006)
Tracey, N., Clark, J., Mander, K., McDermid, J.: An automated framework for structural test-data generation. In: ASE, p. 285. IEEE (1998)
Fraser, G., Arcuri, A., McMinn, P.: A memetic algorithm for whole test suite generation. J. Syst. Softw. 103, 311–327 (2015)
Fraser, G., Arcuri, A.: Whole test suite generation. IEEE Trans. Software Eng. 39(2), 276–291 (2013)
Panichella, A., Kifetew, F.M., Tonella, P.: Reformulating branch coverage as a many-objective optimization problem. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–10. IEEE (2015)
Xanthakis, S., Ellis, C., Skourlas, C., Le Gall, A., Katsikas, S., Karapoulios, K.: Application of genetic algorithms to software testing. In: Proceedings of the 5th International Conference on Software Engineering and Applications, pp. 625–636 (1992)
Tlili, M., Wappler, S., Sthamer, H.: Improving evolutionary real-time testing. In: Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, pp. 1917–1924. ACM (2006)
McMinn, P., Stevenson, M., Harman, M.: Reducing qualitative human oracle costs associated with automatically generated test data. In: Proceedings of the First International Workshop on Software Test Output Validation, pp. 1–4. ACM (2010)
Fraser, G., Zeller, A.: Exploiting common object usage in test case generation. In: 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation, pp. 80–89. IEEE (2011)
Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Egyed, A., Alba, E.: Comparative analysis of classical multi-objective evolutionary algorithms and seeding strategies for pairwise testing of software product lines. In: 2014 IEEE Congress on Evolutionary Computation (CEC), pp. 387–396. IEEE (2014)
Chen, T., Li, M., Yao, X.: On the effects of seeding strategies: a case for search-based multi-objective service composition. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1419–1426. ACM (2018)
Chen, T., Li, M., Yao, X.: Standing on the shoulders of giants: seeding search-based multi-objective optimization with prior knowledge for software service composition. Inf. Softw. Technol. 114, 155–175 (2019)
Alshahwan, N., Harman, M.: Automated web application testing using search based software engineering. In: Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, pp. 3–12. IEEE Computer Society (2011)
Fraser, G., Arcuri, A.: The seed is strong: seeding strategies in search-based software testing. In: 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, pp. 121–130. IEEE (2012)
Rojas, J.M., Fraser, G., Arcuri, A.: Seeding strategies in search-based unit test generation. Softw. Test. Verif. Reliab. 26(5), 366–401 (2016)
Bidgoli, A.M., Haghighi, H.: A new approach for search space reduction and seeding by analysis of the clauses. In: Colanzi, T.E., McMinn, P. (eds.) SSBSE 2018. LNCS, vol. 11036, pp. 343–348. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99241-9_19
bibclean.c (1995). http://www.cs.bham.ac.uk/~wbl/biblio/tools/bibclean.c. Accessed 15 Sept 2019
Acknowledgments
The authors would like to thank Aidan Murphy, Muhammad Hamad Khan and Sehrish Saeed for their help with conceptualization of the idea, graphic designs and benchmark functions, respectively. This work is supported by the Science Foundation of Ireland (SFI) Grant Number 16/IA/4605.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Anjum, M.S., Ryan, C. (2020). Seeding Grammars in Grammatical Evolution to Improve Search Based Software Testing. In: Hu, T., Lourenço, N., Medvet, E., Divina, F. (eds) Genetic Programming. EuroGP 2020. Lecture Notes in Computer Science(), vol 12101. Springer, Cham. https://doi.org/10.1007/978-3-030-44094-7_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-44094-7_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-44093-0
Online ISBN: 978-3-030-44094-7
eBook Packages: Computer ScienceComputer Science (R0)