Skip to main content
SpringerLink
Log in
Book cover

European Conference on Genetic Programming

EuroGP 2019: Genetic Programming pp 3–18Cite as

Ariadne: Evolving Test Data Using Grammatical Evolution

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11451))

Abstract

Software testing is a key component in software quality assurance; it typically involves generating test data that exercises all instructions and tested conditions in a program and, due to its complexity, can consume as much as 50% of overall software development budget. Some evolutionary computing techniques have been successfully applied to automate the process of test data generation but no existing techniques exploit variable interdependencies in the process of test data generation, even though several studies from the software testing literature suggest that the variables examined in the branching conditions of real life programs are often interdependent on each other, for example, if (x == y), etc.

We propose the Ariadne system which uses Grammatical Evolution (GE) and a simple Attribute Grammar to exploit the variable interdependencies in the process of test data generation. Our results show that Ariadne dramatically improves both effectiveness and efficiency when compared with existing techniques based upon well-established criteria, attaining coverage (the standard software testing success metric for these sorts of problems) of 100% on all benchmarks with far fewer program evaluations (often between a third and a tenth of other systems).

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    Where possible we used the same functions, but where the actual source code isn’t available we have tried to reproduce the code as closely as possible and in order to facilitate future comparisons we have made available the source code at http://bds.ul.ie/?page_id=390/.

References

  1. Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing. Wiley, Hoboken (2011)

    Google Scholar 

  2. Beizer, B.: Software Testing Techniques, 2nd edn. Van Nostrand Reinhold Inc., New York (1990). ISBN 0-442-20672-0

    MATH  Google Scholar 

  3. Sauder, R.L.: A general test data generator for COBOL. In: Proceedings of the 1–3 May 1962, Spring Joint Computer Conference, pp. 317–323. ACM (1962)

    Google Scholar 

  4. McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verif. Reliab. 14(2), 105–156 (2004)

    Article  Google Scholar 

  5. Ali, S., Briand, L.C., Hemmati, H., Panesar-Walawege, R.K.: A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans. Softw. Eng. 36(6), 742–762 (2010)

    Article  Google Scholar 

  6. Holland, J.H.: Genetic algorithms. Sci. Am. 267(1), 66–73 (1992)

    Article  Google Scholar 

  7. Aleti, A., Buhnova, B., Grunske, L., Koziolek, A., Meedeniya, I.: Software architecture optimization methods: a systematic literature review. IEEE Trans. Softw. Eng. 39(5), 658–683 (2013)

    Article  Google Scholar 

  8. Michael, C.C., McGraw, G., Schatz, M.A.: Generating software test data by evolution. IEEE Trans. Softw. Eng. 12, 1085–1110 (2001)

    Article  Google Scholar 

  9. Miller, J., Reformat, M., Zhang, H.: Automatic test data generation using genetic algorithm and program dependence graphs. Inf. Softw. Technol. 48(7), 586–605 (2006)

    Article  Google Scholar 

  10. Harman, M., McMinn, P.: A theoretical and empirical study of search-based testing: local, global, and hybrid search. IEEE Trans. Softw. Eng. 36(2), 226–247 (2010)

    Article  Google Scholar 

  11. Cohen, E.I.: A finite domain-testing strategy for computer program testing. Ph.D. thesis, The Ohio State University (1978)

    Google Scholar 

  12. Elshoff, J.L.: An analysis of some commercial PL/I programs. IEEE Trans. Softw. Eng. 2, 113–120 (1976)

    Article  Google Scholar 

  13. Ryan, C., Collins, J.J., Neill, M.O.: Grammatical evolution: evolving programs for an arbitrary language. In: Banzhaf, W., Poli, R., Schoenauer, M., Fogarty, T.C. (eds.) EuroGP 1998. LNCS, vol. 1391, pp. 83–96. Springer, Heidelberg (1998). https://doi.org/10.1007/BFb0055930

    Chapter  Google Scholar 

  14. O’Neill, M., Ryan, C.: Grammatical evolution. IEEE Trans. Evol. Comput. 5(4), 349–358 (2001)

    Article  Google Scholar 

  15. Harman, M., Jia, Y., Zhang, Y.: Achievements, open problems and challenges for search based software testing. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–12. IEEE (2015)

    Google Scholar 

  16. Boyer, R.S., Elspas, B., Levitt, K.N.: Selecta formal system for testing and debugging programs by symbolic execution. ACM SIGPLAN Not. 10(6), 234–245 (1975)

    Article  Google Scholar 

  17. Miller, W., Spooner, D.L.: Automatic generation of floating-point test data. IEEE Trans. Softw. Eng. 3, 223–226 (1976)

    Article  MathSciNet  Google Scholar 

  18. Clarke, L.A.: A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Eng. 3, 215–222 (1976)

    Article  MathSciNet  Google Scholar 

  19. Ramamoorthy, C.V., Ho, S.B., Chen, W.: On the automated generation of program test data. IEEE Trans. Softw. Eng. 4, 293–300 (1976)

    Article  Google Scholar 

  20. Offutt, A.J.: An integrated automatic test data generation system. In: Yeh, R.T. (ed.) Case Technology, pp. 129–147. Springer, Boston (1991). https://doi.org/10.1007/978-1-4615-3644-4_7

    Chapter  Google Scholar 

  21. DeMilli, R., Offutt, A.J.: Constraint-based automatic test data generation. IEEE Trans. Softw. Eng. 17(9), 900–910 (1991)

    Article  Google Scholar 

  22. Offutt, A.J., Jin, Z., Pan, J.: The dynamic domain reduction procedure for test data generation. Softw. Pract. Exp. 29(2), 167–193 (1999)

    Article  Google Scholar 

  23. Korel, B.: Automated software test data generation. IEEE Trans. Softw. Eng. 16(8), 870–879 (1990)

    Article  Google Scholar 

  24. Korel, B.: Dynamic method for software test data generation. Softw. Test. Verif. Reliab. 2(4), 203–213 (1992)

    Article  Google Scholar 

  25. Korel, B.: Automated test data generation for programs with procedures. In: ACM SIGSOFT Software Engineering Notes, vol. 21, pp. 209–215. ACM (1996)

    Google Scholar 

  26. Ferguson, R., Korel, B.: The chaining approach for software test data generation. ACM Trans. Softw. Eng. Methodol. (TOSEM) 5(1), 63–86 (1996)

    Article  Google Scholar 

  27. Tracey, N., Clark, J., Mander, K., McDermid, J.: An automated framework for structural test-data generation. In: ASE, p. 285. IEEE (1998)

    Google Scholar 

  28. Tracey, N., Clark, J.A., Mander, K.: The way forward for unifying dynamic test-case generation: the optimisation-based approach. In: Proceedings of the IFIP International Workshop on Dependable Computing and Its Applications (DCIA), York (1998)

    Google Scholar 

  29. Xanthakis, S., Ellis, C., Skourlas, C., Le Gall, A., Katsikas, S., Karapoulios, K.: Application of genetic algorithms to software testing. In: Proceedings of the 5th International Conference on Software Engineering and Applications, pp. 625–636 (1992)

    Google Scholar 

  30. Jones, B.F., Sthamer, H.H., Eyres, D.E.: Automatic structural testing using genetic algorithms. Softw. Eng. J. 11(5), 299–306 (1996)

    Article  Google Scholar 

  31. Pargas, R.P., Harrold, M.J., Peck, R.R.: Test-data generation using genetic algorithms. Softw. Test. Verif. Reliab. 9(4), 263–282 (1999)

    Article  Google Scholar 

  32. Wegener, J., Baresel, A., Sthamer, H.: Evolutionary test environment for automatic structural testing. Inf. Softw. Technol. 43(14), 841–854 (2001)

    Article  Google Scholar 

  33. Arcuri, A., Yao, X.: Search based software testing of object-oriented containers. Inf. Sci. 178(15), 3075–3095 (2008)

    Article  Google Scholar 

  34. Fraser, G., Arcuri, A., McMinn, P.: A memetic algorithm for whole test suite generation. J. Syst. Softw. 103, 311–327 (2015)

    Article  Google Scholar 

  35. Fraser, G., Arcuri, A.: Whole test suite generation. IEEE Trans. Softw. Eng. 39(2), 276–291 (2013)

    Article  Google Scholar 

  36. Panichella, A., Kifetew, F.M., Tonella, P.: Reformulating branch coverage as a many-objective optimization problem. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pp. 1–10. IEEE (2015)

    Google Scholar 

  37. Karim, M.R., Ryan, C.: Sensitive ants are sensible ants. In: Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation, pp. 775–782. ACM (2012)

    Google Scholar 

  38. bibclean.c (1995). http://www.cs.bham.ac.uk/~wbl/biblio/tools/bibclean.c. Accessed 09 Nov 2018

  39. Reeves, C.R., Rowe, J.E.: Genetic Algorithms Principles and Presentation: A Guide to GA Theory. Springer, New York (2002). https://doi.org/10.1007/b101880

    Book  MATH  Google Scholar 

  40. Mitchell, M., Forrest, S., Holland, J.H.: The royal road for genetic algorithms: fitness landscapes and GA performance. In: Proceedings of the First European Conference on Artificial Life, pp. 245–254 (1992)

    Google Scholar 

  41. DeMillo, R.A., Offutt, A.J.: Experimental results from an automatic test case generator. ACM Trans. Softw. Eng. Methodol. (TOSEM) 2(2), 109–127 (1993)

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank Muhammad Hamad Khan for his help with the graphic designs. This work is supported by Lero, the Irish Software Research Centre, and the Science Foundation of Ireland.

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Systems, University of Limerick, Castletroy, Limerick, Ireland

    Muhammad Sheraz Anjum & Conor Ryan

Authors
  1. Muhammad Sheraz Anjum
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Conor Ryan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Sheraz Anjum .

Editor information

Editors and Affiliations

  1. Brno University of Technology, Brno, Czech Republic

    Lukas Sekanina

  2. Memorial University, St. John's, NL, Canada

    Ting Hu

  3. University of Coimbra, Coimbra, Portugal

    Nuno Lourenço

  4. HTWK Leipzig University of Applied Sciences, Leipzig, Germany

    Hendrik Richter

  5. University of Granada, Granada, Spain

    Pablo García-Sánchez

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Anjum, M.S., Ryan, C. (2019). Ariadne: Evolving Test Data Using Grammatical Evolution. In: Sekanina, L., Hu, T., Lourenço, N., Richter, H., García-Sánchez, P. (eds) Genetic Programming. EuroGP 2019. Lecture Notes in Computer Science(), vol 11451. Springer, Cham. https://doi.org/10.1007/978-3-030-16670-0_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-16670-0_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-16669-4

  • Online ISBN: 978-3-030-16670-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation