Advertisement

Memetic Computing

, Volume 10, Issue 3, pp 257–265 | Cite as

Using genetic programming to evolve action selection rules in traversal-based automated software testing: results obtained with the TESTAR tool

  • Anna I. Esparcia-Alcázar
  • Francisco Almenar
  • Tanja E. J. Vos
  • Urko Rueda
Regular Research Paper

Abstract

Traversal-based automated software testing involves testing an application via its graphical user interface (GUI) and thereby taking the user’s point of view and executing actions in a human-like manner. These actions are decided on the fly, as the software under test (SUT) is being run, as opposed to being set up in the form of a sequence prior to the testing, a sequence that is then used to exercise the SUT. In practice, random choice is commonly used to decide which action to execute at each state (a procedure commonly referred to as monkey testing), but a number of alternative mechanisms have also been proposed in the literature. Here we propose using genetic programming (GP) to evolve such an action selection strategy, defined as a list of IF-THEN rules. Genetic programming has proved to be suited for evolving all sorts of programs, and rules in particular, provided adequate primitives (functions and terminals) are defined. These primitives must aim to extract the most relevant information from the SUT and the dynamics of the testing process. We introduce a number of such primitives suited to the problem at hand and evaluate their usefulness based on various metrics. We carry out experiments and compare the results with those obtained by random selection and also by Q-learning, a reinforcement learning technique. Three applications are used as Software Under Test (SUT) in the experiments. The analysis shows the potential of GP to evolve action selection strategies.

Keywords

Automated software testing via the GUI Genetic programming Action selection for testing Testing metrics 

Notes

Acknowledgements

The authors wish to thank Xara Sharman for her help with the graphics.

References

  1. 1.
    Aho P, Menz N, Rty T (2013) Dynamic reverse engineering of GUI models for testing. In: Proceedings of 2013 international conference on control, decision and information technologies (CoDIT’13)Google Scholar
  2. 2.
    Aho P, Oliveira R, Algroth E, Vos T (2016) Evolution of automated testing of software systems through graphical user interface. In: Procs. of the 1st international conference on advances in computation, communications and services (ACCSE 2016), Valencia, pp 16–21Google Scholar
  3. 3.
    Alegroth E, Feldt R, Ryrholm L (2014) Visual GUI testing in practice: challenges, problems and limitations. Empir Softw Eng 20:694–744.  https://doi.org/10.1007/s10664-013-9293-5 CrossRefGoogle Scholar
  4. 4.
    Barr ET, Harman M, McMinn P, Shahbaz M, Yoo S (2015) The oracle problem in software testing: a survey. IEEE Trans Softw Eng 41(5):507–525CrossRefGoogle Scholar
  5. 5.
    Bauersfeld S, Vos TEJ (2012) A reinforcement learning approach to automated GUI robustness testing. In: Fast abstracts of the 4th symposium on search-based software engineering (SSBSE 2012), pp 7–12Google Scholar
  6. 6.
    Bauersfeld S, de Rojas A, Vos T (2014) Evaluating rogue user testing in industry: an experience report. In: 2014 IEEE eighth international conference on research challenges in information science (RCIS), pp 1–10.  https://doi.org/10.1109/RCIS.2014.6861051
  7. 7.
    Bauersfeld S, Vos TEJ, Condori-Fernández N, Bagnato A, Brosse E (2014) Evaluating the TESTAR tool in an industrial case study. In: 2014 ACM-IEEE international symposium on empirical software engineering and measurement, ESEM 2014, Torino, Italy, September 18–19, 2014, p 4Google Scholar
  8. 8.
    Bauersfeld S, Wappler S, Wegener J (2011) A metaheuristic approach to test sequence generation for applications with a GUI. In: Cohen MB, Ó Cinnéide M (eds) Search based software engineering: third international symposium, SSBSE 2011, Szeged, Hungary, September 10-12, 2011. Proceedings. Springer Berlin Heidelberg, Berlin, Heidelberg, pp 173–187Google Scholar
  9. 9.
    Brameier MF, Banzhaf W (2010) Linear genetic programming, 1st edn. Springer, New YorkMATHGoogle Scholar
  10. 10.
    Chaudhary N, Sangwan O (2016) Metrics for event driven software. Int J Adv Comput Sci Appl 7(1):85–89Google Scholar
  11. 11.
    Esparcia-Alcázar AI, Almenar F, Martínez M, Rueda U, Vos TE (2016) Q-learning strategies for action selection in the TESTAR automated testing tool. In: Proceedings of META 2016 6th international conference on metaheuristics and nature inspired computing, pp 174–180Google Scholar
  12. 12.
    Esparcia-Alcázar AI, Almenar F, Rueda U, Vos TEJ (2017) Evolving rules for action selection in automated testing via genetic programming–a first approach. In: Squillero G, Sim K (eds) Applications of evolutionary computation: 20th European conference, evoapplications 2017, Amsterdam, The Netherlands, April 19–21, 2017, Proceedings, part II. Springer, pp 82–95.  https://doi.org/10.1007/978-3-319-55792-2_6
  13. 13.
    Esparcia-Alcázar AI, Moravec J (2013) Fitness approximation for bot evolution in genetic programming. Soft Comput 17(8):1479–1487.  https://doi.org/10.1007/s00500-012-0965-7 CrossRefGoogle Scholar
  14. 14.
    He W, Zhao R, Zhu Q (2015) Integrating evolutionary testing with reinforcement learning for automated test generation of object-oriented software. Chin J Electron 24(1):38–45CrossRefGoogle Scholar
  15. 15.
    Koza JR (1992) Genetic programming: on the programming of computers by means of natural selection. MIT Press, CambridgeMATHGoogle Scholar
  16. 16.
    Lehman J, Stanley KO (2011) Novelty search and the problem with objectives. In: Riolo R, Vladislavleva E, Moore JH (eds) Genetic programming theory and practice IX, genetic and evolutionary computation. Springer, New York, pp 37–56Google Scholar
  17. 17.
    Memon AM, Soffa ML, Pollack ME (2001) Coverage criteria for GUI testing. In: Proceedings of ESEC/FSE 2001, pp 256–267Google Scholar
  18. 18.
    Rueda U, Vos TEJ, Almenar F, Martínez MO, Esparcia-Alcázar AI (2015) TESTAR: from academic prototype towards an industry-ready tool for automated testing at the user interface level. In: Canos JH, Gonzalez Harbour M (eds) Actas de las XX Jornadas de Ingeniería del Software y Bases de Datos (JISBD 2015), pp 236–245Google Scholar
  19. 19.
    Seesing A, Gross HG (2006) A genetic programming approach to automated test generation for object-oriented software. Int Trans Syst Sci Appl 1(2):127–134Google Scholar
  20. 20.
    Vos TE, Kruse PM, Condori-Fernández N, Bauersfeld S, Wegener J (2015) TESTAR: tool support for test automation at the user interface level. Int J Inf Syst Model Des 6(3):46–83.  https://doi.org/10.4018/IJISMD.2015070103 CrossRefGoogle Scholar
  21. 21.
    Wappler S, Wegener J (2006) Evolutionary unit testing of object-oriented software using strongly-typed genetic programming. In: Proceedings of the 8th annual conference on genetic and evolutionary computation, GECCO’06. ACM, New York, NY, USA, pp 1925–1932. URL  https://doi.org/10.1145/1143997.1144317
  22. 22.
    Watkins C (1989) Learning from delayed rewards. Ph.D. Thesis. Cambridge UniversityGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Systems Engineering and ControlUniversitat Politècnica de ValènciaValenciaSpain
  2. 2.Software Production Methods Research CentreUniversitat Politècnica de ValènciaValenciaSpain
  3. 3.Open UniversiteitHeerlenThe Netherlands
  4. 4.Etra I+DValenciaSpain

Personalised recommendations