Empirical Software Engineering

, Volume 12, Issue 2, pp 183–239 | Cite as

Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques

  • Man Xiao
  • Mohamed El-Attar
  • Marek Reformat
  • James Miller


Software testing is an essential process in software development. Software testing is very costly, often consuming half the financial resources assigned to a project. The most laborious part of software testing is the generation of test-data. Currently, this process is principally a manual process. Hence, the automation of test-data generation can significantly cut the total cost of software testing and the software development cycle in general. A number of automated test-data generation approaches have already been explored. This paper highlights the goal-oriented approach as a promising approach to devise automated test-data generators. A range of optimization techniques can be used within these goal-oriented test-data generators, and their respective characteristics, when applied to these situations remain relatively unexplored. Therefore, in this paper, a comparative study about the effectiveness of the most commonly used optimization techniques is conducted.


Empirical software engineering Optimization techniques Test-data generation Empirical results 


  1. Back T (1996) Evolutionary algorithms in theory and practice. Oxford University Press, New YorkGoogle Scholar
  2. Back T, Hoffmeister F (1991) Extended selection mechanisms in genetic algorithms. Proc. of the Fourth Int. Conf. on Genetic Algorithms 92–99Google Scholar
  3. Back T, Fogel DB, Michalewicz Z (eds) (2000) Evolutionary computations I. Institute of Physics Publishing, BristolGoogle Scholar
  4. Baker JE (1985) Adaptive selection methods for genetic algorithms. Proc. of an International Conference on Genetic Algorithms. Erlbaum, HillsdaleGoogle Scholar
  5. Beizer B (1990) Software testing techniques. Van Nostrand Reinhold, 2nd ednGoogle Scholar
  6. Bird D, Munoz C (1982) Automatic generation of random Se•checking test cases. IBM Syst J 22(3):229–245CrossRefGoogle Scholar
  7. Caruana RA, Schaffer JD (1988) Representation and hidden bias: Gray vs. binary coding for genetic algorithms. In: Proceedings of the Fifth International Conference on Machine Learning, Morgan KaufmannGoogle Scholar
  8. Chang J, Richardson D, Sankar S (1996) Structural specification-based testing with ADL. In Proceedings of ISSTA’96 (International Symposium on Software Testing and Analysis), pp 62–70, San DiegoGoogle Scholar
  9. Change KH, Cross JH, Carlisle WH, Brown DB (1991) A framework for intelligent test data generators. J Intell Robot Syst-Theory and ApplicationGoogle Scholar
  10. Clarke LA (1976) A system to generate test data and symbolically execute programs. IEEE Trans Softw Eng 2(3):215–222Google Scholar
  11. Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Erlbaum, New JerseyMATHGoogle Scholar
  12. Corana A, Marchesi M, Martini C, Ridella S (1987) Minimizing multimodal functions of continuous variables with the “simulated annealing” algorithm. ACM Trans Math Softw 13(3):262–280MATHCrossRefMathSciNetGoogle Scholar
  13. De Jong, K (1975) The analysis of the behavior of a class of genetic adaptive systems, Ph.D. Dissertation, Dept, of Computer Science, University of Michigan, Ann ArborGoogle Scholar
  14. Dowsland KA (1993) Modern heuristic techniques for combinatorial problems. Chapter 2—simulated annealing, McGraw Hill, pp 20–69Google Scholar
  15. Eiben AE, Hinterding R, Michalewicz Z (1999) Parameter control in evolutionary algorithms. IEEE Transactions Computation 3(2):124–139CrossRefGoogle Scholar
  16. Goldberg DE (1989) Genetic algorithms in search, optimization, and machine learning. Addison Wesley, New YorkMATHGoogle Scholar
  17. Greffenstette JJ (1986) Optimization of control parameters for genetic algorithms. IEEE Trans Syst Man Cybern SMC-16 1:122–128CrossRefGoogle Scholar
  18. Hedges LV, Olkin I (1985) Statistical methods for meta-analysis. Academic, Orlando, FLMATHGoogle Scholar
  19. Holland JH (1975) Adaptation in natural and artificial systems. University of Michigan PressGoogle Scholar
  20. Howden WW (1997) Symbolic testing and the DISSECT symbolic evaluation system. IEEE Trans Softw Eng 3(4):266–278Google Scholar
  21. Janikow CZ, Michalewicz Z (1991) An experimental comparison of binary and floating point representations in genetic algorithms. In: Belew RK, Booker LB (eds) Proceedings of the Fourth International Conference on Genetic Algorithms, Morgan Kaufmann, pp 31–36Google Scholar
  22. Jones B, Sthamer H, Eyres D (1995) The automatic generation of software test data sets using adaptive search techniques. Proc. of the 3rd Conference on Software Quality Management 2:435–444Google Scholar
  23. Kirkpatrick S, Gelatt CD Jr, Vecchi MP (1983) Optimization by simulated annealing. Science 220(4598):671–680CrossRefMathSciNetGoogle Scholar
  24. Koakutsu S, Kang M, Dai WW (1995) Genetic simulated annealing and applications to non-slicing floorplan design. (UCSC-CRL-95-52)Google Scholar
  25. Korel B (1990) A dynamic approach of automated test data generation. Conference on Software Maintenance 311–317Google Scholar
  26. Korel B (1996) Automated test data generation for programs with procedures. In: Ziel SJ (ed) Proceedings of the 1996 International Symposium on Software Testing and Analysis (ISSTA), pp 209–215Google Scholar
  27. McGraw KO, Wong SP (1992) A common language effect-size statistic. Psychol Bull 111:361–365CrossRefGoogle Scholar
  28. McMinn P (2004) Search-based software test data generation: a survey. Softw Test Verif Reliab 14(2):105–156CrossRefGoogle Scholar
  29. Michael CC, McGraw G, Schatz MA (2001) Generating software test data by evolution. IEEE Trans Softw Eng 27(12):1085–1110CrossRefGoogle Scholar
  30. Miki M, Hiroyasu T, Ono K (2002) Simulated annealing with advanced adaptive neighborhood. In: Computational intelligence and applications (Proceedings of the 2nd International Workshop on Intelligent Systems Design and Applications: ISDA-02, 113–118 2002Google Scholar
  31. Miller J (2004) Statistical significance testing—a panacea for software technology experiments? J Syst Sofw 73:183–192CrossRefGoogle Scholar
  32. Mills HD, Dyer MD, Linger RC (1987) Cleanroom software engineering. IEEE Softw 4(5):19–25Google Scholar
  33. Osman AH, Kelly JP (1996) Meta-heuristics: an overview meta-heuristics: Theory & applications. 1–22Google Scholar
  34. Ramamoorthy CV, Ho SF, Chen WT (1976) On the generation of program test data. IEEE Trans Softw Eng 2(4):293–300Google Scholar
  35. Schaffer JD (1987) Some effects of selection procedures on hyperplane sampling by genetic algorithms. In: Davis L (ed) Genetic algorithms and simulated annealing. PitmanGoogle Scholar
  36. Schaffer JD, Caruana R, Eshelman L, Das R (1989) A study of control parameters affecting online performance of genetic algorithms for function optimization. In: Proceedings of the 3rd International Conference in Genetic Algorithms, Morgan Kaufmann, San Mateo, California, pp 51–60Google Scholar
  37. Tassey G (2002) The economics impact of inadequate infrastructure for software testing. National Institute of Standards and Technology reportGoogle Scholar
  38. Thevenhod-Fosse P, Waeselynck H (1998) STATEMATE: applied to statistical software testing. Proc. of the 1998 International Symposium on Software Testing and AnalysisGoogle Scholar
  39. Tracy N, Clark J, Mander K (1998) Automated program flaw finding using simulated annealing. Proc. of the International Symposium on Software Testing and AnalysisGoogle Scholar
  40. Voas JM, Morell L, Miller KW (1991) Predicting where faults can hide from testing. IEEE Softw 8(2):41–38CrossRefGoogle Scholar
  41. Whitley D (1993) (ed) An executable model of a simple genetic algorithm. Foundations of Genetic Algorithms (FOGA 2). Morgan KaufmannGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2006

Authors and Affiliations

  • Man Xiao
    • 1
  • Mohamed El-Attar
    • 1
  • Marek Reformat
    • 1
  • James Miller
    • 1
  1. 1.Department of Electrical and Computer Engineering, STEAM LaboratoryUniversity of AlbertaEdmontonCanada

Personalised recommendations