Hypervolume-Based Search for Test Case Prioritization

  • Dario Di NucciEmail author
  • Annibale Panichella
  • Andy Zaidman
  • Andrea De Lucia
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9275)


Test case prioritization (TCP) is aimed at finding an ideal ordering for executing the available test cases to reveal faults earlier. To solve this problem greedy algorithms and meta-heuristics have been widely investigated, but in most cases there is no statistically significant difference between them in terms of effectiveness. The fitness function used to guide meta-heuristics condenses the cumulative coverage scores achieved by a test case ordering using the Area Under Curve (AUC) metric. In this paper we notice that the AUC metric represents a simplified version of the hypervolume metric used in many objective optimization and we propose HGA, a Hypervolume-based Genetic Algorithm, to solve the TCP problem when using multiple test criteria. The results shows that HGA is more cost-effective than the additional greedy algorithm on large systems and on average requires 36 % of the execution time required by the additional greedy algorithm.


Test case prioritization Genetic algorithm Hypervolume 


  1. 1.
    Arcuri, A., Briand, L.C.: A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Proceedings of International Conference on Software Engineering (ICSE), pp. 1–10. ACM (2011)Google Scholar
  2. 2.
    Auger, A., Bader, J., Brockhoff, D., Zitzler, E.: Theory of the hypervolume indicator: optimal \(\mu \)-distributions and the choice of the reference point. In: Proceedings of SIGEVO workshop on Foundations of Genetic Algorithms (FOGA), pp. 87–102. ACM (2009)Google Scholar
  3. 3.
    Bryce, R.C., Colbourn, C.J., Cohen, M.B.: A framework of greedy methods for constructing interaction test suites. In: Proceedings International Conference on Software Engineering (ICSE), pp. 146–155 (2005)Google Scholar
  4. 4.
    Chen, T.Y., Lau, M.F.: Dividing strategies for the optimization of a test suite. Inf. Process. Lett. 60(3), 135–141 (1996)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Cohen, M., Dwyer, M., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach. IEEE Trans. Softw. Eng. 34, 633–650 (2008)CrossRefGoogle Scholar
  6. 6.
    Conover, W.J.: Practical Nonparametric Statistics, 3rd edn. Wiley, New York (1998)Google Scholar
  7. 7.
    De Lucia, A., Di Penta, M., Oliveto, R., Panichella, A.: On the role of diversity measures for multi-objective test case selection. In: Proceedings of International Workshop on Automation of Software Test (AST), pp. 145–151 (2012)Google Scholar
  8. 8.
    Do, H., Rothermel, G., Kinneer, A.: Empirical studies of test case prioritization in a junit testing environment. In: 15th International Symposium on Software Reliability Engineering, pp. 113–124. IEEE Computer Society (2004)Google Scholar
  9. 9.
    Durillo, J.J., Nebro, A.J.: jMetal: a java framework for multi-objective optimization. Adv. Eng. Softw. 42, 760–771 (2011)CrossRefGoogle Scholar
  10. 10.
    Elbaum, S., Malishevsky, A., Rothermel, G.: Incorporating varying test costs and fault severities into test case prioritization. In: Proceedings of International Conference on Software Engineering (ICSE), pp. 329–338. IEEE (2001)Google Scholar
  11. 11.
    Elbaum, S., Malishevsky, A.G., Rothermel, G.: Prioritizing test cases for regression testing. In: Proceedings of International Symposium on Software Testing and Analysis (ISSTA), pp. 102–112. ACM (2000)Google Scholar
  12. 12.
    Elbaum, S., Malishevsky, A.G., Rothermel, G.: Prioritizing test cases for regression testing. Softw. Eng. Notes 25, 102–112 (2000)CrossRefGoogle Scholar
  13. 13.
    Elbaum, S., Malishevsky, A.G., Rothermel, G.: Test case prioritization: a family of empirical studies. IEEE Trans. Softw. Eng. 28(2), 159–182 (2002)CrossRefGoogle Scholar
  14. 14.
    Elbaum, S., Rothermel, G., Kanduri, S., Malishevsky, A.: Selecting a cost-effective test case prioritization technique. Softw. Qual. J. 12(3), 185–210 (2004)CrossRefGoogle Scholar
  15. 15.
    Hao, D., Zhang, L., Zhang, L., Rothermel, G., Mei, H.: A unified test case prioritization approach. ACM Trans. Softw. Eng. Methodol. 24(2), 10:1–10:31 (2014)CrossRefGoogle Scholar
  16. 16.
    Harrold, M.J., Gupta, R., Soffa, M.L.: A methodology for controlling the size of a test suite. ACM Trans. Softw. Eng. Methodol. 2, 270–285 (1993)CrossRefGoogle Scholar
  17. 17.
    Huang, Y.C., Huang, C.Y., Chang, J.R., Chen, T.Y.: Design and analysis of cost-cognizant test case prioritization using genetic algorithm with test history. In: Proceedings of Annual Computer Software and Applications Conference (COMPSAC), pp. 413–418. IEEE (2010)Google Scholar
  18. 18.
    Hughes, E.: Evolutionary many-objective optimisation: many once or one many? IEEE Congr. Evol. Comput. 1, 222–227 (2005)Google Scholar
  19. 19.
    Hyunsook Do, S.G.E., Rothermel, G.: Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact. Empirical Softw. Eng.: Int. J. 10, 405–435 (2005)CrossRefGoogle Scholar
  20. 20.
    Islam, M., Marchetto, A., Susi, A., Scanniello, G.: A multi-objective technique to prioritize test cases based on latent semantic indexing. In: Proceedings of European Conf. on Software Maintenance and Reengineering (CSMR), pp. 21–30. IEEE (2012)Google Scholar
  21. 21.
    Li, Z., Bian, Y., Zhao, R., Cheng, J.: A fine-grained parallel multi-objective test case prioritization on GPU. In: Ruhe, G., Zhang, Y. (eds.) SSBSE 2013. LNCS, vol. 8084, pp. 111–125. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  22. 22.
    Li, Z., Harman, M., Hierons, R.M.: Search algorithms for regression test case prioritization. IEEE Trans. Softw. Eng. 33(4), 225–237 (2007)CrossRefGoogle Scholar
  23. 23.
    Malishevsky, A.G., Ruthruff, J.R., Rothermel, G., Elbaum, S.: Cost-cognizant test case prioritization. Technical report, Department of Computer Science and Engineering (2006)Google Scholar
  24. 24.
    Panichella, A., Oliveto, R., Di Penta, M., De Lucia, A.: Improving multi-objective test case selection by injecting diversity in genetic algorithms. IEEE Trans. Softw. Eng. 41(4), 358–383 (2015)CrossRefGoogle Scholar
  25. 25.
    Rothermel, G., Untch, R., Chu, C., Harrold, M.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27(10), 929–948 (2001)CrossRefGoogle Scholar
  26. 26.
    Rothermel, G., Harrold, M.J., von Ronne, J., Hong, C.: Empirical studies of test-suite reduction. Softw. Test. Verif. Reliab. 12, 219–249 (2002)CrossRefGoogle Scholar
  27. 27.
    Srikanth, H., Williams, L., Osborne, J.: System test case prioritization of new and regression test cases. In: International Symposium on Empirical Software Engineering (2005)Google Scholar
  28. 28.
    Vargha, A., Delaney, H.D.: A critique and improvement of the cl common language effect size statistics of mcgraw and wong. J. Educ. Behav. Stat. 25(2), 101–132 (2000)Google Scholar
  29. 29.
    Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verif. Reliab. 22(2), 67–120 (2012)CrossRefGoogle Scholar
  30. 30.
    Yoo, S., Harman, M.: Pareto efficient multi-objective test case selection. In: Proceedings of International Symposium on Software Testing and Analysis (ISSTA), pp. 140–150. ACM (2007)Google Scholar
  31. 31.
    Yoo, S., Harman, M.: Using hybrid algorithm for Pareto efficient multi-objective test suite minimisation. J. Syst. Softw. 83(4), 689–701 (2010)CrossRefGoogle Scholar
  32. 32.
    Zhang, L., Hao, D., Zhang, L., Rothermel, G., Mei, H.: Bridging the gap between the total and additional test-case prioritization strategies. In: Proceedings of International Conference on Software Engineering (ICSE), pp. 192–201. IEEE (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Dario Di Nucci
    • 1
    Email author
  • Annibale Panichella
    • 2
  • Andy Zaidman
    • 2
  • Andrea De Lucia
    • 1
  1. 1.University of SalernoSalernoItaly
  2. 2.Delft University of TechnologyDelftThe Netherlands

Personalised recommendations