Performance Comparison of Multi-objective Algorithms for Test Case Prioritization During Web Application Testing

  • Munish KhannaEmail author
  • Achint Chaudhary
  • Abhishek Toofani
  • Anil Pawar
Research Article - Computer Engineering and Computer Science


Test case prioritization (TCP) is a widely accepted and extensively used strategy during regression testing. TCP is the permutation of test cases to enhance efficiency in achieving performance goals. These goals can belong to the category of single objective problem or multi-objective problem. This empirical study focuses on three objectives wherein two objectives are to be maximized and the remaining one minimized. During this study, three websites and various versions were created on which non-dominated sorting genetic algorithm-II and variant of non-dominated sorting artificial bee colony algorithm were applied to prioritize sequence of test cases. The problem size varies from small-size fault matrix \((34\times 27)\) to mid-size fault matrix \((157\times 128)\). Performance of the two algorithms was measured on various parameters and also verified on the basis of statistical testing. An alternate approach for solving this multi-objective problem, based on dynamic programming, is also proposed in this study, and it is concluded that performance of this algorithm is at par with other suggested ones.


Multi-objective optimization NSGA-II NSABC Dynamic programming Test case prioritization Search-based software engineering 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Mathur, A.: Foundations of Software Testing, Seventh Impression. Pearson Education, London (2012)Google Scholar
  2. 2.
    Chauhan, N.: Software Testing Principles and Practices, 1st edn. Oxford University Press, Oxford (2010)Google Scholar
  3. 3.
    Singh, Y.: Software Testing, 1st edn. Cambridge University Press, Cambridge (2012)Google Scholar
  4. 4.
    Rothermal, G.; Untch, R.; Harrold, M.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27(10), 929–948 (2001)Google Scholar
  5. 5.
    Malishevsky, A.G.; Ruthruff, J.R.; Rothermel, G.; Elbaum, S.: Cost-cognizant test case prioritization. Technical Report TR-UNL-CSE-2006-0004, Department of Computer Science and Engineering, University of Nebraska-Lincoln, Lincoln (2006)Google Scholar
  6. 6.
    Harman, M.; Li, Z.; Hierons, R.: Search algorithms for regression test case prioritization. IEEE Trans. Softw. Eng. 33(4), 225–237 (2007)Google Scholar
  7. 7.
    Elbaum, S.; Malishevsky, A.; Rothermel, G.: Incorporating varying test costs and fault severities into test case prioritization. In: Proceedings of the 23rd International Conference on Software Engineering, pp. 329–338 (2001)Google Scholar
  8. 8.
    Jiang, B.; Zhang, Z.; Chan, W.K.; Tse, T.H.: Adaptive random test case prioritization. In: ASE2009, 24th IEEE/ACM International Conference on Automated Software Engineering, pp. 233–244 (2009)Google Scholar
  9. 9.
    Noor, T.B.; Hemmati, H.: A similarity-based approach for test case prioritization using historical failure data. In: 26th International Symposium on Software Reliability Engineering ISSRE, pp. 58–68 (2015)Google Scholar
  10. 10.
    Arafeen, M.J.; Do, H.: Test case prioritization using requirements-based clustering. In: 6th International Conference on Software Testing, Verification and Validation, pp. 312–321 (2013)Google Scholar
  11. 11.
    Laali, M.; Liu, H.; Hamilton, M.; Spichkova, M.; Schmidt, H.: Test case prioritization using online fault detection information. In: Ada-Europe International Conference on Reliable Software Technologies, pp. 78–93 (2016)Google Scholar
  12. 12.
    Wang, S.; Nam, J.; Tan, L.: QTEP: quality-aware test case prioritization. In: Foundations of Software Engineering, pp. 523–534. ACM, New York (2017)Google Scholar
  13. 13.
    Mei, H.; Hao, D.; Zhang, L.; Zhou, J.; Rothermel, G.: A static approach to prioritizing Junit test cases. IEEE Trans. Softw. Eng. 38(6), 1258–1275 (2012)Google Scholar
  14. 14.
    Thomas, S.W.; Hemmati, H.; Hassan, A.E.; Blostein, D.: Static test case prioritization using topic models. Empir. Softw. Eng. 19(1), 182–212 (2012)Google Scholar
  15. 15.
    Mohanty, R.; Suman, S.; Das, S.K.: Modelling the pull-out capacity of ground anchors using multi-objective feature selection. Arab. J. Sci. Eng. 42(3), 1231–1241 (2017)Google Scholar
  16. 16.
    Shapiai, M.I.; Ibrahim, Z.; Adam, A.: Pareto optimality concept for incorporating prior knowledge for system identification problem with insufficient samples. Arab. J. Sci. Eng. 42(7), 2697–2710 (2017)Google Scholar
  17. 17.
    Zhang, Y.; Harman, M.; Mansouri, S.A.: The multi-objective next release problem. In: GECCO’07, pp. 1129–1137. ACM, London (2007)Google Scholar
  18. 18.
    Ruiz, M.; Roderiguez, D.; Riquelme, J.; Harrison, R.: Multi-objective Simulation Optimization in Software Project Management. Oxford Brookes University, Oxford (2011)Google Scholar
  19. 19.
    Wang, Z.; Tang, K.; Yao, X.: Multi-objective approaches to optimal testing resource allocation in modular software systems. IEEE Trans. Reliab. 59(3), 563–575 (2000)Google Scholar
  20. 20.
    Choudhary, K.; Purohit, G.: A Multi-objective optimization algorithm for uniformly distributed generation of test cases. In: IEEE International Conference on Computing for Sustainable Global Development, pp. 455–457 (2014)Google Scholar
  21. 21.
    Mondal, D.; Hemmati, H.; Durocher, S.: Exploring test suite diversification and code coverage in multi-objective test case selection. In: IEEE Conference on Software Testing, Verification and Validation, pp. 1–10 (2015)Google Scholar
  22. 22.
    Yoo, S.; Harman, M.: Pareto efficient multi-objective test case selection. In: ISSTA 2007, pp. 140–150. ACM, London (2007)Google Scholar
  23. 23.
    Marchetto, A.; Islam, M.; Scanniello, G.; Susi, A.: A multi-objective technique for test suite reduction. In: The 8th International Conference on Software Engineering Advances. IARIA (2013)Google Scholar
  24. 24.
    Zheng, W.; Hierons, R.; Li, M.; Liu, X.; Vinciotti, V.: Multi-objective optimization for regression testing. Inf. Sci. 334, 1–16 (2015)Google Scholar
  25. 25.
    Canfora, G.; Lucia, A.D.; Penta, M.D.; Oliveto, R.; Panichella, A.; Panichella, S.: Defect prediction as a multi-objective optimization problem. Softw. Test. Verif. Reliab. 25(4), 426–459 (2015)Google Scholar
  26. 26.
    Marchetto, A.; Islam, M.; Scanniello, G.; Asghar, W.; Susi, A.: A multi-objective technique to prioritize test cases. IEEE Trans. Softw. Eng. 42(10), 918–940 (2016)Google Scholar
  27. 27.
    Karaboga, D.; Goremli, B.; Ozturk, C.; Karaboga, N.: A comprehensive survey: artificial bee colony (ABC) and applications. Artif. Intell. Rev. 42(1), 21–57 (2014)Google Scholar
  28. 28.
    Karaboga, D.; Beyza, G.: A combinatorial artificial bee colony algorithm for travelling salesman problem. In: INISTA IEEE International Symposium, pp. 50–53 (2011)Google Scholar
  29. 29.
    Lam, S.S.B.; Raju, M.LH.P.; Kiran, U.M.; Swaraj, C.; Srivastava, P.R.: Automated generations of independent paths and test suite optimization using artificial bee colony. In: ICCTSD2011, Procedia Engineering 30, pp. 191–200 (2012)Google Scholar
  30. 30.
    Chong, C.S.; Low, M.Y.H.; Sivakumar, A.I, Lay.: A bee colony optimization for job shop scheduling. In: IEEE Proceedings of the 38th Conference on Winter Simulation, pp. 1954–1961 (2006)Google Scholar
  31. 31.
    Kaur, A.; Goyal, S.: A bee colony optimization algorithm for code coverage test suite prioritization. IJEST 3(4), 2786–2795 (2011)Google Scholar
  32. 32.
    Srikanth; Kulkarni, N.J.; Naveen, K.V.; Singh, P.; Srivastava, P.R.: Test case optimization using artificial bee colony algorithm. In: International Conference on Advances in Computing and Communication, pp. 570–579 (2011)Google Scholar
  33. 33.
    Joseph, A.K.; RadhaMani, G.: A hybrid model of particle swarm optimization and artificial bee colony algorithm for test case optimization. IJCSE 3(5), 459–471 (2011)Google Scholar
  34. 34.
    Mala, D.J.; Mohan, V.; kamalapriya, M.: Automated software test optimization framework and artificial bee colony optimization based approach. IET Softw. 4(5), 334–348 (2010)Google Scholar
  35. 35.
    Dahiya, S.K.; Chhabra, J.K.; Kumar, S.: Application of artificial bee colony algorithm to software testing. In: 21st IEEE Australian Software Engineering Conference, pp. 149–154 (2010)Google Scholar
  36. 36.
    Konsaard, P.; Ramingwong, L.: Using artificial bee colony for code coverage based test suite prioritization. In: 2nd International Conference on Information Science and Security, pp. 1–4 (2015)Google Scholar
  37. 37.
    Aghdam, Z.K.; Arasteh, B.: An efficient method to generate test data for software structural testing using artificial bee colony optimization algorithm. Int. J. Softw. Eng. Knowl. Eng. 27(6), 951–966 (2017)Google Scholar
  38. 38.
    Li, X.; Li, Z.; Lin, L.: An artificial bee colony algorithm for multi-objective optimization. In: 2nd International Conference on Intelligent Systems Design and Engineering Application, pp. 153–156 (2012)Google Scholar
  39. 39.
    Amarjeet, P.; Chhabra, J.K.: Many-objective artificial bee colony algorithm for large-scale software module clustering problem. Soft Comput. 22(19), 6341–6361 (2017). Google Scholar
  40. 40.
    Mann, M.; Tomar, P.; Sangwan, O.P.: Bio-inspired meta heuristics: evolving and prioritizing software test data. Appl. Intell. 48(3), 687–702 (2017). Google Scholar
  41. 41.
    Deb, K.; Pratap, A.; Agarwal, S.; Meyarivan, T.: A fast and elitist multi-objective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)Google Scholar
  42. 42.
    Deb, K.: Multi-objective Optimization Using Evolutionary Algorithms, 1st edn. Wiley, London (2010)zbMATHGoogle Scholar
  43. 43.
    Coreman, T.H.: Introduction to Algorithms, 2nd edn. PHI Printing Press, New York (2001)Google Scholar
  44. 44.
    Elbaum, S.; Rothermal, G.; Karre, S.; FisherII, M.: Leveraging user-session data to support web application testing. IEEE Trans. Softw. Eng. 31(3), 187–202 (2005)Google Scholar
  45. 45.
    Elbaum, S.; Malishevsky, A.G.; Rothermal, G.: Test case prioritization: a family of empirical studies. IEEE Trans. Softw. Eng. 28(2), 159–182 (2002)Google Scholar
  46. 46.
    Hutchins, M.; Foster, H.; Goradia, T.; Ostrand, T.: Experiments on the effectiveness of dataflow and control flow based test adequacy criteria. In: International Conference Software Engineering, pp. 191–200 (1994)Google Scholar
  47. 47.
    Wong, W.; Horgan, J.; London, S.; Mathur, A.: Effect of test set minimization on fault detection effectiveness. In: Proceedings 17th International Conference on Software Engineering, pp. 41–50 (1995)Google Scholar

Copyright information

© King Fahd University of Petroleum & Minerals 2019

Authors and Affiliations

  1. 1.Hindustan College of Science and TechnologyMathuraIndia
  2. 2.Indian Institute of SciencesBengaluruIndia

Personalised recommendations