Advertisement

Enhanced regression testing technique for agile software development and continuous integration strategies

  • Sadia Ali
  • Yaser Hafeez
  • Shariq Hussain
  • Shunkun YangEmail author
Article

Abstract

To survive in competitive marketplaces, most organizations have adopted agile methodologies to facilitate continuous integration and faster application delivery and rely on regression testing during application development to validate the quality and reliability of the software after changes have been made. Consequently, for large projects with cost and time constraints, it is extremely difficult to determine which test cases to run at the end of each release. In this paper, a test case prioritization and selection approach is proposed to improve the quality of releases. From existing literature, we analyzed prevailing problems and proposed solution relevant to regression testing in agile practices. The proposed approach is based on two phases. First, test cases are prioritized by clustering those test cases that frequently change. In case of a tie, test cases are prioritized based on their respective failure frequencies and coverage criteria. Second, test cases with a higher frequency of failure or coverage criteria are selected. The proposed technique was validated by an empirical study on three industrial subjects. The results show that the method successfully selects an optimal test suite and increases the fault detection rate (i.e., more than 90% in the case of proposed technique and less than 50% in other techniques), which reduces the number of irrelevant test cases and avoids detecting duplicate faults. The results of evaluation metrics illustrate that the proposed technique significantly outperform (i.e., between 91 and 97%) as compared to other existing regression testing techniques (i.e., between 52 and 68%). Therefore, our model enhances the test case prioritization and selection with the ability for earlier and high fault detection. Thus, pruning out irrelevant test cases and redundant faults and enhancing the regression testing process for agile applications.

Keywords

Regression testing Agile methodology Agile environment Test case prioritization Test suite selection Frequency of change 

Notes

Funding information

The work reported in this paper was supported by the National Natural Science Foundation of China (Grant No. 61672080).

Compliance with ethical standards

Conflict of interest

The authors declare that there is no conflict of interest.

References

  1. Abu Hasan, M., Abdur Rahman, M., & Saeed Siddik, M. (2017). Test case prioritization based on dissimilarity clustering using historical data analysis. Information, Communication and Computing Technology, 750, 269–281.  https://doi.org/10.1007/978-981-10-6544-6_25.CrossRefGoogle Scholar
  2. Agren, S. M., Knauss, E., Heldal, R., Pelliccione, P., Malmqvist, G., & Boden, J. (2018). The manager perspective on requirements impact on automotive systems development speed. 2018 IEEE 26th International Requirements Engineering Conference (RE), 17–28.  https://doi.org/10.1109/RE.2018.00-55.
  3. Al-Hajjaji, M., Thüm, T., Lochau, M., Meinicke, J., & Saake, G. (2019). Effective product-line testing using similarity-based product prioritization. Software and Systems Modeling, 18(1), 499–521.  https://doi.org/10.1007/s10270-016-0569-2.CrossRefGoogle Scholar
  4. Alkharabsheh, K., Crespo, Y., Manso, E., & Taboada, J. A. (2018). Software design smell detection: a systematic mapping study. Software Quality Journal.  https://doi.org/10.1007/s11219-018-9424-8.
  5. Almasri, N., Tahat, L., & Korel, B. (2017). Toward automatically quantifying the impact of a change in systems. Software Quality Journal, 25(3), 601–640.  https://doi.org/10.1007/s11219-016-9316-8.CrossRefGoogle Scholar
  6. Aman, H., Nakano, T., Ogasawara, H., & Kawahara, M. (2018). A topic model and test history-based test case recommendation method for regression testing. 2018 IEEE international conference on software testing, verification and validation workshops (ICSTW), 392–397.  https://doi.org/10.1109/ICSTW.2018.00079.
  7. Anand, R. V., & Dinakaran, M. (2017). Handling stakeholder conflict by agile requirement prioritization using Apriori technique. Computers and Electrical Engineering, 61, 126–136.  https://doi.org/10.1016/j.compeleceng.2017.06.022.CrossRefGoogle Scholar
  8. Anderson, J., Salem, S., & Do, H. (2014). Improving the effectiveness of test suite through mining historical data. Proceedings of the 11th Working Conference on Mining Software Repositories - MSR 2014, 142–151.  https://doi.org/10.1145/2597073.2597084.
  9. Anita, & Chauhan, N. (2014). A regression test selection technique by optimizing user stories in an agile environment. IEEE International Advance Computing Conference (IACC), 2014, 1454–1458.  https://doi.org/10.1109/IAdCC.2014.6779540.Google Scholar
  10. Ansari, A., Khan, A., Khan, A., & Mukadam, K. (2016). Optimized regression test using test case prioritization. Procedia Computer Science, 79, 152–160.  https://doi.org/10.1016/j.procs.2016.03.020.CrossRefGoogle Scholar
  11. Arafeen, M. J., & Do, H. (2013). Test case prioritization using requirements-based clustering. 2013 IEEE sixth international conference on software testing, verification and validation, 312–321.  https://doi.org/10.1109/ICST.2013.12.
  12. Azizi, M., & Do, H. (2018). A collaborative filtering recommender system for test case prioritization in web applications. Proceedings of the 33rd Annual ACM Symposium on Applied Computing - SAC ‘18, 1560–1567.  https://doi.org/10.1145/3167132.3167299.
  13. Chen, J., Zhu, L., Chen, T. Y., Towey, D., Kuo, F.-C., Huang, R., & Guo, Y. (2018). Test case prioritization for object-oriented software: an adaptive random sequence approach based on clustering. Journal of Systems and Software, 135, 107–125.  https://doi.org/10.1016/j.jss.2017.09.031.CrossRefGoogle Scholar
  14. del Sagrado, J., & del Águila, I. M. (2018). Stability prediction of the software requirements specification. Software Quality Journal, 26(2), 585–605.  https://doi.org/10.1007/s11219-017-9362-x.CrossRefGoogle Scholar
  15. Do, H. (2016). Recent advances in regression testing techniques. In Advances in Computers (Vol. 103, pp. 53–77).  https://doi.org/10.1016/bs.adcom.2016.04.004.
  16. Elbaum, S., Rothermel, G., & Penix, J. (2014). Techniques for improving regression testing in continuous integration development environments. Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering - FSE 2014, 235–245.  https://doi.org/10.1145/2635868.2635910.
  17. Felderer, M., & Herrmann, A. (2019). Comprehensibility of system models during test design: a controlled experiment comparing UML activity diagrams and state machines. Software Quality Journal, 27(1), 125–147.  https://doi.org/10.1007/s11219-018-9407-9.CrossRefGoogle Scholar
  18. Fischer, S., Lopez-Herrejon, R. E., & Egyed, A. (2018). Towards a fault-detection benchmark for evaluating software product line testing approaches. Proceedings of the 33rd Annual ACM Symposium on Applied Computing - SAC ‘18, 2034–2041.  https://doi.org/10.1145/3167132.3167350.
  19. Flemström, D., Potena, P., Sundmark, D., Afzal, W., & Bohlin, M. (2018). Similarity-based prioritization of test case automation. Software Quality Journal, 26(4), 1421–1449.  https://doi.org/10.1007/s11219-017-9401-7.CrossRefGoogle Scholar
  20. Gultepe, E., & Makrehchi, M. (2018). Improving clustering performance using independent component analysis and unsupervised feature learning. Human-centric Computing and Information Sciences, 8(1), 25.  https://doi.org/10.1186/s13673-018-0148-3.CrossRefGoogle Scholar
  21. Gupta, A., Mishra, N., Tripathi, A., Vardhan, M., & Kushwaha, D. S. (2015). An improved history-based test prioritization technique using code coverage. In Advanced Computer and Communication Engineering Technology (Vol. 315, pp. 437–448).  https://doi.org/10.1007/978-3-319-07674-4_43.
  22. Haghighatkhah, A., Mäntylä, M., Oivo, M., & Kuvaja, P. (2018). Test prioritization in continuous integration environments. Journal of Systems and Software, 146, 80–98.  https://doi.org/10.1016/j.jss.2018.08.061.CrossRefGoogle Scholar
  23. Heck, P., & Zaidman, A. (2018). A systematic literature review on quality criteria for agile requirements specifications. Software Quality Journal, 26(1), 127–160.  https://doi.org/10.1007/s11219-016-9336-4.CrossRefGoogle Scholar
  24. Hettiarachchi, C., Do, H., & Choi, B. (2016). Risk-based test case prioritization using a fuzzy expert system. Information and Software Technology, 69, 1–15.  https://doi.org/10.1016/j.infsof.2015.08.008.CrossRefGoogle Scholar
  25. Horváth, F., Gergely, T., Beszédes, Á., Tengeri, D., Balogh, G., & Gyimóthy, T. (2019). Code coverage differences of Java bytecode and source code instrumentation tools. Software Quality Journal, 27(1), 79–123.  https://doi.org/10.1007/s11219-017-9389-z.CrossRefGoogle Scholar
  26. Huang, Y.-C., Peng, K.-L., & Huang, C.-Y. (2012). A history-based cost-cognizant test case prioritization technique in regression testing. Journal of Systems and Software, 85(3), 626–637.  https://doi.org/10.1016/j.jss.2011.09.063.MathSciNetCrossRefGoogle Scholar
  27. Kandil, P., Moussa, S., & Badr, N. (2014). Regression testing approach for large-scale systems. IEEE International Symposium on Software Reliability Engineering Workshops, 2014, 132–133.  https://doi.org/10.1109/ISSREW.2014.96.Google Scholar
  28. Kandil, P., Moussa, S., & Badr, N. (2015). A methodology for regression testing reduction and prioritization of agile releases. 2015 5th international conference on Information & Communication Technology and accessibility (ICTA), 1–6.  https://doi.org/10.1109/ICTA.2015.7426903.
  29. Kandil, P., Moussa, S., & Badr, N. (2016). Cluster-based test cases prioritization and selection technique for agile regression testing: cluster-based technique for agile regression testing. Journal of Software: Evolution and Process, 29(6), e1794.  https://doi.org/10.1002/smr.1794.Google Scholar
  30. Knauss, E., Staron, M., Meding, W., Soder, O., Nilsson, A., & Castell, M. (2015). Supporting continuous integration by code-churn based test selection. 2015 IEEE/ACM 2nd International Workshop on Rapid Continuous Software Engineering, 19–25.  https://doi.org/10.1109/RCoSE.2015.11.
  31. Lachmann, R., Lity, S., Lischke, S., Beddig, S., Schulze, S., & Schaefer, I. (2015). Delta-oriented test case prioritization for integration testing of software product lines. Proceedings of the 19th international conference on software product line - SPLC ‘15, 81–90.  https://doi.org/10.1145/2791060.2791073.
  32. Li, X., Wong, W. E., Gao, R., Hu, L., & Hosono, S. (2018). Genetic algorithm-based test generation for software product line with the integration of fault localization techniques. Empirical Software Engineering, 23(1), 1–51.  https://doi.org/10.1007/s10664-016-9494-9.CrossRefGoogle Scholar
  33. Lin, C.-T., Chen, C.-D., Tsai, C.-S., & Kapfhammer, G. M. (2013). History-based test case prioritization with software version awareness 2013 18th international conference on engineering of complex computer systems, 171–172.  https://doi.org/10.1109/ICECCS.2013.33.
  34. Lu, Y., Lou, Y., Cheng, S., Zhang, L., Hao, D., Zhou, Y., & Zhang, L. (2016). How does regression test prioritization perform in real-world software evolution? Proceedings of the 38th International Conference on Software Engineering - ICSE ‘16, 535–546.  https://doi.org/10.1145/2884781.2884874.
  35. Ma, C., & Provost, J. (2017). A model-based testing framework with reduced set of test cases for programmable controllers. 2017 13th IEEE Conference on Automation Science and Engineering (CASE), 944–949.  https://doi.org/10.1109/COASE.2017.8256225.
  36. Ma, T., Ali, S., Yue, T., & Elaasar, M. (2019). Testing self-healing cyber-physical systems under uncertainty: a fragility-oriented approach. Software Quality Journal, 27(2), 615–649.  https://doi.org/10.1007/s11219-018-9437-3.CrossRefGoogle Scholar
  37. Magalhães, C., Andrade, J., Perrusi, L., & Mota, A. (2017). Evaluating an automatic text-based test case selection using a non-instrumented code coverage analysis. Proceedings of the 2nd Brazilian Symposium on Systematic and Automated Software Testing - SAST, 1–9.  https://doi.org/10.1145/3128473.3128478.
  38. Mahali, P., & Mohapatra, D. P. (2018). Model based test case prioritization using UML behavioural diagrams and association rule mining. International Journal of Systems Assurance Engineering and Management, 9(5), 1063–1079.  https://doi.org/10.1007/s13198-018-0736-7.CrossRefGoogle Scholar
  39. Miranda, B., & Bertolino, A. (2017). Scope-aided test prioritization, selection and minimization for software reuse. Journal of Systems and Software, 131, 528–549.  https://doi.org/10.1016/j.jss.2016.06.058.CrossRefGoogle Scholar
  40. Miranda, B., & Bertolino, A. (2018). An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing. Software Quality Journal, 26(4), 1571–1594.  https://doi.org/10.1007/s11219-017-9388-0.CrossRefGoogle Scholar
  41. Ni, C., Liu, W.-S., Chen, X., Gu, Q., Chen, D.-X., & Huang, Q.-G. (2017). A cluster based feature selection method for cross-project software defect prediction. Journal of Computer Science and Technology, 32(6), 1090–1107.  https://doi.org/10.1007/s11390-017-1785-0.CrossRefGoogle Scholar
  42. Noor, T. B., & Hemmati, H. (2015). A similarity-based approach for test case prioritization using historical failure data. 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), 58–68.  https://doi.org/10.1109/ISSRE.2015.7381799.
  43. Ouni, A., Kessentini, M., ÓCinnéide, M., Sahraoui, H., Deb, K., & Inoue, K. (2017). MORE: a multi-objective refactoring recommendation approach to introducing design patterns and fixing code smells: MORE: a multi-objective refactoring recommendation approach. Journal of Software: Evolution and Process, 29(5), e1843.  https://doi.org/10.1002/smr.1843.Google Scholar
  44. Ouriques, J. F. S., Cartaxo, E. G., & Machado, P. D. L. (2018). Test case prioritization techniques for model-based testing: a replicated study. Software Quality Journal, 26(4), 1451–1482.  https://doi.org/10.1007/s11219-017-9398-y.CrossRefGoogle Scholar
  45. Özdağoğlu, G., & Kavuncubaşı, E. (2019). Monitoring the software bug-fixing process through the process mining approach. Journal of Software: Evolution and Process, e2162.  https://doi.org/10.1002/smr.2162.
  46. Panichella, A., Oliveto, R., Penta, M. D., & De Lucia, A. (2015). Improving multi-objective test case selection by injecting diversity in genetic algorithms. IEEE Transactions on Software Engineering, 41(4), 358–383.  https://doi.org/10.1109/TSE.2014.2364175.CrossRefGoogle Scholar
  47. Rosero, R. H., Gómez, O. S., & Rodríguez, G. (2016). 15 years of software regression testing techniques — a survey. International Journal of Software Engineering and Knowledge Engineering, 26(05), 675–689.  https://doi.org/10.1142/S0218194016300013.CrossRefGoogle Scholar
  48. Rosero, R. H., Gomez, O. S., & Rodriguez, G. (2017). Regression testing of database applications under an incremental software development setting. IEEE Access, 5, 18419–18428.  https://doi.org/10.1109/ACCESS.2017.2749502.CrossRefGoogle Scholar
  49. Shin, S. Y., Nejati, S., Sabetzadeh, M., Briand, L. C., & Zimmer, F. (2018). Test case prioritization for acceptance testing of cyber physical systems: a multi-objective search-based approach. Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis - ISSTA 2018, 49–60.  https://doi.org/10.1145/3213846.3213852.
  50. Silva, D., Rabelo, R., Campanha, M., Neto, P. S., Oliveira, P. A., & Britto, R. (2016). A hybrid approach for test case prioritization and selection. IEEE Congress on Evolutionary Computation (CEC), 2016, 4508–4515.  https://doi.org/10.1109/CEC.2016.7744363.Google Scholar
  51. Souto, S., & d’Amorim, M. (2017). Time-space efficient regression testing for configurable systems. ArXiv:1702.03457 [Cs]. Retrieved from http://arxiv.org/abs/1702.03457
  52. Spieker, H., Gotlieb, A., Marijan, D., & Mossige, M. (2017). Reinforcement learning for automatic test case prioritization and selection in continuous integration. Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis - ISSTA 2017, 12–22.  https://doi.org/10.1145/3092703.3092709.
  53. Thangiah, M., & Basri, S. (2016). A preliminary analysis of various testing techniques in agile development - a systematic literature review. 2016 3rd international conference on computer and information sciences (ICCOINS), 600–605.  https://doi.org/10.1109/ICCOINS.2016.7783283.
  54. Wang, X., & Zeng, H. (2014). Dynamic test case prioritization based on multi-objective. 15th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), 1–6.  https://doi.org/10.1109/SNPD.2014.6888744.
  55. Wang, X., & Zeng, H. (2016). History-based dynamic test case prioritization for requirement properties in regression testing. Proceedings of the International Workshop on Continuous Software Evolution and Delivery - CSED ‘16, 41–47.  https://doi.org/10.1145/2896941.2896949.
  56. Wang, X., Zeng, H., Gao, H., Miao, H., & Lin, W. (2019). Location-based test case prioritization for software embedded in mobile devices using the law of gravitation. Mobile Information Systems, 2019, 1–14.  https://doi.org/10.1155/2019/9083956.Google Scholar
  57. Zhao, D., Lin, H., Ran, L., Han, M., Tian, J., Lu, L., & Xiang, J. (2019). CVSkSA: cross-architecture vulnerability search in firmware based on kNN-SVM and attributed control flow graph. Software Quality Journal.  https://doi.org/10.1007/s11219-018-9435-5.

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • Sadia Ali
    • 1
  • Yaser Hafeez
    • 1
  • Shariq Hussain
    • 2
  • Shunkun Yang
    • 3
    Email author
  1. 1.University Institute of Information TechnologyPir Mehr Ali Shah Arid Agriculture UniversityRawalpindiPakistan
  2. 2.Department of Software EngineeringFoundation University IslamabadRawalpindiPakistan
  3. 3.School of Reliability and Systems EngineeringBeihang UniversityBeijingChina

Personalised recommendations