Enhanced regression testing technique for agile software development and continuous integration strategies
To survive in competitive marketplaces, most organizations have adopted agile methodologies to facilitate continuous integration and faster application delivery and rely on regression testing during application development to validate the quality and reliability of the software after changes have been made. Consequently, for large projects with cost and time constraints, it is extremely difficult to determine which test cases to run at the end of each release. In this paper, a test case prioritization and selection approach is proposed to improve the quality of releases. From existing literature, we analyzed prevailing problems and proposed solution relevant to regression testing in agile practices. The proposed approach is based on two phases. First, test cases are prioritized by clustering those test cases that frequently change. In case of a tie, test cases are prioritized based on their respective failure frequencies and coverage criteria. Second, test cases with a higher frequency of failure or coverage criteria are selected. The proposed technique was validated by an empirical study on three industrial subjects. The results show that the method successfully selects an optimal test suite and increases the fault detection rate (i.e., more than 90% in the case of proposed technique and less than 50% in other techniques), which reduces the number of irrelevant test cases and avoids detecting duplicate faults. The results of evaluation metrics illustrate that the proposed technique significantly outperform (i.e., between 91 and 97%) as compared to other existing regression testing techniques (i.e., between 52 and 68%). Therefore, our model enhances the test case prioritization and selection with the ability for earlier and high fault detection. Thus, pruning out irrelevant test cases and redundant faults and enhancing the regression testing process for agile applications.
KeywordsRegression testing Agile methodology Agile environment Test case prioritization Test suite selection Frequency of change
The work reported in this paper was supported by the National Natural Science Foundation of China (Grant No. 61672080).
Compliance with ethical standards
Conflict of interest
The authors declare that there is no conflict of interest.
- Agren, S. M., Knauss, E., Heldal, R., Pelliccione, P., Malmqvist, G., & Boden, J. (2018). The manager perspective on requirements impact on automotive systems development speed. 2018 IEEE 26th International Requirements Engineering Conference (RE), 17–28. https://doi.org/10.1109/RE.2018.00-55.
- Alkharabsheh, K., Crespo, Y., Manso, E., & Taboada, J. A. (2018). Software design smell detection: a systematic mapping study. Software Quality Journal. https://doi.org/10.1007/s11219-018-9424-8.
- Aman, H., Nakano, T., Ogasawara, H., & Kawahara, M. (2018). A topic model and test history-based test case recommendation method for regression testing. 2018 IEEE international conference on software testing, verification and validation workshops (ICSTW), 392–397. https://doi.org/10.1109/ICSTW.2018.00079.
- Anderson, J., Salem, S., & Do, H. (2014). Improving the effectiveness of test suite through mining historical data. Proceedings of the 11th Working Conference on Mining Software Repositories - MSR 2014, 142–151. https://doi.org/10.1145/2597073.2597084.
- Arafeen, M. J., & Do, H. (2013). Test case prioritization using requirements-based clustering. 2013 IEEE sixth international conference on software testing, verification and validation, 312–321. https://doi.org/10.1109/ICST.2013.12.
- Azizi, M., & Do, H. (2018). A collaborative filtering recommender system for test case prioritization in web applications. Proceedings of the 33rd Annual ACM Symposium on Applied Computing - SAC ‘18, 1560–1567. https://doi.org/10.1145/3167132.3167299.
- Chen, J., Zhu, L., Chen, T. Y., Towey, D., Kuo, F.-C., Huang, R., & Guo, Y. (2018). Test case prioritization for object-oriented software: an adaptive random sequence approach based on clustering. Journal of Systems and Software, 135, 107–125. https://doi.org/10.1016/j.jss.2017.09.031.CrossRefGoogle Scholar
- Do, H. (2016). Recent advances in regression testing techniques. In Advances in Computers (Vol. 103, pp. 53–77). https://doi.org/10.1016/bs.adcom.2016.04.004.
- Elbaum, S., Rothermel, G., & Penix, J. (2014). Techniques for improving regression testing in continuous integration development environments. Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering - FSE 2014, 235–245. https://doi.org/10.1145/2635868.2635910.
- Fischer, S., Lopez-Herrejon, R. E., & Egyed, A. (2018). Towards a fault-detection benchmark for evaluating software product line testing approaches. Proceedings of the 33rd Annual ACM Symposium on Applied Computing - SAC ‘18, 2034–2041. https://doi.org/10.1145/3167132.3167350.
- Gupta, A., Mishra, N., Tripathi, A., Vardhan, M., & Kushwaha, D. S. (2015). An improved history-based test prioritization technique using code coverage. In Advanced Computer and Communication Engineering Technology (Vol. 315, pp. 437–448). https://doi.org/10.1007/978-3-319-07674-4_43.
- Kandil, P., Moussa, S., & Badr, N. (2015). A methodology for regression testing reduction and prioritization of agile releases. 2015 5th international conference on Information & Communication Technology and accessibility (ICTA), 1–6. https://doi.org/10.1109/ICTA.2015.7426903.
- Knauss, E., Staron, M., Meding, W., Soder, O., Nilsson, A., & Castell, M. (2015). Supporting continuous integration by code-churn based test selection. 2015 IEEE/ACM 2nd International Workshop on Rapid Continuous Software Engineering, 19–25. https://doi.org/10.1109/RCoSE.2015.11.
- Lachmann, R., Lity, S., Lischke, S., Beddig, S., Schulze, S., & Schaefer, I. (2015). Delta-oriented test case prioritization for integration testing of software product lines. Proceedings of the 19th international conference on software product line - SPLC ‘15, 81–90. https://doi.org/10.1145/2791060.2791073.
- Lin, C.-T., Chen, C.-D., Tsai, C.-S., & Kapfhammer, G. M. (2013). History-based test case prioritization with software version awareness 2013 18th international conference on engineering of complex computer systems, 171–172. https://doi.org/10.1109/ICECCS.2013.33.
- Lu, Y., Lou, Y., Cheng, S., Zhang, L., Hao, D., Zhou, Y., & Zhang, L. (2016). How does regression test prioritization perform in real-world software evolution? Proceedings of the 38th International Conference on Software Engineering - ICSE ‘16, 535–546. https://doi.org/10.1145/2884781.2884874.
- Ma, C., & Provost, J. (2017). A model-based testing framework with reduced set of test cases for programmable controllers. 2017 13th IEEE Conference on Automation Science and Engineering (CASE), 944–949. https://doi.org/10.1109/COASE.2017.8256225.
- Magalhães, C., Andrade, J., Perrusi, L., & Mota, A. (2017). Evaluating an automatic text-based test case selection using a non-instrumented code coverage analysis. Proceedings of the 2nd Brazilian Symposium on Systematic and Automated Software Testing - SAST, 1–9. https://doi.org/10.1145/3128473.3128478.
- Noor, T. B., & Hemmati, H. (2015). A similarity-based approach for test case prioritization using historical failure data. 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), 58–68. https://doi.org/10.1109/ISSRE.2015.7381799.
- Ouni, A., Kessentini, M., ÓCinnéide, M., Sahraoui, H., Deb, K., & Inoue, K. (2017). MORE: a multi-objective refactoring recommendation approach to introducing design patterns and fixing code smells: MORE: a multi-objective refactoring recommendation approach. Journal of Software: Evolution and Process, 29(5), e1843. https://doi.org/10.1002/smr.1843.Google Scholar
- Özdağoğlu, G., & Kavuncubaşı, E. (2019). Monitoring the software bug-fixing process through the process mining approach. Journal of Software: Evolution and Process, e2162. https://doi.org/10.1002/smr.2162.
- Shin, S. Y., Nejati, S., Sabetzadeh, M., Briand, L. C., & Zimmer, F. (2018). Test case prioritization for acceptance testing of cyber physical systems: a multi-objective search-based approach. Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis - ISSTA 2018, 49–60. https://doi.org/10.1145/3213846.3213852.
- Souto, S., & d’Amorim, M. (2017). Time-space efficient regression testing for configurable systems. ArXiv:1702.03457 [Cs]. Retrieved from http://arxiv.org/abs/1702.03457
- Spieker, H., Gotlieb, A., Marijan, D., & Mossige, M. (2017). Reinforcement learning for automatic test case prioritization and selection in continuous integration. Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis - ISSTA 2017, 12–22. https://doi.org/10.1145/3092703.3092709.
- Thangiah, M., & Basri, S. (2016). A preliminary analysis of various testing techniques in agile development - a systematic literature review. 2016 3rd international conference on computer and information sciences (ICCOINS), 600–605. https://doi.org/10.1109/ICCOINS.2016.7783283.
- Wang, X., & Zeng, H. (2014). Dynamic test case prioritization based on multi-objective. 15th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), 1–6. https://doi.org/10.1109/SNPD.2014.6888744.
- Wang, X., & Zeng, H. (2016). History-based dynamic test case prioritization for requirement properties in regression testing. Proceedings of the International Workshop on Continuous Software Evolution and Delivery - CSED ‘16, 41–47. https://doi.org/10.1145/2896941.2896949.
- Zhao, D., Lin, H., Ran, L., Han, M., Tian, J., Lu, L., & Xiang, J. (2019). CVSkSA: cross-architecture vulnerability search in firmware based on kNN-SVM and attributed control flow graph. Software Quality Journal. https://doi.org/10.1007/s11219-018-9435-5.