Does Live Regression Testing Help?
Regression testing is an expensive, yet crucial part of the software development process. As regression test suites grow in size, the time required for their execution increases proportionally, and their execution is often either delegated to a specialized testing environment out of developers reach, or they are omitted completely. This could have a variety of negative effects on the developers’ productivity, including interruptions and slowdown of developers’ workflow. We propose a method of live regression unit testing to address these issues via incorporating Regression Test Selection and Test Case Prioritization techniques and an automatized change detection mechanism to run the regression testing in the background automatically. By combining the test results with source code changes and code coverage information, we are able to precisely identify source code changes responsible for test failures. By the paired two-sample t-test we proved, that our method is able to increase the speed of fault detection and to fix changes responsible for incorrect behaviour almost 2 times (p-value = 0.001, \(\alpha \) = 0.05).
KeywordsRegression testing Regression test selection Test case prioritization
This work was partially supported by the Slovak Research and Development Agency under the contract No. APVV-15-0508, and by the Scientific Grant Agency of the Slovak Republic, grant No. VG 1/0759/19.
- 2.Beszédes, A., Gergely, T., Schrettner, L., Jász, J., Langó, L., Gyimóthy, T.: Code coverage-based regression test selection and prioritization in Webkit. In: 2012 28th IEEE International Conference on SW Maintenance (ICSM), pp. 46–55 (2012)Google Scholar
- 3.Biswas, S., Mall, R., Satpathy, M., Sukumaran, S.: Regression test selection techniques: a survey. Informatica 35, 289–321 (2011)Google Scholar
- 5.Elbaum, S., Rothermel, G., Penix, J.: Techniques for improving regression testing in continuous integration development environments. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of SW Engineering, FSE 2014, pp. 235–245. ACM, New York (2014)Google Scholar
- 8.NCSS, LLC: Paired t-test (2019). https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/NCSS/Paired_T-Test.pdf. Accessed 27 Aug 2019