Advertisement

An efficient regression testing approach for PHP Web applications using test selection and reusable constraints

  • Ravi EdaEmail author
  • Hyunsook Do
Regular Submission
  • 21 Downloads

Abstract

Web applications undergo frequent changes. These changes can be due to the addition of new features or the modification of existing features to support customer requests or to patch faults in the system. Given that Web applications have a large surface area subject to attack, changes often include security fixes either in response to malicious attacks or to forestall such attacks. Effective regression testing should ensure that any change does not disable existing features or compromise security. Executing the entire regression test suite takes time and consumes many resources. One approach is to focus regression test efforts only on code paths that were modified in the new version. Such code paths can be identified using tools such as PHP Analysis and Regression Testing Engine (PARTE). In this paper, we extend this approach to test selection where a subset of existing tests that cover the modified code paths can be detected. To further reduce the amount of regression testing needed, we used PARTE’s reusable constraint value information to identify tests that can be reused against the new version without having to modify the input test values. We performed an empirical study to determine whether test selection data combined with reusable constraint values would further improve the turnaround time for regression tests. Results from the experiment conducted on four Hypertext Preprocessor (PHP) web applications demonstrate that this approach is effective in reducing the cost of regression testing of frequently patched Web applications.

Keywords

Regression testing Test selection Test reuse PHP Web applications Empirical studies 

Notes

Funding information

This work was supported, in part, by NSF CAREER Award CCF-1564238 to University of North Texas.

References

  1. Alshahwan, N., & Harman, M. (2012). State aware test case regeneration for improving web application test suite coverage and fault detection. In Proceedings of the 2012 international symposium on software testing and analysis, ISSTA 2012 (pp. 45–55). Minneapolis: ACM.Google Scholar
  2. Andrews, J.H., Briand, L.C., Labiche, Y. (2005). Is mutation an appropriate tool for testing experiments?. In Proceedings of the 27th international conference on software engineering, ICSE ’05 (pp. 402–411). St. Louis: ACM.Google Scholar
  3. Apiwattanapong, T., Santelices, R., Chittimalli, P.K., Orso, A., Harrold, M.J. (2006). Matrix: maintenance-oriented testing requirements identifier and examiner. In Proceedings of the testing: academic & industrial conference on practice and research techniques, TAIC-PART ’06 (pp. 137–146).Google Scholar
  4. Arnold, R.S., & Bohner, S.A. (1993). Impact analysis - towards a framework for comparison. In Proceedings of the conference on software maintenance, ICSM ’93 (pp. 292–301). IEEE.Google Scholar
  5. Artzi, S., Dolby, J., Tip, F., Pistoia, M. (2010). Practical fault localization for dynamic web applications. In Proceedings of the 32Nd ACM/IEEE international conference on software engineering - vol. 1, ICSE ’10 (pp. 265–274). ACM.Google Scholar
  6. Bell, J., Legunsen, O., Hilton, M., Eloussi, L., Yung, T., Marinov, D. (2018). Deflaker: automatically detecting flaky tests. In Proceedings of the 40th international conference on software engineering, ICSE’18 (pp. 433–444). Gothenburg: ACM.Google Scholar
  7. Boggs, R., Bozman, J., Perry, R. (2018). Reducing downtime and business loss: addressing business risk with effective technology. Technical report, International Data Corporation (IDC), 2009 Accessed October 2018.Google Scholar
  8. Chen, Y., Probert, R.L., Ural, H. (2007). Model-based regression test suite generation using dependence analysis. In Proceedings of the 3rd international workshop on advances in model-based testing, A-MOST ’07 (pp. 54–62). ACM.Google Scholar
  9. Choco. (2018). http://www.emn.fr/z-info/choco-solver/. Accessed October 2018.
  10. Dallmeier, V., Burger, M., Orth, T., Zeller, A. (2012). Webmate: a tool for testing web 2.0 applications. In Proceedings of the workshop on javascript tools, JSTools’12 (pp. 11–15). Beijing: ACM.Google Scholar
  11. Deng, Y., Frankl, P., Chays, D. (2005). Testing database transactions with agenda. In Proceedings of the 27th international conference on software engineering, ICSE ’05 (pp. 78–87): ACM.Google Scholar
  12. Do, H., & Hossain, Md. (2014). An efficient regression testing approach for php web applications: a controlled experiment. Journal of Software Testing, Verification and Reliability, 24, 367–385.CrossRefGoogle Scholar
  13. Elbaum, S., Malishevsky, A., Rothermel, G. (2001). Incorporating varying test costs and fault severities into test case prioritization. In Proceedings of the 23rd international conference on software engineering, ICSE ’01 (pp. 329–338). IEEE.Google Scholar
  14. Emmi, M., Majumdar, R., Sen, K. (2007). Dynamic test input generation for database applications. In Proceedings of the 2007 international symposium on software testing and analysis, ISSTA ’07 (pp. 151–162). ACM.Google Scholar
  15. Engström, E., Skoglund, M., Runeson, P. (2008). Empirical evaluations of regression test selection techniques: a systematic review. In Proceedings of the second ACM-IEEE international symposium on empirical software engineering and measurement, ESEM ’08 (pp. 22–31). ACM.Google Scholar
  16. Faqforge. (2018). Web page. https://sourceforge.net/projects/faqforge/. Accessed October 2018.
  17. Halfond, W.G.J., Anand, S., Orso, A. (2009). Precise interface identification to improve testing and analysis of web applications. In Proceedings of the eighteenth international symposium on software testing and analysis, ISSTA ’09 (pp. 285–296). ACM.Google Scholar
  18. Herzig, K., Greiler, M., Czerwonka, J., Murphy, B. (2015). The art of testing less without sacrificing quality. In Proceedings of the 37th international conference on software engineering - vol. 1, ICSE’15 (pp. 483–493). Florence: IEEE.Google Scholar
  19. Hooimeijer, P., & Weimer, W. (2010). Solving string constraints lazily. In Proceedings of the IEEE/ACM international conference on automated software engineering, ASE ’10 (pp. 377–386). ACM.Google Scholar
  20. Hossain, Md., Do, H., Eda, R. (2014). Regression testing for web applications using reusable constraint values. In Proceedings of the fourth international workshop on regression testing (Regression 2014), Cleveland, USA (pp. 312–321).Google Scholar
  21. Humbug. (2018). Mutation testing for php. https://github.com/humbug/humbug/. Accessed October 2018.
  22. Jia, X., Ghezzi, C., Ying, S. (2015). Enhancing reuse of constraint solutions to improve symbolic execution. In Proceedings of the 2015 international symposium on software testing and analysis, ISSTA 2015 (pp. 177–187). ACM.Google Scholar
  23. Just, R., Jalali, D., Inozemtseva, L., Ernst, M.D., Holmes, R., Fraser, G. (2014). Are mutants a valid substitute for real faults in software testing?. In Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, FSE 2014 (pp. 654–665). Hong Kong: ACM.Google Scholar
  24. Kazmi, R., Jawawi, D.N.A., Mohamad, R., Ghani, I. (2017). Effective regression test case selection: a systematic literature review. ACM Computing Surveys, 50(2), 29:1–29:32.CrossRefGoogle Scholar
  25. Kiezun, A., Ganesh, V., Artzi, S., Guo, P.J., Hooimeijer, P., Ernst, M.D. (2013). Hampi: a solver for word equations over strings, regular expressions, and context-free grammars. ACM Transactions on Software Engineering and Methodology, 21(4), 25:1–25:28.Google Scholar
  26. Kim, M., Cobb, J., Harrold, M.J., Kurc, T., Orso, A., Saltz, J., Post, A., Malhotra, K., Navathe, S.B. (2012). Efficient regression testing of ontology-driven systems. In Proceedings of the 2012 international symposium on software testing and analysis, ISSTA 2012 (pp. 320–330). ACM.Google Scholar
  27. Kitchenham, B.A., Pfleeger, S.L., Pickard, L.M., Jones, P.W., Hoaglin, D.C., El Emam, K., Rosenberg, J. (2002). Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering, 28(8), 721–734.CrossRefGoogle Scholar
  28. Klarlund, N. (1998). Mona & fido: the logic-automaton connection in practice. In Selected papers from the11th international workshop on computer science logic, CSL’97 (pp. 311–326). Springer.Google Scholar
  29. Legunsen, O., Hariri, F., Shi, A., Lu, Y., Zhang, L., Marinov, D. (2016). An extensive study of static regression test selection in modern software evolution. In Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, FSE 2016 (pp. 583–594). Seattle: ACM.Google Scholar
  30. Leung, H.K.N., & White, L. (1989). Insights into regression testing. In Proceedings of the conference on software maintenance (pp. 60–69). IEEE.Google Scholar
  31. Log4php. (2018a). Web page. http://logging.apache.org/log4php/. Accessed October 2018.
  32. Log4php. (2018b). Issue 133. https://issues.apache.org/jira/browse/LOG4PHP-133/. Accessed October 2018.
  33. Mambo. (2018). Web page. http://mambo-foundation.org/. Accessed October 2018.
  34. Mantis. (2018). Web page. http://www.mantisbt.org/. Accessed October 2018.
  35. Mirarab, S., Akhlaghi, S., Tahvildari, L. (2012). Size-constrained regression test case selection using multicriteria optimization. IEEE Transactions on Software Engineering, 38(4), 936–956.CrossRefGoogle Scholar
  36. Öqvist, J., Hedin, G., Magnusson, B. (2016). Extraction-based regression test selection. In Proceedings of the 13th international conference on principles and practices of programming on the Java platform: virtual machines, languages, and tools, PPPJ’16 (pp. 5:1–5:10). Lugano: ACM.Google Scholar
  37. Oscommerce. (2018). Web page. https://www.oscommerce.com/. Accessed October 2018.
  38. Papadakis, M., Shin, D., Yoo, S., Bae, D.-H. (2018). Are mutation scores correlated with real fault detection?: a large scale empirical study on the relationship between mutants and real faults. In Proceedings of the 40th international conference on software engineering, ICSE’18 (pp. 537–548). Gothenburg: ACM.Google Scholar
  39. Phpscheduleit. (2018a). Web page. https://sourceforge.net/projects/phpscheduleit/. Accessed October 2018.
  40. Phpscheduleit. (2018b). Issue 18. https://sourceforge.net/p/phpscheduleit/bugs/18/. Accessed October 2018.
  41. Phpunit. (2018). Official site. http://phpunit.de/. Accessed October 2018.
  42. Ren, X., Shah, F., Tip, F., Ryder, B.G., Chesley, O. (2004). Chianti: a tool for change impact analysis of java programs. SIGPLAN Notices, 39(10), 432–448.CrossRefGoogle Scholar
  43. Ricca, F., & Tonella, P. (2001). Analysis and testing of web applications. In Proceedings of the 23rd international conference on software engineering, ICSE ’01 (pp. 25–34). IEEE.Google Scholar
  44. Rothermel, G., & Harrold, M.J. (1996). Analyzing regression test selection techniques. IEEE Transactions on Software Engineering, 22(8), 529–551.CrossRefGoogle Scholar
  45. Rothermel, G., & Harrold, M.J. (1994). A framework for evaluating regression test selection techniques. In Proceedings of the 16th international conference on software engineering, ICSE ’94 (pp. 201–210): IEEE.Google Scholar
  46. Rothermel, G., & Harrold, M.J. (1997). A safe, efficient regression test selection technique. ACM Transactions on Software Engineering and Methodology, 6(2), 173–210.CrossRefGoogle Scholar
  47. Rubinov, K., & Wuttke, J. (2012). Augmenting test suites automatically. In Proceedings of the 34th international conference on software engineering, ICSE ’12 (pp. 1433–1434). IEEE.Google Scholar
  48. Santelices, R., & Harrold, M.J. (2011). Applying aggressive propagation-based strategies for testing changes. In Proceedings of the 2011 fourth IEEE international conference on software testing, verification and validation, ICST ’11 (pp. 11–20). IEEE.Google Scholar
  49. Seleniumhq. (2018). Official site. http://www.seleniumhq.org/. Accessed October 2018.
  50. Shi, A., Yung, T., Gyori, A., Marinov, D. (2015). Comparing and combining test-suite reduction and regression test selection. In Proceedings of the 2015 10th joint meeting on foundations of software engineering, ESEC/FSE 2015 (pp. 237–247). ACM.Google Scholar
  51. Sun, X., Li, B., Leung, H., Li, B., Zhu, J. (2015). Static change impact analysis techniques. Journal of Systems and Software, 109(C), 137–149.CrossRefGoogle Scholar
  52. Taneja, K., Xie, T., Tillmann, N., de Halleux, J. (2011). Express: guided path exploration for efficient regression test generation. In Proceedings of the 2011 international symposium on software testing and analysis, ISSTA ’11 (pp. 1–11). ACM.Google Scholar
  53. Tonella, P. (2003). Using a concept lattice of decomposition slices for program understanding and impact analysis. IEEE Transactions on Software Engineering, 29 (6), 495–509.CrossRefGoogle Scholar
  54. Visser, W., Geldenhuys, J., Dwyer, M.B. (2012). Green: reducing, reusing and recycling constraints in program analysis. In Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering, FSE ’12 (pp. 58:1–58:11). ACM.Google Scholar
  55. Wassermann, G., & Su, Z. (2007). Sound and precise analysis of web applications for injection vulnerabilities. SIGPLAN Notices, 42(6), 32–41.CrossRefGoogle Scholar
  56. Wassermann, G., Yu, D., Chander, A., Dhurjati, D., Inamura, H., Su, Z. (2008). Dynamic test input generation for web applications. In Proceedings of the 2008 international symposium on software testing and analysis, ISSTA ’08 (pp. 249–260). ACM.Google Scholar
  57. Xu, L., Xu, B., Jiang, J. (2005). Testing web applications focusing on their specialties. SIGSOFT Software Engineering Notes, 30(1), 10.CrossRefGoogle Scholar
  58. Xu, Z., Kim, Y., Kim, M., Rothermel, G. (2011). A hybrid directed test suite augmentation technique. In Proceedings of the 2011 IEEE 22nd international symposium on software reliability engineering, ISSRE ’11 (pp. 150–159). IEEE.Google Scholar
  59. Xu, Z., Kim, Y., Kim, M., Rothermel, G., Cohen, M.B. (2010). Directed test suite augmentation: techniques and tradeoffs. In Proceedings of the eighteenth ACM SIGSOFT international symposium on foundations of software engineering, FSE ’10 (pp. 257–266). ACM.Google Scholar
  60. Yoo, S., & Harman, M. (2012). Regression testing minimization, selection and prioritization: a survey. Software Test Verification Reliability, 22(2), 67–120.CrossRefGoogle Scholar
  61. Ziftci, C., & Reardon, J. (2017). Who broke the build?: automatically identifying changes that induce test failures in continuous integration at google scale. In Proceedings of the 39th international conference on software engineering: software engineering in practice track, ICSE-SEIP ’17 (pp. 113–122). Buenos Aires: IEEE.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.North Dakota State UniversityFargoUSA
  2. 2.Microsoft CorporationFargoUSA
  3. 3.University of North TexasDentonUSA

Personalised recommendations