Advertisement

Do Null-Type Mutation Operators Help Prevent Null-Type Faults?

  • Ali ParsaiEmail author
  • Serge Demeyer
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11376)

Abstract

The null-type is a major source of faults in Java programs, and its overuse has a severe impact on software maintenance. Unfortunately traditional mutation testing operators do not cover null-type faults by default, hence cannot be used as a preventive measure. We address this problem by designing four new mutation operators which model null-type faults explicitly. We show how these mutation operators are capable of revealing the missing tests, and we demonstrate that these mutation operators are useful in practice. For the latter, we analyze the test suites of 15 open-source projects to describe the trade-offs related to the adoption of these operators to strengthen the test suite.

Keywords

Software maintenance Software testing Mutation testing Null-type Test quality 

References

  1. 1.
    Alexander, I.: Misuse cases: use cases with hostile intent. IEEE Softw. 20(1), 58–66 (2003).  https://doi.org/10.1109/ms.2003.1159030CrossRefGoogle Scholar
  2. 2.
    Ammann, P., Delamaro, M.E., Offutt, J.: Establishing theoretical minimal sets of mutants. In: 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation, pp. 21–30. IEEE, March 2014.  https://doi.org/10.1109/icst.2014.13
  3. 3.
    Bradbury, J.S., Cordy, J.R., Dingel, J.: Mutation operators for concurrent Java (J2SE 5.0). In: MUTATION 2006, Proceedings of the Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006), School of Computing, Queen’s University, Kingston, p. 11. IEEE, Washington, DC, November 2006.  https://doi.org/10.1109/mutation.2006.10
  4. 4.
    Coles, H., Laurent, T., Henard, C., Papadakis, M., Ventresque, A.: PIT: a practical mutation testing tool for Java (Demo). In: Proceedings of the 25th International Symposium on Software Testing and Analysis - ISSTA 2016, pp. 449–452. ACM Press, New York (2016).  https://doi.org/10.1145/2931037.2948707
  5. 5.
    DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. Computer 11(4), 34–41 (1978).  https://doi.org/10.1109/c-m.1978.218136CrossRefGoogle Scholar
  6. 6.
    Deng, L., Offutt, J., Ammann, P., Mirzaei, N.: Mutation operators for testing Android apps. Inf. Softw. Technol. 81, 154–168 (2017).  https://doi.org/10.1016/j.infsof.2016.04.012CrossRefGoogle Scholar
  7. 7.
    Ferrari, F.C., Maldonado, J., Rashid, A.: Mutation testing for aspect-oriented programs. In: 2008 International Conference on Software Testing, Verification, and Validation. ICST 2008, Department of Computer System, Sao Paulo University, Sao Carlos, pp. 52–61. IEEE, Washington, DC, April 2008.  https://doi.org/10.1109/icst.2008.37
  8. 8.
    Fowler, M.: Refactoring: Improving the Design of Existing Code. Addison-Wesley, Boston (1999)zbMATHGoogle Scholar
  9. 9.
    Fraser, G., Zeller, A.: Mutation-driven generation of unit tests and oracles. IEEE Trans. Softw. Eng. 38(2), 278–292 (2012).  https://doi.org/10.1109/tse.2011.93CrossRefGoogle Scholar
  10. 10.
    Gligoric, M., Groce, A., Zhang, C., Sharma, R., Alipour, M.A., Marinov, D.: Comparing non-adequate test suites using coverage criteria. In: Proceedings of the 2013 International Symposium on Software Testing and Analysis - ISSTA 2013, pp. 302–313. ACM Press, New York (2013).  https://doi.org/10.1145/2483760.2483769
  11. 11.
    Gosling, J., Joy, B., Steele, G., Bracha, G., Buckley, A.: The Java Language Specification (Java SE). Oracle, 8th edn. Addison-Wesley, Boston (2014)Google Scholar
  12. 12.
    Inozemtseva, L., Holmes, R.: Coverage is not strongly correlated with test suite effectiveness. In: Proceedings of the 36th International Conference on Software Engineering - ICSE 2014, pp. 435–445. ACM Press, New York (2014).  https://doi.org/10.1145/2568225.2568271
  13. 13.
    Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011).  https://doi.org/10.1109/tse.2010.62CrossRefGoogle Scholar
  14. 14.
    Just, R., Jalali, D., Inozemtseva, L., Ernst, M.D., Holmes, R., Fraser, G.: Are mutants a valid substitute for real faults in software testing? In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering - FSE 2014, pp. 654–665. ACM Press, New York (2014).  https://doi.org/10.1145/2635868.2635929
  15. 15.
    Kimura, S., Hotta, K., Higo, Y., Igaki, H., Kusumoto, S.: Does return null matter? In: 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE), pp. 244–253. IEEE, February 2014.  https://doi.org/10.1109/csmr-wcre.2014.6747176
  16. 16.
    King, K.N., Offutt, A.J.: A fortran language system for mutation-based software testing. Softw.: Pract. Exp. 21(7), 685–718 (1991).  https://doi.org/10.1002/spe.4380210704CrossRefGoogle Scholar
  17. 17.
    Kuhn, D.R.: Fault classes and error detection capability of specification-based testing. ACM Trans. Softw. Eng. Methodol. 8(4), 411–424 (1999).  https://doi.org/10.1145/322993.322996CrossRefGoogle Scholar
  18. 18.
    Kurtz, B., Ammann, P., Offutt, J.: Static analysis of mutant subsumption. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–10. IEEE, April 2015.  https://doi.org/10.1109/icstw.2015.7107454
  19. 19.
    Ma, Y.S., Kwon, Y.R., Offutt, J.: Inter-class mutation operators for Java. In: 13th International Symposium on Software Reliability Engineering, 2002. Proceedings, pp. 352–363. IEEE Computer Society (2002).  https://doi.org/10.1109/issre.2002.1173287
  20. 20.
    Nanavati, J., Wu, F., Harman, M., Jia, Y., Krinke, J.: Mutation testing of memory-related operators. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–10. IEEE, April 2015.  https://doi.org/10.1109/icstw.2015.7107449
  21. 21.
    Offutt, A.J., Lee, A., Rothermel, G., Untch, R.H., Zapf, C.: An experimental determination of sufficient mutant operators. ACM Trans. Softw. Eng. Methodol. 5(2), 99–118 (1996).  https://doi.org/10.1145/227607.227610CrossRefGoogle Scholar
  22. 22.
    Offutt, A.J., Pan, J.: Automatically detecting equivalent mutants and infeasible paths. Softw. Test. Verif. Reliab. 7(3), 165–192 (1997).  https://doi.org/10.1002/(SICI)1099-1689(199709)7:3<165::AID-STVR143>3.0.CO;2-UCrossRefGoogle Scholar
  23. 23.
    Oliveira, R.A., Alegroth, E., Gao, Z., Memon, A.: Definition and evaluation of mutation operators for GUI-level mutation analysis. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–10. IEEE, April 2015.  https://doi.org/10.1109/icstw.2015.7107457
  24. 24.
    Osman, H., Leuenberger, M., Lungu, M., Nierstrasz, O.: Tracking null checks in open-source java systems. In: 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), vol. 1, pp. 304–313. IEEE, March 2016.  https://doi.org/10.1109/saner.2016.57
  25. 25.
    Osman, H., Lungu, M., Nierstrasz, O.: Mining frequent bug-fix code changes. In: 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE), pp. 343–347. IEEE, February 2014.  https://doi.org/10.1109/csmr-wcre.2014.6747191
  26. 26.
    Papadakis, M., Henard, C., Harman, M., Jia, Y., Traon, Y.L.: Threats to the validity of mutation-based test assessment. In: Proceedings of the 25th International Symposium on Software Testing and Analysis - ISSTA 2016, pp. 354–365. ACM Press, New York (2016).  https://doi.org/10.1145/2931037.2931040
  27. 27.
    Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Traon, Y.L., Harman, M.: Mutation testing advances: an analysis and survey. In: Advances in Computers (2018).  https://doi.org/10.1016/bs.adcom.2018.03.015Google Scholar
  28. 28.
  29. 29.
    Parsai, A., Demeyer, S., De Busser, S.: C++11/14 mutation operators based on common fault patterns. In: Medina-Bulo, I., Merayo, M.G., Hierons, R. (eds.) ICTSS 2018. LNCS, vol. 11146, pp. 102–118. Springer, Cham (2018).  https://doi.org/10.1007/978-3-319-99927-2_9CrossRefGoogle Scholar
  30. 30.
    Parsai, A., Murgia, A., Demeyer, S.: Evaluating random mutant selection at class-level in projects with non-adequate test suites. In: Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering - EASE 2016, pp. 11:1–11:10. ACM Press, New York (2016).  https://doi.org/10.1145/2915970.2915992
  31. 31.
    Parsai, A., Murgia, A., Demeyer, S.: A model to estimate first-order mutation coverage from higher-order mutation coverage. In: 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS), pp. 365–373. IEEE, August 2016.  https://doi.org/10.1109/qrs.2016.48
  32. 32.
    Parsai, A., Murgia, A., Demeyer, S.: LittleDarwin: a feature-rich and extensible mutation testing framework for large and complex Java systems. In: Dastani, M., Sirjani, M. (eds.) FSEN 2017. LNCS, vol. 10522, pp. 148–163. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-68972-2_10CrossRefGoogle Scholar
  33. 33.
    Parsai, A., Murgia, A., Soetens, Q.D., Demeyer, S.: Mutation testing as a safety net for test code refactoring. In: Scientific Workshop Proceedings of the XP2015 on - XP 2015 workshops, pp. 8:1–8:7. ACM Press, New York (2015).  https://doi.org/10.1145/2764979.2764987
  34. 34.
    Parsai, A., Soetens, Q.D., Murgia, A., Demeyer, S.: Considering polymorphism in change-based test suite reduction. In: Dingsøyr, T., Moe, N.B., Tonelli, R., Counsell, S., Gencel, C., Petersen, K. (eds.) XP 2014. LNBIP, vol. 199, pp. 166–181. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-14358-3_14CrossRefGoogle Scholar
  35. 35.
    Schuler, D., Zeller, A.: Javalanche: efficient mutation testing for Java. In: Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering on European Software Engineering Conference and Foundations of Software Engineering Symposium - ESEC/FSE 2009, pp. 297–298. ACM Press, New York (2009).  https://doi.org/10.1145/1595696.1595750
  36. 36.
    Yin, R.K.: Case Study Research: Design and Methods. Applied Social Research Methods. SAGE Publications, Thousand Oaks (2003)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University of AntwerpAntwerpBelgium
  2. 2.University of Antwerp and Flanders MakeAntwerpBelgium

Personalised recommendations