Skip to main content
SpringerLink
Log in

Evaluation of the Prediction-Based Approach to Cost Reduction in Mutation Testing

  • Conference paper
  • First Online:
Information Systems Architecture and Technology: Proceedings of 39th International Conference on Information Systems Architecture and Technology – ISAT 2018 (ISAT 2018)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 853))

  • 1084 Accesses

Abstract

Mutation testing is the most effective technique for assessing the quality of test suites, but it is also very expensive in terms of computational costs. The cost arises from the need to generate and execute a large number of so called mutants. The paper presents and evaluates a machine learning approach to dealing with the issue of limiting the number of executed mutants. The approach uses classification algorithm to predict mutants execution results for a subset of the generated mutants without their execution. The evaluation of the approach takes into consideration two aspects: accuracy of the predicted results and stability of prediction. In the paper the details of the evaluation experiment and its results are presented and discussed. The approach is tested on four examples having different number of mutants ranging from 90 to over 300. The obtained results indicate that the predicted value of the mutation score is consistently higher then the actual one thus allowing for using the results with high confidence.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Acree, A.T.: On Mutation. PhD Thesis, Georgia Institute of Technology, Atlanta, Georgia (1980)

    Google Scholar 

  2. Andrews, J.H., Briand, L.C., Labiche, Y.: Is mutation an appropriate tool for testing experiments? In: Proceedings of the ICSE05, pp. 402–411. IEEE (2005)

    Google Scholar 

  3. DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. Computer 11(4), 34–41 (1978)

    Article  Google Scholar 

  4. Howden, W.E.: Weak mutation testing and completeness of test sets. IEEE Trans. Softw. Eng. 8, 371–379 (1982)

    Article  Google Scholar 

  5. Ji, C., Chen, Z., Xu, B., Zhao, Z.: A novel method of mutation clustering based on domain analysis. In: Proceedings of the 21st ICSEKE09, Boston, USA (2009)

    Google Scholar 

  6. Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37, 649–678 (2011)

    Article  Google Scholar 

  7. Just, R., Jalali, D., Inozemtseva, L., Ernst, M.D., Holmes, R., Fraser, G.: Are mutants a valid substitute for real faults in software testing? In: Proceedings of ACM SIGSOFT Symposium on the Foundations of Software Engineering, Hong Kong, China, pp. 654–665 (2014)

    Google Scholar 

  8. Ma, Y.-S., Kim, S.-W.: Mutation testing cost reduction by clustering overlapped mutants. J. Syst. Softw. 115, 18–30 (2016)

    Article  Google Scholar 

  9. Ma, Y., Offutt, J., Kwon, Y.R.: MuJava: a mutation system for Java. In: Proceedings of ICSE06, Shanghai, China, pp. 827–830 (2006)

    Google Scholar 

  10. Mathur, A.P., Krauser, E.W.: Mutant Unification for Improved Vectorization. Purdue University, West Lafayette, Indiana, Technical report SERC-TR-14-P (1988)

    Google Scholar 

  11. Mitchell, T.: Machine Learning. Mcgraw-Hill Education, New York City (1997)

    MATH  Google Scholar 

  12. Myers, G., Sandler, C., Badgett, T.: The Art of Software Testing. Wiley, Hoboken (2011)

    Google Scholar 

  13. Offutt, A.J., Rothermel, G., Zapf, C.: An experimental evaluation of selective mutation. In: Proceedings of ICSE, pp. 100–107 (1993)

    Google Scholar 

  14. Strug, J., Strug, B.: Machine learning approach in mutation testing. In: LNCS, vol. 764, pp. 200–214 (2012)

    Google Scholar 

  15. Strug, J., Strug, B.: Classifying mutants with decomposition Kernel. In: Proceedings of ICAISC2016, LNCS, vol. 9692, pp. 644–654 (2016)

    Google Scholar 

  16. Strug, J., Strug, B.: Using classification for cost reduction of applying mutation testing. In: Proceedings of FedCSIS2017, pp. 99–108 (2017)

    Google Scholar 

  17. Strug, J., Strug, B.: Cost reduction in mutation testing with bytecode-level mutants classification. In: Proceedings of ICAISC2018, LNCS, vol. 10841, pp. 714–723. Springer (2018)

    Google Scholar 

  18. Thierry, T.C., Papadakis, M., Traon, Y.L., Harman, M.: Empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption. In: Proceedings of IEEE/ACM ICSE17, Buenos Aires, Argentina, pp. 597–608 (2017)

    Google Scholar 

  19. Usaola, M.P., Mateo, P.R.: Mutation testing cost reduction techniques: a survey. IEEE Softw. 27(3), 80–86 (2010)

    Article  Google Scholar 

  20. Wong, W.E., Mathur, A.P.: Reducing the cost of mutation testing: an empirical study. JSS 31(3), 185–196 (1995)

    Google Scholar 

  21. Zhang, J., Zhang, L., Harman, M., Hao, D., Jia, Y., Zhang, L.: Predictive mutation testing. In: Proceedings of ISSTA2016, Saarbrücken, Germany, pp. 342–353 (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Electrical and Computer Engineering, Cracow University of Technology, ul. Warszawska 24, 31-155, Krakow, Poland

    Joanna Strug

  2. Department of Physics, Astronomy and Applied Computer Science, Jagiellonian University, Lojasiewicza 11, 30-348, Krakow, Poland

    Barbara Strug

Authors
  1. Joanna Strug
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Barbara Strug
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joanna Strug .

Editor information

Editors and Affiliations

  1. Faculty of Computer Science and Management, Wrocław University of Science and Technology, Wrocław, Poland

    Jerzy Świątek

  2. Faculty of Computer Science and Management, Wrocław University of Science and Technology, Wrocław, Poland

    Leszek Borzemski

  3. University of Applied Sciences in Nysa, Nysa, Poland

    Zofia Wilimowska

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Strug, J., Strug, B. (2019). Evaluation of the Prediction-Based Approach to Cost Reduction in Mutation Testing. In: Świątek, J., Borzemski, L., Wilimowska, Z. (eds) Information Systems Architecture and Technology: Proceedings of 39th International Conference on Information Systems Architecture and Technology – ISAT 2018. ISAT 2018. Advances in Intelligent Systems and Computing, vol 853. Springer, Cham. https://doi.org/10.1007/978-3-319-99996-8_31

Download citation

Publish with us

Policies and ethics

Search

Navigation