Object-Oriented Testing Capabilities and Performance Evaluation of the C# Mutation System

  • Anna Derezińska
  • Anna Szustek
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7054)


The main purpose of mutation testing approach is to check a test suite quality in terms of the adequacy to killing programs with inserted programming faults. We present advances in the C# mutation testing system that supports object-oriented mutation operators. The system enhancements related to functional requirements (mutation operators, avoiding generation of invalid and partially of equivalent mutants) and non-functional ones (speed-up using a new parser and reflection, space reduction storing mutant updates). Mutation testing of six widely used open source programs is discussed. The quality of the tests supplied with these programs was experimentally determined. Performance measures were evaluated to assess system enhancements (2-4 faster mutants creation, 10-100 times disk space reduction, tradeoff of time overhead for storing mutants of different size in a local or remote repository).


mutation testing object-oriented mutation operators C# system evolution 


  1. 1.
    Voas, J.M., McGraw, G.: Software Fault Injection, Inoculating Programs Against Errors. John Wiley & Sons Inc. (1998)Google Scholar
  2. 2.
    Andrews, J.H., Briand, C., Labiche, Y., Namin, A.S.: Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria. IEEE Transactions on Software Engineering 342(8), 608–624 (2006)CrossRefGoogle Scholar
  3. 3.
    Offut, J.: A Mutation Carol Past, Present and Future. In: Proc. of the 4th International Workshop on Mutation Analysis, Mutation 2009, Denver, Colorado (2009)Google Scholar
  4. 4.
    Chevalley, P.: Applying Mutation Analysis for Object-oriented Programs Using a Reflective Approach. In: Proc. of the 8th Asia-Pacific Software Engineering Conference, ASPEC, pp. 267–270 (2001)Google Scholar
  5. 5.
    Chevalley, P., Thevenod-Fosse, P.: A Mutation Analysis Tool for Java Programs. Journal on Software Tools for Techn. Transfer (STTT) 5(1), 90–103 (2003)CrossRefGoogle Scholar
  6. 6.
    Kim, S., Clark, J., McDermid, J.A.: Class Mutation: Mutation Testing for Object-oriented Programs. In: Proc. of Conference on Object-Oriented Software Systems, Erfurt, Germany (2000)Google Scholar
  7. 7.
    Ma, Y.-S., Kwon, Y.-R., Offutt, J.: Inter-class Mutation Operators for Java. In: Proc. of International Symposium on Software Reliability Engineering, ISSRE 2002. IEEE Computer Soc. (2002)Google Scholar
  8. 8.
    Derezińska, A.: Advanced Mutation Operators Applicable in C# programs. In: Sacha, K. (ed.) Software Engineering Techniques: Design for Quality. IFIP, vol. 227, pp. 283–288. Springer, Boston (2006)CrossRefGoogle Scholar
  9. 9.
    Derezińska, A.: Quality Assessment of Mutation Operators Dedicated for C# Programs. In: Proc. of the 6th International Conference on Quality Software, QSIC 2006, pp. 227–234. IEEE Computer Soc. Press, California (2006)CrossRefGoogle Scholar
  10. 10.
    Derezińska, A., Szustek, A.: CREAM - a System for Object-oriented Mutation of C# Programs. In: Szczepański, S., Kłosowski, M., Felendzer, Z. (eds.) Annals Gdansk University of Technology Faculty of ETI, No 5, Information Technologies, vol. 13, pp. 389–406. Gdańsk (2007)Google Scholar
  11. 11.
    Derezińska, A., Szustek, A.: Tool-supported Mutation Approach for Verification of C# programs. In: Zamojski, W., Mazurkiewicz, J., Sugier, J., Walkowiak, T. (eds.) Proc. of International Conference on Dependability of Computer Systems, DepCoS-RELCOMEX 2008, pp. 261–268. IEEE Comp. Soc., USA (2008)CrossRefGoogle Scholar
  12. 12.
    Derezińska, A., Sarba, K.: Distributed Environment Integrating Tools for Software Testing. In: Elleithy, K. (ed.) Advanced Techniques in Computing Sciences and Software Engineering, pp. 545–550. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  13. 13.
  14. 14.
    Delmaro, M., Maldonado, J.: Proteum - a Tool for the Assessment of Test Adequacy for C Programs. In: Proc. of Conference on Performability in Computing Sys., PCS 1996, pp. 79–95 (1996)Google Scholar
  15. 15.
    Namin, S., Andrews, J.H.: On Sufficiency of Mutants. In: Proc. of the 29th International Conference on Software Engineering, ICSE 2007 (2007)Google Scholar
  16. 16.
    Jia, Y., Harman, M.: MiLu: a Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language. TAIC-Part (2008)Google Scholar
  17. 17.
    Irvine, S.A., et al.: Jumble Java Byte Code to Measure the Effectiveness of Unit Tests. In: Mutation 2007 at TAIC.Part 2007, 3th Inter. Workshop on Mutation Analysis, pp. 169–175. Cumberland Lodge, Windsor UK (2007)Google Scholar
  18. 18.
    Kim, S.-W., Harrold, M.J., Kwon, Y.-R.: MuGamma: Mutation Analysis of Deployed Software to Increase Confidence and Assist Evolution. In: Proc. of the 2nd Workshop on Mutation Analysis, Mutation 2006, Releigh, North Carolina (November 2006)Google Scholar
  19. 19.
    Madeyski, L.: On the Effects of Pair Programming on Thoroughness and Fault-finding Effectiveness of Unit Tests. In: Münch, J., Abrahamsson, P. (eds.) PROFES 2007. LNCS, vol. 4589, pp. 207–221. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  20. 20.
    Grün, B.J.M., Schuler, D., Zeller, A.: The Impact of Equivalent Mutants. In: Proc. of the 4th International Workshop on Mutation Analysis, Mutation 2009, Denver, Colorado (2009)Google Scholar
  21. 21.
    Moore, I.: Jester a JUnit Test Tester. eXtreme Programming and Flexible Process in Software Engineering – XP 2000 (2000)Google Scholar
  22. 22.
    Do, H., Rothermel, G.: A Controlled Experiment Assessing Test Case Prioritization Techniques via Mutation Faults. In: Proc. of the 21st IEEE International Confe on Software Maintenance, ICSM 2005, pp. 411–420. IEEE Comp. Soc. (2005)Google Scholar
  23. 23.
    Ma, Y.-S., Offutt, J., Kwon, Y.-R.: MuJava: an Automated Class Mutation System, Software Testing, Verification and Reliability 15(2) (2005)Google Scholar
  24. 24.
  25. 25.
    Lee, H.-J., Ma, Y.-S., Kwon, Y.-R.: Empirical Evaluation of Orthogonality of Class Mutation Operators. In: 11th Asia-Pacific Software Engineering Conference. IEEE Computer Society (2004)Google Scholar
  26. 26.
    Smith, B.H., Williams, L.: A Empirical Evaluation of the MuJava Mutation Operators. In: Mutation 2007 at TAIC.Part 2007, 3th International Workshop on Mutation Analysis, pp. 193–202. Cumberland Lodge, Windsor UK (2007)Google Scholar
  27. 27.
    Baudry, B., Fleurey, F., Jezequel, J.-M., Traon, Y.L.: From Genetic to Bacteriological Algorithms for Mutation-based Testing. Software Testing, Verification and Reliability 15(2), 73–96 (2005)CrossRefGoogle Scholar
  28. 28.
    Kowalski, K.: Implementing Object Mutations into Intermediate Code for C# programs, Bach. Thesis, Inst. of Comp. Science, Warsaw Univ. of Technology (2008) (in polish)Google Scholar
  29. 29.
    Offut, J., Rothermel, G., Zapf, C.: An Experimental Evaluation of Selective Mutation. In: Proc. of the 15th International Conference on Software Engineering, pp. 100–107 (1993)Google Scholar
  30. 30.
    NRefactory, http// Scholar
  31. 31.

Copyright information

© IFIP International Federation for Information Processing 2012

Authors and Affiliations

  • Anna Derezińska
    • 1
  • Anna Szustek
    • 1
  1. 1.Institute of Computer ScienceWarsaw University of TechnologyWarsawPoland

Personalised recommendations