The Science of Killing Bugs in a Black Box

(Keynote Talk)
  • Bernhard Klaus Aichernig
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7554)


In this talk I will discuss the combination of model-based testing and mutation testing. Model-based testing is a black-box testing technique that avoids the labour of manually writing hundreds of test cases, but instead advocates the capturing of the expected behaviour in a model of the system under test. The test cases are automatically generated from this model. The technique is receiving growing interest in the embedded-systems domain, where models are the rule rather than the exception.

Mutation testing is a technique for assessing and improving a test suite. A number of faulty versions of a program under test are produced by injecting bugs into its source code. These faulty programs are called mutants. A tester analyses if his test suite can ”kill” all mutants. We say that a test kills a mutant if it is able to distinguish it from the original. The tester improves his test suite until all faulty mutants get killed.

In model-based mutation testing, we combine the central ideas of model-based testing and mutation testing: we inject bugs in a model and generate a test suite that will kill these bugs. In this talk, I will discuss its scientific foundations, tools, and results. The foundations include semantics and conformance relations; the supporting tools involve model checkers, constraint solvers and SMT solvers; our experimental results are taken from two European projects on embedded-systems. I will conclude with a proposal how model-based mutation testing can be integrated into an agile, iterative development process.


Test Suite Mutation Testing System Under Test Generate Test Case Test Oracle 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aichernig, B.K.: Mutation Testing in the Refinement Calculus. Formal Aspects of Computing 15(2-3), 280–295 (2003)zbMATHCrossRefGoogle Scholar
  2. 2.
    Aichernig, B.K., Brandl, H., Jöbstl, E., Krenn, W.: Efficient mutation killers in action. In: IEEE Fourth International Conference on Software Testing, Verification and Validation, ICST 2011, Berlin, Germany, March 21-25, pp. 120–129. IEEE Computer Society (2011)Google Scholar
  3. 3.
    Aichernig, B.K., Delgado, C.C.: From Faults Via Test Purposes to Test Cases: On the Fault-Based Testing of Concurrent Systems. In: Baresi, L., Heckel, R. (eds.) FASE 2006. LNCS, vol. 3922, pp. 324–338. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  4. 4.
    Aichernig, B.K., He, J.: Mutation testing in UTP. Formal Aspects of Computing 21(1-2), 33–64 (2009)zbMATHCrossRefGoogle Scholar
  5. 5.
    Aichernig, B.K., Jöbstl, E.: Efficient refinement checking for model-based mutation testing. In: Proceedings of the 12th International Conference on Quality Software (QSIC 2012). IEEE Computer Society (in press, 2012)Google Scholar
  6. 6.
    Aichernig, B.K., Jöbstl, E.: Towards symbolic model-based mutation testing: Combining reachability and refinement checking. In: 7th Workshop on Model-Based Testing (MBT 2012). EPTCS, vol. 80, pp. 88–102 (2012)Google Scholar
  7. 7.
    Aichernig, B.K., Jöbstl, E.: Towards symbolic model-based mutation testing: Pitfalls in expressing semantics as constraints. In: Workshops Proc. of the 5th Int. Conf. on Software Testing, Verification and Validation (ICST 2012), pp. 752–757. IEEE Computer Society (2012)Google Scholar
  8. 8.
    Aichernig, B.K., Salas, P.A.P.: Test case generation by OCL mutation and constraint solving. In: Cai, K.-Y., Ohnishi, A. (eds.) Fifth International Conference on Quality Software, QSIC 2005, Melbourne, Australia, September 19-21, pp. 64–71. IEEE Computer Society (2005)Google Scholar
  9. 9.
    Arbab, F.: Reo: A Channel-based Coordination Model for Component Composition. Mathematical Structures in Computer Science 14(3), 329–366 (2004)MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Back, R.-J., Kurki-Suonio, R.: Decentralization of process nets with centralized control. In: 2nd ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing, PODC 1983, pp. 131–142. ACM (1983)Google Scholar
  11. 11.
    Brandl, H., Weiglhofer, M., Aichernig, B.K.: Automated conformance verification of hybrid systems. In: Wang, J., Chan, W.K., Kuo, F.-C. (eds.) Proceedings of the 10th International Conference on Quality Software, QSIC 2010, Zhangjiajie, China, July 14-15, pp. 3–12. IEEE Computer Society (2010)Google Scholar
  12. 12.
    Gries, D.: The Science of Programming. Texts and Monographs in Computer Science. Springer (1981)Google Scholar
  13. 13.
    Jard, C., Jéron, T.: TGV: theory, principles and algorithms. International Journal on Software Tools for Technology Transfer (STTT) 7(4), 297–315 (2005)CrossRefGoogle Scholar
  14. 14.
    Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering 37(5), 649–678 (2011)CrossRefGoogle Scholar
  15. 15.
    Krenn, W., Aichernig, B.K.: Test case generation by contract mutation in Spec#. In: Finkbeiner, B., Gurevich, Y., Petrenko, A.K. (eds.) Proceedings of Fifth Workshop on Model Based Testing (MBT 2009), York, England, March 22. Electronic Notes in Theoretical Computer Science, vol. 253(2), pp. 71–86. Elsevier (October 2009)Google Scholar
  16. 16.
    Krenn, W., Schlick, R., Aichernig, B.K.: Mapping UML to Labeled Transition Systems for Test-Case Generation – A Translation via Object-Oriented Action Systems. In: de Boer, F.S., Bonsangue, M.M., Hallerstede, S., Leuschel, M. (eds.) FMCO 2009. LNCS, vol. 6286, pp. 186–207. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  17. 17.
    Meng, S., Arbab, F., Aichernig, B.K., Astefanoaei, L., de Boer, F.S., Rutten, J.: Connectors as designs: Modeling, refinement and test case generation. Science of Computer Programming 77(7-8), 799–822 (2012)zbMATHCrossRefGoogle Scholar
  18. 18.
    Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers (2007)Google Scholar
  19. 19.
    Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. Software Testing, Verification and Reliability (2011)Google Scholar
  20. 20.
    Weiglhofer, M., Aichernig, B.K., Wotawa, F.: Fault-based conformance testing in practice. International Journal of Software and Informatics 3(2-3), 375–411 (2009); Special double issue on Formal Methods of Program Development edited by Dines BjoernerGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Bernhard Klaus Aichernig
    • 1
  1. 1.Institute for Software TechnologyGraz University of TechnologyAustria

Personalised recommendations