The Science of Killing Bugs in a Black Box
In this talk I will discuss the combination of model-based testing and mutation testing. Model-based testing is a black-box testing technique that avoids the labour of manually writing hundreds of test cases, but instead advocates the capturing of the expected behaviour in a model of the system under test. The test cases are automatically generated from this model. The technique is receiving growing interest in the embedded-systems domain, where models are the rule rather than the exception.
Mutation testing is a technique for assessing and improving a test suite. A number of faulty versions of a program under test are produced by injecting bugs into its source code. These faulty programs are called mutants. A tester analyses if his test suite can ”kill” all mutants. We say that a test kills a mutant if it is able to distinguish it from the original. The tester improves his test suite until all faulty mutants get killed.
In model-based mutation testing, we combine the central ideas of model-based testing and mutation testing: we inject bugs in a model and generate a test suite that will kill these bugs. In this talk, I will discuss its scientific foundations, tools, and results. The foundations include semantics and conformance relations; the supporting tools involve model checkers, constraint solvers and SMT solvers; our experimental results are taken from two European projects on embedded-systems. I will conclude with a proposal how model-based mutation testing can be integrated into an agile, iterative development process.
KeywordsTest Suite Mutation Testing System Under Test Generate Test Case Test Oracle
Unable to display preview. Download preview PDF.
- 2.Aichernig, B.K., Brandl, H., Jöbstl, E., Krenn, W.: Efficient mutation killers in action. In: IEEE Fourth International Conference on Software Testing, Verification and Validation, ICST 2011, Berlin, Germany, March 21-25, pp. 120–129. IEEE Computer Society (2011)Google Scholar
- 5.Aichernig, B.K., Jöbstl, E.: Efficient refinement checking for model-based mutation testing. In: Proceedings of the 12th International Conference on Quality Software (QSIC 2012). IEEE Computer Society (in press, 2012)Google Scholar
- 6.Aichernig, B.K., Jöbstl, E.: Towards symbolic model-based mutation testing: Combining reachability and refinement checking. In: 7th Workshop on Model-Based Testing (MBT 2012). EPTCS, vol. 80, pp. 88–102 (2012)Google Scholar
- 7.Aichernig, B.K., Jöbstl, E.: Towards symbolic model-based mutation testing: Pitfalls in expressing semantics as constraints. In: Workshops Proc. of the 5th Int. Conf. on Software Testing, Verification and Validation (ICST 2012), pp. 752–757. IEEE Computer Society (2012)Google Scholar
- 8.Aichernig, B.K., Salas, P.A.P.: Test case generation by OCL mutation and constraint solving. In: Cai, K.-Y., Ohnishi, A. (eds.) Fifth International Conference on Quality Software, QSIC 2005, Melbourne, Australia, September 19-21, pp. 64–71. IEEE Computer Society (2005)Google Scholar
- 10.Back, R.-J., Kurki-Suonio, R.: Decentralization of process nets with centralized control. In: 2nd ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing, PODC 1983, pp. 131–142. ACM (1983)Google Scholar
- 11.Brandl, H., Weiglhofer, M., Aichernig, B.K.: Automated conformance verification of hybrid systems. In: Wang, J., Chan, W.K., Kuo, F.-C. (eds.) Proceedings of the 10th International Conference on Quality Software, QSIC 2010, Zhangjiajie, China, July 14-15, pp. 3–12. IEEE Computer Society (2010)Google Scholar
- 12.Gries, D.: The Science of Programming. Texts and Monographs in Computer Science. Springer (1981)Google Scholar
- 15.Krenn, W., Aichernig, B.K.: Test case generation by contract mutation in Spec#. In: Finkbeiner, B., Gurevich, Y., Petrenko, A.K. (eds.) Proceedings of Fifth Workshop on Model Based Testing (MBT 2009), York, England, March 22. Electronic Notes in Theoretical Computer Science, vol. 253(2), pp. 71–86. Elsevier (October 2009)Google Scholar
- 16.Krenn, W., Schlick, R., Aichernig, B.K.: Mapping UML to Labeled Transition Systems for Test-Case Generation – A Translation via Object-Oriented Action Systems. In: de Boer, F.S., Bonsangue, M.M., Hallerstede, S., Leuschel, M. (eds.) FMCO 2009. LNCS, vol. 6286, pp. 186–207. Springer, Heidelberg (2010)CrossRefGoogle Scholar
- 18.Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers (2007)Google Scholar
- 19.Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. Software Testing, Verification and Reliability (2011)Google Scholar
- 20.Weiglhofer, M., Aichernig, B.K., Wotawa, F.: Fault-based conformance testing in practice. International Journal of Software and Informatics 3(2-3), 375–411 (2009); Special double issue on Formal Methods of Program Development edited by Dines BjoernerGoogle Scholar