Multiple Mutation Testing from FSM

  • Alexandre Petrenko
  • Omer Nguena TimoEmail author
  • S. Ramesh
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9688)


Fault model based testing receives constantly growing interest of both, researchers and test practitioners. A fault model is typically a tuple of a specification, fault domain, and conformance relation. In the context of testing from finite state machines, the specification is an FSM of a certain type. Conformance relation is specific to the type of FSM and for complete deterministic machines it is equivalence relation. Fault domain is a set of implementation machines each of which models some faults, such as output, transfer or transition faults. In the traditional checking experiment theory the fault domain is the universe of all machines with a given number of states and input and output sets of the specification. Another way of defining fault domains similar to the one used in classical program mutation is to list a number of FSM mutants obtained by changing transitions of the specification. We follow in this paper the approach of defining fault domain as a set of all possible deterministic submachines of a given nondeterministic FSM, called a mutation machine, proposed in our previous work. The mutation machine contains a specification machine and extends it with a number of mutated transitions modelling potential faults. Thus, a single mutant represents multiple mutations and mutation machine represents numerous mutants. We propose a method for analyzing mutation coverage of tests which we cast as a constraint satisfaction problem. The approach is based on logical encoding and SMT-solving, it avoids enumeration of mutants while still offering a possibility to estimate the test adequacy (mutation score). The preliminary experiments performed on an industrial controller indicate that the approach scales sufficiently well.


FSM Conformance testing Mutation testing Fault modelling Fault model-based test generation Test coverage Fault coverage analysis 



This work is supported in part by GM R&D and the MEIE of Gouvernement du Québec. The authors would like to thank the reviewers for their useful comments.


  1. 1.
    Pomeranz, I., Sudhakar, M.R.: Test generation for multiple state-table faults in finite-state machines. IEEE Trans. Comput. 46(7), 783–794 (1997)CrossRefGoogle Scholar
  2. 2.
    Poage, J.F., McCluskey, Jr., E.J.: Derivation of optimal test sequences for sequential machines. In: Proceedings of the IEEE 5th Symposium on Switching Circuits Theory and Logical Design, pp. 121–132 (1964)Google Scholar
  3. 3.
    DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: help for the practicing programmer. IEEE Comput. 11(4), 34–41 (1978)CrossRefGoogle Scholar
  4. 4.
    DeMilli, R.A., Offutt, J.A.: Constraint-based automatic test data generation. IEEE Trans. Softw. Eng. 17(9), 900–910 (1991)CrossRefGoogle Scholar
  5. 5.
    Grunsky, I.S., Petrenko, A.: Design of checking experiments with automata describing protocols. Automatic Control and Computer Sciences. Allerton Press Inc. USA. No. 4 (1988)Google Scholar
  6. 6.
    Hennie, F.C.: Fault detecting experiments for sequential circuits. In: Proceedings of the IEEE 5th Annual Symposium on Switching Circuits Theory and Logical Design, pp. 95–110. Princeton (1964)Google Scholar
  7. 7.
    Koufareva, I., Petrenko, A., Yevtushenko, N.: Test generation driven by user-defined fault models. In: Csopaki, G., Dibuz, S., Tarnay, K. (eds.) Testing of Communicating Systems. IFIP — The International Federation for Information Processing, vol. 21, pp. 215–233. Springer, New York (1999)CrossRefGoogle Scholar
  8. 8.
    Lee, D., Yannakakis, M.: Principles and methods of testing finite-state machines - a survey. Proc. IEEE 84(8), 1090–1123 (1996)CrossRefGoogle Scholar
  9. 9.
    Moore, E.F.: Gedanken - Experiments on sequential machines. In: Automata Studies. Princeton University Press, pp. 129–153 (1956)Google Scholar
  10. 10.
    Petrenko, A., Yevtushenko, N.: Test suite generation for a FSM with a given type of implementation errors. In: Proceedings of IFIP 12th International Symposium on Protocol Specification, Testing, and Verification, pp. 229–243 (1992)Google Scholar
  11. 11.
    Petrenko, A., Yevtushenko, N., Bochmann, G.V.: Fault models for testing in context. In: Gotzhein, R., Bredereke, J. (eds.) Formal Description Techniques IX, pp. 163–178. Springer, USA (1996)CrossRefGoogle Scholar
  12. 12.
    Vasilevskii, M.P.: Failure diagnosis of automata. Cybernetics 4, 653–665 (1973). Plenum Publishing Corporation. New YorkGoogle Scholar
  13. 13.
    Chow, T.S.: Testing software design modeled by finite-state machines. IEEE Trans. Softw. Eng. 4(3), 178–187 (1978)CrossRefzbMATHGoogle Scholar
  14. 14.
    Vuong, S.T., Ko, K.C.: A novel approach to protocol test sequence generation. In: Global Telecommunications Conference, vol. 3, pp. 2–5. IEEE (1990)Google Scholar
  15. 15.
    Godskesen, J.C.: Fault models for embedded systems. In: Pierre, L., Kropf, T. (eds.) CHARME 1999. LNCS, vol. 1703, pp. 354–359. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  16. 16.
    Cheng, K.T., Jou, J.Y.: A functional fault model for sequential machines. IEEE Trans. Comput. Aided Des. Integr. Circ. Syst. 11(9), 1065–1073 (1992)CrossRefGoogle Scholar
  17. 17.
    Petrenko, A., Boroday, S., Groz, R.: Confirming configurations in EFSM testing. IEEE Trans. Softw. Eng. 30(1), 29–42 (2004)CrossRefzbMATHGoogle Scholar
  18. 18.
    Gordon, F., Wotawa, F., Ammann, P.: Testing with model checkers: a survey. Softw. Test. Verification Reliab. 19(3), 215–261 (2009)CrossRefGoogle Scholar
  19. 19.
    Anand, S., et al.: An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw. 86(8), 1978–2001 (2013)CrossRefGoogle Scholar
  20. 20.
    Petrenko, A.: Fault model-driven test derivation from finite state models: Annotated bibliography. Modeling and verification of parallel processes, pp. 196–205. Springer, Heidelberg (2001)zbMATHGoogle Scholar
  21. 21.
    Petrenko, A., Bochmann, G.V., Yao, M.: On fault coverage of tests for finite state specifications. Comput. Netw. ISDN Syst. 29(1), 81–106 (1996)CrossRefGoogle Scholar
  22. 22.
    Simao, A., Petrenko, A., Maldonado, J.C.: Comparing finite state machine test coverage criteria. IET Softw. 3(2), 91–105 (2009)CrossRefGoogle Scholar
  23. 23.
    De Moura, L., Dutertre B.: Yices 1.0: An efficient SMT solver. In: The Satisfiability Modulo Theories Competition (SMT-COMP) (2006)Google Scholar
  24. 24.
    de Moura, L., Bjørner, N.S.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  25. 25.
    Rӧsch, S., Ulewicz, S., Provost, J., Vogel-Heuser, B.: Review of model-based testing approaches in production automation and adjacent domains - current challenges and research gaps. J. Softw. Eng. Appl. 8, 499–519 (2015)CrossRefGoogle Scholar
  26. 26.
    Bochmann, G.V., et al.: Fault models in testing. In: Proceedings of the IFIP TC6/WG6. 1 Fourth International Workshop on Protocol Test Systems, pp. 17–30. North-Holland Publishing Co. (1991)Google Scholar
  27. 27.
    Petrenko, A., Yevtushenko, N.: Testing from partial deterministic FSM specifications. IEEE Trans. Comput. 54(9), 1154–1165 (2005)CrossRefGoogle Scholar
  28. 28.
    Korel, B., Tahat, L.H., Harman M.: Test prioritization using system models. In: Proceedings of the 21st IEEE International Conference on Software Maintenance, pp. 559–568 (2005)Google Scholar
  29. 29.
    Petrenko, A., Dury, A., Ramesh, S., Mohalik, S.: A method and tool for test optimization for automotive controllers. In: ICST Workshops, pp. 198–207 (2013)Google Scholar
  30. 30.
    Parr, T.: The Definitive ANTLR 4 Reference, vol. 2. Pragmatic Bookshelf, Raleigh (2013)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2016

Authors and Affiliations

  • Alexandre Petrenko
    • 1
  • Omer Nguena Timo
    • 1
    Email author
  • S. Ramesh
    • 2
  1. 1.Computer Research Institute of Montreal, CRIMMontrealCanada
  2. 2.GM Global R&DWarrenUSA

Personalised recommendations