Mutation Testing of Smart Contracts at Scale

  • Pieter HartelEmail author
  • Richard Schumi
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12165)


It is crucial that smart contracts are tested thoroughly due to their immutable nature. Even small bugs in smart contracts can lead to huge monetary losses. However, testing is not enough; it is also important to ensure the quality and completeness of the tests. There are already several approaches that tackle this challenge with mutation testing, but their effectiveness is questionable since they only considered small contract samples. Hence, we evaluate the quality of smart contract mutation testing at scale. We choose the most promising of the existing (smart contract specific) mutation operators, analyse their effectiveness in terms of killability and highlight severe vulnerabilities that can be injected with the mutations. Moreover, we improve the existing mutation methods by introducing a novel killing condition that is able to detect a deviation in the gas consumption, i.e., in the monetary value that is required to perform transactions.


Mutation testing Ethereum Smart contracts Solidity Gas limit as a killing criterion Vulnerability injection Modifier issues 



This work was supported in part by the National Research Foundation (NRF), Prime Minister’s Office, Singapore, under its National Cybersecurity R&D Programme (Award No. NRF2016NCR-NCR002-028) and administered by the National Cybersecurity R&D Directorate.

We thank Maarten Everts, Joran Honig, Sun Jun, and the anonymous reviewers for their comments on our work.

The replication package for the experiments can be found at


  1. 1.
    Andesta, E., Faghih, F., Fooladgar, M.: Testing smart contracts gets smarter. Technical report, Department of Electrical and Computer Engineering University of Tehran, December 2019.
  2. 2.
    Atzei, N., Bartoletti, M., Cimoli, T.: A survey of attacks on ethereum smart contracts (SoK). In: Maffei, M., Ryan, M. (eds.) POST 2017. LNCS, vol. 10204, pp. 164–186. Springer, Heidelberg (2017). Scholar
  3. 3.
    Bugrara, S.: User experience with language-independent formal verification. Technical report, ConsenSys, December 2019.
  4. 4.
    Chapman, P.: Deviant: a mutation testing tool for Solidity smart contracts. Master thesis 1593, Boise State University, August 2019.
  5. 5.
    Chia, V., et al.: Rethinking blockchain security: position paper. In: Atiquzzaman, M., Li, J., Meng, W. (eds.) Confs on Internet of Things, Green Computing and Communications, Cyber, Physical and Social Computing, Smart Data, Blockchain, Computer and Information Technology, Congress on Cybermatics, pp. 1273–1280. IEEE, Halifax, Canada, July 2018.
  6. 6.
    Fu, Y., Ren, M., Ma, F., Jiang, Y., Shi, H., Sun, J.: Evmfuzz: differential fuzz testing of Ethereum virtual machine. Technical report, Tsinghua University, China, April 2019.
  7. 7.
    Gopinath, R., Jensen, C., Groce, A.: Code coverage for suite evaluation by developers. In: 36th International Conference on Software Engineering (ICSE), pp. 72–82. ACM, New York, Hyderabad, India, May 2014.
  8. 8.
    Groce, A., Holmes, J., Marinov, D., Shi, A., Zhang, L.: An extensible, regular-expression-based tool for multi-language mutant generation. In: 40th International Conference on Software Engineering: Companion Proceeedings (ICSE), pp. 25–28. ACM, New York, Gothenburg, Sweden, May 2018.
  9. 9.
    Grün, B.J.M., Schuler, D., Zeller, A.: The impact of equivalent mutants. In: Second International Conference on Software Testing Verification and Validation, ICST 2009, Denver, Colorado, USA, 1–4 April 2009, Workshops Proceedings, pp. 192–199. IEEE Computer Society (2009).
  10. 10.
    Hartel, P., van Staalduinen, M.: Truffle tests for free - replaying Ethereum smart contracts for transparency. Technical report, Singapore University of Technology and Design, Singapore, July 2019.
  11. 11.
    Hierons, R.M., Harman, M., Danicic, S.: Using program slicing to assist in the detection of equivalent mutants. Softw. Test. Verif. Reliab. 9(4), 233–262 (1999).<233::aid-stvr191>;2-3
  12. 12.
    Honig, J.J., Everts, M.H., Huisman, M.: Practical mutation testing for smart contracts. In: Pérez-Solà, C., Navarro-Arribas, G., Biryukov, A., Garcia-Alfaro, J. (eds.) DPM/CBT -2019. LNCS, vol. 11737, pp. 289–303. Springer, Cham (2019). Scholar
  13. 13.
    Inozemtseva, L., Holmes, R.: Coverage is not strongly correlated with test suite effectiveness. In: 36th International Conference on Software Engineering (ICSE), pp. 435–445. ACM, New York, Hyderabad (2014).
  14. 14.
    Jabbarvand, R., Malek, S.: \(\rm \mu \)droid: an energy-aware mutation testing framework for android. In: Bodden, E., Schäfer, W., van Deursen, A., Zisman, A. (eds.) Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, 4–8 September 2017. pp. 208–219. ACM (2017).
  15. 15.
    Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011). Scholar
  16. 16.
    King, K.N., Offutt, A.J.: A fortran language system for mutation-ased software testing. Softw.-Pract. Experience 21(7), 685–718 (1991). Scholar
  17. 17.
    Kintis, M., Papadakis, M., Jia, Y., Malevris, N., Traon, Y.L., Harman, M.: Detecting trivial mutant equivalences via compiler optimisations. IEEE Trans. Softw. Eng. 44(4), 308–333 (2018). Scholar
  18. 18.
    Mehar, M.I., et al.: Understanding a revolutionary and flawed grand experiment in blockchain: the dao attack. J. Cases Inf. Technol. 21(1), 19–32 (2019). Scholar
  19. 19.
    Nilsson, R., Offutt, J., Mellin, J.: Test case generation for mutation-based testing of timeliness. Electron. Notes Theor. Comput. Sci. 164(4), 97–114 (2006). Scholar
  20. 20.
    O’Connor, R.: Simplicity: a new language for blockchains. In: Proceedings of the 2017 Workshop on Programming Languages and Analysis for Security, PLAS@CCS 2017, Dallas, TX, USA, 30 October 2017. pp. 107–120. ACM (2017).
  21. 21.
    Offutt, A.J., Craft, W.M.: Using compiler optimization techniques to detect equivalent mutants. Softw. Test. Verif. Reliab. 4(3), 131–154 (1994). Scholar
  22. 22.
    Offutt, A.J., Pan, J.: Automatically detecting equivalent mutants and infeasible paths. Softw. Test. Verif. Reliab. 7(3), 165–192 (1997).<165::aid-stvr143>;2-u
  23. 23.
    Offutt, A.J., Untch, R.H.: Mutation 2000: uniting the orthogonal. In: Wong, E.W. (ed.) Mutation Testing for the New Century, pp. 34–44. Springer, Boston (2001). Scholar
  24. 24.
    Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Traon, Y.L., Harman, M.: Mutation testing advances: an analysis and survey. Adv. Comput. 112, 275–378 (2019). ElsivierCrossRefGoogle Scholar
  25. 25.
    Peng, C., Rajan, A.: Sif: a framework for Solidity code instrumentation and analysis. Technical report, University of Edinburgh, UK, May 2019.
  26. 26.
    Schuler, D., Zeller, A.: (un-)covering equivalent mutants. In: Third International Conference on Software Testing, Verification and Validation, ICST 2010, Paris, France, 7–9 April 2010, pp. 45–54. IEEE Computer Society (2010).
  27. 27.
    SmartContractSecurity: Smart contract weakness classification registry (2019).
  28. 28.
    Tengeri, D., et al.: Relating code coverage, mutation score and test suite reducibility to defect density. In: Ninth IEEE International Conference on Software Testing, Verification and Validation Workshops, ICST Workshops 2016, Chicago, IL, USA, 11–15 April 2016, pp. 174–179. IEEE Computer Society (2016).
  29. 29.
    Wang, H., Li, Y., Lin, S.W., Artho, C., Ma, L., Liu, Y.: Oracle-supported dynamic exploit generation for smart contracts. Technical report, Nanyang Technological University, Singapore, September 2019.
  30. 30.
    Wang, X., Xie, Z., He, J., Zhao, G., Ruihua, N.: Basis path coverage criteria for smart contract application testing. Technical report, School of Computer Science, South China Normal University Guangzhou, China, Noveember 2019.
  31. 31.
    Wood, G.: Ethereum: a secure decentralised generalised transaction ledger - EIP-150 revision. Technical report 759dccd,, August 2017.
  32. 32.
    Wu, H., Wang, X., Xu, J., Zou, W., Zhang, L., Chen, Z.: Mutation testing for Ethereum smart contract. Technical report, Nanjing University, China, August 2019.
  33. 33.
    Zhang, L., Hou, S.S., Hu, J.J., Xie, T., Mei, H.: Is operator-based mutant selection superior to random mutant selection? In: 32nd International Conference on Software Engineering (ICSE), pp. 435–444. ACM, New York, Cape Town, May 2010.
  34. 34.
    Zhu, Q., Panichella, A., Zaidman, A.: A systematic literature review of how mutation testing supports quality assurance processes. J. Softw. Test. Verif. Reliab. 28(6), e1675:1–e1675:39 (2018). Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Singapore University of Technology and DesignSingaporeSingapore
  2. 2.Delft University of TechnologyDelftThe Netherlands
  3. 3.Singapore Management UniversitySingaporeSingapore

Personalised recommendations