Mutation Testing of Smart Contracts at Scale
- 29 Downloads
It is crucial that smart contracts are tested thoroughly due to their immutable nature. Even small bugs in smart contracts can lead to huge monetary losses. However, testing is not enough; it is also important to ensure the quality and completeness of the tests. There are already several approaches that tackle this challenge with mutation testing, but their effectiveness is questionable since they only considered small contract samples. Hence, we evaluate the quality of smart contract mutation testing at scale. We choose the most promising of the existing (smart contract specific) mutation operators, analyse their effectiveness in terms of killability and highlight severe vulnerabilities that can be injected with the mutations. Moreover, we improve the existing mutation methods by introducing a novel killing condition that is able to detect a deviation in the gas consumption, i.e., in the monetary value that is required to perform transactions.
KeywordsMutation testing Ethereum Smart contracts Solidity Gas limit as a killing criterion Vulnerability injection Modifier issues
This work was supported in part by the National Research Foundation (NRF), Prime Minister’s Office, Singapore, under its National Cybersecurity R&D Programme (Award No. NRF2016NCR-NCR002-028) and administered by the National Cybersecurity R&D Directorate.
We thank Maarten Everts, Joran Honig, Sun Jun, and the anonymous reviewers for their comments on our work.
The replication package for the experiments can be found at https://doi.org/10.5281/zenodo.3726691.
- 1.Andesta, E., Faghih, F., Fooladgar, M.: Testing smart contracts gets smarter. Technical report, Department of Electrical and Computer Engineering University of Tehran, December 2019. https://arxiv.org/abs/1912.04780
- 3.Bugrara, S.: User experience with language-independent formal verification. Technical report, ConsenSys, December 2019. https://arxiv.org/abs/1912.02951
- 4.Chapman, P.: Deviant: a mutation testing tool for Solidity smart contracts. Master thesis 1593, Boise State University, August 2019. https://doi.org/10.18122/td/1593/boisestate
- 5.Chia, V., et al.: Rethinking blockchain security: position paper. In: Atiquzzaman, M., Li, J., Meng, W. (eds.) Confs on Internet of Things, Green Computing and Communications, Cyber, Physical and Social Computing, Smart Data, Blockchain, Computer and Information Technology, Congress on Cybermatics, pp. 1273–1280. IEEE, Halifax, Canada, July 2018. https://doi.org/10.1109/Cybermatics_2018.2018.00222
- 6.Fu, Y., Ren, M., Ma, F., Jiang, Y., Shi, H., Sun, J.: Evmfuzz: differential fuzz testing of Ethereum virtual machine. Technical report, Tsinghua University, China, April 2019. https://arxiv.org/abs/1903.08483
- 7.Gopinath, R., Jensen, C., Groce, A.: Code coverage for suite evaluation by developers. In: 36th International Conference on Software Engineering (ICSE), pp. 72–82. ACM, New York, Hyderabad, India, May 2014. https://doi.org/10.1145/2568225.2568278
- 8.Groce, A., Holmes, J., Marinov, D., Shi, A., Zhang, L.: An extensible, regular-expression-based tool for multi-language mutant generation. In: 40th International Conference on Software Engineering: Companion Proceeedings (ICSE), pp. 25–28. ACM, New York, Gothenburg, Sweden, May 2018. https://doi.org/10.1145/3183440.3183485
- 9.Grün, B.J.M., Schuler, D., Zeller, A.: The impact of equivalent mutants. In: Second International Conference on Software Testing Verification and Validation, ICST 2009, Denver, Colorado, USA, 1–4 April 2009, Workshops Proceedings, pp. 192–199. IEEE Computer Society (2009). https://doi.org/10.1109/ICSTW.2009.37
- 10.Hartel, P., van Staalduinen, M.: Truffle tests for free - replaying Ethereum smart contracts for transparency. Technical report, Singapore University of Technology and Design, Singapore, July 2019. https://arxiv.org/abs/1907.09208
- 11.Hierons, R.M., Harman, M., Danicic, S.: Using program slicing to assist in the detection of equivalent mutants. Softw. Test. Verif. Reliab. 9(4), 233–262 (1999). https://doi.org/10.1002/(sici)1099-1689(199912)9:4<233::aid-stvr191>3.0.co;2-3
- 12.Honig, J.J., Everts, M.H., Huisman, M.: Practical mutation testing for smart contracts. In: Pérez-Solà, C., Navarro-Arribas, G., Biryukov, A., Garcia-Alfaro, J. (eds.) DPM/CBT -2019. LNCS, vol. 11737, pp. 289–303. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-31500-9_19CrossRefGoogle Scholar
- 13.Inozemtseva, L., Holmes, R.: Coverage is not strongly correlated with test suite effectiveness. In: 36th International Conference on Software Engineering (ICSE), pp. 435–445. ACM, New York, Hyderabad (2014). https://doi.org/10.1145/2568225.2568271
- 14.Jabbarvand, R., Malek, S.: \(\rm \mu \)droid: an energy-aware mutation testing framework for android. In: Bodden, E., Schäfer, W., van Deursen, A., Zisman, A. (eds.) Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, 4–8 September 2017. pp. 208–219. ACM (2017). https://doi.org/10.1145/3106237.3106244
- 20.O’Connor, R.: Simplicity: a new language for blockchains. In: Proceedings of the 2017 Workshop on Programming Languages and Analysis for Security, PLAS@CCS 2017, Dallas, TX, USA, 30 October 2017. pp. 107–120. ACM (2017). https://doi.org/10.1145/3139337.3139340
- 22.Offutt, A.J., Pan, J.: Automatically detecting equivalent mutants and infeasible paths. Softw. Test. Verif. Reliab. 7(3), 165–192 (1997). https://doi.org/10.1002/(sici)1099-1689(199709)7:3<165::aid-stvr143>3.0.co;2-u
- 25.Peng, C., Rajan, A.: Sif: a framework for Solidity code instrumentation and analysis. Technical report, University of Edinburgh, UK, May 2019. https://arxiv.org/abs/1905.01659
- 26.Schuler, D., Zeller, A.: (un-)covering equivalent mutants. In: Third International Conference on Software Testing, Verification and Validation, ICST 2010, Paris, France, 7–9 April 2010, pp. 45–54. IEEE Computer Society (2010). https://doi.org/10.1109/ICST.2010.30
- 27.SmartContractSecurity: Smart contract weakness classification registry (2019). https://github.com/SmartContractSecurity/SWC-registry/
- 28.Tengeri, D., et al.: Relating code coverage, mutation score and test suite reducibility to defect density. In: Ninth IEEE International Conference on Software Testing, Verification and Validation Workshops, ICST Workshops 2016, Chicago, IL, USA, 11–15 April 2016, pp. 174–179. IEEE Computer Society (2016). https://doi.org/10.1109/ICSTW.2016.25
- 29.Wang, H., Li, Y., Lin, S.W., Artho, C., Ma, L., Liu, Y.: Oracle-supported dynamic exploit generation for smart contracts. Technical report, Nanyang Technological University, Singapore, September 2019. https://arxiv.org/abs/1909.06605
- 30.Wang, X., Xie, Z., He, J., Zhao, G., Ruihua, N.: Basis path coverage criteria for smart contract application testing. Technical report, School of Computer Science, South China Normal University Guangzhou, China, Noveember 2019. https://arxiv.org/abs/1911.10471
- 31.Wood, G.: Ethereum: a secure decentralised generalised transaction ledger - EIP-150 revision. Technical report 759dccd, Ethcore.io, August 2017. https://ethereum.github.io/yellowpaper/paper.pdf
- 32.Wu, H., Wang, X., Xu, J., Zou, W., Zhang, L., Chen, Z.: Mutation testing for Ethereum smart contract. Technical report, Nanjing University, China, August 2019. https://arxiv.org/abs/1908.03707
- 33.Zhang, L., Hou, S.S., Hu, J.J., Xie, T., Mei, H.: Is operator-based mutant selection superior to random mutant selection? In: 32nd International Conference on Software Engineering (ICSE), pp. 435–444. ACM, New York, Cape Town, May 2010. https://doi.org/10.1145/1806799.1806863