Combining Multiple Coverage Criteria in Search-Based Unit Test Generation
Automated test generation techniques typically aim at maximising coverage of well-established structural criteria such as statement or branch coverage. In practice, generating tests only for one specific criterion may not be sufficient when testing object oriented classes, as standard structural coverage criteria do not fully capture the properties developers may desire of their unit test suites. For example, covering a large number of statements could be easily achieved by just calling the main method of a class; yet, a good unit test suite would consist of smaller unit tests invoking individual methods, and checking return values and states with test assertions. There are several different properties that test suites should exhibit, and a search-based test generator could easily be extended with additional fitness functions to capture these properties.
However, does search-based testing scale to combinations of multiple criteria, and what is the effect on the size and coverage of the resulting test suites? To answer these questions, we extended the EvoSuite unit test generation tool to support combinations of multiple test criteria, defined and implemented several different criteria, and applied combinations of criteria to a sample of 650 open source Java classes. Our experiments suggest that optimising for several criteria at the same time is feasible without increasing computational costs: When combining nine different criteria, we observed an average decrease of only 0.4 % for the constituent coverage criteria, while the test suites may grow up to 70 %.
KeywordsFitness Function Test Suite Unit Test Coverage Criterion Branch Distance
Supported by the National Research Fund, Luxembourg (FNR/P10/03) and the EPSRC project “EXOGEN” (EP/K030353/1).
- 1.Alshahwan, N., Harman, M.: Coverage and fault detection of the output-uniqueness test selection criteria. In: Proceedings of ISSTA 2014, pp. 181–192. ACM (2014)Google Scholar
- 4.Fraser, G., Arcuri, A.: EvoSuite: automatic test suite generation for object-oriented software. In: Proceedings of FSE 2011, pp. 416–419. ACM (2011)Google Scholar
- 7.Fraser, G., Arcuri, A.: Achieving scalable mutation-based generation of whole test suites. Empirical Softw. Eng. 20(3), 1–30 (2014)Google Scholar
- 8.Fraser, G., Staats, M., McMinn, P., Arcuri, A., Padberg, F.: Does automated white-box test generation really help software testers? In: Proceedings of ISSTA 2013, pp. 291–301. ACM (2013)Google Scholar
- 9.Harman, M., Lakhotia, K., McMinn, P.: A multi-objective approach to search-based test data generation. In: Proceedings of GECCO 2007, pp. 1098–1105. ACM (2007)Google Scholar
- 13.Li, N., Meng, X., Offutt, J., Deng, L.: Is bytecode instrumentation as good as source code instrumentation: an empirical study with industrial tools (experience report). In: Proceedings of ISSRE 2013, pp. 380–389. IEEE (2013)Google Scholar
- 16.Yoo, S., Harman, M.: Pareto efficient multi-objective test case selection. In: Proceedings of ISSTA 2007, pp. 140–150. ACM (2007)Google Scholar