State Coverage: An Empirical Analysis Based on a User Study

Abstract

State coverage is a relatively new metric to evaluate the quality of test suites. While most existing test adequacy criteria measure the degree of exploration of the code under test, state coverage estimates the strength of the assertions in the test suite. Initial experiments have shown that there is a correlation between state coverage and mutation adequacy, and that expert users can discover new faults by modifying the test suite to increase state coverage. Since the faults injected by mutation testing are relatively simple, it is not clear whether these experiment are valid in a broader setting. Furthermore, these results may not be reproducible by average users, since they usually lack full understanding of the internals of the tool.

This paper presents a user-based experiment to evaluate whether the state coverage of a test suite correlates with the number defects it discovers. While the results of the experiments fail to confirm this hypothesis, they do raises important questions. First, test suites with high state coverage should be good in finding logical software faults, but these faults occur less frequently than structural faults. Second, state coverage is not monotonic in the size of the test suite. Therefore, adding new test cases which check new properties and detect new faults can often reduce state coverage. Based on this, we suggest a number of improvements.