Extending Coverage Criteria by Evaluating Their Robustness to Code Structure Changes
Code coverage is usually used as a measurement of testing quality and as adequacy criterion. Unfortunately, code coverage is very sensitive to modifications of the code structure, and, therefore, the same test suite can achieve different degrees of coverage on the same program written in two syntactically different ways. For this reason, code coverage can provide the tester with misleading information.
In order to understand how a testing criterion is affected by code structure modifications, we introduce a way to measure the sensitivity of coverage to code changes. We formalize the modifications of the code structure using semantic preserving code-to-code transformations and we propose a framework to evaluate coverage robustness to these transformations, extending actual existing coverage criteria.
This allows us to define which programs and which test suites can be considered robust with respect to a certain set of transformations. We can identify when the obtained coverage is fragile and we extend the concept of coverage criterion by introducing an index that measures the fragility of the coverage of a given test suite. We show how to compute the fragility index and we evidence that also well-written industrial code and realistic test suites can be fragile. Moreover, we suggest how to deal with this kind of testing fragility.
KeywordsTest Suite Coverage Criterion Code Coverage Testability Transformation Code Transformation
- 1.Refactoring catalog - website, http://www.refactoring.com/catalog/
- 2.Aho, A.V., Sethi, R., Ullman, J.D.: Compilers: principles, techniques, and tools. Addison-Wesley Longman Publishing Co., Inc., Boston (1986)Google Scholar
- 4.ARP 4761, Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne Systems and Equipment. Aerospace Recommended Practice, Society of Automotive Engineers, Detroit, USA (1996)Google Scholar
- 6.Dijkstra, E.W.: Notes on structured programming. In: Dahl, O.J., Dijkstra, E.W., Hoare, C.A.R. (eds.) Structured Programming. Academic Press (1972)Google Scholar
- 8.Fowler, M.: Refactoring: Improving the Design of Existing Code. Addison-Wesley (August 1999)Google Scholar
- 9.Fraser, G., Arcuri, A.: Evosuite: Automatic test suite generation for object-oriented software. In: Proc. of ACM SIGSOFT ESEC/FSE, pp. 416–419 (2011)Google Scholar
- 12.Kirner, R.: Towards preserving model coverage and structural code coverage. EURASIP J. Emb. Sys., 1–16 (2009)Google Scholar
- 13.Marick, B., Smith, J., Jones, M.: How to misuse code coverage. In: Proc. of ICTCS 1999 (June 1999)Google Scholar
- 15.Myers, G.J.: The art of software testing, 2nd edn. Wiley (2004)Google Scholar
- 16.Namin, A.S., Andrews, J.H.: The influence of size and coverage on test suite effectiveness. In: Proc. of ISSTA 2009, pp. 57–68. ACM (2009)Google Scholar
- 17.Rajan, A., Whalen, M.W., Heimdahl, M.P.: The effect of program and model structure on mc/dc test adequacy coverage. In: Proc. of ICSE, pp. 161–170 (2008)Google Scholar
- 21.Staats, M., Pǎsǎreanu, C.: Parallel symbolic execution for structural test generation. In: Proc. of ISSTA, pp. 183–194. ACM, New York (2010)Google Scholar
- 22.Weißleder, S.: Simulated satisfaction of coverage criteria on uml state machines. In: Proc. of ICST 2010, pp. 117–126. IEEE Computer Society (2010)Google Scholar