Abstract
Given a grammar (or other sorts of meta-data), one can trivially derive combinatorially exhaustive test-data sets up to a specified depth. Without further efforts, such test-data sets would be huge at the least and explosive most of the time. Fortunately, scenarios of grammar-based testing tend to admit non-explosive approximations of naive combinatorial coverage.
In this paper, we describe the notion of controllable combinatorial coverage and a corresponding algorithm for test-data generation. The approach is based on a suite of control mechanisms to be used for the characterization of test-data sets as well-defined and understandable approximations of full combinatorial coverage.
The approach has been implemented in the C#-based test-data generator Geno, which has been successfully used in projects that required differential testing, stress testing and conformance testing of grammar-driven functionality.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Andoni, A., Daniliuc, D., Khurshid, S., Marinov, D.: Evaluating the Small Scope Hypothesis (unpublished, September 2002), Available at: http://sdg.lcs.mit.edu/publications.html
Bernot, G., Gaudel, M.C., Marre, B.: Software testing based on formal specifications: a theory and a tool. Software Engineering Journal 6(6), 387–405 (1991)
Bird, D.L., Munoz, C.U.: Automatic generation of random self-checking test cases. IBM Systems Journal 22(3), 229–245 (1983)
Bouge, L., Choquet, N., Fribourg, L., Gaudel, M.-C.: Test sets generation from algebraic specifications using logic programming. Journal of Systems and Software 6(4), 343–360 (1986)
Boyapati, C., Khurshid, S., Marinov, D.: Korat: automated testing based on java predicates. In: Proc. International Symposium on Software testing and analysis, pp. 123–133. ACM Press, New York (2002)
Burgess, C.: The Automated Generation of Test Cases for Compilers. Software Testing, Verification and Reliability 4(2), 81–99 (1994)
Burgess, C.J., Saidi, M.: The automatic generation of test cases for optimizing Fortran compilers. Information and Software Technology 38(2), 111–119 (1996)
Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: ICFP 2000: Proceedings of the fifth ACM SIGPLAN international conference on Functional programming, pp. 268–279. ACM Press, New York (2000)
Cohen, D., Dalal, S., Fredman, M., Patton, G.: The AETG system: An approach to testing based on combinatorial design. IEEE Transactions on Software Engineering 23(7), 437–443 (1997)
Foundations of Software Engineering, Microsoft Research. AsmL—Abstract State Machine Language (2005), http://research.microsoft.com/fse/AsmL/
Gurevich, Y., Wallace, C.: Specification and Verification of the Windows Card Runtime Environment Using Abstract State Machines. Technical report, Microsoft Research (February 1999) MSR-TR-99-07
Harm, J., Lämmel, R.: Two-dimensional Approximation Coverage. Informatica 24(3), 355–369 (2000)
Jackson, D., Damon, C.A.: Elements of Style: Analyzing a Software Design Feature with a Counterexample Detector. IEEE Transactions on Software Engineering 22(7), 484–495 (1996)
King, J.C.: Symbolic execution and program testing. Communications of the ACM 19(7), 385–394 (1976)
Knuth, D.: Semantics of context-free languages. Mathematical Systems Theory 2, 127–145 (1968), Corrections in 5, 95–96 (1971)
Lämmel, R.: Grammar Testing. In: Hussmann, H. (ed.) FASE 2001. LNCS, vol. 2029, pp. 201–216. Springer, Heidelberg (2001)
Lämmel, R., Harm, J.: Test case characterisation by regular path expressions. In: Brinksma, E., Tretmans, J. (eds.) Proc. Formal Approaches to Testing of Software (FATES 2001), August 2001. Notes Series NS-01-4, pp. 109–124. BRICS (2001)
Lei, Y., Tai, K.-C.: In-parameter-order: A test generation strategy for pairwise testing. In: HASE, pp. 254–261. IEEE Computer Society, Los Alamitos (1998)
Maurer, P.: Generating test data with enhanced context-free grammars. IEEE Software 7(4), 50–56 (1990)
McKeeman, W.: Differential testing for software. Digital Technical Journal of Digital Equipment Corporation 10(1), 100–107 (1998)
Purdom, P.: A sentence generator for testing parsers. BIT 12(3), 366–375 (1972)
Schlimmer, J., et al.: Web Services Policy Framework (September 2004), Available at: http://www-128.ibm.com/developerworks/library/specification/ws-polfram/
Sirer, E.G., Bershad, B.N.: Using production grammars in software testing. In: USENIX (ed.) Proceedings of the 2nd Conference on Domain-Specific Languages (DSL 1999), Austin, Texas, USA, October 3–5, pp. 1–13. USENIX, Berkeley (1999)
Slutz, D.: Massive Stochastic Testing for SQL. Technical Report MSR-TR-98-21, Microsoft Research, Redmond (1998); A shorter form of the paper appeared in the Proc. of the 24th VLDB Conference, New York, USA (1998)
Stepney, S.: High Integrity Compilation: A Case Study. Prentice Hall, Englewood Cliffs (1993)
Tai, K., Lei, Y.: A Test Generation Strategy for Pairwise Testing. IEEE Transactions on Software Engineering 28(1), 109–111 (2002)
Tillmann, N., Schulte, W., Grieskamp, W.: Parameterized Unit Tests. Technical report, Microsoft Research, MSR-TR-2005-64 (2005); also appeared in FSE/ESEC 2005
W3C. XML Path Language (XPath) Version 1.0 (November 1999), http://www.w3.org/TR/xpath
W3C. XML Schema (2000–2003), http://www.w3.org/XML/Schema
Zaytsev, V.V.: Combinatorial test set generation: Concepts, implementation, case study. Master’s thesis, Universiteit Twente, Enschede, The Netherlands (June 2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 IFIP International Federation for Information Processing
About this paper
Cite this paper
Lämmel, R., Schulte, W. (2006). Controllable Combinatorial Coverage in Grammar-Based Testing. In: Uyar, M.Ü., Duale, A.Y., Fecko, M.A. (eds) Testing of Communicating Systems. TestCom 2006. Lecture Notes in Computer Science, vol 3964. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11754008_2
Download citation
DOI: https://doi.org/10.1007/11754008_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-34184-0
Online ISBN: 978-3-540-34185-7
eBook Packages: Computer ScienceComputer Science (R0)