Evaluation of Tie-Breaking and Parameter Ordering for the IPO Family of Algorithms Used in Covering Array Generation
The IPO (In-Parameter-Order) family of algorithms is a popular set of greedy methods for the construction of covering arrays. Aspects such as tie-breaking behavior or parameter ordering can have major impact on the quality of the resulting arrays but have so far not been studied in a systematic manner. In this paper, we survey and present a general framework for the IPO family of algorithms (i.e. IPOG, IPOG-F and IPOG-F2) and present ways to instantiate these abstract components. Then, we evaluate the performance of these variations on a large set of instances, in an extensive experimental setting in terms of covering array sizes.
KeywordsCovering arrays IPO family Tie-breaks Parameter ordering Experiments
The research presented in this paper has been funded in part by the Austrian Research Promotion Agency (FFG) under grants 851205 (Security ProtocoL Interaction Testing in Practice - SPLIT) and 865248 (SECuring Web technologies with combinatorial Interaction Testing - SecWIT).
Part of this research has also been carried out in the context of the Austrian COMET K1 program which is publicly funded by the Austrian Research Promotion Agency (FFG) and the Vienna Business Agency (WAW).
This work was made possible by the facilities of the Shared Hierarchical Academic Research Computing Network (SHARCNET) and Compute/Calcul Canada.
- 1.Bryce, R.C., Colbourn, C.J., Cohen, M.B.: A framework of greedy methods for constructing interaction test suites. In: Proceedings of the 27th International Conference on Software Engineering, ICSE 2005, pp. 146–155. ACM (2005)Google Scholar
- 3.Cohen, M.B., Gibbons, P.B., Mugridge, W.B., Colbourn, C.J.: Constructing test suites for interaction testing. In: Proceedings of the 25th International Conference on Software Engineering, ICSE 2003, pp. 38–48. IEEE Computer Society (2003)Google Scholar
- 4.Duan, F., Lei, Y., Yu, L., Kacker, R.N., Kuhn, D.R.: Improving IPOG’s vertical growth based on a graph coloring scheme. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–8 (2015)Google Scholar
- 7.Gao, S., Lv, J., Du, B., Jiang, Y., Ma, S.: General optimization strategies for refining the in-parameter-order algorithm. In: 2014 14th International Conference on Quality Software (QSIC), pp. 21–26. IEEE (2014)Google Scholar
- 12.Kuhn, D., Kacker, R., Lei, Y.: Introduction to Combinatorial Testing. Chapman & Hall/CRC Innovations in Software Engineering and Software Development Series. Taylor & Francis, London (2013)Google Scholar
- 13.Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: IPOG: a general strategy for T-way software testing. In: 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems, ECBS 2007, pp. 549–556. IEEE (2007)Google Scholar
- 15.Lei, Y., Tai, K.C.: In-parameter-order: a test generation strategy for pairwise testing. In: Proceedings of Third IEEE International High-Assurance Systems Engineering Symposium, pp. 254–261. IEEE (1998)Google Scholar
- 17.Yu, L., Duan, F., Lei, Y., Kacker, R.N., Kuhn, D.R.: Constraint handling in combinatorial test generation using forbidden tuples. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–9 (2015)Google Scholar
- 18.Yu, L., Lei, Y., Nourozborazjany, M., Kacker, R.N., Kuhn, D.R.: An efficient algorithm for constraint handling in combinatorial test generation. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, pp. 242–251 (2013)Google Scholar