Evaluation of Tie-Breaking and Parameter Ordering for the IPO Family of Algorithms Used in Covering Array Generation

  • Kristoffer Kleine
  • Ilias Kotsireas
  • Dimitris E. SimosEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10979)


The IPO (In-Parameter-Order) family of algorithms is a popular set of greedy methods for the construction of covering arrays. Aspects such as tie-breaking behavior or parameter ordering can have major impact on the quality of the resulting arrays but have so far not been studied in a systematic manner. In this paper, we survey and present a general framework for the IPO family of algorithms (i.e. IPOG, IPOG-F and IPOG-F2) and present ways to instantiate these abstract components. Then, we evaluate the performance of these variations on a large set of instances, in an extensive experimental setting in terms of covering array sizes.


Covering arrays IPO family Tie-breaks Parameter ordering Experiments 



The research presented in this paper has been funded in part by the Austrian Research Promotion Agency (FFG) under grants 851205 (Security ProtocoL Interaction Testing in Practice - SPLIT) and 865248 (SECuring Web technologies with combinatorial Interaction Testing - SecWIT).

Part of this research has also been carried out in the context of the Austrian COMET K1 program which is publicly funded by the Austrian Research Promotion Agency (FFG) and the Vienna Business Agency (WAW).

This work was made possible by the facilities of the Shared Hierarchical Academic Research Computing Network (SHARCNET) and Compute/Calcul Canada.


  1. 1.
    Bryce, R.C., Colbourn, C.J., Cohen, M.B.: A framework of greedy methods for constructing interaction test suites. In: Proceedings of the 27th International Conference on Software Engineering, ICSE 2005, pp. 146–155. ACM (2005)Google Scholar
  2. 2.
    Cheng, C.T.: The test suite generation problem: optimal instances and their implications. Discrete Appl. Math. 155(15), 1943–1957 (2007)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Cohen, M.B., Gibbons, P.B., Mugridge, W.B., Colbourn, C.J.: Constructing test suites for interaction testing. In: Proceedings of the 25th International Conference on Software Engineering, ICSE 2003, pp. 38–48. IEEE Computer Society (2003)Google Scholar
  4. 4.
    Duan, F., Lei, Y., Yu, L., Kacker, R.N., Kuhn, D.R.: Improving IPOG’s vertical growth based on a graph coloring scheme. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–8 (2015)Google Scholar
  5. 5.
    Forbes, M., Lawrence, J., Lei, Y., Kacker, R.N., Kuhn, D.R.: Refining the in-parameter-order strategy for constructing covering arrays. J. Res. Nat. Inst. Stan. Technol. 113(5), 287 (2008)CrossRefGoogle Scholar
  6. 6.
    Gao, S.W., Lv, J.H., Du, B.L., Colbourn, C.J., Ma, S.L.: Balancing frequencies and fault detection in the in-parameter-order algorithm. J. Comput. Sci. Technol. 30(5), 957–968 (2015)CrossRefGoogle Scholar
  7. 7.
    Gao, S., Lv, J., Du, B., Jiang, Y., Ma, S.: General optimization strategies for refining the in-parameter-order algorithm. In: 2014 14th International Conference on Quality Software (QSIC), pp. 21–26. IEEE (2014)Google Scholar
  8. 8.
    Hartman, A.: Software and hardware testing using combinatorial covering suites. In: Golumbic, M., Hartman, I.A. (eds.) Graph Theory, Combinatorics and Algorithms, Operations Research/Computer Science Interfaces Series, vol. 34, pp. 237–266. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  9. 9.
    Hartman, A., Raskin, L.: Problems and algorithms for covering arrays. Discrete Math. 284(1–3), 149–156 (2004)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Kampel, L., Simos, D.E.: Set-based algorithms for combinatorial test set generation. In: Wotawa, F., Nica, M., Kushik, N. (eds.) ICTSS 2016. LNCS, vol. 9976, pp. 231–240. Springer, Cham (2016). Scholar
  11. 11.
    Kleine, K., Simos, D.E.: An efficient design and implementation of the in-parameter-order algorithm. Math. Comput. Sci. 12(1), 51–67 (2018)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Kuhn, D., Kacker, R., Lei, Y.: Introduction to Combinatorial Testing. Chapman & Hall/CRC Innovations in Software Engineering and Software Development Series. Taylor & Francis, London (2013)Google Scholar
  13. 13.
    Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: IPOG: a general strategy for T-way software testing. In: 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems, ECBS 2007, pp. 549–556. IEEE (2007)Google Scholar
  14. 14.
    Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J.: IPOG/IPOG-D: efficient test generation for multi-way combinatorial testing. Softw. Test. Verification Reliab. 18(3), 125–148 (2008)CrossRefGoogle Scholar
  15. 15.
    Lei, Y., Tai, K.C.: In-parameter-order: a test generation strategy for pairwise testing. In: Proceedings of Third IEEE International High-Assurance Systems Engineering Symposium, pp. 254–261. IEEE (1998)Google Scholar
  16. 16.
    Younis, M.I., Zamli, K.Z.: MIPOG-an efficient t-way minimization strategy for combinatorial testing. Int. J. Comput. Theory Eng. 3(3), 388 (2011)CrossRefGoogle Scholar
  17. 17.
    Yu, L., Duan, F., Lei, Y., Kacker, R.N., Kuhn, D.R.: Constraint handling in combinatorial test generation using forbidden tuples. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–9 (2015)Google Scholar
  18. 18.
    Yu, L., Lei, Y., Nourozborazjany, M., Kacker, R.N., Kuhn, D.R.: An efficient algorithm for constraint handling in combinatorial test generation. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, pp. 242–251 (2013)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Kristoffer Kleine
    • 1
  • Ilias Kotsireas
    • 2
  • Dimitris E. Simos
    • 1
    Email author
  1. 1.SBA ResearchViennaAustria
  2. 2.Wilfrid Laurier UniversityWaterlooCanada

Personalised recommendations