Bi-criteria Test Suite Reduction by Cluster Analysis of Execution Profiles

  • Alireza Khalilian
  • Saeed Parsa
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7054)


The aim has been to minimize regression test suites while retaining fault detection capability of the test suite admissible. An appropriate minimized test suite should exercise different execution paths within a program. However, minimization of test suites may result in significant fault detection loss. To alleviate the loss, a new bi-criteria heuristic algorithm, using cluster analysis of test cases execution profiles is proposed in this paper. Cluster analysis of execution profiles categorizes test cases according to their similarity in terms of exercising a certain coverage criterion. Considering additional coverage criteria the proposed algorithm samples some test cases from each cluster. These additional criteria exercise execution paths, different from those covered by the main testing criteria. Experiments on the Siemens suite manifest the applicability of the proposed approach and present interesting insights into the use of cluster analysis to the bi-criteria test suite reduction.


Software regression testing testing criteria test suite minimization test suite reduction fault detection effectiveness 


  1. 1.
    Rothermel, G., Harrold, M.J., von Ronne, J., Hong, C.: Empirical Studies of Test-Suite Reduction. Journal of Software Testing, Verification, and Reliability 12(4), 219–249 (2002)CrossRefGoogle Scholar
  2. 2.
    Rothermel, G., Harrold, M.J., Ostrin, J., Hong, C.: An Empirical Study of the Effects of Minimization on the Fault Detection Capabilities of Test Suites. In: Proceedings of the International Conference on Software Maintenance. IEEE Computer Society (1998)Google Scholar
  3. 3.
    Harrold, M.J., Gupta, R., Soffa, M.L.: A methodology for Controlling the Size of a Test Suite. ACM Transactions on Software Engineering and Methodology 2, 270–285 (1993)CrossRefGoogle Scholar
  4. 4.
    Chen, T.Y., Lau, M.F.: Heuristics Toward the Optimization of the Size of a Test Suite. In: Proc. of the 3rd International Conference on Software Quality Management, vol. 2, pp. 415–424. Seville, Spain (1995)Google Scholar
  5. 5.
    Jones, J.A., Harrold, M.J.: Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage. IEEE Transactions on Software Engineering 29, 195–209 (2003)CrossRefGoogle Scholar
  6. 6.
    McMaster, S., Memon, A.: Call-Stack Coverage for GUI Test Suite Reduction. IEEE Transactions on Software Engineering 34, 99–115 (2008)CrossRefGoogle Scholar
  7. 7.
    Tallam, S., Gupta, N.: A Concept Analysis Inspired Greedy Algorithm for Test Suite Minimization. In: Proc. of the 6th ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering. ACM, Lisbon (2005)Google Scholar
  8. 8.
    Leon, D., Podgurski, A.: A Comparison of Coverage-Based and Distribution-Based Techniques for Filtering and Prioritizing Test Cases. In: Proc. of the 14th International Symposium on Software Reliability Engineering. IEEE Computer Society (2003)Google Scholar
  9. 9.
    Chen, Z., Xu, B., Zhang, X., Nie, C.: A Novel Approach for Test Suite Reduction Based on Requirement Relation Contraction. In: Proc. of the 2008 ACM Symposium on Applied Computing. ACM, Fortaleza (2008)Google Scholar
  10. 10.
    Jeffrey, D., Gupta, N.: Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction. IEEE Transactions on Software Engineering 33, 108–123 (2007)CrossRefGoogle Scholar
  11. 11.
    Marick, B.: The Craft of Software Testing: Subsystem Testing. Prentice Hall, Englewood Cliffs (1995)Google Scholar
  12. 12.
    Leon, D., Podgurski, A., White, L.J.: Multivariate Visualization in Observation-based Testing. In: Proc. of the 22nd International Conference on Software Engineering. ACM, Limerick (2000)Google Scholar
  13. 13.
    Dickinson, W., Leon, D., Podgurski, A.: Finding Failures by Cluster Analysis of Execution Profiles. In: Proc. of the 23rd International Conference on Software Engineering. IEEE Computer Society, Toronto (2001)Google Scholar
  14. 14.
    Dickinson, W., Leon, D., Podgurski, A.: Pursuing Failure: the Distribution of Program Failures in a Profile Space. In: Proceedings of the 8th European Software Engineering Conference Held Jointly with 9th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, Vienna (2001)Google Scholar
  15. 15.
    Zhong, H., Zhang, L., Mei, H.: An Experimental Study of Four Typical Test Suite Reduction Techniques. Information and Software Technology 50, 534–546 (2008)CrossRefGoogle Scholar
  16. 16.
    Chen, T.Y., Lau, M.: A New Heuristic for Test Suite Reduction. Information and Software Technology 40(5-6) (1998)Google Scholar
  17. 17.
    Wong, W.E., Horgan, J.R., London, S., Mathur, A.P.: Effect of Test Set Minimization on Fault Detection Effectiveness. Software-Practice and Experience 28, 347–369 (1998)CrossRefGoogle Scholar
  18. 18.
    Mansour, N., El-Fakih, K.: Simulated Annealing and Genetic Algorithms for Optimal Regression Testing. Journal of Software Maintenance 11, 19–34 (1999)CrossRefGoogle Scholar
  19. 19.
    Black, J., Melachrinoudis, E., Kaeli, D.: Bi-Criteria Models for All-Uses Test Suite Reduction. In: Proc. of the 26th International Conference on Software Engineering. IEEE Computer Society (2004)Google Scholar
  20. 20.
    Harrold, M.J., Rothermel, G., Wu, R., Yi, L.: An Empirical Investigation of Program Spectra. In: Proc. of the 1998 ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering. ACM, Montreal (1998)Google Scholar
  21. 21.
    Rothermel, G., Elbaum, S., Kinneer, A., Do, H.: Software-artifact Infrastructure Repository,
  22. 22.
    Witten, H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Kaufmann series in Data Management Systems (2005)Google Scholar
  23. 23.
    SAS 9.1.3 Documentation, SAS/GRAPH 9.1 Reference,
  24. 24.
    Yang, Y., Guan, X., You, J.: CLOPE: a Fast and Effective Clustering Algorithm for Transactional Data. In: Proc. of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, Edmonton (2002)Google Scholar
  25. 25.
    Horgan, J.R., London, S.A.: ATAC: A Data Flow Coverage Testing Tool for C. In: Proc. Symp. Assessment of Quality Software Development Tools, pp. 2–10 (1992)Google Scholar
  26. 26.
    Freund, J.E.: Mathematical Statistics, 5th edn. Prentice-Hall (1992)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2012

Authors and Affiliations

  • Alireza Khalilian
    • 1
  • Saeed Parsa
    • 1
  1. 1.Iran University of Science and TechnologyTehranIran

Personalised recommendations