Advertisement

Reducing the Cost of Model-Based Testing through Test Case Diversity

  • Hadi Hemmati
  • Andrea Arcuri
  • Lionel Briand
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6435)

Abstract

Model-based testing (MBT) suffers from two main problems which in many real world systems make MBT impractical: scalability and automatic oracle generation. When no automated oracle is available, or when testing must be performed on actual hardware or a restricted-access network, for example, only a small set of test cases can be executed and evaluated. However, MBT techniques usually generate large sets of test cases when applied to real systems, regardless of the coverage criteria. Therefore, one needs to select a small enough subset of these test cases that have the highest possible fault revealing power. In this paper, we investigate and compare various techniques for rewarding diversity in the selected test cases as a way to increase the likelihood of fault detection. We use a similarity measure defined on the representation of the test cases and use it in several algorithms that aim at maximizing the diversity of test cases. Using an industrial system with actual faults, we found that rewarding diversity leads to higher fault detection compared to the techniques commonly reported in the literature: coverage-based and random selection. Among the investigated algorithms, diversification using Genetic Algorithms is the most cost-effective technique.

Keywords

Test case selection Model-based testing Search-based testing Clustering algorithms Similarity measure Genetic Algorithms Adaptive Random Testing Jaccard Index UML state machines 

References

  1. 1.
    Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan-Kaufmann, San Francisco (2006)Google Scholar
  2. 2.
    Hemmati, H., Briand, L., Arcuri, A., Ali, S.: An Enhanced Test Case Selection Approach for Model-Based Testing: An Industrial Case Study. In: 18th ACM International Symposium on Foundations of Software Engineering, FSE (2010)Google Scholar
  3. 3.
    Hemmati, H., Briand, L., Arcuri, A.: Investigation of Similarity Measures for Model-Based Test Case Selection. Simula Research Laboratory, Technical Report (2010-05) (2010)Google Scholar
  4. 4.
    Teknomo, K.: Similarity Measurement, http://people.revoledu.com/kardi/tutorial/Similarity
  5. 5.
    Goldberg, D.E.: Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley Professional, Reading (2001)Google Scholar
  6. 6.
    Xu, R., Wunsch II, D.C.: Survey of Clustering Algorithms. IEEE Transactions on Neural Netwoks 16, 645–678 (2005)CrossRefGoogle Scholar
  7. 7.
    Yoo, S., Harman, M., Tonella, P., Susi, A.: Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge. In: 18th ACM International Symposium on Software Testing and Analysis, ISSTA (2009)Google Scholar
  8. 8.
    Chen, T.Y., Kuoa, F.-C., Merkela, R.G., Tseb, T.H.: Adaptive Random Testing: The ART of test case diversity. Journal of Systems and Software 83, 60–66 (2010)CrossRefGoogle Scholar
  9. 9.
    Ciupa, I., Leitner, A., Oriol, M., Meyer, B.: ARTOO: Adaptive Random Testing for Object-Oriented Software. In: 30th IEEE International Conference on Software Engineering (ICSE) (2008)Google Scholar
  10. 10.
    Harman, M.: The Current State and Future of Search Based Software Engineering. In: Future of Software Engineering, pp. 342–357. IEEE Computer Society, Los Alamitos (2007)CrossRefGoogle Scholar
  11. 11.
    Binder, R.V.: Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley Professional, Reading (1999)Google Scholar
  12. 12.
    Elbaum, S.G., Malishevsky, A.G., Rothermel, G.: Test Case Prioritization: A Family of Empirical Studies. IEEE Transactions on Software Engineering 28, 159–182 (2002)CrossRefGoogle Scholar
  13. 13.
    Li, Z., Harman, M., Hierons, R.M.: Search Algorithms for Regression Test Case Prioritization. IEEE Transactions on Software Engineering 33, 225–237 (2007)CrossRefGoogle Scholar
  14. 14.
    Ma, X.Y., Sheng, B.K., Ye, C.Q.: Test-Suite Reduction Using Genetic Algorithm. In: Cao, J., Nejdl, W., Xu, M. (eds.) APPT 2005. LNCS, vol. 3756, pp. 253–262. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  15. 15.
    Leon, D., Podgurski, A.: A Comparison of Coverage-Based and Distribution-Based Techniques for Filtering and Prioritizing Test Cases. In: 14th IEEE International Symposium on Software Reliability Engineering, ISSRE (2003)Google Scholar
  16. 16.
    Masri, W., Podgurski, A., Leon, D.: An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows. IEEE Transactions on Software Engineering 33 (2007)Google Scholar
  17. 17.
    Jiang, B., Zhang, Z., Chan, W.K., Tse, T.H.: Adaptive random test case prioritization. In: 25th IEEE/ACM International Conference on Automated Software Engineering, ASE (2009)Google Scholar
  18. 18.
    Simão, A.d.S., Mello, R.F.d., Senger, L.J.: A Technique to Reduce the Test Case Suites for Regression Testing Based on a Self-Organizing Neural Network Architecture. In: 30th Annual International Computer Software and Applications Conference, COMPSAC (2006)Google Scholar
  19. 19.
    Ramanathan, M.K., Koyutürk, M., Grama, A., Jagannathan, S.: PHALANX: a graph-theoretic framework for test case prioritization. In: 23rd Annual ACM Symposium on Applied Computing (2008)Google Scholar
  20. 20.
    Ledru, Y., Petrenko, A., Boroday, S.: Using String Distances for Test Case Prioritisation. In: 24th IEEE/ACM International Conference on Automated Software Engineering, ASE (2009)Google Scholar
  21. 21.
    Cartaxo, E.G., Machado, P.D.L., Neto, F.G.O.: On the use of a similarity function for test case selection in the context of model-based testing. In: Software Testing, Verification and Reliability (2009)Google Scholar
  22. 22.
    Ali, S., Hemmati, H., Holt, N.E., Arisholm, E., Briand, L.: Model Transformations as a Strategy to Automate Model-Based Testing - A Tool and Industrial Case Studies. Simula Research Laboratory, Technical Report (2010-01) (2010)Google Scholar
  23. 23.
    Ali, S., Briand, L.C., Hemmati, H., Panesar-Walawege, R.K.: A Systematic Review of the Application and Empirical Investigation of Search-based Test-Case Generation. IEEE Transactions on Software Engineering, Special issue on Search-Based Software Engineering, SBSE (in press, 2010)Google Scholar
  24. 24.
    Harman, M., McMinn, P.: A Theoretical and Empirical Study of Search Based Testing: Local, Global and Hybrid Search. IEEE Transactions on Software Engineering 36, 226–247 (2010)CrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2010

Authors and Affiliations

  • Hadi Hemmati
    • 1
    • 2
  • Andrea Arcuri
    • 1
  • Lionel Briand
    • 1
    • 2
  1. 1.Simula Research LaboratoryUniversity of OsloNorway
  2. 2.Department of InformaticsUniversity of OsloNorway

Personalised recommendations