Using search-based techniques for testing executable software models specified through graph transformations

Abstract

Design by contract is a software development methodology that uses contracts for defining interfaces among interacting components of a software system. Graph transformation system is used to specify the behavioral aspects of software components by defining the pre- and post-conditions of methods as contracts. In this paper, we focus on testing executable software models specified by a graph transformation system. A set of model-specific coverage criteria and a cost-aware search-based test generation approach are introduced. To evaluate the effectiveness of the proposed coverage criteria and the test generation approach, a type of mutation analysis is presented at the model level. Furthermore, a couple of fault-detection methods are used to assess the quality of the generated tests in the model-level mutation analysis. The proposed approach is implemented in GROOVE, a toolset for model checking graph transformation systems. The empirical results based on some well-known case studies demonstrate the efficiency and scalability of each proposed coverage criterion and testing approach. The comparison of the proposed test generation approach with state-of-the-art techniques indicates a significant improvement in terms of fault-detection capability and testing costs.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  1. 1.

    Liu S (2016) Validating formal specifications using testing-based specification animation. In: Proceedings of the 4th FME workshop on formal methods in software engineering. ACM

  2. 2.

    Belli F et al (2016) Model-based mutation testing—approach and case studies. Sci Comput Program 120:25–48

    Article  Google Scholar 

  3. 3.

    Sargent RG (2013) Verification and validation of simulation models. J Simul 7(1):12–24

    MathSciNet  Article  Google Scholar 

  4. 4.

    Granda M et al (2017) Effectiveness assessment of an early testing technique using model-level mutants. In: Proceedings of the 21st international conference on evaluation and assessment in software engineering. ACM

  5. 5.

    Wong PY et al (2015) Testing abstract behavioral specifications. Int J Softw Tools Technol Transf 17(1):107–119

    Article  Google Scholar 

  6. 6.

    Kemmerer RA (1985) Testing formal specifications to detect design errors. IEEE Trans Softw Eng 1:32–43

    Article  Google Scholar 

  7. 7.

    González CA et al (2018) Enabling model testing of cyber-physical systems. In: Proceedings of the 21th ACM/IEEE international conference on model driven engineering languages and systems

  8. 8.

    Araujo W, Briand LC, Labiche Y (2014) On the effectiveness of contracts as test oracles in the detection and diagnosis of functional faults in concurrent object-oriented software. IEEE Trans Softw Eng 40(10):971–992

    Article  Google Scholar 

  9. 9.

    Meyer B (1988) Object-oriented software construction, vol 2. Prentice Hall, New York

    Google Scholar 

  10. 10.

    Ehrig H, Ehrig K (2006) Overview of formal concepts for model transformations based on typed attributed graph transformation. Electron Notes Theor Comput Sci 152:3–22

    Article  Google Scholar 

  11. 11.

    Heckel R (2006) Graph transformation in a nutshell. Electron Notes Theor Comput Sci 148(1):187–198

    Article  Google Scholar 

  12. 12.

    Taentzer G (2003) AGG: a graph transformation environment for modeling and validation of software. In: International workshop on applications of graph transformations with industrial relevance. Springer

  13. 13.

    König B et al (2018) A tutorial on graph transformation. In: Graph transformation, specifications, and nets. Springer, pp 83–104

  14. 14.

    Machado R, Ribeiro L, Heckel R (2015) Rule-based transformation of graph rewriting rules: towards higher-order graph grammars. Theor Comput Sci 594:1–23

    MathSciNet  Article  Google Scholar 

  15. 15.

    Kalaee A, Rafe V (2018) Model-based test suite generation for graph transformation system using model simulation and search-based techniques. Information and Software Technology

  16. 16.

    Baier C, Katoen J-P (2008) Principles of model checking. MIT Press, Cambridge

    Google Scholar 

  17. 17.

    Xu D et al (2016) Automated integration testing using logical contracts. IEEE Trans Reliab 65(3):1205–1222

    Article  Google Scholar 

  18. 18.

    Rafe V, Rahmani M, Rashidi K (2013) A survey on coping with the state space explosion problem in model checking. Int Res J Appl Basic Sci 4(6):1379–1384

    Google Scholar 

  19. 19.

    Enoiu EP et al (2016) Automated test generation using model checking: an industrial evaluation. Int J Softw Tools Technol Transf 18(3):335–353

    Article  Google Scholar 

  20. 20.

    Villani E et al (2019) Integrating model checking and model based testing for industrial software development. Comput Ind 104:88–102

    Article  Google Scholar 

  21. 21.

    Fraser G, Wotawa F, Ammann P (2009) Issues in using model checkers for test case generation. J Syst Softw 82(9):1403–1418

    Article  Google Scholar 

  22. 22.

    Ferrer J et al (2015) Search based algorithms for test sequence generation in functional testing. Inf Softw Technol 58:419–432

    Article  Google Scholar 

  23. 23.

    Khan TA, Runge O, Heckel R (2012) Testing against visual contracts: model-based coverage. In: International conference on graph transformation. Springer

  24. 24.

    Runge O, Khan TA, Heckel R (2013) Test case generation using visual contracts. In: Electronic Communications of the EASST, vol 58

  25. 25.

    de Bruijn V (2013) Model-based testing with graph grammars

  26. 26.

    Rensink A (2003) The GROOVE simulator: a tool for state space generation. In: International workshop on applications of graph transformations with industrial relevance. Springer

  27. 27.

    Rozenberg G (1997) Handbook of graph grammars and computing by graph transformation, vol 1. World Scientific, Singapore

    Google Scholar 

  28. 28.

    Kastenberg H, Rensink A (2006) Model checking dynamic states in GROOVE. In: International SPIN workshop on model checking of software. Springer

  29. 29.

    Varró D, Balogh A (2007) The model transformation language of the VIATRA2 framework. Sci Comput Program 68(3):214–234

    MathSciNet  Article  Google Scholar 

  30. 30.

    Heckel R, Khan TA, Machado R (2011) Towards test coverage criteria for visual contracts. In: Electronic Communications of the EASST, vol 41

  31. 31.

    Koopman P, Achten P, Plasmeijer R (2008) Testing and validating the quality of specifications. In: 2008 IEEE international conference on software testing verification and validation workshop. IEEE

  32. 32.

    Jard C, Bochmann GV (1983) An approach to testing specifications. J Syst Softw 3(4):315–323

    Article  Google Scholar 

  33. 33.

    Anand S et al (2013) An orchestrated survey of methodologies for automated software test case generation. J Syst Softw 86(8):1978–2001

    Article  Google Scholar 

  34. 34.

    Fraser G, Wotawa F, Ammann PE (2009) Testing with model checkers: a survey. Softw Test Verif Reliab 19(3):215–261

    Article  Google Scholar 

  35. 35.

    Harman M, Jia Y, Zhang Y (2015) Achievements, open problems and challenges for search based software testing. In: 2015 IEEE 8th international conference on software testing, verification and validation (ICST). IEEE

  36. 36.

    Khari M, Kumar P (2017) An extensive evaluation of search-based software testing: a review. Soft Comput 1–14

  37. 37.

    Fraser G, Arcuri A (2013) Whole test suite generation. IEEE Trans Softw Eng 39(2):276–291

    Article  Google Scholar 

  38. 38.

    Heckel R, Mariani L (2004) Component integration testing by graph transformations. In: International conference on computer science, software engineering, information technology, e-Business, and applications, Cairo. Citeseer

  39. 39.

    Gönczy L, Heckel R, Varró D (2007) Model-based testing of service infrastructure components. In: Testing of software and communicating systems. Springer. pp 155–170

  40. 40.

    Legay A et al (2017) On featured transition systems. In: International conference on current trends in theory and practice of informatics. Springer

  41. 41.

    Harman M et al (2010) Optimizing for the number of tests generated in search based test data generation with an application to the oracle cost problem. In: 2010 third international conference on software testing, verification, and validation workshops (ICSTW). IEEE

  42. 42.

    Arqub OA, Abo-Hammour Z (2014) Numerical solution of systems of second-order boundary value problems using continuous genetic algorithm. Inf Sci 279:396–415

    MathSciNet  Article  Google Scholar 

  43. 43.

    Grechanik M, Devanla G (2016) Mutation integration testing. In: 2016 IEEE international conference on software quality, reliability and security (QRS). IEEE

  44. 44.

    Offutt AJ (1992) Investigations of the software testing coupling effect. ACM Trans Softw Eng Methodol (TOSEM) 1(1):5–20

    Article  Google Scholar 

  45. 45.

    Ma Y-S, Offutt J, Kwon Y-R (2006) MuJava: a mutation system for Java. In: Proceedings of the 28th international conference on Software engineering. ACM

  46. 46.

    Schmidt A, Varró D (2003) CheckVML: a tool for model checking visual modeling languages. In: International conference on the unified modeling language. Springer

  47. 47.

    Engels G, Güldali B, Lohmann M (2006) Towards model-driven unit testing. In: International conference on model driven engineering languages and systems. Springer

  48. 48.

    Lobo F, Lima CF, Michalewicz Z (2007) Parameter setting in evolutionary algorithms, vol 54. Springer, Berlin

    Google Scholar 

  49. 49.

    Lobo FG, Lima CF (2007) Adaptive population sizing schemes in genetic algorithms. In: Parameter setting in evolutionary algorithms. Springer, pp 185–204

  50. 50.

    Arqub OA (2017) Adaptation of reproducing kernel algorithm for solving fuzzy Fredholm-Volterra integrodifferential equations. Neural Comput Appl 28(7):1591–1610

    Article  Google Scholar 

  51. 51.

    Arqub OA et al (2016) Numerical solutions of fuzzy differential equations using reproducing kernel Hilbert space method. Soft Comput 20(8):3283–3302

    Article  Google Scholar 

  52. 52.

    Arqub OA et al (2017) Application of reproducing kernel algorithm for solving second-order, two-point fuzzy boundary value problems. Soft Comput 21(23):7191–7206

    Article  Google Scholar 

  53. 53.

    Malburg J, Fraser G (2011) Combining search-based and constraint-based testing. In: Proceedings of the 2011 26th IEEE/ACM international conference on automated software engineering. IEEE Computer Society

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Vahid Rafe.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Algorithm 4 shows how coverage goals concerning the introduced data-dependencies in a given test case are detected in GROOVE toolset. It receives a test sequence as input and keeps track of data-flow along the execution path. Lines 1–3 of the algorithm initializes needed data structures including four empty sets to maintain detected coverage goals and put the SUC to the start state. Then each transformation step of the test sequence is applied to the system state and experienced data-flow, i.e. def-use, are recorded through the while loop (Lines 4–26). In each transformation step, the applied rule was registered for created objects (def step, Lines 5–8) to detect the source of the data object in the use step. Next, all data-dependencies experienced by the applied transformation step extracted as Create_Read (Lines 9–13), Create_Delete (Lines 14–18), Create_Update (Lines 19–23), and Delete_NAC (Lines 24 and 25). To specify the Delete_NAC relations, as shown in Algorithm 5, we should check for each transition t how the elements deleted in the previous transitions enable transition t through the provision of its NAC conditions.

figured
figuree

Algorithm 6 shows how we calculate the number of covered test objectives by all test cases of a test suite as a fitness of the test suite. In this algorithm initially, we have four empty sets (CreatRead, CreateDelete, CreateUpdate, DeleteNAC) to save test goals covered by the test suite (Line 1). Then, data-dependencies covered by each test case are extracted by Algorithm 1 (Line 3) and added to the corresponding sets of the test suite (Lines 4–7). Finally, the fitness value is calculated as a sum of the size of all introduced sets which selected as coverage criteria (Line 8, in this algorithm all of the introduced data-flow coverage criteria are considered).

figuref

Algorithm 7 is designed for generating a random test path. This algorithm launches in the start state and initializes a new empty test path (Lines 1 and 2), then selects randomly an outgoing transition from the current state and goes to the next state through the selected transition (Lines 4–6). The number of the selected transition added to the encoded test path (Line 7).

figureg

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Bahrampour, A., Rafe, V. Using search-based techniques for testing executable software models specified through graph transformations. Int. J. Mach. Learn. & Cyber. (2020). https://doi.org/10.1007/s13042-020-01149-9

Download citation

Keywords

  • Model testing
  • Graph transformation specification
  • Specification testing
  • Design by contract
  • Coverage criteria