Skip to main content
SpringerLink
Log in

Learning-Based Testing: Recent Progress and Future Prospects

  • Chapter
  • First Online:
Machine Learning for Dynamic Software Analysis: Potentials and Limits

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 11026))

Abstract

We present a survey of recent progress in the area of learning-based testing (LBT). The emphasis is primarily on fundamental concepts and theoretical principles, rather than applications and case studies. After surveying the basic principles and a concrete implementation of the approach, we describe recent directions in research such as: quantifying the hardness of learning problems, over-approximation methods for learning, and quantifying the power of model checker generated test cases. The common theme underlying these research directions is seen to be metrics for model convergence. Such metrics enable a precise, general and quantitative approach to both speed of learning and test coverage. Moreover, quantitative approaches to black-box test coverage serve to distinguish LBT from alternative approaches such as random and search-based testing. We conclude by outlining some prospects for future research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 16.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For certain models of computation, such as neural networks, it seems that no static analysis techniques are currently known. So this paradigm of machine learning seems less useful in LBT at the present time.

  2. 2.

    Infinite counter-examples can be approximated by finite truncations, and in this way warning verdicts can be generated.

  3. 3.

    The SUT may also be inherently more complex then the model, e.g. having the structure of a pushdown automaton that is not reducible to any finite automaton.

  4. 4.

    This problem is most acute where the SUT latency, i.e. the time to execute one test case, is high.

  5. 5.

    Recall that a Moore automaton is a finite state machine which has an output value associated with each state.

  6. 6.

    An accessor string is an input sequence needed to reach a specific state in the SUT state space. A distinguishing string is suffix that can distinguish two SUT states in terms of their output behaviour. See e.g. [19].

  7. 7.

    Fast random generation is important to measure average performance over large sample sizes.

  8. 8.

    These results where obtained using a glass-box equivalence checker, so that the complexity of finding equivalence counter-examples is constant.

  9. 9.

    The prefix tree consists of all input/output observations of the SUT stored as an n-ary tree, where n is the size of the input alphabet.

  10. 10.

    In practise rule (3) is not completely relaxed.

References

  1. Angluin, D.: Learning regular sets from queries and counterexamples. Inf. Comput. 75, 87–106 (1987)

    Article  MathSciNet  Google Scholar 

  2. Arcuri, A., Iqbal, M.Z., Briand, L.: Formal analysis of the effectiveness and predictability of random testing. In: ACM International Symposium on Software Testing and Analysis (ISSTA) (2010)

    Google Scholar 

  3. Balcázar, J.L., Dáaz, J., Gavaldà, R.: Algorithms for learning finite automata from queries: a unified view. In: Du, D.Z., Ko, K.I. (eds.) Advances in Algorithms, Languages, and Complexity, pp. 53–72. Springer, Boston (1997). https://doi.org/10.1007/978-1-4613-3394-4_2

    Chapter  Google Scholar 

  4. Berg, T., Jonsson, B., Leucker, M., Saksena, M.: Insights to Angluin’s learning. Electr. Notes Theor. Comput. Sci. 118, 3–18 (2005)

    Article  Google Scholar 

  5. Bergadano, F., Gunetti, D.: Testing by means of inductive program learning. ACM Trans. Softw. Eng. Methodol. 5(2), 119–145 (1996)

    Article  Google Scholar 

  6. Cavada, R., et al.: The nuXmv symbolic model checker. In: Biere, A., Bloem, R. (eds.) CAV 2014. LNCS, vol. 8559, pp. 334–342. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-08867-9_22

    Chapter  Google Scholar 

  7. Broy, M., Jonsson, B., Katoen, J.-P., Leucker, M., Pretschner, A. (eds.): Model-Based Testing of Reactive Systems. LNCS, vol. 3472. Springer, Heidelberg (2005). https://doi.org/10.1007/b137241

    Book  MATH  Google Scholar 

  8. Chauhan, P., Clarke, E., Kukula, J., Sapra, S., Veith, H., Wang, D.: Automated abstraction refinement for model checking large state spaces using SAT based conflict analysis. In: Aagaard, M.D., O’Leary, J.W. (eds.) FMCAD 2002. LNCS, vol. 2517, pp. 33–51. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-36126-X_3

    Chapter  Google Scholar 

  9. Cherniavsky, J.C., Smith, C.H.: A recursion theoretic approach to program testing. IEEE Trans. Softw. Eng. SE–13(7), 777–784 (1987)

    Article  MathSciNet  Google Scholar 

  10. Chow, T.S.: Testing software design model by finite state machines. IEEE Trans. Softw. Eng. SE–4(3), 178–187 (1978)

    Article  Google Scholar 

  11. Cimatti, A., et al.: NuSMV 2: an OpenSource tool for symbolic model checking. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 359–364. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45657-0_29

    Chapter  Google Scholar 

  12. Clarke, E., Gupta, A., Kukula, J., Strichman, O.: SAT based abstraction-refinement using ILP and machine learning techniques. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 265–279. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45657-0_20

    Chapter  Google Scholar 

  13. Czerny, M.X.: Learning-based software testing: evaluation of Angluin’s L* algorithm and adaptations in practice. Batchelors thesis, Karlsruhe Institute of Technology, Department of Informatics Institute for Theoretical Computer Science (2014)

    Google Scholar 

  14. Feng, L., Lundmark, S., Meinke, K., Niu, F., Sindhu, M.A., Wong, P.Y.H.: Case studies in learning-based testing. In: Yenigün, H., Yilmaz, C., Ulrich, A. (eds.) ICTSS 2013. LNCS, vol. 8254, pp. 164–179. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-41707-8_11

    Chapter  Google Scholar 

  15. Fisher, M.: An Introduction to Practical Formal Methods Using Temporal Logic. Wiley, Chichester (2011)

    Book  Google Scholar 

  16. Fiterău-Broştean, P., Howar, F.: Learning-based testing the sliding window behavior of TCP implementations. In: Petrucci, L., Seceleanu, C., Cavalcanti, A. (eds.) FMICS/AVoCS -2017. LNCS, vol. 10471, pp. 185–200. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67113-0_12

    Chapter  Google Scholar 

  17. Fraser, G., Wotawa, F., Ammann, P.E.: Testing with model checkers: a survey. SNA Technical report, 2007-P2-04, TU Graz (2007)

    Google Scholar 

  18. Groce, A., Peled, D., Yannakakis, M.: Adaptive model checking. Log. J. IGPL 14(5), 729–744 (2006)

    Article  MathSciNet  Google Scholar 

  19. de la Higuera, C.: Grammatical Inference. Cambridge University Press, Cambridge (2010)

    Book  Google Scholar 

  20. Howar, F., Bauer, O., Merten, M., Steffen, B., Margaria, T.: The teachers’ crowd: the impact of distributed oracles on active automata learning. In: Hähnle, R., Knoop, J., Margaria, T., Schreiner, D., Steffen, B. (eds.) ISoLA 2011. CCIS, pp. 232–247. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-34781-8_18

    Chapter  Google Scholar 

  21. Hungar, H., Margaria, T., Steffen, B.: Test-based model generation for legacy systems. In: Proceedings of the 2003 International Test Conference (ITC 2003), pp. 971–980 (2003)

    Google Scholar 

  22. Kearns, M.J., Vazirani, U.: An Introduction to Computational Learning Theory. MIT Press, Cambridge (1994)

    Google Scholar 

  23. Lang, K.J., Pearlmutter, B.A., Price, R.A.: Results of the Abbadingo one DFA learning competition and a new evidence-driven state merging algorithm. In: Honavar, V., Slutzki, G. (eds.) ICGI 1998. LNCS, vol. 1433, pp. 1–12. Springer, Heidelberg (1998). https://doi.org/10.1007/BFb0054059

    Chapter  Google Scholar 

  24. Li, K., Groz, R., Shahbaz, M.: Integration testing of components guided by incremental state machine learning. In: TAIC PART, pp. 59–70 (2006)

    Google Scholar 

  25. Lucent Technologies Inc.: Directly verifying a black box system Patent, US Patent Number 6,526,544, September 1999

    Google Scholar 

  26. McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verif. Reliab. 14(2), 105–156 (2004)

    Article  Google Scholar 

  27. Meinke, K.: Automated black-box testing of functional correctness using function approximation. In: Rothermel, G. (ed) Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2004, pp. 143–153. ACM Press (2004). Softw. Eng. Not. 29(4)

    Google Scholar 

  28. Meinke, K.: CGE: a sequential learning algorithm for Mealy automata. In: Sempere, J.M., García, P. (eds.) ICGI 2010. LNCS (LNAI), vol. 6339, pp. 148–162. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15488-1_13

    Chapter  Google Scholar 

  29. Meinke, K., Niu, F.: A learning-based approach to unit testing of numerical software. In: Petrenko, A., Simão, A., Maldonado, J.C. (eds.) ICTSS 2010. LNCS, vol. 6435, pp. 221–235. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-16573-3_16

    Chapter  Google Scholar 

  30. Meinke, K., Niu, F.: Learning-based testing for reactive systems using term rewriting technology. In: Wolff, B., Zaïdi, F. (eds.) ICTSS 2011. LNCS, vol. 7019, pp. 97–114. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24580-0_8

    Chapter  Google Scholar 

  31. Meinke, K., Sindhu, M.A.: Incremental learning-based testing for reactive systems. In: Gogolla, M., Wolff, B. (eds.) TAP 2011. LNCS, vol. 6706, pp. 134–151. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21768-5_11

    Chapter  Google Scholar 

  32. Meinke, K., Niu, F., Sindhu, M.: Learning-based software testing: a tutorial. In: Hähnle, R., Knoop, J., Margaria, T., Schreiner, D., Steffen, B. (eds.) ISoLA 2011. CCIS, pp. 200–219. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-34781-8_16

    Chapter  Google Scholar 

  33. Meinke, K., Niu, F.: An incremental learning algorithm for hybrid automata. Internal report, School of Computer Science and Communication, KTH Royal Institute of Technology (2013)

    Google Scholar 

  34. Meinke, K., Sindhu, M.: LBTest: a learning-based testing tool for reactive systems. In: Proceedings of the Sixth International IEEE Conference on Software Testing, Verification and Validation (ICST 2013), pp. 447–454 (2013)

    Google Scholar 

  35. Meinke, K., Nycander, P.: Learning-based testing of distributed microservice architectures: correctness and fault injection. In: Bianculli, D., Calinescu, R., Rumpe, B. (eds.) SEFM 2015. LNCS, vol. 9509, pp. 3–10. Springer, Heidelberg (2015). https://doi.org/10.1007/978-3-662-49224-6_1

    Chapter  Google Scholar 

  36. Meinke, K.: LBTest user manual. Internal report, School of Computer Science and Communication, KTH Royal Institute of Technology (2016)

    Google Scholar 

  37. Khosrowjerdi, H., Meinke, K., Rasmusson, A.: Learning-based testing for safety critical automotive applications. In: Bozzano, M., Papadopoulos, Y. (eds.) IMBSA 2017. LNCS, vol. 10437, pp. 197–211. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-64119-5_13

    Chapter  Google Scholar 

  38. Meinke, K.: Learning-based testing of cyber-physical systems-of-systems: a platooning study. In: Reinecke, P., Di Marco, A. (eds.) EPEW 2017. LNCS, vol. 10497, pp. 135–151. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66583-2_9

    Chapter  Google Scholar 

  39. Peled, D., Vardi, M.Y., Yannakakis, M.: Black-box checking. In: Wu, J., et al. (eds.) Formal Methods for Protocol Engineering and Distributed Systems. FORTE/PSTV, pp. 225–240. Kluwer, Beijing (1999)

    Chapter  Google Scholar 

  40. Raffelt, H., Steffen, B., Margaria, T.: Dynamic testing via automata learning. In: Yorav, K. (ed.) HVC 2007. LNCS, vol. 4899, pp. 136–152. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-77966-7_13

    Chapter  Google Scholar 

  41. Raffelt, H., Merten, M., Steffen, B., Margaria, T.: Dynamic testing via automata learning. STTT 11(4), 307–324 (2009)

    Article  Google Scholar 

  42. Romanik, K.: Approximate testing and its relationship to learning. Theor. Comp. Sci. 188, 79–99 (1997)

    Article  MathSciNet  Google Scholar 

  43. Romanik, K., Vitter, J.S.: Using Vapnik-Chervonenkis dimension to analyze the testing complexity of program, segments. Inf. Comput. 128(2), 87–108 (1996)

    Article  MathSciNet  Google Scholar 

  44. Dupont, P.: StaMiNa regular inference competition (2010). www.stamina.chefbe.net

  45. Shahbaz, M., Groz, R.: Analysis and testing of black-box component-based systems by inferring partial models. Softw. Test. Verif. Reliab. 24, 253–288 (2014)

    Article  Google Scholar 

  46. Vasilewskii, M.P.: Failure diagnosis of automata. Kibertetika 9(4), 98–108 (1973)

    Google Scholar 

  47. Verwer, S., de Weerdt, M., Witteveen, C.: A likelihood-ratio test for identifying probabilistic deterministic real-time automata from positive data. In: Sempere, J.M., García, P. (eds.) ICGI 2010. LNCS (LNAI), vol. 6339, pp. 203–216. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15488-1_17

    Chapter  Google Scholar 

  48. Walkinshaw, N., Bogdanov, K., Derrick, J., Paris, J.: Increasing functional coverage by inductive testing: a case study. In: Petrenko, A., Simão, A., Maldonado, J.C. (eds.) ICTSS 2010. LNCS, vol. 6435, pp. 126–141. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-16573-3_10

    Chapter  Google Scholar 

  49. Weyuker, E.: Assessing test data adequacy through program inference. ACM Trans. Program. Lang. Syst. 5(4), 641–655 (1983)

    Article  Google Scholar 

  50. Whalen, M., Rajan, A., Heimdahl, M., Miller, S.: Coverage metrics for requirements-based testing. In: Proceedings of the International Symposium on Software Testing and Analysis, ISSTA 2006, pp. 25–35. ACM Press (2006)

    Google Scholar 

  51. Zhu, H., Hall, P., May, J.: Inductive inference and software testing. J. Softw. Test. Verif. Reliab. 2(2), 3–31 (1992)

    Google Scholar 

  52. Zhu, H.: A formal interpretation of software testing as inductive inference. J. Softw. Test. Verif. Reliab. 6(1), 3–31 (1996)

    Article  Google Scholar 

  53. Khosrowjerdi, H., Meinke, K.: Learning-based testing for autonomous systems using spatial and temporal requirements. Technical report, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology (2018)

    Google Scholar 

Download references

Acknowledgements

We wish to acknowledge the financial support of several research grants and agencies that allowed us to reach this point in our research. These include: EU project FP7-231620 HATS, ARTEMIS project JU 269335 MBAT, VINNOVA project 2013-05608 VIRTUES, EU ECSEL project 692529 SafeCOP and ITEA 3 project 16032 TESTOMAT.

Author information

Authors and Affiliations

  1. School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, 100-44, Stockholm, Sweden

    Karl Meinke

Authors
  1. Karl Meinke
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Karl Meinke .

Editor information

Editors and Affiliations

  1. The Open University, Milton Keynes, United Kingdom

    Amel Bennaceur

  2. Technische Universität Darmstadt, Darmstadt, Germany

    Reiner Hähnle

  3. KTH Royal Institute of Technology, Stockholm, Sweden

    Karl Meinke

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Meinke, K. (2018). Learning-Based Testing: Recent Progress and Future Prospects. In: Bennaceur, A., Hähnle, R., Meinke, K. (eds) Machine Learning for Dynamic Software Analysis: Potentials and Limits. Lecture Notes in Computer Science(), vol 11026. Springer, Cham. https://doi.org/10.1007/978-3-319-96562-8_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-96562-8_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-96561-1

  • Online ISBN: 978-3-319-96562-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation