Skip to main content
SpringerLink
Log in

Statistical Model Checking of Response Times for Different System Deployments

  • Conference paper
  • First Online:
Book cover Dependable Software Engineering. Theories, Tools, and Applications (SETTA 2018)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 10998))

Abstract

Performance testing is becoming increasingly important for interactive systems. Evaluating their performance with respect to user expectations is complex, especially for different system deployments. Various load-testing approaches and performance-simulation methods aim at such analyses. However, these techniques have certain disadvantages, like a high testing effort for load testing, and a questionable model accuracy for simulation methods. Hence, we propose a combination of both techniques. We apply statistical model checking with a learned timed model and evaluate the results on the real system with hypothesis testing. Moreover, we check the established hypotheses of a reference system on various system deployments (configurations), like different hardware or network settings, and analyse the influence on the performance. Our method is realised with a property-based testing tool that is extended with algorithms from statistical model checking. We illustrate the feasibility of our technique with an industrial case study of a web application.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://pypi.python.org/pypi/hypothesis

  2. 2.

    https://fscheck.github.io/FsCheck

  3. 3.

    https://www.r-project.org

  4. 4.

    https://www.avl.com

References

  1. Agha, G., Palmskog, K.: A survey of statistical model checking. ACM Trans. Model. Comput. Simul. 28(1), 6:1–6:39 (2018)

    Article  MathSciNet  Google Scholar 

  2. Aichernig, B.K., Schumi, R.: Property-based testing with FsCheck by deriving properties from business rule models. In: ICSTW, pp. 219–228. IEEE (2016)

    Google Scholar 

  3. Aichernig, B.K., Schumi, R.: Towards integrating statistical model checking into property-based testing. In: MEMOCODE, pp. 71–76. IEEE (2016)

    Google Scholar 

  4. Aichernig, B.K., Schumi, R.: Property-based testing of web services by deriving properties from business-rule models. Softw. Syst. Model. 1–23 (2017). https://doi.org/10.1007/s10270-017-0647-0

  5. Aichernig, B.K., Schumi, R.: Statistical model checking meets property-based testing. In: ICST, pp. 390–400. IEEE (2017)

    Google Scholar 

  6. Alur, R., Dill, D.L.: A theory of timed automata. Theor. Comput. Sci. 126(2), 183–235 (1994)

    Article  MathSciNet  Google Scholar 

  7. Arts, T.: On shrinking randomly generated load tests. In: Erlang 2014, pp. 25–31. ACM (2014)

    Google Scholar 

  8. Ballarini, P., Bertrand, N., Horváth, A., Paolieri, M., Vicario, E.: Transient analysis of networks of stochastic timed automata using stochastic state classes. In: Joshi, K., Siegle, M., Stoelinga, M., D’Argenio, P.R. (eds.) QEST 2013. LNCS, vol. 8054, pp. 355–371. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40196-1_30

    Chapter  Google Scholar 

  9. Banga, G., Druschel, P.: Measuring the capacity of a web server under realistic loads. World Wide Web 2(1–2), 69–83 (1999)

    Article  Google Scholar 

  10. Becker, S., Koziolek, H., Reussner, R.H.: The Palladio component model for model-driven performance prediction. J. Syst. Softw. 82(1), 3–22 (2009)

    Article  Google Scholar 

  11. Blair, L., Jones, T., Blair, G.: Stochastically enhanced timed automata. In: Smith, S.F., Talcott, C.L. (eds.) FMOODS 2000. IAICT, vol. 49, pp. 327–347. Springer, Boston, MA (2000). https://doi.org/10.1007/978-0-387-35520-7_17

    Chapter  Google Scholar 

  12. Book, M., Gruhn, V., Hülder, M., Köhler, A., Kriegel, A.: Cost and response time simulation for web-based applications on mobile channels. In: QSIC, pp. 83–90. IEEE (2005)

    Google Scholar 

  13. Bulychev, P.E., et al.: UPPAAL-SMC: statistical model checking for priced timed automata. In: QAPL. EPTCS, vol. 85, pp. 1–16. Open Publishing Association (2012)

    Google Scholar 

  14. Chen, X., Mohapatra, P., Chen, H.: An admission control scheme for predictable server response time for web accesses. In: WWW, pp. 545–554. ACM (2001)

    Google Scholar 

  15. Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: ICFP, pp. 268–279. ACM (2000)

    Google Scholar 

  16. Claessen, K., et al.: Finding race conditions in Erlang with QuickCheck and PULSE. In: ICFP, pp. 149–160. ACM (2009)

    Google Scholar 

  17. Draheim, D., Grundy, J.C., Hosking, J.G., Lutteroth, C., Weber, G.: Realistic load testing of web applications. In: CSMR, pp. 57–70. IEEE (2006)

    Google Scholar 

  18. Govindarajulu, Z.: Sequential Statistics. World Scientific (2004)

    Google Scholar 

  19. Harel, D.: Statecharts: a visual formalism for complex systems. Sci. Comput. Program. 8(3), 231–274 (1987)

    Article  MathSciNet  Google Scholar 

  20. Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963)

    Article  MathSciNet  Google Scholar 

  21. Hughes, J.: QuickCheck testing for fun and profit. In: Hanus, M. (ed.) PADL 2007. LNCS, vol. 4354, pp. 1–32. Springer, Heidelberg (2006). https://doi.org/10.1007/978-3-540-69611-7_1

    Chapter  Google Scholar 

  22. Hughes, J., Pierce, B.C., Arts, T., Norell, U.: Mysteries of Dropbox: property-based testing of a distributed synchronization service. In: ICST, pp. 135–145. IEEE (2016)

    Google Scholar 

  23. Kalaji, A.S., Hierons, R.M., Swift, S.: Generating feasible transition paths for testing from an extended finite state machine. In: ICST, pp. 230–239. IEEE (2009)

    Google Scholar 

  24. Kwiatkowska, M., Norman, G., Segala, R., Sproston, J.: Verifying quantitative properties of continuous probabilistic timed automata. In: Palamidessi, C. (ed.) CONCUR 2000. LNCS, vol. 1877, pp. 123–137. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-44618-4_11

    Chapter  Google Scholar 

  25. Legay, A., Delahaye, B., Bensalem, S.: Statistical model checking: an overview. In: Barringer, H., et al. (eds.) RV 2010. LNCS, vol. 6418, pp. 122–135. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-16612-9_11

    Chapter  Google Scholar 

  26. Legay, A., Sedwards, S.: On statistical model checking with PLASMA. In: TASE, pp. 139–145. IEEE (2014)

    Google Scholar 

  27. Lu, Y., Nolte, T., Bate, I., Cucu-Grosjean, L.: A statistical response-time analysis of real-time embedded systems. In: RTSS, pp. 351–362. IEEE (2012)

    Google Scholar 

  28. Malik, H., Shakshuki, E.M.: Classification of post-deployment performance diagnostic techniques for large-scale software systems. Procedia Comput. Sci. 37, 244–251 (2014)

    Article  Google Scholar 

  29. Menascé, D.A.: Load testing of web sites. IEEE Internet Comput. 6(4), 70–74 (2002)

    Article  Google Scholar 

  30. Nilsson, R.: ScalaCheck: The Definitive Guide. IT Pro, Artima Incorporated (2014)

    Google Scholar 

  31. Norell, U., Svensson, H., Arts, T.: Testing blocking operations with QuickCheck’s component library. In: Erlang 2013, pp. 87–92. ACM (2013)

    Google Scholar 

  32. Nourikhah, H., Akbari, M.K., Kalantari, M.: Modeling and predicting measured response time of cloud-based web services using long-memory time series. J. Supercomput. 71(2), 673–696 (2015)

    Article  Google Scholar 

  33. Papadakis, M., Sagonas, K.: A PropEr integration of types and function specifications with property-based testing. In: Erlang 2011, pp. 39–50. ACM (2011)

    Google Scholar 

  34. Rencher, A., Christensen, W.: Methods of Multivariate Analysis. Wiley, Hoboken (2012)

    Book  Google Scholar 

  35. Rina, S.T.: A comparative study of performance testing tools. Int. J. Adv. Res. Comp. Sci. Softw. Eng. IJARCSSE 3(5), 1300–1307 (2013)

    Google Scholar 

  36. Roloff, E., Diener, M., Carissimi, A., Navaux, P.O.A.: High performance computing in the cloud: Deployment, performance and cost efficiency. In: CloudCom, pp. 371–378. IEEE Computer Society (2012)

    Google Scholar 

  37. Schumi, R., Lang, P., Aichernig, B.K., Krenn, W., Schlick, R.: Checking response-time properties of web-service applications under stochastic user profiles. In: Yevtushenko, N., Cavalli, A.R., Yenigün, H. (eds.) ICTSS 2017. LNCS, vol. 10533, pp. 293–310. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67549-7_18

    Chapter  Google Scholar 

  38. Wald, A.: Sequential Analysis. Courier Corporation (1973)

    Google Scholar 

  39. Yu, J., Han, J., Schneider, J., Hine, C.M., Versteeg, S.: A Petri-Net-based virtual deployment testing environment for enterprise software systems. Comput. J. 60(1), 27–44 (2017)

    Article  MathSciNet  Google Scholar 

  40. Zhang, F., et al.: Modeling and evaluation of wireless sensor network protocols by stochastic timed automata. Electr. Notes Theor. Comput. Sci. 296, 261–277 (2013)

    Article  Google Scholar 

Download references

Acknowledgements

This research was funded by the Austrian Research Promotion Agency (FFG), project TRUCONF, No. 845582. We are grateful to the project team and to the anonymous reviewers for their remarks.

Author information

Authors and Affiliations

  1. Institute of Software Technology, Graz University of Technology, Graz, Austria

    Bernhard K. Aichernig & Richard Schumi

  2. AVL List GmbH, Graz, Austria

    Severin Kann

Authors
  1. Bernhard K. Aichernig
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Severin Kann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Richard Schumi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richard Schumi .

Editor information

Editors and Affiliations

  1. Nanjing University, Nanjing, China

    Xinyu Feng

  2. Westfälische Wilhelms-Universität Münster, Münster, Germany

    Markus Müller-Olm

  3. Western Michigan University, Kalamazoo, Michigan, USA

    Zijiang Yang

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Aichernig, B.K., Kann, S., Schumi, R. (2018). Statistical Model Checking of Response Times for Different System Deployments. In: Feng, X., Müller-Olm, M., Yang, Z. (eds) Dependable Software Engineering. Theories, Tools, and Applications. SETTA 2018. Lecture Notes in Computer Science(), vol 10998. Springer, Cham. https://doi.org/10.1007/978-3-319-99933-3_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-99933-3_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-99932-6

  • Online ISBN: 978-3-319-99933-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation