Abstract
Including performance tests as a part of unit testing is technically more difficult than including functional tests – besides the usual challenges of performance measurement, specifying and testing the correctness conditions is also more complex. In earlier work, we have proposed a formalism for expressing these conditions, the Stochastic Performance Logic. In this paper, we evaluate our formalism in the context of performance unit testing of JDOM, an open source project for working with XML data. We focus on the ability to capture and test developer assumptions and on the practical behavior of the built in hypothesis testing when the formal assumptions of the tests are not met.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Bergmann, V.: ContiPerf 2 (2013), http://databene.org/contiperf.html
Bulej, L., Bures, T., Keznikl, J., Koubkova, A., Podzimek, A., Tuma, P.: Capturing Performance Assumptions using Stochastic Performance Logic. In: Proc. ICPE 2012. ACM (2012)
Clark, M.: JUnitPerf (2013), http://www.clarkware.com/software/JUnitPerf
Foo, K., Jiang, Z.M., Adams, B., Hassan, A., Zou, Y., Flora, P.: Mining performance regression testing repositories for automated performance analysis. In: Proc. QSIC 2010. IEEE (2010)
Ghaith, S., Wang, M., Perry, P., Murphy, J.: Profile-based, load-independent anomaly detection and analysis in performance regression testing of software systems. In: Proc. CSMR 2013. IEEE (2013)
Heger, C., Happe, J., Farahbod, R.: Automated root cause isolation of performance regressions during software development. In: Proc. ICPE 2013. ACM (2013)
JDOM (2013), http://www.jdom.org
hunterhacker/jdom [Git] (2013), https://github.com/hunterhacker/jdom
hunterhacker/jdom: Verifier performance (2013), https://github.com/hunterhacker/jdom/wiki/Verifier-Performance
JUnit (April 2013), http://junit.org
Kalibera, T., Bulej, L., Tůma, P.: Benchmark Precision and Random Initial State. In: Proc. SPECTS 2005. SCS (2005)
Kalibera, T., Tůma, P.: Precise Regression Benchmarking with Random Effects: Improving Mono Benchmark Results. In: Horváth, A., Telek, M. (eds.) EPEW 2006. LNCS, vol. 4054, pp. 63–77. Springer, Heidelberg (2006)
Oliveira, A., Petkovich, J.-C., Reidemeister, T., Fischmeister, S.: Datamill: Rigorous performance evaluation made easy. In: Proc. ICPE 2013. ACM (2013)
Porter, A., Yilmaz, C., Memon, A.M., Schmidt, D.C., Natarajan, B.: Skoll: A process and infrastructure for distributed continuous quality assurance. IEEE Trans. Softw. Eng. 33(8), 510–525 (2007)
Puchko, T.: Retrotranslator (2013), http://retrotranslator.sourceforge.net
SPL Tools (2013), http://d3s.mff.cuni.cz/software/spl-java
Welch, B.L.: The Generalization of Student’s Problem when Several Different Population Variances are Involved. Biometrika 34(1/2), 28–35 (1947)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Horký, V., Haas, F., Kotrč, J., Lacina, M., Tůma, P. (2013). Performance Regression Unit Testing: A Case Study. In: Balsamo, M.S., Knottenbelt, W.J., Marin, A. (eds) Computer Performance Engineering. EPEW 2013. Lecture Notes in Computer Science, vol 8168. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40725-3_12
Download citation
DOI: https://doi.org/10.1007/978-3-642-40725-3_12
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-40724-6
Online ISBN: 978-3-642-40725-3
eBook Packages: Computer ScienceComputer Science (R0)