Abstract
This study presents an experimental study comparing the testing quality of a Hybrid Testing (HT) process with the commonly used approaches in industry: Scripted Testing (ST) and Exploratory Testing (ET). The study was conducted in an international IT service company in Sweden with the involvement of six experienced testers. Two measures were used for comparison: 1) defect detection effectiveness (DDE) and 2) functionality coverage (FC). The results indicated that HT performed better in terms of DDE than ST and worse than ET. In terms of FC, HT performed better than ET, while no significant differences were observed between the HT and ST. Furthermore, HT performed best for experienced testers, but worse with less experienced testers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
ISO/IEC: 29119-2: Test processes, international software testing standard. Technical report (2013)
Agruss, C., Johnson, B.: Ad hoc software testing, a perspective on exploration and improvisation. Technical report, Florida Institute of Technology, USA (April 2000)
Itkonen, J., Mäntylä, M., Lassenius, C.: How do testers do it? an exploratory study on manual testing practices. In: Proceedings of the Third International Symposium on Empirical Software Engineering and Measurement (ESEM 2009), pp. 494–497 (2009)
Ahonen, J.J., Junttila, T., Sakkinen, M.: Impacts of the organizational model on testing: Three industrial cases. Empirical Software Engineering 9(4), 275–296 (2004)
Andersson, C., Runeson, P.: Verification and validation in industry – a qualitative survey on the state of practice. In: International Symposium on Empirical Software Engineering (ISESE 2002), pp. 37–47 (2002)
Itkonen, J., Rautiainen, K.: Exploratory testing: a multiple case study. In: International Symposium on Empirical Software Engineering (ISESE 2005), pp. 84–93 (2005)
Itkonen, J.: Do test cases really matter? An experiment comparing test case based and exploratory testing. PhD thesis, Helsinki University of Technology, Finland (2008)
Kaner, C., Falk, J., Nguyen, H.Q.: Testing computer software, 2nd edn. Van Nostrand Reinhold, New York (1993)
Crispin, L., Gregory, J.: Agile Testing: A Practical Guide for Testers and Agile Teams. Addison-Wesley (2009)
Itkonen, J., Rautiainen, K., Lassenius, C.: Toward an understanding of quality assurance in agile software development. International Journal of Agile Manufacturing 8(2), 39–49 (2005)
Bourque, P., Dupuis, R.: Guide to the software engineering body of knowledge (swebok). Technical report. IEEE Computer Society, Los Alamitos, California (2004)
Bach, J.: Exploratory testing. In: Veenendal, E.V. (ed.) The Testing Practitioner. UTN Publishers (2005)
Bach, J.: Session-based test management. Software Testing and Quality Engineering Magazine 2 (2000)
Shoaib, L., Nadeem, A., Akbar, A.: An empirical evaluation of the influence of human personality on exploratory software testing. In: Proceedings of the IEEE 13th International Multitopic Conference (INMIC 2009), pp. 1–6 (2009)
Copeland, L.: A practitioner’s guide to software test design. Artech House, Boston (2004)
Itkonen, J., Mäntylä, M., Lassenius, C.: Defect detection efficiency: Test case based vs. exploratory testing. In: Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007), pp. 61–70 (2007)
Shah, S.M.A., Gencel, C., Alvi, U.S., Petersen, K.: Towards a hybrid testing process unifying exploratory testing and scripted testing. Journal of Software Maintenance and Evolution: Research and Practice (2013)
Tahat, L.H., Bader, A., Vaysburg, B., Korel, B.: Requirement-based automated black-box test generation. In: Proceedings of the 25th International Computer Software and Applications Conference (COMPSAC 2001), pp. 489–495 (2001)
Basili, V.R., Caldiera, G., Rombach, H.D.: The goal question metric approach. Encyclopedia of Software Engineering 2, 528–532 (1994)
Wohlin, C.: Experimentation in software engineering: an introduction. Kluwer, Boston (2000)
Dybå, T., Kampenes, V.B., Sjøberg, D.I.K.: A systematic review of statistical power in software engineering experiments. Information & Software Technology 48(8), 745–755 (2006)
David, H.A.: Upper 5 and 1% points of the maximum f-ratio. Biometrika 39(3), 422–424 (1952)
Cohen, J.: Statistical power analysis for the behavioral sciences, 2nd edn. L. Erlbaum Associates, Hillsdale (1988)
Juristo, N., Moreno, A.M., Vegas, S.: Reviewing 25 years of testing technique experiments. Empirical Software Engineering 9(1-2), 7–44 (2004)
Sim, S.E., Ratanotayanon, S., Aiyelokun, O., Morris, E.: An initial study to develop an empirical test for software engineering expertise. Institute for Software Research, University of California, Irvine, CA, USA, Technical Report# UCI-ISR-06-6 (2006)
Petersen, K., Wohlin, C.: A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case. Journal of Systems and Software 82(9), 1479–1490 (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Shah, S.M.A., Alvi, U.S., Gencel, C., Petersen, K. (2014). Comparing a Hybrid Testing Process with Scripted and Exploratory Testing: An Experimental Study with Practitioners. In: Cantone, G., Marchesi, M. (eds) Agile Processes in Software Engineering and Extreme Programming. XP 2014. Lecture Notes in Business Information Processing, vol 179. Springer, Cham. https://doi.org/10.1007/978-3-319-06862-6_13
Download citation
DOI: https://doi.org/10.1007/978-3-319-06862-6_13
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-06861-9
Online ISBN: 978-3-319-06862-6
eBook Packages: Computer ScienceComputer Science (R0)