Comparing a Hybrid Testing Process with Scripted and Exploratory Testing: An Experimental Study with Practitioners

  • Syed Muhammad Ali Shah
  • Usman Sattar Alvi
  • Cigdem Gencel
  • Kai Petersen
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 179)


This study presents an experimental study comparing the testing quality of a Hybrid Testing (HT) process with the commonly used approaches in industry: Scripted Testing (ST) and Exploratory Testing (ET). The study was conducted in an international IT service company in Sweden with the involvement of six experienced testers. Two measures were used for comparison: 1) defect detection effectiveness (DDE) and 2) functionality coverage (FC). The results indicated that HT performed better in terms of DDE than ST and worse than ET. In terms of FC, HT performed better than ET, while no significant differences were observed between the HT and ST. Furthermore, HT performed best for experienced testers, but worse with less experienced testers.


Exploratory Testing Scripted Testing Hybrid Testing Experiment Industrial 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    ISO/IEC: 29119-2: Test processes, international software testing standard. Technical report (2013)Google Scholar
  2. 2.
    Agruss, C., Johnson, B.: Ad hoc software testing, a perspective on exploration and improvisation. Technical report, Florida Institute of Technology, USA (April 2000)Google Scholar
  3. 3.
    Itkonen, J., Mäntylä, M., Lassenius, C.: How do testers do it? an exploratory study on manual testing practices. In: Proceedings of the Third International Symposium on Empirical Software Engineering and Measurement (ESEM 2009), pp. 494–497 (2009)Google Scholar
  4. 4.
    Ahonen, J.J., Junttila, T., Sakkinen, M.: Impacts of the organizational model on testing: Three industrial cases. Empirical Software Engineering 9(4), 275–296 (2004)CrossRefGoogle Scholar
  5. 5.
    Andersson, C., Runeson, P.: Verification and validation in industry – a qualitative survey on the state of practice. In: International Symposium on Empirical Software Engineering (ISESE 2002), pp. 37–47 (2002)Google Scholar
  6. 6.
    Itkonen, J., Rautiainen, K.: Exploratory testing: a multiple case study. In: International Symposium on Empirical Software Engineering (ISESE 2005), pp. 84–93 (2005)Google Scholar
  7. 7.
    Itkonen, J.: Do test cases really matter? An experiment comparing test case based and exploratory testing. PhD thesis, Helsinki University of Technology, Finland (2008)Google Scholar
  8. 8.
    Kaner, C., Falk, J., Nguyen, H.Q.: Testing computer software, 2nd edn. Van Nostrand Reinhold, New York (1993)zbMATHGoogle Scholar
  9. 9.
    Crispin, L., Gregory, J.: Agile Testing: A Practical Guide for Testers and Agile Teams. Addison-Wesley (2009)Google Scholar
  10. 10.
    Itkonen, J., Rautiainen, K., Lassenius, C.: Toward an understanding of quality assurance in agile software development. International Journal of Agile Manufacturing 8(2), 39–49 (2005)Google Scholar
  11. 11.
    Bourque, P., Dupuis, R.: Guide to the software engineering body of knowledge (swebok). Technical report. IEEE Computer Society, Los Alamitos, California (2004)Google Scholar
  12. 12.
    Bach, J.: Exploratory testing. In: Veenendal, E.V. (ed.) The Testing Practitioner. UTN Publishers (2005)Google Scholar
  13. 13.
    Bach, J.: Session-based test management. Software Testing and Quality Engineering Magazine 2 (2000)Google Scholar
  14. 14.
    Shoaib, L., Nadeem, A., Akbar, A.: An empirical evaluation of the influence of human personality on exploratory software testing. In: Proceedings of the IEEE 13th International Multitopic Conference (INMIC 2009), pp. 1–6 (2009)Google Scholar
  15. 15.
    Copeland, L.: A practitioner’s guide to software test design. Artech House, Boston (2004)Google Scholar
  16. 16.
    Itkonen, J., Mäntylä, M., Lassenius, C.: Defect detection efficiency: Test case based vs. exploratory testing. In: Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007), pp. 61–70 (2007)Google Scholar
  17. 17.
    Shah, S.M.A., Gencel, C., Alvi, U.S., Petersen, K.: Towards a hybrid testing process unifying exploratory testing and scripted testing. Journal of Software Maintenance and Evolution: Research and Practice (2013)Google Scholar
  18. 18.
    Tahat, L.H., Bader, A., Vaysburg, B., Korel, B.: Requirement-based automated black-box test generation. In: Proceedings of the 25th International Computer Software and Applications Conference (COMPSAC 2001), pp. 489–495 (2001)Google Scholar
  19. 19.
    Basili, V.R., Caldiera, G., Rombach, H.D.: The goal question metric approach. Encyclopedia of Software Engineering 2, 528–532 (1994)Google Scholar
  20. 20.
    Wohlin, C.: Experimentation in software engineering: an introduction. Kluwer, Boston (2000)CrossRefzbMATHGoogle Scholar
  21. 21.
    Dybå, T., Kampenes, V.B., Sjøberg, D.I.K.: A systematic review of statistical power in software engineering experiments. Information & Software Technology 48(8), 745–755 (2006)CrossRefGoogle Scholar
  22. 22.
    David, H.A.: Upper 5 and 1% points of the maximum f-ratio. Biometrika 39(3), 422–424 (1952)CrossRefzbMATHMathSciNetGoogle Scholar
  23. 23.
    Cohen, J.: Statistical power analysis for the behavioral sciences, 2nd edn. L. Erlbaum Associates, Hillsdale (1988)zbMATHGoogle Scholar
  24. 24.
    Juristo, N., Moreno, A.M., Vegas, S.: Reviewing 25 years of testing technique experiments. Empirical Software Engineering 9(1-2), 7–44 (2004)CrossRefGoogle Scholar
  25. 25.
    Sim, S.E., Ratanotayanon, S., Aiyelokun, O., Morris, E.: An initial study to develop an empirical test for software engineering expertise. Institute for Software Research, University of California, Irvine, CA, USA, Technical Report# UCI-ISR-06-6 (2006)Google Scholar
  26. 26.
    Petersen, K., Wohlin, C.: A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case. Journal of Systems and Software 82(9), 1479–1490 (2009)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Syed Muhammad Ali Shah
    • 1
  • Usman Sattar Alvi
    • 2
  • Cigdem Gencel
    • 3
  • Kai Petersen
    • 4
  1. 1.Politecnico di TorinoTorinoItaly
  2. 2.Seamless ABStockholmSweden
  3. 3.Free University of Bolzano-BozenBolzanoItaly
  4. 4.Blekinge Institute of TechnologyKarlskronaSweden

Personalised recommendations