Skip to main content
SpringerLink
Log in

Does the Performance of TDD Hold Across Software Companies and Premises? A Group of Industrial Experiments on TDD

  • Conference paper
  • First Online:
Product-Focused Software Process Improvement (PROFES 2018)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 11271))

Abstract

Test-Driven Development (TDD) has been claimed to increase external software quality. However, the extent to which TDD increases external quality has been seldom studied in industrial experiments. We conduct four industrial experiments in two different companies to evaluate the performance of TDD on external quality. We study whether the performance of TDD holds across the premises of the same company and across companies. We identify participant-level characteristics impacting results. Iterative-Test Last (ITL), the reverse approach of TDD, outperforms TDD in three out of four premises. ITL outperforms TDD in both companies. The larger the experience with unit testing and testing tools, the larger the difference in performance between ITL and TDD (in favour of ITL). Technological environment (i.e., programming language and testing tool) seems not to impact results. Evaluating participant-level characteristics impacting results in industrial experiments may ease the understanding of TDD’s performance in realistic settings.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For simplicity’s sake, along the rest of the article we refer to external quality and quality interchangeably.

  2. 2.

    The survey and its results were published elsewhere [36].

  3. 3.

    For simplicity’s sake, we consider the variables measured along the survey as continuous. This approach is commonly followed in other disciplines [37].

  4. 4.

    The normality assumption is met in all experiments according to the Shapiro-Wilk test [38].

  5. 5.

    The regression lines for “Programming” and “Language” partially overlap and thus, only that of “Language” is visible.

References

  1. Beck, K.: Test-Driven Development: By Example. Addison-Wesley Professional (2003)

    Google Scholar 

  2. ISO: ISO/IEC 25010:2011 (2011)

    Google Scholar 

  3. Munir, H., Moayyed, M., Petersen, K.: Considering rigor and relevance when evaluating test driven development: a systematic review. Inf. Softw. Technol. 56(4), 375–394 (2014)

    Article  Google Scholar 

  4. Rafique, Y., Mišić, V.B.: The effects of test-driven development on external quality and productivity: a meta-analysis. IEEE Trans. Softw. Eng. 39(6), 835–856 (2013)

    Article  Google Scholar 

  5. Bissi, W., Neto, A.G.S.S., Emer, M.C.F.P.: The effects of test driven development on internal quality, external quality and productivity: a systematic review. Inf. Softw. Technol. 74, 45–54 (2016)

    Article  Google Scholar 

  6. Kollanus, S.: Test-driven development-still a promising approach? In: Seventh International Conference on the Quality of Information and Communications Technology (QUATIC), pp. 403–408. IEEE (2010)

    Google Scholar 

  7. Shull, F., Melnik, G., Turhan, B., Layman, L., Diep, M., Erdogmus, H.: What do we know about test-driven development? IEEE Softw. 27(6), 16–19 (2010)

    Article  Google Scholar 

  8. Causevic, A., Sundmark, D., Punnekkat, S.: Factors limiting industrial adoption of test driven development: a systematic review. In: IEEE Fourth International Conference on Software Testing, Verification and Validation (ICST), pp. 337–346. IEEE (2011)

    Google Scholar 

  9. Mäkinen, S., Münch, J.: Effects of test-driven development: a comparative analysis of empirical studies. In: Winkler, D., Biffl, S., Bergsmann, J. (eds.) SWQD 2014. LNBIP, vol. 166, pp. 155–169. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-03602-1_10

    Chapter  Google Scholar 

  10. McDaid, K., Rust, A., Bishop, B.: Test-driven development: can it work for spreadsheets? In: Proceedings of the 4th International Workshop on End-User Software Engineering, pp. 25–29. ACM (2008)

    Google Scholar 

  11. Maximilien, E.M., Williams, L.: Assessing test-driven development at IBM. In: Proceedings of the 25th International Conference on Software Engineering, pp. 564–569. IEEE (2003)

    Google Scholar 

  12. Sjoberg, D.I., et al.: Conducting realistic experiments in software engineering. In: Proceedings of the International Symposium on Empirical Software Engineering, pp. 17–26. IEEE (2002)

    Google Scholar 

  13. Sjøberg, D.I.K., et al.: Challenges and recommendations when increasing the realism of controlled software engineering experiments. In: Conradi, R., Wang, A.I. (eds.) Empirical Methods and Studies in Software Engineering. LNCS, vol. 2765, pp. 24–38. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-540-45143-3_3

    Chapter  Google Scholar 

  14. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29044-2

    Book  MATH  Google Scholar 

  15. George, B., Williams, L.: A structured experiment of test-driven development. Inf. Softw. Technol. 46(5), 337–342 (2004)

    Article  Google Scholar 

  16. Geras, A., Smith, M., Miller, J.: A prototype empirical evaluation of test driven development. In: Proceedings of the 10th International Symposium on Software Metrics, pp. 405–416. IEEE (2004)

    Google Scholar 

  17. Tosun, A., et al.: An industry experiment on the effects of test-driven development on external quality and productivity. Empir. Softw. Eng. 22(6), 2763–2805 (2017)

    Article  Google Scholar 

  18. Borenstein, M., Hedges, L.V., Higgins, J.P., Rothstein, H.R.: Introduction to Meta-analysis. Wiley, Chichester (2011)

    MATH  Google Scholar 

  19. Bannerman, S., Martin, A.: A multiple comparative study of test-with development product changes and their effects on team speed and product quality. Empir. Softw. Eng. 16(2), 177–210 (2011)

    Article  Google Scholar 

  20. Siniaalto, M., Abrahamsson, P.: Does test-driven development improve the program code? Alarming results from a comparative case study. In: Meyer, B., Nawrocki, J.R., Walter, B. (eds.) CEE-SET 2007. LNCS, vol. 5082, pp. 143–156. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-85279-7_12

    Chapter  Google Scholar 

  21. Bhat, T., Nagappan, N.: Evaluating the efficacy of test-driven development: industrial case studies. In: Proceedings of the 2006 ACM/IEEE International Symposium on Empirical Software Engineering, pp. 356–363. ACM (2006)

    Google Scholar 

  22. Ynchausti, R.A.: Integrating unit testing into a software development teams process. In: XP 2001, pp. 84–87 (2001)

    Google Scholar 

  23. Damm, L.-O., Lundberg, L.: Quality impact of introducing component-level test automation and test-driven development. In: Abrahamsson, P., Baddoo, N., Margaria, T., Messnarz, R. (eds.) EuroSPI 2007. LNCS, vol. 4764, pp. 187–199. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-75381-0_17

    Chapter  Google Scholar 

  24. Nagappan, N., Maximilien, E.M., Bhat, T., Williams, L.: Realizing quality improvement through test driven development: results and experiences of four industrial teams. Empir. Softw. Eng. 13(3), 289–302 (2008)

    Article  Google Scholar 

  25. Damm, L.O., Lundberg, L.: Results from introducing component-level test automation and test-driven development. J. Syst. Softw. 79(7), 1001–1014 (2006)

    Article  Google Scholar 

  26. Lui, K.M., Chan, K.C.C.: Test driven development and software process improvement in China. In: Eckstein, J., Baumeister, H. (eds.) XP 2004. LNCS, vol. 3092, pp. 219–222. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24853-8_27

    Chapter  Google Scholar 

  27. Williams, L., Maximilien, E.M., Vouk, M.: Test-driven development as a defect-reduction practice. In: 14th International Symposium on Software Reliability Engineering, ISSRE 2003, pp. 34–45. IEEE (2003)

    Google Scholar 

  28. Dogša, T., Batič, D.: The effectiveness of test-driven development: an industrial case study. Softw. Qual. J. 19(4), 643–661 (2011)

    Article  Google Scholar 

  29. Slyngstad, O.P.N., Li, J., Conradi, R., Rønneberg, H., Landre, E., Wesenberg, H.: The impact of test driven development on the evolution of a reusable framework of components-an industrial case study. In: The Third International Conference on Software Engineering Advances, ICSEA 2008, pp. 214–223. IEEE (2008)

    Google Scholar 

  30. Aniche, M.F., Gerosa, M.A.: Most common mistakes in test-driven development practice: results from an online survey with developers. In: Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), pp. 469–478. IEEE (2010)

    Google Scholar 

  31. Sanchez, J.C., Williams, L., Maximilien, E.M.: On the sustained use of a test-driven development practice at IBM. In: Agile Conference (AGILE), pp. 5–14. IEEE (2007)

    Google Scholar 

  32. Dybå, T., Dingsøyr, T.: Strength of evidence in systematic reviews in software engineering. In: Proceedings of the Second ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 178–187. ACM (2008)

    Google Scholar 

  33. Guyatt, G.H., et al.: Grade: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ (Clinical research ed.) 336(7650), 924–926 (2008)

    Article  Google Scholar 

  34. Gomez, O.S., Juristo, N., Vegas, S.: Understanding replication of experiments in software engineering: a classification. Inf. Softw. Technol. 56(8), 1033–1048 (2014)

    Article  Google Scholar 

  35. Erdogmus, H., Morisio, M., Torchiano, M.: On the effectiveness of the test-first approach to programming. IEEE Trans. Softw. Eng. 31(3), 226–237 (2005)

    Article  Google Scholar 

  36. Dieste, O., Aranda, A.M., Uyaguari, F., Turhan, B., Tosun, A., Fucci, D., Oivo, M., Juristo, N.: Empirical evaluation of the effects of experience on code quality and programmer productivity: an exploratory study. Empir. Softw. Eng. 22(5), 2457–2542 (2017)

    Article  Google Scholar 

  37. Norman, G.: Likert scales, levels of measurement and the laws of statistics. Adv. Health Sci. Educ. 15(5), 625–632 (2010)

    Article  Google Scholar 

  38. Field, A.: Discovering Statistics Using IBM SPSS Statistics. Sage (2013)

    Google Scholar 

  39. Burke, D.L., Ensor, J., Riley, R.D.: Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ. Stat. Med. 36(5), 855–875 (2017)

    Article  MathSciNet  Google Scholar 

  40. Fisher, D., Copas, A., Tierney, J., Parmar, M.: A critical review of methods for the assessment of patient-level interactions in individual participant data meta-analysis of randomized trials, and guidance for practitioners. J. Clin. Epidemiol. 64(9), 949–967 (2011)

    Article  Google Scholar 

  41. Briand, L., Bianculli, D., Nejati, S., Pastore, F., Sabetzadeh, M.: The case for context-driven software engineering research: generalizability is overrated. IEEE Softw. 34(5), 72–75 (2017)

    Article  Google Scholar 

  42. Kraemer, H.C.: Pitfalls of multisite randomized clinical trials of efficacy and effectiveness. Schizophr. Bull. 26(3), 533 (2000)

    Article  Google Scholar 

  43. Cooper, H., Patall, E.A.: The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. Psychol. Methods 14(2), 165 (2009)

    Article  Google Scholar 

  44. Offutt, J.: Why don’t we publish more TDD research papers? Softw. Test. Verif. Reliab. 28(4), e1670 (2018)

    Article  Google Scholar 

Download references

Acknowledgments

This research was developed with the support of the Spanish Ministry of Science and Innovation project TIN2014-60490-P.

Author information

Authors and Affiliations

  1. M3S (M-Group), ITEE University of Oulu, Oulu, Finland

    Adrian Santos & Markku Oivo

  2. F-Secure, Oulu, Finland

    Janne Järvinen

  3. Bittium, Oulu, Finland

    Jari Partanen

  4. Escuela Técnica Superior de Ingenieros Informáticos, Universidad Politécnica de Madrid, Madrid, Spain

    Natalia Juristo

Authors
  1. Adrian Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Janne Järvinen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jari Partanen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Markku Oivo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Natalia Juristo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adrian Santos .

Editor information

Editors and Affiliations

  1. Technische Universität Clausthal, Goslar, Germany

    Marco Kuhrmann

  2. Leibniz University Hannover, Hannover, Germany

    Kurt Schneider

  3. University of Tartu, Tartu, Estonia

    Dietmar Pfahl

  4. Okayama Prefectural University, Okayama, Japan

    Sousuke Amasaki

  5. QAware GmbH, Munich, Germany

    Marcus Ciolkowski

  6. Chalmers | University of Gothenburg, Gothenburg, Sweden

    Regina Hebig

  7. IT University of Copenhagen, Copenhagen, Denmark

    Paolo Tell

  8. Leibniz University Hannover, Hannover, Germany

    Jil Klünder

  9. Clausthal University of Technology, Goslar, Germany

    Steffen Küpper

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Santos, A., Järvinen, J., Partanen, J., Oivo, M., Juristo, N. (2018). Does the Performance of TDD Hold Across Software Companies and Premises? A Group of Industrial Experiments on TDD. In: Kuhrmann, M., et al. Product-Focused Software Process Improvement. PROFES 2018. Lecture Notes in Computer Science(), vol 11271. Springer, Cham. https://doi.org/10.1007/978-3-030-03673-7_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-03673-7_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-03672-0

  • Online ISBN: 978-3-030-03673-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation