Abstract
In our recent academic experiments, an existence of positive test bias, that is lack of negative test cases, was identified when a test driven development approach was used. At the same time, when defect detecting ability of individual test cases was calculated, it was noted that the probability of a negative test case to detect a defect was substantially higher than that of a positive test case.
The goal of this study is to investigate the existence of positive test bias in test driven development within an industrial context, and measure defect detecting ability of both positive and negative test cases. An industrial experiment was conducted at Infosys Ltd. India, whose employees voluntarily signed up to participate in the study and were randomly assigned to groups utilizing test driven development, test driven development with negative testing, and test last development. Source code and test cases created by each participant during the study were collected and analysed.
The collected data indicate a statistically significant difference between the number of positive and negative test cases created by industrial participants, confirming the existence of positive test bias. The difference in defect detecting ability of positive and negative test cases is also statistically significant. As a result, similarly to our previous academic study, 29% of all test cases were negative, contributing by revealing as much as 71% of all the defects found by all test cases. With this industrial experiment, we confirmed the existence of a positive test bias in an industrial context, as well as significantly higher defect detecting ability of negative test cases.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Beck, K.: Extreme programming explained: embrace change. Addison-Wesley Longman Publishing Co., Inc., Boston (2000)
Koskela, L.: Test driven: practical tdd and acceptance tdd for java developers. Manning Publications Co., Greenwich (2007)
Causevic, A., Sundmark, D., Punnekkat, S.: An industrial survey on contemporary aspects of software testing. In: Proceedings of the 3rd International Conference on Software Testing, Verification and Validation (ICST), pp. 393–401 (2010)
George, B., Williams, L.: A structured experiment of test-driven development. Information and Software Technology 46(5), 337–342 (2003)
Erdogmus, H., Morisio, M., Torchiano, M.: On the effectiveness of the test-first approach to programming. IEEE Transactions on Software Engineering 31, 226–237 (2005)
Janzen, D.S., Saiedian, H.: On the influence of test-driven development on software design. In: Conference on Software Engineering Education and Training, pp. 141–148 (2006)
Gupta, A., Jalote, P.: An experimental evaluation of the effectiveness and efficiency of the test driven development. In: Proceedings of the First International Symposium on Empirical Software Engineering and Measurement, ESEM 2007, pp. 285–294. IEEE Computer Society, Washington, DC (2007)
Vu, J.H., Frojd, N., Shenkel-Therolf, C., Janzen, D.S.: Evaluating test-driven development in an industry-sponsored capstone project. In: Proceedings of the 2009 Sixth International Conference on Information Technology: New Generations, pp. 229–234. IEEE Computer Society, Washington, DC (2009)
Causevic, A., Sundmark, D., Punnekkat, S.: Factors limiting industrial adoption of test driven development: A systematic review. In: 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation (ICST), pp. 337–346 (March 2011)
Causevic, A., Punnekkat, S., Sundmark, D.: Quality of testing in test driven development. In: 2012 Eight International Conference on the Quality of Information and Communications Technology (QUATIC) (September 2012)
Teasley, B.E., Leventhal, L.M., Mynatt, C.R., Rohlman, D.S.: Why Software Testing Is Sometimes Ineffective: Two Applied Studies of Positive Test Strategy. Journal of Applied Psychology 79(1), 142–155 (1994)
Leventhal, L.M., Teasley, B., Rohlman, D.S., Instone, K.: Positive Test Bias in Software Testing Among Professionals: A Review. In: Bass, L.J., Unger, C., Gornostaev, J. (eds.) EWHCI 1993. LNCS, vol. 753, pp. 210–218. Springer, Heidelberg (1993)
Madeyski, L.: The impact of test-first programming on branch coverage and mutation score indicator of unit tests: An experiment. Inf. Softw. Technol. 52, 169–184 (2010)
Shelton, W., Li, N., Ammann, P., Offutt, J.: Adding criteria-based tests to test driven development. In: Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, ICST 2012, pp. 878–886. IEEE Computer Society, Washington, DC (2012)
Fenton, N.E., Pfleeger, S.L.: Software Metrics: A Rigorous and Practical Approach, 2nd edn. PWS Publishing Co., Boston (1998)
Causevic, A., Sundmark, D., Punnekkat, S.: Test case quality in test driven development: A study design and a pilot experiment. In: 16th International Conference on Evaluation Assessment in Software Engineering (EASE 2012), pp. 223–227 (May 2012)
Eclipse, http://www.eclipse.org
jUnit Framework, http://www.junit.org
Kollanus, S., Isomöttönen, V.: Understanding tdd in academic environment: experiences from two experiments. In: Proceedings of the 8th International Conference on Computing Education Research, Koli 2008, pp. 25–31. ACM, New York (2008)
Flohr, T., Schneider, T.: Lessons learned from an XP experiment with students: Test-first needs more teachings. In: Münch, J., Vierimaa, M. (eds.) PROFES 2006. LNCS, vol. 4034, pp. 305–318. Springer, Heidelberg (2006)
R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2012) ISBN 3-900051-07-0
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Causevic, A., Shukla, R., Punnekkat, S., Sundmark, D. (2013). Effects of Negative Testing on TDD: An Industrial Experiment. In: Baumeister, H., Weber, B. (eds) Agile Processes in Software Engineering and Extreme Programming. XP 2013. Lecture Notes in Business Information Processing, vol 149. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-38314-4_7
Download citation
DOI: https://doi.org/10.1007/978-3-642-38314-4_7
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-38313-7
Online ISBN: 978-3-642-38314-4
eBook Packages: Computer ScienceComputer Science (R0)