Skip to main content
SpringerLink
Log in
Book cover

International Conference on Product-Focused Software Process Improvement

PROFES 2017: Product-Focused Software Process Improvement pp 111–127Cite as

The Choice of Code Review Process: A Survey on the State of the Practice

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 10611))

Abstract

Code review has been known to be an effective quality assurance technique for decades. In the last years, industrial code review practices were observed to converge towards “change-based/modern code review”, but with a lot of variation in the details of the processes. Recent research also proposed hypotheses on factors that influence the choice of process. However, all current research in this area is based on small and largely non-random samples of cases. Therefore, we set out to assess the current state of the practice and to test some of these hypotheses with a survey among commercial software development teams. We received responses from 240 teams. They support many of the stated hypotheses, e.g., that change-based code review is the dominating style of code review in the industry, and that teams doing change-based code review have a lower risk that review use fades away. However, other hypotheses could not be confirmed, mainly that the balance of effects a team tries to reach with code reviews acts as a mediator in determining the details of the review process. Apart from these findings, we contribute the survey data set as a foundation for future research.

This is a preview of subscription content, log in via an institution.

Notes

  1. 1.

    Also called “modern code review”, “differential code review” or “patch review” in other publications.

  2. 2.

    More precisely, 240 respondents answered at least the questions about being part of the target population and about their team’s review use, which were the only obligatory questions in the survey.

  3. 3.

    The remaining 15 did not answer this question. Unless otherwise noted, we only include respondents that answered the respective questions in our analyses; consequently, the total sum of respondents will differ between analyses.

  4. 4.

    This number is likely biased, see Sect. 5.

  5. 5.

    I.e. we did not check every possible combination, but only those where the prior research gave reason to expect an influence.

  6. 6.

    The numbers are not fully comparable: Rigby and Bird looked at the actual number of reviewers in a large sample of reviews, whereas we asked our participants for the usual number of reviewers in a review.

  7. 7.

    A weakness in the used questionnaire is that there was no explicit “We do not use any tool” choice available. Therefore, the distinction between non-response and non-use of tools cannot be reliably made.

References

  1. Asundi, J., Jayant, R.: Patch review processes in open source software development communities: a comparative case study. In: 40th Annual Hawaii International Conference on System Sciences, HICSS 2007, p. 166c. IEEE (2007)

    Google Scholar 

  2. Bacchelli, A., Bird, C.: Expectations, outcomes, and challenges of modern code review. In: Proceedings of the 2013 International Conference on Software Engineering, pp. 712–721. IEEE (2013)

    Google Scholar 

  3. Baker Jr., R.A.: Code reviews enhance software quality. In: Proceedings of the 19th International Conference on Software Engineering, pp. 570–571. ACM (1997)

    Google Scholar 

  4. Basili, V., Caldiera, G., Lanubile, F., Shull, F.: Studies on reading techniques. In: Proceedings of the Twenty-First Annual Software Engineering Workshop, vol. 96, p. 002 (1996)

    Google Scholar 

  5. Basili, V.R., Green, S., Laitenberger, O., Lanubile, F., Shull, F., Sørumgård, S., Zelkowitz, M.V.: The empirical investigation of perspective-based reading. Empirical Softw. Eng. 1(2), 133–164 (1996)

    Article  Google Scholar 

  6. Baum, T.: Detailed table with review effects (team level) and their connections to contextual factors and process variants for “factors influencing code review processes in industry” (2016). http://dx.doi.org/10.6084/m9.figshare.5104111

  7. Baum, T., Leßmann, H., Schneider, K.: Online material for survey on code review use. http://dx.doi.org/10.6084/m9.figshare.5104249

  8. Baum, T., Liskin, O., Niklas, K., Schneider, K.: A faceted classification scheme for change-based industrial code review processes. In: 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, Vienna (2016)

    Google Scholar 

  9. Baum, T., Liskin, O., Niklas, K., Schneider, K.: Factors influencing code review processes in industry. In: Proceedings of the ACM SIGSOFT 24th International Symposium on the Foundations of Software Engineering. ACM, Seattle (2016)

    Google Scholar 

  10. Baysal, O., Kononenko, O., Holmes, R., Godfrey, M.W.: Investigating technical and non-technical factors influencing modern code review. Empirical Softw. Eng. 21, 932–959 (2016)

    Article  Google Scholar 

  11. Bernhart, M., Strobl, S., Mauczka, A., Grechenig, T.: Applying continuous code reviews in airport operations software. In: 2012 12th International Conference on Quality Software (QSIC), pp. 214–219. IEEE (2012)

    Google Scholar 

  12. Bosu, A., Carver, J.C.: Impact of peer code review on peer impression formation: a survey. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 133–142. IEEE (2013)

    Google Scholar 

  13. Ciolkowski, M., Laitenberger, O., Biffl, S.: Software reviews: the state of the practice. IEEE Softw. 20(6), 46–51 (2003)

    Article  Google Scholar 

  14. Fagan, M.E.: Design and code inspections to reduce errors in program development. IBM Syst. J. 15(3), 182–211 (1976)

    Article  Google Scholar 

  15. Forrester Research Inc.: The value and importance of code reviews, March 2010. http://embedded-computing.com/white-papers/white-paper-value-importance-code-reviews/. Accessed 13 June 2017

  16. Gousios, G., Pinzger, M., Deursen, A.V.: An exploratory study of the pull-based software development model. In: Proceedings of the 36th International Conference on Software Engineering, pp. 345–355. ACM, Hyderabad (2014)

    Google Scholar 

  17. Harjumaa, L., Tervonen, I., Huttunen, A.: Peer reviews in real life-motivators and demotivators. In: Fifth International Conference on Quality Software (QSIC 2005). IEEE (2005)

    Google Scholar 

  18. Jacob, R., Heinz, A., Décieux, J.P.: Umfrage: Einführung in die Methoden der Umfrageforschung. Walter de Gruyter (2013)

    Google Scholar 

  19. Kollanus, S., Koskinen, J.: Software inspections in practice: six case studies. In: Münch, J., Vierimaa, M. (eds.) PROFES 2006. LNCS, vol. 4034, pp. 377–382. Springer, Heidelberg (2006). doi:10.1007/11767718_31

    Chapter  Google Scholar 

  20. Kuhrmann, M., Diebold, P., Münch, J., Tell, P., Garousi, V., Felderer, M., Trektere, K., Linssen, O., Hanser, E., Prause, C.R.: Hybrid software and system development in practice: waterfall, scrum, and beyond. In: ICSSP 2017 (2017)

    Google Scholar 

  21. Laitenberger, O., Vegas, S., Ciolkowski, M.: The state of the practice of review and review technologies in Germany. Tech. report 011.02, Virtual Softw. Eng. Competence Center (VISEK) (2002)

    Google Scholar 

  22. Laitenberger, O., DeBaud, J.M.: An encompassing life cycle centric survey of software inspection. J. Syst. Softw. 50(1), 5–31 (2000)

    Article  Google Scholar 

  23. March, J.G., Simon, H.A.: Organizations. Wiley, London (1958)

    Google Scholar 

  24. Rigby, P.C., Bird, C.: Convergent contemporary software peer review practices. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pp. 202–212. ACM, Saint Petersburg (2013)

    Google Scholar 

  25. Rigby, P.C., Storey, M.A.: Understanding broadcast based peer review on open source software projects. In: Proceedings of the 33rd International Conference on Software Engineering, pp. 541–550. ACM (2011)

    Google Scholar 

  26. SmartBear: The state of code quality 2016. https://smartbear.com/resources/ebooks/state-of-code-quality-2016/. Accessed 13 June 2017

  27. Sudman, S., Bradburn, N.M.: Asking Questions: A Practical Guide to Questionnaire Design. Jossey-Bass Publishers, San Francisco (1982)

    Google Scholar 

  28. Thongtanunam, P., McIntosh, S., Hassan, A.E., Iida, H.: Investigating code review practices in defective files: an empirical study of the QT system. In: Proceedings of the 12th Working Conference on Mining Software Repositories, MSR 2015 (2015)

    Google Scholar 

  29. Wang, J., Shih, P.C., Wu, Y., Carroll, J.M.: Comparative case studies of open source software peer review practices. Inf. Softw. Technol. 67, 1–12 (2015)

    Article  Google Scholar 

  30. Winter, M., Vosseberg, K., Spillner, A.: Umfrage 2016 “Softwaretest in Praxis und Forschung”. dpunkt.verlag (2016)

    Google Scholar 

Download references

Acknowledgments

The authors would like to thank all pre-testers and all participants of the survey for the time and effort they donated. We would further like to thank Philipp Diebold and Paul Clarke for providing questions from their surveys for reuse.

Author information

Authors and Affiliations

  1. FG Software Engineering, Leibniz Universität Hannover, Hannover, Germany

    Tobias Baum, Hendrik Leßmann & Kurt Schneider

Authors
  1. Tobias Baum
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hendrik Leßmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kurt Schneider
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tobias Baum .

Editor information

Editors and Affiliations

  1. University of Innsbruck, Innsbruck, Austria

    Michael Felderer

  2. Technical University Munich, Garching, Germany

    Daniel Méndez Fernández

  3. Brunel University London, Uxbridge, United Kingdom

    Burak Turhan

  4. Pontifical Catholic University of Rio de, Rio de Janeiro, Brazil

    Marcos Kalinowski

  5. University College London, London, United Kingdom

    Federica Sarro

  6. Vienna University of Technology, Vienna, Austria

    Dietmar Winkler

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Baum, T., Leßmann, H., Schneider, K. (2017). The Choice of Code Review Process: A Survey on the State of the Practice. In: Felderer, M., Méndez Fernández, D., Turhan, B., Kalinowski, M., Sarro, F., Winkler, D. (eds) Product-Focused Software Process Improvement. PROFES 2017. Lecture Notes in Computer Science(), vol 10611. Springer, Cham. https://doi.org/10.1007/978-3-319-69926-4_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-69926-4_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-69925-7

  • Online ISBN: 978-3-319-69926-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation