Skip to main content
SpringerLink
Log in

Understanding the impact of rapid releases on software quality

The case of firefox

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Many software companies are shifting from the traditional multi-month release cycle to shorter release cycles. For example, Google Chrome and Mozilla Firefox release new versions every 6 weeks. These shorter release cycles reduce the users’ waiting time for a new release and offer better feedback and marketing opportunities to companies, but it is unclear if the quality of the software product improves as well, since developers and testers are under more pressure. In this paper, we extend our previous empirical study of Mozilla Firefox on the impact of rapid releases on quality assurance with feedback by Mozilla project members. The study compares crash rates, median uptime, and the proportion of pre- and post-release bugs in traditional releases with those in rapid releases, and we also analyze the source code changes made by developers to identify potential changes in the development process. We found that (1) with shorter release cycles, users do not experience significantly more pre- or post-release bugs (percentage-wise) and (2) bugs are fixed faster, yet (3) users experience these bugs earlier during software execution (the program crashes earlier). Increased integration activity and propagation of harder bugs to later versions account for some of these findings. Overall, our case study suggests that a clear release engineering process with thorough automation is one of the major challenges when switching to rapid releases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. http://www.campwoodsw.com/

References

  • HP (2011) Shorten release cycles by bringing developers to application lifecycle management. HP Applications Handbook, Retrieved on Febuary 08, 2012

  • Mozilla (2011) Mozilla puts out firefox 5.0 web browser which carries over 1,000 improvements in just about 3 months of development. InvestmentWatch on June 25th, 2011. Retrieved on January 12, 2012

  • Beck K, Andres C (2004) Extreme programming explained: embrace change 2nd edn. Addison-Wesley

  • Shankland S (2011) Rapid-release firefox meets corporate backlash. http://cnet.co/ktBsUU

  • Kaply M (2011) Why do companies stay on old technology? Retrieved on January 12, 2012

  • Shankland S (2011) Mozilla proposes not-so-rapid-release firefox. CNET. Retrieved on February 08, 2012

  • Vaughan-Nichols SJ (2012) The truth about goobuntu: Google’s in-house desktop ubuntu linux. http://www.zdnet.com/the-truth-about-goobuntu-googles-in-house-desktop-ubuntu-linux-7000003462/

  • Baysal O, Davis I, Godfrey MW (2011) A tale of two browsers. In: Proceedings of the 8th working conference on mining software repositories (MSR). pp. 238–241

  • Porter A, Yilmaz C, Memon AM, Krishna AS, Schmidt DC, Gokhale A (2006) Techniques and processes for improving the quality and performance of open-source software. Softw Process Improv Pract 11:163–176

    Article  Google Scholar 

  • Downer T (2011) Some clarification and musings. Accessed 6 Jan 2012

  • Li PL, Kivett R, Zhan Z, Jeon Se, Nagappan N, Murphy B, Ko AJ (2012) Characterizing the differences between pre- and post- release versions of software. In: Proceedings of the 33rd international conference on software engineering (ICSE). pp 716–725

  • Khomh F, Dhaliwal T, Zou Y, Adams B (2012) Do faster releases improve software quality? an empirical case study of mozilla firefox. In: Proceedings of the 9th working conference on mining software repositories (MSR). pp. 179–188

  • Ltd. RS (2012) Web browsers (global marketshare). Roxr Software Ltd. Retrieved on January 12, 2012

  • Shankland S (2010) Google ethos speeds up chrome release cycle. http://cnet.co/wlS24U

  • Sicore D (2011) New channels for firefox rapid releases. The Mozilla Blog. 2011-04-13. Retrieved on January 12, 2012

  • Rouget P (2012) Shaping a firefox feature - how does it work? http://paulrouget.com/e/featuredesign/

  • Mozilla (2013) Auto-tools/automation development. https://wiki.mozilla.org/QA/Automation_Services

  • Mäntylä M, Khomh F, Adams B, Engstrom E, Petersen K (2013) On rapid releases and software testing. In: Proceedings of the 29th IEEE international conference on software maintenance (ICSM). Eindhoven, The Netherlands. To appear

  • Paul R (2011) Mozilla outlines 16-week firefox development cycle. Accessed 6 Jan 2012

  • Mozilla (2011) Socorro: Mozilla’s crash reporting system. https://crash-stats.mozilla.com/home/products/firefox. Accessed 29 March 2011

  • Khomh F, Chan B, Zou Y, Hassan AE (2011) An entropy evaluation approach for triaging field crashes: A case study of mozilla firefox. In: Proceedings of the 18th working conference on reverse engineering (WCRE)

  • Herraiz I, Shihab E, Nguyen THD, Hassan AE (2011) Impact of installation counts on perceived quality: a case study on debian. In: Proceedings of the 18th working conference on reverse engineering (WCRE). pp 219–228

  • Hollander M, Wolfe DA (1999) Nonparametric statistical methods, 2nd edn. Wiley

  • Marschall M (2007) Transforming a six month release cycle to continuous flow. In: Proceedings of the conference on AGILE. pp 395–400

  • Śliwerski J, Zimmermann T, Zeller A (2005) When do changes induce fixes? In: Proceedings of the 2005 international workshop on mining software repositories (MSR). pp 1–5

  • Fox J (2008) Applied regression analysis and generalized linear, 2nd edn. Sage Publications

  • Johnson JW (2000) A heuristic method for estimating the relative weight of predictor variables in multiple regression. Multivar Behav Res 35:1–19

    Article  Google Scholar 

  • Yin RK (2002) Case study research: design and methods, 3rd edn. SAGE Publications

  • Software C. (2012) Sourcemonitor. Accessed 12 Jan 2012

  • Otte T, Moreton R, Knoell HD (2008) Applied quality assurance methods under the open source development model. In: Proceedings of the 32nd Annual IEEE international computer software and applications conference (COMPSAC). pp 1247–1252

  • Brown AW (2011) A case study in agile-at-scale delivery. In: Proceedings of the 12th international conference on agile processes in software engineering and extreme programming (XP), vol 77. pp 266–281

  • Jenkins J (2011) Velocity culture (the unmet challenge in ops). Presentation at O’Reilly velocity conference

  • Gamma E (2005) Agile, open source, distributed, and on-time – inside the eclipse development process. Keynote at the 27th international conference on software engineering (ICSE)

  • van der Storm T (2005) Continuous release and upgrade of component-based software. In: Proceedings of the 12th international workshop on software configuration management (SCM). pp 43–57

  • Dolstra E, de Jonge M, Visser E (2004) Nix: a safe and policy-free system for software deployment. In: Proceedings of the 18th USENIX conference on system admin. pp 79–92

  • Duvall P, Matyas SM, Glover A (2007) Continuous Integration: improving software quality and reducing risk. Addison-Wesley Professional

  • Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test, and deployment automation, 1st edn. Addison-Wesley Professional

  • Kong S, Kendall JE, Kendall KE (2009) The challenge of improving software quality: developers’ beliefs about the contribution of agile practices. In: Proceedings of the Americas conference on information systems (AMCIS). 12p

  • VersionOne (2009) 4th annual state of agile survey. http://bit.ly/6BPw5

  • Hodgetts P., Phillips D. (2002) 30. in: extreme adoption experiences of a B2B start up. Addison-Wesley Longman Publishing Co. Inc. Extreme programming perspectives

  • Kuppuswami S, Vivekanandan K, Ramaswamy P, Rodrigues P (2003) The effects of individual xp practices on software development effort. SIGSOFT Softw Eng Notes 28:6

    Article  Google Scholar 

  • Stewart KJ, Darcy DP, Daniel SL (2005) Observations on patterns of development in open source software projects. SIGSOFT Softw Eng Notes 30:1–5

    Article  Google Scholar 

  • Jansen S, Brinkkemper S (2006) Ten misconceptions about product software release management explained using update cost/value functions. In: Proceedings of the international workshop on software product management. pp 44–50

  • Zaman S, Adams B, Hassan AE (2012) A qualitative study on performance bugs. In: Proceedings of the 9th IEEE working conference on mining software repositories (MSR). Zurich, pp 199–208

  • Lu S, Park S, Seo E, Zhou Y (2008) Learning from mistakes: a comprehensive study on real world concurrency bug characteristics. In: Proceedings of the 13th international conference on architectural support for programming languages and operating systems (ASPLOS). pp. 329–339

  • Thung F, Wang S, Lo D, Jiang L (2012) An empirical study of bugs in machine learning systems. In: Proceedings of the 23rd IEEE international symposium on software reliability engineering (ISSRE). 271–280

  • Sahoo SK, Criswell J, Adve V (2010) An empirical study of reported bugs in server software with implications for automated bug diagnosis. In: Proceedings of the 32nd ACM/IEEE international conference on software engineering (ICSE), vol 1. pp 485–494

Download references

Acknowledgments

We would like to thank the two Mozilla QA engineers who provided feedback on our findings. Their statements are accounts of personal experience and opinion, and are in no means whatsoever an official statement from Mozilla.

Author information

Authors and Affiliations

  1. SWAT, Polytechnique Montréal, Québec, Canada

    Foutse Khomh

  2. MCIS, Polytechnique Montréal, Québec, Canada

    Bram Adams

  3. Department of Electrical and Computer Engineering, Queen’s University, Kingston, ON, Canada

    Tejinder Dhaliwal & Ying Zou

Authors
  1. Foutse Khomh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bram Adams
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tejinder Dhaliwal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ying Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Foutse Khomh.

Additional information

Communicated by: Maximillano di Penta and Tao Xie

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khomh, F., Adams, B., Dhaliwal, T. et al. Understanding the impact of rapid releases on software quality. Empir Software Eng 20, 336–373 (2015). https://doi.org/10.1007/s10664-014-9308-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-014-9308-x

Keywords

Search

Navigation