Abstract
The release frequency of software projects has increased in recent years. Adopters of so-called rapid releases—short release cycles, often on the order of weeks, days, or even hours—claim that they can deliver fixed issues (i.e., implemented bug fixes and new features) to users more quickly. However, there is little empirical evidence to support these claims. In fact, our prior work shows that code integration phases may introduce delays for rapidly releasing projects—98% of the fixed issues in the rapidly releasing Firefox project had their integration delayed by at least one release. To better understand the impact that rapid release cycles have on the integration delay of fixed issues, we perform a comparative study of traditional and rapid release cycles. Our comparative study has two parts: (i) a quantitative empirical analysis of 72,114 issue reports from the Firefox project, and a (ii) qualitative study involving 37 participants, who are contributors of the Firefox, Eclipse, and ArgoUML projects. Our study is divided into quantitative and qualitative analyses. Quantitative analyses reveal that, surprisingly, fixed issues take a median of 54% (57 days) longer to be integrated in rapid Firefox releases than the traditional ones. To investigate the factors that are related to integration delay in traditional and rapid release cycles, we train regression models that model whether a fixed issue will have its integration delayed or not. Our explanatory models achieve good discrimination (ROC areas of 0.80–0.84) and calibration scores (Brier scores of 0.05–0.16) for rapid and traditional releases. Our explanatory models indicate that (i) traditional releases prioritize the integration of backlog issues, while (ii) rapid releases prioritize issues that were fixed in the current release cycle. Complementary qualitative analyses reveal that participants’ perception about integration delay is tightly related to activities that involve decision making, risk management, and team collaboration. Moreover, the allure of shipping fixed issues faster is a main motivator for adopting rapid release cycles among participants (although this motivation is not supported by our quantitative analysis). Furthermore, to explain why traditional releases deliver fixed issues more quickly, our participants point out the rush for integration in traditional releases and the increased time that is invested on polishing issues in rapid releases. Our results suggest that rapid release cycles may not be a silver bullet for the rapid delivery of new content to users. Instead, our results suggest that the benefits of rapid releases are increased software stability and user feedback.
Similar content being viewed by others
Notes
From 2013 to 2016 by the time of this study.
We did not observe a statistically significant difference in integration delays between issues that are fixed by the reporters themselves and issues that are fixed by a different team member.
References
Adams B, McIntosh S (2016) Modern release engineering in a nutshell: why researchers should care. In: Proceedings of the 23rd international conference on software analysis, evolution, and reengineering (SANER), pp 78–90
AlGhamdi HM, Syer MD, Shang W, Hassan AE (2016) An automated approach for recommending when to stop performance tests. In: Proceedings of the international conference on software maintenance and evolution. IEEE, Piscataway, pp 279–289
Antoniol G, Ayari K, Penta MD, Khomh F, Guéhéneuc Y (2008) Is it a bug or an enhancement?: a text-based approach to classify change requests. In: Proceedings of the 2008 conference of the centre for advanced studies on collaborative research (CASCON), pp 23–37
Baskerville R, Pries-Heje J (2004) Short cycle time systems development. Inf Syst J 14:237–264
Baysal O, Davis I, Godfrey MW (2011) A tale of two browsers. In: Proceedings of the 8th working conference on mining software repositories (MSR). ACM, New York, pp 238–241
Beck K (2000) Extreme programming explained: embrace change. Addison-Wesley Professional, Reading
Beller M, Gousios G, Zaidman A (2015) How (much) do developers test?. In: 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol 2. IEEE, Piscataway, pp 559–562
Boehm BW (1988) A spiral model of software development and enhancement. Computer 21(5):61–72
Charmaz K (2014) Constructing grounded theory. SAGE, Newbury Park
Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114:494–509
da Costa DA, Abebe SL, McIntosh S, Kulesza U, Hassan AE (2014) An empirical study of delays in the integration of addressed issues. In: Proceedings of the 30th international conference on software maintenance and evolution (ICSME), pp 281–290
da Costa DA, McIntosh S, Kulesza U, Hassan AE (2016) The impact of switching to a rapid release cycle on the integration delay of addressed issues: an empirical study of the mozilla firefox project. In: Proceedings of the 13th international workshop on mining software repositories. ACM, New York, pp 374–385
Dunn OJ (1964) Multiple comparisons using rank sums. Technometrics 6 (3):241–252
Efron B (1986) How biased is the apparent error rate of a prediction rule?. In: Journal of the american statistical association, Taylor & Francis, Milton Park, vol 81, pp 461–470
Fisher RA (1925) Statistical methods for research workers. Genesis Publishing Pvt Ltd, New Delhi
Giger E, Pinzger M, Gall H (2010) Predicting the fix time of bugs. In: Proceedings of the 2nd international workshop on recommendation systems for software engineering (RSSE). ACM, New York, pp 52–56
Greer D, Ruhe G (2004) Software release planning: an evolutionary and iterative approach. Inf Softw Technol 46:243–253
Harrell FE (2001) Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis. Springer, New York
Herraiz I, German DM, Gonzalez-Barahona JM, Robles G (2008) Towards a simplification of the bug report form in eclipse. In: Proceedings of the 2008 international working conference on mining software repositories (MSR), pp 145–148
Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6:65–70
Howell DC (2005) Median absolute deviation. In: Encyclopedia of statistics in behavioral science. Wiley Online Library, Hoboken
Iasonos A, Schrag D, Raj GV, Panageas KS (2008) How to build and interpret a nomogram for cancer prognosis. In: Journal of clinical oncology, american society of clinical oncology, vol 26, pp 1364–1370
Jeong G, Kim S, Zimmermann T (2009) Improving bug triage with bug tossing graphs. In: Proceedings of the 7th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (ESEC/FSE). ACM, New York, pp 111–120
Jiang Y, Adams B (2014) How much does integrating this commit cost? - a position paper. In: 2nd International Workshop on Release Engineering (RELENG)
Jiang Y, Adams B, German DM (2013) Will my patch make it? and how fast?: case study on the linux kernel. In: Proceedings of the 10th working conference on mining software repositories (MSR), pp 101–110
Kampstra P et al. (2008) Beanplot: a boxplot alternative for visual comparison of distributions. J Stat Softw 28:1–9
Karlsson J, Ryan K (1997) A cost-value approach for prioritizing requirements. IEEE Softw 14:67–74
Khomh F, Dhaliwal T, Zou Y, Adams B (2012) Do faster releases improve software quality? an empirical case study of mozilla firefox. In: Proceedings of the 9th IEEE working conference on mining software repositories (MSR). IEEE, Piscataway, pp 179–188
Kruskal WH, Wallis WA (1952) Use of ranks in one-criterion variance analysis. J Am Stat Assoc 47(260):583–621
Leys C, Ley C, Klein O, Bernard P, Licata L (2013) Detecting outliers: do not use standard deviation around the mean, use absolute deviation around the median. J Exp Soc Psychol 49:764–766
Mäntylä MV, Adams B, Khomh F, Engström E, Petersen K (2014) On rapid releases and software testing: a case study and a semi-systematic literature review. In: Journal of Empirical Software Engineering. Springer, Berlin, pp 1–42
McIntosh S, Kamei Y, Adams B, Hassan AE (2016) An empirical study of the impact of modern code review practices on software quality. Empir Softw Eng 21 (5):2146–2189
Morakot C, Hoa Khanh D, Truyen T, Aditya G (2015a) Characterization and prediction of issue-related risks in software projects. In: 12th international conference on mining software repositories (MSR), pp 280–291
Morakot C, Hoa Khanh D, Truyen T, Aditya G (2015b) Predicting delays in software projects using networked classification. In: 30th international conference on automated software engineering (ASE)
Nagappan N, Ball T (2005) Use of relative code churn measures to predict system defect density. In: Proceedings of the 27th international conference on software engineering (ICSE). IEEE, Piscataway, pp 284–292
Paetsch F, Eberlein A, Maurer F (2003) Requirements engineering and agile software development. In: Twelfth IEEE international workshops on enabling technologies: infrastructure for collaborative enterprises, pp 308–313
Panjer LD (2007) Predicting eclipse bug lifetimes. In: Proceedings of the 4th international workshop on mining software repositories (MSR), p 29
Rahman MT, Rigby PC (2015) Release stabilization on linux and chrome. In: IEEE software journal. vol 2. IEEE, Piscataway, pp 81–88
Regnell B, Brinkkemper S (2005) Market-driven requirements engineering for software products. In: Engineering and managing software requirements, pp 287–308
Sandelowski M (1995) Sample size in qualitative research. Res Nurs Health 18 (2):179–183
Schroter A, Bettenburg N, Premraj R (2010) Do stack traces help developers fix bugs?. In: 2010 7th IEEE working conference on mining software repositories (MSR). IEEE, Piscataway, pp 118–121
Schwaber K (1997) Scrum development process. In: Business object design and implementation. Springer, Berlin, pp 117–134
Sheehan KB (2001) E-mail survey response rates: a review. J Comput-Mediat Commun 6(2) 10.1111/j.1083-6101.2001.tb00117.x 10.1111/j.1083-6101.2001.tb00117.x
Shihab E, Ihara A, Kamei Y, Ibrahim WM, Ohira M, Adams B, Hassan AE, Matsumoto K (2010) Predicting re-opened bugs: a case study on the eclipse project. In: Proceedings of 17th working conference on reverse engineering (WCRE). IEEE, Piscataway, pp 249–258
Shimagaki J, Kamei Y, McIntosh S, Pursehouse D, Ubayashi N (2016) Why are commits being reverted?: a comparative study of industrial and open source projects. In: 2016 IEEE international conference on software maintenance and evolution (ICSME). IEEE, Piscataway, pp 301–311
Smith E, Loftin R, Murphy-Hill E, Bird C, Zimmermann T (2013) Improving developer participation rates in surveys. In: 6th international workshop on cooperative and human aspects of software engineering (CHASE), pp 89–92
Souza R, Chavez C, Bittencourt RA (2014) Do rapid releases affect bug reopening? a case study of firefox. In: Proceedings of the brazilian symposium on software engineering (SBES). IEEE, Piscataway, pp 31–40
Souza R, Chavez C, Bittencourt R (2015) Rapid releases and patch backouts: a software analytics approach. In: IEEE software journal, vol 32. IEEE, Piscataway, pp 89–96
Spearman C (1904) The proof and measurement of association between two things. Am J Psychol 15(1):72–101
Subramaniam C, Sen R, Nelson ML (2009) Determinants of open source software project success: a longitudinal study. In: Journal of decision support systems, vol 46. Elsevier, Amsterdam, pp 576–585
Tian Y, Ali N, Lo D, Hassan AE (2015) On the unreliability of bug severity data. Empir Softw Eng 21:1–26
Wilks DS (2011) Statistical methods in the atmospheric sciences, vol 100. Academic Press, Cambridge
Zaman S, Adams B, Hassan AE (2011) Security versus performance bugs: a case study on firefox. In: Proceedings of the 8th working conference on mining software repositories. ACM, New York, pp 93–102
Acknowledgments
This work was partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the National Institute of Science and Technology for Software Engineering (INES), CNPq grant 465614/2014-0. We also thank all the participants from the Firefox, Eclipse, and ArgoUML projects for giving their time to respond our surveys and participate in our interviews.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Romain Robbes, Christian Bird, and Emily Hill
Appendices
Appendix A: Firefox Survey
Appendix B: ArgoUML Survey
Appendix C: Eclipse Survey
Appendix D: Methodology Web Page I
Appendix E: Methodology Web Page II
Appendix F: Invitation Letter
Appendix G: Interview Script
Rights and permissions
About this article
Cite this article
Costa, D., McIntosh, S., Treude, C. et al. The impact of rapid release cycles on the integration delay of fixed issues. Empir Software Eng 23, 835–904 (2018). https://doi.org/10.1007/s10664-017-9548-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-017-9548-7