Feature Toggling is a technique to control the execution of features in a software project. For example, practitioners using feature toggles can experiment with new features in a production environment by exposing them to a subset of users. Some of these toggles require additional maintainability efforts and are expected to be removed, whereas others are meant to remain for a long time. However, to date, very little is known about the removal of feature toggles, which is why we focus on this topic in our paper. We conduct an empirical study that focuses on the removal of feature toggles. We use source code analysis techniques to analyze 12 Python open source projects and surveyed 61 software practitioners to provide deeper insights on the topic. Our study shows that 75% of the toggle components in the studied Python projects are removed within 49 weeks after introduction. However, eventually practitioners remove feature toggles to follow the life cycle of a feature when it becomes stable in production. We also find that not all long-term feature toggles are designed to live that long and not all feature toggles are removed from the source code, opening the possibilities to unwanted risks. Our study broadens the understanding of feature toggles by identifying reasons for their survival in practice and aims to help practitioners make better decisions regarding the way they manage and remove feature toggles.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Toggles are also called flags, bits, flippers, switches or gates. In this paper, we use the term toggles.
A replication package is available on https://github.com/elhoyos/toggles_package
Through our personal correspondence with Mozilla Engineers.
Contacted via the slack channel of OpenedX.
Contacted via personal correspondence.
Via personal correspondence with a Software Engineer of the Center for Open Science.
Via our personal correspondence with the project’s Engineering Manager at Mozilla.
Center for Open Science (2013) https://github.com/CenterForOpenScience/osf.io. Accessed 2019-01-09
Django Packages : Feature Flipping (2018) https://djangopackages.org/grids/g/feature-flip/. Accessed 2018-11-05
Django Waffle — django-waffle 0.14.0 documentation (2018) https://waffle.readthedocs.io/en/stable/. Accessed 2019-01-11
edx-platform (2011) https://github.com/edx/edx-platform. Accessed 2019-01-09
Libraries.io - The Open Source Discovery Service (2015) https://libraries.io/. Accessed 2019-01-09
The Django template language — Django documentation — Django (2019) https://docs.djangoproject.com/en/2.1/ref/templates/language/. Accessed 2019-06-16
The Web framework for perfectionists with deadlines — Django (2019) https://www.djangoproject.com/. Accessed 2019-03-05
PEP 318 – Decorators for Functions and Methods — Python.org (2003) https://www.python.org/dev/peps/pep-0318/. Accessed 2019-01-23
Adams B, Bellomo S, Bird C, Marshall-Keim T, Khomh F, Moir K (2015) The practice and future of release engineering: a roundtable with three release engineers. IEEE Softw 32(2):42–49
Adams B, McIntosh S (2016) Modern release engineering in a nutshell – why researchers should care. In: Leaders of tomorrow: future of software engineering, proceedings of the 23rd IEEE international conference on software analysis, evolution, and reengineering (SANER), Osaka, Japan, pp 78–90
Asthagiri N (2018) OEP-17: Feature toggles. https://open-edx-proposals.readthedocs.io/en/latest/oep-0017-bp-feature-toggles.html. Accessed 2020-01-10
Bosworth A (2012) Building and testing at Facebook. https://www.facebook.com/notes/facebook-engineering/building-and-testing-at-facebook/10151004157328920https://www.facebook.com/notes/facebook-engineering/building-and-testing-at-facebook/10151004157328920. Accessed 2019-03-25
Claps GG, Berntsson Svensson R, Aurum AA (2015) On the journey to continuous deployment: Technical and social challenges along the way. Inf Softw Technol 57(1):21–31
Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46
Corbin J, Strauss A et al (2008) Basics of qualitative research: techniques and procedures for developing grounded theory
Feitelson DG, Frachtenberg E, Beck KL (2013) Development and deployment at facebook. IEEE Internet Comput 17(4):8–17
GitHub (2011) GitHub API v3 — GitHub Developer Guide. https://developer.github.com/v3/. Accessed 2019-01-09
GitHub (2011) Search — GitHub Developer Guide. https://developer.github.com/v3/search. Accessed 2019-01-03
GitHub (2013) Searching code - User Documentation. https://help.github.com/articles/searching-code. Accessed 2019-01-03
Harry B (2012) Announcing Continuous Deployment to Azure with Team Foundation Service — Brian Harrys blog. https://bit.ly/2MvKEkT. Accessed 2019-01-22
Hodgson P (2016) Feature Toggles. https://martinfowler.com/articles/feature-toggles.html. Accessed 2017-08-30
Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test and deployment
Kalliamvakou E, Gousios G, Blincoe K, Singer L, German DM, Damian D (2014) The promises and perils of mining github. In: Proceedings of the 11th working conference on mining software repositories, MSR 2014, ACM, pp 92–101
Kaplan EL, Meier P (1958) Nonparametric estimation from incomplete observations. J Am Stat Assoc 53(282):457–481
Kästner C (2019) Feature flags vs configuration options — same difference? https://www.cs.cmu.edu/ckaestne/featureflags/. (Accessed on 03/30/2020)
LaunchDarkly (2015) Feature flags - feature flags, toggles, controls. http://featureflags.io/feature-flags/. Accessed 2018-11-03
Mäntylä MV, Adams B, Khomh F, Engström E, Petersen K (2015) On rapid releases and software testing: a case study and a semi-systematic literature review. Emp Softw Eng 20(5):1384–1425
Neely S, Stolt S (2013) Continuous delivery? Easy! Just change everything (well, maybe it is not that easy). In: Proceeding - AGILE 2013, pp 121–128
Osherove R (2016) Feature toggles – enterprise devOps. http://enterprisedevops.org/feature-toggle-frameworks-list/. Accessed 2018-12-11
Rahman AAU, Helms E, Williams L, Parnin C (2015) Synthesizing continuous deployment practices used in software development. 2015 Agile Conference, 1–10
Rahman MT, Querel L-P, Rigby PC, Adams B (2016) Feature toggles: practitioner practices and a case study. In: Proceedings of the 13th international conference on mining software repositories, MSR ’16. ACM, New York, pp 201–211
Rahman MT, Rigby PC, Shihab E (2018) The modular and feature toggle architectures of Google Chrome. Emp Softw Eng, 1–28
Ray B, Posnett D, Filkov V, Devanbu P (2014) A large scale study of programming languages and code quality in github. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering - FSE 2014. ACM Press, New York, pp 155–165
Sayagh M, Kerzazi N, Adams B, Petrillo F (2018) Software configuration engineering in practice: interviews, survey, and systematic literature review. IEEE Trans Softw Eng
Schermann G, Cito J, Leitner P, Zdun U, Gall H (2016) An empirical study on principles and practices of continuous delivery and deployment. Peer J Preprints 4:e1889v1
Securities and E. Commission (2013) Administrative and cease-and-desist proceedings against knight capital americas LLC. https://www.sec.gov/litigation/admin/2013/34-70694.pdf. Accessed 2019-01-19
Smith E, Loftin R, Murphy-Hill E, Bird C, Zimmermann T (2013) Improving developer participation rates in surveys. In: 2013 6Th international workshop on cooperative and human aspects of software engineering (CHASE). IEEE, pp 89–92
Yin RK (2009) Case study research: design and methods (applied social research methods). London and Singapore: Sage
Zapata D (2014) Going from 3 week to daily releases at netflix. USENIX Association, Philadelphia
We would like to thank the practitioners who devoted their time and effort to respond to our online survey, and to all the practitioners of Mozilla, Université catholique de Louvain, Open edX, Tennessee Data Commons and the Center for Open Science, that responded back to our call and contributed their valuable experience using feature toggles in their projects.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Communicated by: Sarah Nadi
About this article
Cite this article
Hoyos, J., Abdalkareem, R., Mujahid, S. et al. On the Removal of Feature Toggles. Empir Software Eng 26, 15 (2021). https://doi.org/10.1007/s10664-020-09902-y
- Feature toggles
- Continuous integration
- Continuous delivery
- Empirical studies