On the Removal of Feature Toggles

A Study of Python Projects and Practitioners Motivations


Feature Toggling is a technique to control the execution of features in a software project. For example, practitioners using feature toggles can experiment with new features in a production environment by exposing them to a subset of users. Some of these toggles require additional maintainability efforts and are expected to be removed, whereas others are meant to remain for a long time. However, to date, very little is known about the removal of feature toggles, which is why we focus on this topic in our paper. We conduct an empirical study that focuses on the removal of feature toggles. We use source code analysis techniques to analyze 12 Python open source projects and surveyed 61 software practitioners to provide deeper insights on the topic. Our study shows that 75% of the toggle components in the studied Python projects are removed within 49 weeks after introduction. However, eventually practitioners remove feature toggles to follow the life cycle of a feature when it becomes stable in production. We also find that not all long-term feature toggles are designed to live that long and not all feature toggles are removed from the source code, opening the possibilities to unwanted risks. Our study broadens the understanding of feature toggles by identifying reasons for their survival in practice and aims to help practitioners make better decisions regarding the way they manage and remove feature toggles.

This is a preview of subscription content, access via your institution.

Fig. 1
Listing 1
Listing 2
Listing 3
Listing 4
Fig. 2
Fig. 3
Fig. 4
Fig. 5


  1. 1.

    Toggles are also called flags, bits, flippers, switches or gates. In this paper, we use the term toggles.

  2. 2.

    A replication package is available on https://github.com/elhoyos/toggles_package

  3. 3.


  4. 4.

    Through our personal correspondence with Mozilla Engineers.

  5. 5.

    Contacted via the slack channel of OpenedX.

  6. 6.

    Contacted via personal correspondence.

  7. 7.

    Via personal correspondence with a Software Engineer of the Center for Open Science.

  8. 8.

    Via our personal correspondence with the project’s Engineering Manager at Mozilla.


  1. Center for Open Science (2013) https://github.com/CenterForOpenScience/osf.io. Accessed 2019-01-09

  2. Django Packages : Feature Flipping (2018) https://djangopackages.org/grids/g/feature-flip/. Accessed 2018-11-05

  3. Django Waffle — django-waffle 0.14.0 documentation (2018) https://waffle.readthedocs.io/en/stable/. Accessed 2019-01-11

  4. edx-platform (2011) https://github.com/edx/edx-platform. Accessed 2019-01-09

  5. Libraries.io - The Open Source Discovery Service (2015) https://libraries.io/. Accessed 2019-01-09

  6. The Django template language — Django documentation — Django (2019) https://docs.djangoproject.com/en/2.1/ref/templates/language/. Accessed 2019-06-16

  7. The Web framework for perfectionists with deadlines — Django (2019) https://www.djangoproject.com/. Accessed 2019-03-05

  8. PEP 318 – Decorators for Functions and Methods — Python.org (2003) https://www.python.org/dev/peps/pep-0318/. Accessed 2019-01-23

  9. Adams B, Bellomo S, Bird C, Marshall-Keim T, Khomh F, Moir K (2015) The practice and future of release engineering: a roundtable with three release engineers. IEEE Softw 32(2):42–49

    Article  Google Scholar 

  10. Adams B, McIntosh S (2016) Modern release engineering in a nutshell – why researchers should care. In: Leaders of tomorrow: future of software engineering, proceedings of the 23rd IEEE international conference on software analysis, evolution, and reengineering (SANER), Osaka, Japan, pp 78–90

  11. Asthagiri N (2018) OEP-17: Feature toggles. https://open-edx-proposals.readthedocs.io/en/latest/oep-0017-bp-feature-toggles.html. Accessed 2020-01-10

  12. Bosworth A (2012) Building and testing at Facebook. https://www.facebook.com/notes/facebook-engineering/building-and-testing-at-facebook/10151004157328920https://www.facebook.com/notes/facebook-engineering/building-and-testing-at-facebook/10151004157328920. Accessed 2019-03-25

  13. Claps GG, Berntsson Svensson R, Aurum AA (2015) On the journey to continuous deployment: Technical and social challenges along the way. Inf Softw Technol 57(1):21–31

    Article  Google Scholar 

  14. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46

    Article  Google Scholar 

  15. Corbin J, Strauss A et al (2008) Basics of qualitative research: techniques and procedures for developing grounded theory

  16. Feitelson DG, Frachtenberg E, Beck KL (2013) Development and deployment at facebook. IEEE Internet Comput 17(4):8–17

    Article  Google Scholar 

  17. GitHub (2011) GitHub API v3 — GitHub Developer Guide. https://developer.github.com/v3/. Accessed 2019-01-09

  18. GitHub (2011) Search — GitHub Developer Guide. https://developer.github.com/v3/search. Accessed 2019-01-03

  19. GitHub (2013) Searching code - User Documentation. https://help.github.com/articles/searching-code. Accessed 2019-01-03

  20. Harry B (2012) Announcing Continuous Deployment to Azure with Team Foundation Service — Brian Harrys blog. https://bit.ly/2MvKEkT. Accessed 2019-01-22

  21. Hodgson P (2016) Feature Toggles. https://martinfowler.com/articles/feature-toggles.html. Accessed 2017-08-30

  22. Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test and deployment

  23. Kalliamvakou E, Gousios G, Blincoe K, Singer L, German DM, Damian D (2014) The promises and perils of mining github. In: Proceedings of the 11th working conference on mining software repositories, MSR 2014, ACM, pp 92–101

  24. Kaplan EL, Meier P (1958) Nonparametric estimation from incomplete observations. J Am Stat Assoc 53(282):457–481

    MathSciNet  Article  Google Scholar 

  25. Kästner C (2019) Feature flags vs configuration options — same difference? https://www.cs.cmu.edu/ckaestne/featureflags/. (Accessed on 03/30/2020)

  26. LaunchDarkly (2015) Feature flags - feature flags, toggles, controls. http://featureflags.io/feature-flags/. Accessed 2018-11-03

  27. Mäntylä MV, Adams B, Khomh F, Engström E, Petersen K (2015) On rapid releases and software testing: a case study and a semi-systematic literature review. Emp Softw Eng 20(5):1384–1425

    Article  Google Scholar 

  28. Neely S, Stolt S (2013) Continuous delivery? Easy! Just change everything (well, maybe it is not that easy). In: Proceeding - AGILE 2013, pp 121–128

  29. Osherove R (2016) Feature toggles – enterprise devOps. http://enterprisedevops.org/feature-toggle-frameworks-list/. Accessed 2018-12-11

  30. Rahman AAU, Helms E, Williams L, Parnin C (2015) Synthesizing continuous deployment practices used in software development. 2015 Agile Conference, 1–10

  31. Rahman MT, Querel L-P, Rigby PC, Adams B (2016) Feature toggles: practitioner practices and a case study. In: Proceedings of the 13th international conference on mining software repositories, MSR ’16. ACM, New York, pp 201–211

  32. Rahman MT, Rigby PC, Shihab E (2018) The modular and feature toggle architectures of Google Chrome. Emp Softw Eng, 1–28

  33. Ray B, Posnett D, Filkov V, Devanbu P (2014) A large scale study of programming languages and code quality in github. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering - FSE 2014. ACM Press, New York, pp 155–165

  34. Sayagh M, Kerzazi N, Adams B, Petrillo F (2018) Software configuration engineering in practice: interviews, survey, and systematic literature review. IEEE Trans Softw Eng

  35. Schermann G, Cito J, Leitner P, Zdun U, Gall H (2016) An empirical study on principles and practices of continuous delivery and deployment. Peer J Preprints 4:e1889v1

  36. Securities and E. Commission (2013) Administrative and cease-and-desist proceedings against knight capital americas LLC. https://www.sec.gov/litigation/admin/2013/34-70694.pdf. Accessed 2019-01-19

  37. Smith E, Loftin R, Murphy-Hill E, Bird C, Zimmermann T (2013) Improving developer participation rates in surveys. In: 2013 6Th international workshop on cooperative and human aspects of software engineering (CHASE). IEEE, pp 89–92

  38. Yin RK (2009) Case study research: design and methods (applied social research methods). London and Singapore: Sage

  39. Zapata D (2014) Going from 3 week to daily releases at netflix. USENIX Association, Philadelphia

    Google Scholar 

Download references


We would like to thank the practitioners who devoted their time and effort to respond to our online survey, and to all the practitioners of Mozilla, Université catholique de Louvain, Open edX, Tennessee Data Commons and the Center for Open Science, that responded back to our call and contributed their valuable experience using feature toggles in their projects.

Author information



Corresponding author

Correspondence to Juan Hoyos.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by: Sarah Nadi

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hoyos, J., Abdalkareem, R., Mujahid, S. et al. On the Removal of Feature Toggles. Empir Software Eng 26, 15 (2021). https://doi.org/10.1007/s10664-020-09902-y

Download citation


  • Feature toggles
  • Continuous integration
  • Continuous delivery
  • Empirical studies