On being ‘systematic’ in literature reviews

  • Sebastian K. Boell
  • Dubravka Cecez-Kecmanovic


General guidelines for conducting literature reviews often do not address the question of literature searches and dealing with a potentially large number of identified sources. These issues are specifically addressed by so-called systematic literature reviews (SLR) that propose a strict protocol for the search and appraisal of literature. Moreover, SLR are claimed to be a ‘standardized method’ for literature reviews, that is, replicable, transparent, objective, unbiased, and rigorous, and thus superior to other approaches for conducting literature reviews. These are significant and consequential claims that — 2014; despite increasing adoption of SLR — 2014; remained largely unnoticed in the information systems (IS) literature. The objective of this debate is to draw attention of the IS community to SLR’s claims, to question their justification and reveal potential risks of their adoption. This is achieved by first examining the origins of SLR and the prescribed systematic literature review process and then by critically assessing their claims and implications. In this debate, we show that SLR are applicable and useful for a very specific kind of literature review, a meta study that identifies and summarizes evidence from earlier research. We also demonstrate that the claims that SLR provide superior quality are not justified. More importantly, we argue that SLR as a general approach to conducting literature reviews is highly questionable, concealing significant perils. The paper cautions that SLR could undermine critical engagement with literature and what it means to be scholarly in academic work.


Systematic Literature Reviews SLR Conducting Literature Reviews Literature Review Narrative Literature Review Database Searches 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Agrawal, A., Boese, M. and Sarker, S. (2010). A Review of the HCI Literature in IS: The Missing Links of Computer-mediated Communication, Culture, and Interaction, In AMCIS 2010 Proceedings, Paper 523.Google Scholar
  2. Alvesson, M. and Sandberg, J. (2011). Generating Research Questions Through Problematization, Academy of Management Review 36(2): 247–271.Google Scholar
  3. Amrollahi, A., Ghapanchi, A. H. and Talaei-Khoei, A. (2013). A Systematic Literature Review on Strategic Information Systems Planning: Insights from the Past Decade, Pacific Asia Journal of the Association for Information Systems 5(2): 39–66.Google Scholar
  4. Atkins, C. and Louw, G. (2000). Reclaiming Knowledge: A Case for Evidence Based Information Systems Reclaiming Knowledge: A Case for Evidence-Based Information Systems, In ECIS 2000, Paper 28.Google Scholar
  5. Bandara, W., Miskon, S. and Fielt, E. (2011). A Systematic, Tool-supported Method for Conducting Literature Reviews in Information Systems, In ECIS 2011 Proceedings, Paper 221.Google Scholar
  6. Basten, D. and Sunyaev, A. (2011). The Nature of Adherence to Planning as Criterion for Information System Project Success, Sprouts: Working Papers on Information Systems 11(142).Google Scholar
  7. Basten, D. and Sunyaev, A. (2014). A Systematic Mapping of Factors Affecting Accuracy of Software Development Effort Estimation, Communications of the Association for Information Systems 34: 51–86.Google Scholar
  8. Baumeister, R. E. and Leary, M. R. (1997). Writing Narrative Literature Reviews, Review of General Psychology 1(3): 311–320.Google Scholar
  9. Blair, D. (2006). Wittgenstein, Language and Information. Back to the Rough Ground!, Dordrecht: Springer.Google Scholar
  10. Bodoff, D. (2009). Emergence of Terminological Conventions as a Searcherindexer Coordination Game, Journal of the American Society for Information Science and Technology 60(12): 2509–2529.Google Scholar
  11. Boell, S. K. and Cecez-Kecmanovic, D. (2014). A Hermeneutic Approach for Conducting Literature Reviews and Literature Searches, Communications of the Association for Information Systems 34: 257–286.Google Scholar
  12. Boote, D. N. and Beile, P. (2005). Scholars Before Researchers: On the Centrality of the Dissertation Literature Review in Research Preparation, Educational Researcher 34(6): 3–15.Google Scholar
  13. Brocke, J., Simons, A., Niehaves, B., Reimer, K., Plattfaut, R. and Cleven, A. (2009). Reconstructing the Giant: On the Importance of Rigour in Documenting the Literature Search Process, In ECIS 2009 Proceedings, Paper 161.Google Scholar
  14. Buckland, M. and Gey, F. (1994). The Relationship Between Recall and Precision, Journal of the American Society for Information Science 45(1): 12–19.Google Scholar
  15. Campbell Collaboration (2007). Retrieved on 10 May 2014 from
  16. Chalmers, I. and Altman, D. G. (1995). Systematic Reviews, London: BMJ.Google Scholar
  17. Checkland, P. and Holwell, S. (1998). Information, Systems and Information Systems: Making Sense of the Field, Chichester: Wiley.Google Scholar
  18. Cheung, C. M. K. and Thadani, D. R. (2010). The Effectiveness of Electronic Word-of-Mouth Communication: A Literature Analysis, In BLED 2010 Proceedings, Paper 18.Google Scholar
  19. Clarke, M. J. and Stewart, L. A. (1995). Obtaining Data from Randomized Controlled Trials: How Much Do We Need for Reliable and Informative Meta-analyses?, In Systematic Reviews, London: BMJ, pp. 37–47.Google Scholar
  20. Combs, J. P., Bustamante, R. M. and Onwuegbuzie, A. J. (2010).An Interactive Model for Facilitating Development of Literature Reviews, International Journal of Multiple Research Approaches 4(2): 159–182.Google Scholar
  21. Constantinides, P., Chiasson, M. W. and Introna, L. D. (2012). The Ends of Information Systems Research: A Pragmatic Framework, MIS Quarterly 36(1): 1–19.Google Scholar
  22. Cruzes, D. S. and Dybå, T. (2011). Research Synthesis in Software Engineering: A Tertiary Study, Information and Software Technology 53(5): 440–455.Google Scholar
  23. Davies, W. M. and Beaumont, T. J. (2007). Literature Reviews, Business, Melbourne.Google Scholar
  24. Davis, F. D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology, MIS Quarterly 13(3): 319–339.Google Scholar
  25. Dellinger, A. B. (2005). Validity and the Review of the Literature, Research in the Schools 12(2): 41–54.Google Scholar
  26. Denyer, D. and Tranfield, D. (2009). Producing a Systematic Review, In D. A. Buchanan & A. Bryman (Eds.), The Sage Handbook of Organizational Research Methods, Thousand Oaks, CA, USA: Sage, pp. 671–689.Google Scholar
  27. DiMaggio, P. J. (1995). Comments on “What Theory is Not”, Administrative Science Quarterly 40(3): 391–397.Google Scholar
  28. Dong, Y. R. (1996). Learning How to Use Citations for Knowledge Transformation: Non-Native Doctoral Students’ Dissertation Writing in Science, Research in Teaching of English 30(4): 428–457.Google Scholar
  29. Dwivedi, Y., Williams, M. D., Lal, B. and Schwarz, A. (2008). Profiling Adoption, Acceptance and Diffusion Research in the Information Systems Discipline, In ECIS 2008 Proceedings, Paper 112.Google Scholar
  30. Evidence-based Medicine Working Group. (1992). Evidence-based Medicine: A New Approach to Teaching the Practice of Medicine, Journal of the American Medical Association 268(17): 2420–2425.Google Scholar
  31. Eysenck, H. J. (1995). Problems with Meta-Analysis, In I. Chalmers & D. G. Altman (Eds.), Systematic Reviews, London: BMJ, pp. 64–74.Google Scholar
  32. Feak, C. B. and Swales, J. M. (2009). Telling a Research Story: Writing a Literature Review, Ann Arbor, Mich: University of Michigan Press.Google Scholar
  33. Finfgeld-Connett, D. and Johnson, E. D. (2013).Literature Search Strategies for Conducting Knowledge-Building and Theory-Generating Qualitative Systematic Reviews, Journal of Advanced Nursing 69(1): 194–204.Google Scholar
  34. Finn, J. A. (2005). Getting a PhD: An Action Plan to Help Manage Your Research, Your Supervisor and Your Project, London: Routledge.Google Scholar
  35. Fugmann, R. (2007). Informationstheorie: Der Jahrhundertbluff. Eine Zeitkritische Betrachtung (Teil 1), Information Wissenschaft und Praxis 58(8): 449–458.Google Scholar
  36. Gacenga, F., Cater-Steel, A., Toleman, M. and Tan, W.-G. (2011). Measuring the Performance of Service Orientated IT Management, Sprouts: Working Papers on Information Systems 11(162).Google Scholar
  37. Goodfellow, J. (1998). Constructing a Narrative, In J. Higgs (Ed.), Writing Qualitative Research, Sydney: Hampden Press, pp. 175–187.Google Scholar
  38. Grahlmann, K. R., Helms, R. W., Hilhorst, C., Brinkkemper, S. and Van Amerongen, S. (2012). Reviewing Enterprise Content Management: A Functional Framework, European Journal of Information Systems 21(3): 268–286.Google Scholar
  39. Gräning, A., Felden, C. and Piechocki, M. (2011). Status Quo and Potential of XBRL for Business and Information Systems Engineering, Business and Information Systems Engineering 3(4): 231–239.Google Scholar
  40. Green, B. N., Johnson, C. D. and Adams, A. (2006). Writing Narrative Literature Reviews for Peer-reviewed Journals: Secrets of the Trade, Journal of Chiropractic Medicine 5(3): 101–117.Google Scholar
  41. Greenhalgh, T. and Peacock, R. (2005). Effectiveness and Efficiency of Search Methods in Systematic Reviews of Complex Evidence: Audit of Primary Sources, British Medical Journal 331(7524): 1064–1065.Google Scholar
  42. Guillemette, M. G. and Paré, G. (2012). Toward a New Theory of the Contribution of the IT Function in Organizations, MIS Quarterly 36(2): 529–551.Google Scholar
  43. Hammersley, M. (2001). On “Systematic” Reviews of Research Literatures: A “Narrative” Response to Evans & Benefield, British Educational Research Journal 27(5): 543–554.Google Scholar
  44. Hart, C. (1998). Doing a Literature Review. Releasing the Social Science Research Imagination, Thousand Oaks: SAGE Publications.Google Scholar
  45. Hartley, J. and Betts, L. (2009). Common Weaknesses in Traditional Abstracts in the Social Sciences, Journal of the American Society for Information Science and Technology 60(10): 2010–2018.Google Scholar
  46. Hjorland, B. (2011). Evidence-Based Practice: An Analysis Based on the Philosophy of Science, Journal of the American Society for Information Science 62(7): 1301–1310.Google Scholar
  47. Holmes, D., Murray, S. J., Perron, A. and McCabe, J. (2008). Nursing Best Practice Guidelines: Reflecting on the Obscene Rise of the Void, Journal of Nursing Management 16(4): 394–403.Google Scholar
  48. Hood, W. W. and Wilson, C. S. (2001). The Scatter of Documents Over Databases in Different Subject Domains: How Many Databases Are Needed?, Journal of the American Society for Information Science and Technology 52(14): 1242–1254.Google Scholar
  49. Hummel, M., Rosenkranz, C. and Holten, R. (2012). The Role of Communication in Agile Systems Development: An Analysis of the State of the Art, Business and Information Systems Engineering 5(5): 343–355.Google Scholar
  50. Jalali, S. and Wohlin, C. (2012). Systematic Literature Studies: Database Searches vs. Backward Snowballing, In International Conference on Empirical Software Engineering and Measurement, Lund, Sweden: ACM, 29–38.Google Scholar
  51. Khoo, C. S. G., Na, J.-C. and Jaidka, K. (2011). Analysis of the Macro-level Discourse Structure of Literature Reviews, Online Information Review 35(2): 255–271.Google Scholar
  52. Kitchenham, B. (2004). Procedures for Performing Systematic Reviews, Keele, Eversleigh: Keele University and NICTA, Technical Report.Google Scholar
  53. Kitchenham, B. and Charters, S. (2007). Guidelines for Performing Systematic Literature Reviews in Software Engineering. Retrieved on 10 May 2014 from–8_OnlinePDF.pdf.Google Scholar
  54. Knipschild, P. (1995). Some Examples of Systematic Reviews, In I. Chalmers & D. G. Altman (Eds.), Systematic Reviews, London: BMJ, pp. 9–16.Google Scholar
  55. Kuhn, T. S. 1962. The Structure of Scientific Revolutions, Chicago: University of Chicago Press.Google Scholar
  56. Kwan, B. S. C. (2008). The Nexus of Reading, Writing and Researching in the Doctoral Undertaking of Humanities and Social Sciences: Implications for Literature Reviewing, English for Specific Purposes 27(1): 42–56.Google Scholar
  57. Kwan, B. S. C., Chan, H. and Lam, C. (2012). Evaluating Prior Scholarship in Literature Reviews of Research Articles: A Comparative Study of Practices in Two Research Paradigms, English for Specific Purposes 31(3): 188–201.Google Scholar
  58. Lacity, M. C., Solomon, S., Yan, A. and Willcocks, L. P. (2011). Business Process Outsourcing Studies: A Critical Review and Research Directions, Journal of Information Technology 26(4): 221–258.Google Scholar
  59. Leonardi, P. M. and Barley, S. R. (2010). What’s Under Construction Here? Social Action, Materiality, and Power in Constructivist Studies of Technology and Organizing, The Academy of Management Annals 4(1): 1–51.Google Scholar
  60. Leite, J. C. S. do P. and Cappelli, C. (2010). Software Transparency, Business and Information Systems Engineering 2(3): 127–139.Google Scholar
  61. LePine, J. A. and Wilcox-King, A. (2010). Editors’ Comments: Developing Novel Theoretical Insight from Reviews of Existing Theory and Research, Academy of Management Review 35(4): 506–509.Google Scholar
  62. Levy, Y. and Ellis, T. J. (2006). A Systems Approach to Conduct an Effective Literature Review in Support of Information Systems Research, Informing Science Journal 9: 181–212.Google Scholar
  63. Machi, L. A. and McEvoy, B. T. (2009). The Literature Review: Six Steps To Success, Thousand Oaks: Corwin Press.Google Scholar
  64. MacLure, M. (2005). “Clarity Bordering on Stupidity”: Where’s the Quality in Systematic Review?, Journal of Education Policy 20(4): 393–416.Google Scholar
  65. Matthias, G. and Patas, J. (2010). Evidence-Based Structuring and Evaluation of Empirical Research in Requirements Engineering: Fundamentals, Framework, Research Map, Business and Information Systems Engineering 2(3): 175–185.Google Scholar
  66. Merschbrock, C. and Munkvold, B. E. (2012). A Research Review on Building Information Modeling in Construction — 2014; An Area Ripe for IS Research, Communications of the Association for Information Systems 31: 307–228.Google Scholar
  67. Mettler, T., Eurich, M. and Winter, R. (2014). On the Use of Experiments in Design Science Research: A Proposition of an Evaluation Framework, Communications of the Association for Information Systems 34: 223–240.Google Scholar
  68. MISQ. (2006). Objectives of the MISQ Theory and Review. Retrieved on 10 May 2014 from Scholar
  69. Mohan, K. and Ahlemann, F. (2011). Understanding Acceptance of Information System Development and Management Methodologies by actual Users: A Review and Assessment of Existing Literature, In Wirtschaftsinformatik Proceedings 2011, Paper 41.Google Scholar
  70. Morrell, K. (2008). The Narrative of “Evidence Based” Management: A Polemic, Journal of Management Studies 45(3): 613–635.Google Scholar
  71. Mulrow, C. D. (1995). Rationale for Systematic Reviews, In I. Chalmers & D. G. Altman (Eds.), Systematic Reviews, London: BMJ, pp. 1–8.Google Scholar
  72. Murray, S. J., Holmes, D., Perron, A. and Rail, G. (2007). No Exit? Intellectual Integrity Under the Regime of “Evidence” and “Best-practices”, Journal of Clinical Practice 13: 512–516.Google Scholar
  73. Neuman, W. L. (2011). Social Research Methods: Quantitative and Qualitative Approaches, 7th ed., Boston, Mass, USA: Pearson.Google Scholar
  74. Oancea, A. and Pring, R. (2008). The Importance of Being Thorough: On Systematic Accumulations of “What Works” in Education Research, Journal of Philosophy of Education 42: 15–39.Google Scholar
  75. Oates, B. (2011). Evidence Based Information Systems: A Decade Later, In ECIS 2011 Proceedings, Paper 222.Google Scholar
  76. Oates, B. J., Edwards, H. M. and Wainwright, D. W. (2012). A Model-Driven Method for the Systematic Literature Review of Qualitative Empirical Research, In ICIS 2012 Proceedings, pp. 1–18.Google Scholar
  77. Okoli, C. and Schabram, K. (2009). Protocol for a Systematic Literature Review of Research on the Wikipedia, Sprouts: Working Papers on Information Systems 9(65).Google Scholar
  78. Okoli, C. and Schabram, K. (2010). A Guide to Conducting a Systematic Literature Review of Information Systems Research, Sprouts: Working Papers on Information Systems 10(26).Google Scholar
  79. Onwuegbuzie, Anthony J Collins, K. M. T., Leech, N. L., Dellinger, A. B. and Jiao, Q. G. (2007). Mixed Methods + Literature Reviews = Mixed Research Syntheses: A Framework for Conducting and Writing Rigorous, Comprehensive, and Insightful Literature Reviews, In The World of Educational Quality. Annual Meeting of the American Educational Research Association, Chicago.Google Scholar
  80. Oxman, A. D. (1995). Checklists for Review Articles, In Systematic Reviews, London: BMJ, pp. 75–85.Google Scholar
  81. Pawson, R. (2006). Digging for Nuggets: How “Bad” Research Can Yield “Good” Evidence, International Journal of Social Research Methodology 9(2): 127–142.Google Scholar
  82. Perry, C. (1998). A Structured Approach for Presenting Theses, Australasian Marketing Journal 6(1): 63–85.Google Scholar
  83. Petticrew, M. and Roberts, H. (2006). Systematic Reviews in the Social Sciences. A Practical Guide, Malden: Blackwell.Google Scholar
  84. Ramsay, C. R., Grant, A. M., Wallace, S. A., Garthwaite, P. H., Monk, A. F. and Russell, I. T. (2000).Assessment of the Learning Curve in Health Technologies — 2014; A Systematic Review, International Journal of Technology Assessment in Health Care 16(4): 1095–1108.Google Scholar
  85. Ridley, D. (2008). The Literature Review. A Step-by-Step Guide for Students, Thousand Oaks: SAGE.Google Scholar
  86. Rose, S., Bisson, J., Wessely, S. (2003). A Systematic Review of Single-Session Psychological Interventions (‘Debriefing’) Following Trauma, Psychotherapy and Psychosomatics 72(4): 176–184.Google Scholar
  87. Roztocki, N. and Weistroffer, H. R. (2008). Event Studies in Information Systems Research: A Review, In AMCIS 2008 Proceedings, Paper 248.Google Scholar
  88. Salton, G. and McGill, M. J. (1983). Introduction to Modern Information Retrieval, New York: McGraw-Hill.Google Scholar
  89. Sandelowski, M. (2008). Reading, Writing and Systematic Review, Journal of Advanced Nursing 64(1): 104–10.Google Scholar
  90. Sandelowski, M., Voils, C. I. and Barroso, J. (2007). Comparability Work and the Management of Difference in Research Synthesis Studies, Social Science & Medicine (1982) 64(1): 236–47.Google Scholar
  91. Schultze, U. and Leidner, D. E. (2002). Studying Knowledge Management in Information Systems Research: Discourses and Theoretical Assumptions, MIS Quarterly 26(3): 213–242.Google Scholar
  92. Schwarz, A., Mehta, M., Johnson, N. and Chin, W. W. (2007). Understanding Frameworks and Reviews: A Commentary to Assist us in Moving Our Field Forward by Analyzing Our Past, DATA BASE for Advances in Information Systems 38(3): 29–50.Google Scholar
  93. Shiffman, R. N., Liaw, Y., Brandt, C. A. and Corb, G. J. (1999). Computer-based Guideline Implementation Systems: A Systematic Review of Functionality and Effectiveness, Journal of the American Medical Informatics Association 6(2): 104–114.Google Scholar
  94. Staples, M. and Niazi, M. (2007). Experiences Using Systematic Review Guidelines, Journal of Systems and Software 80(9): 1425–1437.Google Scholar
  95. Stol, K.-J., Babar, M. A., Russo, B. and Fitzgerald, B. (2009). The Use of Empirical Methods in Open Source Software Research: Facts, Trends and Future Directions, In FLOSS’09, May 18, 2009, Vancouver, Canada, pp. 19–24.Google Scholar
  96. Sutton, R. and Staw, B. M. (1995). Forum What Theory is Not, Administrative Science Quarterly 40(3): 371–384.Google Scholar
  97. Tamm, T., Seddon, P. B., Shanks, G. and Reynolds, P. (2011). How Does Enterprise Architecture Add Value to Organisations?, Communications of the Association for Information Systems 28: 141–168.Google Scholar
  98. Thompson, S. G. (1995). Why Sources of Heterogenity in Meta-Analysis Should be Investigated, In I. Chalmers & D. G. Altman (Eds.), Systematic Reviews, London: BMJ, pp. 48–63.Google Scholar
  99. Tranfield, D., Denyer, D. and Smart, P. (2003). Towards a Methodology for Developing Evidence-Informed Management Knowledge by Means of Systematic Review, British Journal of Management 14(3): 207–222.Google Scholar
  100. Wang, Q. E., Myers, M. D. and Sundaram, D. (2013). Digital Natives and Digital Immigrants: Towards a Model of Digital Fluency, Business and Information Systems Engineering 5(6): 409–419.Google Scholar
  101. Webster, J. and Watson, R. T. (2002). Analyzing the Past to Prepare for the Future: Writing a Literature Review, MIS Quarterly 26(2): xiii–xxiii.Google Scholar
  102. Whetten, D. A. (1989). What Constitutes a Theoretical Contribution?, Academy of Management Review 14(4): 490–495.Google Scholar
  103. Williams, M. D., Dwivedi, Y. K., Lal, B. and Schwarz, A. (2009). Contemporary Trends and Issues in IT Adoption and Diffusion research, Journal of Information Technology 24(1): 1–10.Google Scholar
  104. Wolfswinkel, J. F., Furtmueller, E. and Wilderom, C. P. M. (2013). Using Grounded Theory as a Method for Rigorously Reviewing Literature, European Journal of Information Systems 22(1): 45–55.Google Scholar
  105. Wolfswinkel, J., Furtmueller, E. and Wilderom, C. (2010). Reflecting on E-Recruiting Research Using Grounded Theory, In ECIS 2010 Proceedings, Paper 52.Google Scholar
  106. Wright Mills, C. (1978[1959]) The Sociological Imagination, Oxford: Oxford University Press.Google Scholar
  107. Zhang, P., Li, N. (Lina), Scialdone, M. J. and Carey, J. (2008). The Intellectual Advancement of Human-Computer Interaction Research: A Critical Assessment of the MIS Literature (1990–2008), AIS Transactions on Human-Computer Interaction 1(3): 55–107.Google Scholar

Copyright information

© Journal of Information Technology (JIT) 2015

Authors and Affiliations

  • Sebastian K. Boell
    • 1
  • Dubravka Cecez-Kecmanovic
    • 2
  1. 1.The University Sydney Business SchoolAustralia
  2. 2.Australian School of Business School of Information Systems and TechnologyUniversity of New South WalesAustralia

Personalised recommendations