Advertisement

Search. Review. Repeat? An empirical study of threats to replicating SLR searches

  • Jacob KrügerEmail author
  • Christian Lausberger
  • Ivonne von Nostitz-Wallwitz
  • Gunter Saake
  • Thomas Leich
Article

Abstract

A systematic literature review (SLR) is an empirical method used to provide an overview of existing knowledge and to aggregate evidence within a domain. For computer science, several threats to the completeness of such reviews have been identified, leading to recommendations and guidelines on how to improve their quality. However, few studies address to what extent researchers can replicate an SLR. To conduct a replication, researchers have to first understand how the set of primary studies has been identified in the original study, and can ideally retrieve the same set when following the reported protocol. In this article, we focus on this initial step of a replication and report a two-fold empirical study: Initially, we performed a tertiary study using a sample of SLRs in computer science and identified what information that is needed to replicate the searches is reported. Based on the results, we conducted a descriptive, multi-case study on digital libraries to investigate to what extent these allow replications. The results reveal two threats to replications of SLRs: First, while researchers have improved the quality of their reports, relevant details are still missing—we refer to a reporting threat. Second, we found that some digital libraries are inconsistent in their query results—we refer to a searching threat. While researchers conducting a review can only overcome the first threat and the second may not be an issue for all kinds of replications, researchers should be aware of both threats when conducting, reviewing, and building on SLRs.

Keywords

Tertiary study Systematic literature review Software engineering Threats to validity Replication Digital library 

Notes

Acknowledgments

This research has been supported by the German Research Foundation (DFG) project EXPLANT grants LE 3382/2-1, LE 3382/2-3, SA 465/49-1, and SA 465/49-3.

References

  1. Abdelmaboud A, Jawawi D N, Ghani I, Elsafi A (2015) A comparative evaluation of cloud migration optimization approaches: a systematic literature review. J Theor Appl Inf Technol 79(3):395–414Google Scholar
  2. Afzal W, Alone S, Glocksien K, Torkar R (2016) Software test process improvement approaches: a systematic literature review and an industrial case study. J Syst Softw 111:1–33.  https://doi.org/10.1016/j.jss.2015.08.048 CrossRefGoogle Scholar
  3. Babar MA, Zhang H (2009) Systematic Literature Reviews in Software Engineering: Preliminary Results from Interviews with Researchers. In: International Symposium on Empirical Software Engineering and Measurement. IEEE, pp 346–355.  https://doi.org/10.1109/ESEM.2009.5314235
  4. Badampudi D, Wohlin C, Petersen K (2015) Experiences from Using Snowballing and Database Searches in Systematic Literature Studies. In: International Conference on Evaluation and Assessment in Software Engineering. ACM, pp 17:1–17:10.  https://doi.org/10.1145/2745802.2745818
  5. Bailey J, Zhang C, Budgen D, Turner M, Charters S (2007) Search Engine Overlaps: Do They Agree or Disagree? In: International Workshop on Realising Evidence-Based Software Engineering. IEEE, pp 2–2.  https://doi.org/10.1109/REBSE.2007.4
  6. Bandara W, Miskon S, Fielt E (2011) A systematic Tool-Supported method for conducting literature reviews in information systems. In: European Conference on Information SystemsGoogle Scholar
  7. Biolchini J, Mian PG, Natali ACC, Travassos GH (2005) Systematic Review in Software Engineering. Technical Report. RT - ES 679 / 05, Federal University of Rio de JaneiroGoogle Scholar
  8. Boell S K, Cecez-Kecmanovic D (2015) Debating Systematic Literature Reviews (SLR) and their Ramifications for IS: A Rejoinder to Mike Chiasson, Briony Oates, Ulrike Schultze, and Richard Watson. J Inf Technol 30(2):188–193.  https://doi.org/10.1057/jit.2015.15 CrossRefGoogle Scholar
  9. Booth A, Sutton A, Papaioannou D (2012) Systematic Approaches to a Successful Literature Review. Sage, Newbury ParkGoogle Scholar
  10. Brereton O P, Kitchenham B A, Budgen D, Turner M, Khalil M (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80(4):571–583.  https://doi.org/10.1016/j.jss.2006.07.009 CrossRefGoogle Scholar
  11. Budgen D, Brereton OP (2006) Performing Systematic Literature Reviews in Software Engineering. In: International Conference on Software Engineering. ACM, pp 1051–1052.  https://doi.org/10.1145/1134285.1134500
  12. Budgen D, Brereton O P, Drummond S, Williams N (2018a) Reporting systematic reviews: Some lessons from a tertiary study. Inf Softw Technol 95:62–74.  https://doi.org/10.1016/j.infsof.2017.10.017
  13. Budgen D, Brereton O P, Williams N, Drummond S (2018b) The Contribution that Empirical Studies Performed in Industry make to the Findings of Systematic Reviews: A Tertiary Study. Inf Softw Technol 94:234–244.  https://doi.org/10.1016/j.infsof.2017.10.012
  14. Carver J C (2010) Towards reporting guidelines for experimental replications: a proposal. In: International Workshop on Replication in Empirical Software EngineeringGoogle Scholar
  15. Collins J A, Fauser B C (2005) Balancing the strengths of systematic and narrative reviews. Hum Reprod Update 11(2):103–104.  https://doi.org/10.1093/humupd/dmh058 CrossRefGoogle Scholar
  16. da Silva F Q, Santos A L, Soares S, França A C C, Monteiro C V, Maciel F F (2011) Six years of systematic literature reviews in software engineering: an updated tertiary study. Inf Softw Technol 53(9):899–913.  https://doi.org/10.1016/j.infsof.2011.04.004 CrossRefGoogle Scholar
  17. Dresch A, Lacerda D P, Antunes JAV Jr (2014) Design Science Research: A Method for Science and Technology Advancement. Springer, Berlin.  https://doi.org/10.1007/978-3-319-07374-3
  18. Dybå T, Dingsøyr T (2008) Strength of Evidence in Systematic Reviews in Software Engineering. In: International Symposium on Empirical Software Engineering and Measurement. ACM, pp 178–187,  https://doi.org/10.1145/1414004.1414034
  19. Dybå T, Kitchenham BA, Jørgensen M (2005) Evidence-Based Software engineering for practitioners. IEEE Softw 22(1):58–65.  https://doi.org/10.1109/MS.2005.6 CrossRefGoogle Scholar
  20. Dybå T, Dingsøyr T, Hanssen GK (2007) Applying Systematic Reviews to Diverse Study Types: An Experience Report. In: International Symposium on Empirical Software Engineering and Measurement. IEEE, pp 225–234,  https://doi.org/10.1109/ESEM.2007.59
  21. Falagas M E, Pitsouni E I, Malietzis G A, Pappas G (2008) Comparison of PubMed, Scopus, Web of Science, and Google Scholar: Strengths and Weaknesses. FASEB J 22(2):338–342.  https://doi.org/10.1096/fj.07-9492LSF CrossRefGoogle Scholar
  22. Felizardo KR, Mendes E, Kalinowski M, Souza EF, Vijaykumar NL (2016) Using Forward Snowballing to Update Systematic Reviews in Software Engineering. In: International Symposium on Empirical Software Engineering and Measurement. ACM, pp 53:1–53:6.  https://doi.org/10.1145/2961111.2962630
  23. Fuhr N, Tsakonas G, Aalberg T, Agosti M, Hansen P, Kapidakis S, Klas C P, Kovács L, Landoni M, Micsik A, Papatheodorou C, Peters C, Sølvberg I (2007) Evaluation of Digital Libraries. Int J Digit Libr 8(1):21–38.  https://doi.org/10.1007/s00799-007-0011-z CrossRefGoogle Scholar
  24. Galvan J L, Galvan MC (2016) Writing Literature Reviews: A Guide for Students of the Social and Behavioral Sciences. Routledge, EvanstonGoogle Scholar
  25. Giles CL (2006) The Future of Citeseer: CiteseerX. In: European Conference on Principle and Practice of Knowledge Discovery in Databases. Springer, pp 2–2,  https://doi.org/10.1007/11871637_2
  26. Gómez O S, Juristo N, Vegas S (2014) Understanding replication of experiments in software engineering: a classification. Inf Softw Technol 56(8):1033–1048CrossRefGoogle Scholar
  27. Harzing A W, Alakangas S (2017) Microsoft Academic: Is the Phoenix Getting Wings?. Scientometrics 110(1):371–383.  https://doi.org/10.1007/s11192-016-2185-x CrossRefGoogle Scholar
  28. Hassler E, Carver J C, Hale D, Al-Zubidy A (2016) Identification of SLR tool needs – results of a community workshop. Inf Softw Technol 70:122–129.  https://doi.org/10.1016/j.infsof.2015.10.011 CrossRefGoogle Scholar
  29. Ilyas M, Khan S U (2015) Software integration in global software development: Success factors for GSD vendors. In: International conference on software engineering, artificial intelligence, Networking and Parallel/Distributed Computing. IEEE, pp 1–6.  https://doi.org/10.1109/SNPD.2015.7176187
  30. Imtiaz S, Bano M, Ikram N, Niazi M (2013) A Tertiary Study: Experiences of Conducting Systematic Literature Reviews in Software Engineering. In: International Conference on Evaluation and Assessment in Software Engineering. ACM, pp 177–182.  https://doi.org/10.1145/2460999.2461025
  31. Jalali S, Wohlin C (2012) Systematic Literature Studies: Database Searches vs. Backward Snowballing. In: International Symposium on Empirical Software Engineering and Measurement. ACM, pp 29–38.  https://doi.org/10.1145/2372251.2372257
  32. Khan K S, Kunz R, Kleijnen J, Antes G (2003) Five steps to conducting a systematic review. J R Soc Med 96(3):118–121.  https://doi.org/10.1177/014107680309600304 CrossRefGoogle Scholar
  33. Kitchenham B A, Brereton O P (2013) A systematic review of systematic review process research in software engineering. Inf Softw Technol 55(12):2049–2075.  https://doi.org/10.1016/j.infsof.2013.07.010 CrossRefGoogle Scholar
  34. Kitchenham BA, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Tech. Rep EBSE-2007-01. Keele University, University of DurhamGoogle Scholar
  35. Kitchenham B A, Brereton O P, Budgen D, Turner M, Bailey J, Linkman S (2009) Systematic literature reviews in software engineering - a systematic literature review. Inf Softw Technol 51(1):7–15.  https://doi.org/10.1016/j.infsof.2008.09.009 CrossRefGoogle Scholar
  36. Kitchenham B A, Pretorius R, Budgen D, Brereton O P, Turner M, Niazi M, Linkman S (2010) Systematic literature reviews in software engineering – a tertiary study. Inf Softw Technol 52(8):792–805.  https://doi.org/10.1016/j.infsof.2010.03.006 CrossRefGoogle Scholar
  37. Kitchenham BA, Brereton OP, Li Z, Budgen D, Burn A (2011) Repeatability of Systematic Literature Reviews. In: International Conference on Evaluation and Assessment in Software Engineering. IET, pp 46–55Google Scholar
  38. Kitchenham B A, Budgen D, Brereton OP (2016) Evidence-Based Software Engineering and Systematic Reviews. CRC Press, Boca RatonGoogle Scholar
  39. Levy Y, Ellis T J (2006) A systems approach to conduct an effective literature review in support of information systems research. Inform Sci Int J Emerging Transdiscip 9:181–212.  https://doi.org/10.28945/479 CrossRefGoogle Scholar
  40. Lex A, Gehlenborg N, Strobelt H, Vuillemot R, Pfister H (2014) Upset: Visualization of Intersecting Sets. IEEE Trans Vis Comput Graph 20(12):1983–1992.  https://doi.org/10.1109/TVCG.2014.2346248 CrossRefGoogle Scholar
  41. Lindsay R M, Ehrenberg A S C (1993) The design of replicated studies. Amer Stat 47(3):217–228.  https://doi.org/10.1080/00031305.1993.10475983 CrossRefGoogle Scholar
  42. MacDonell S, Shepperd M, Kitchenham B A, Mendes E (2010) How Reliable Are Systematic Reviews in Empirical Software Engineering?. IEEE Trans Softw Eng 36(5):676–687.  https://doi.org/10.1109/TSE.2010.28 CrossRefGoogle Scholar
  43. Mbiydzenyuy G (2015) Arrival Times with Hours of Service Regulations for Truck Drivers-Tracks and Gaps from Current Research. In: International Conference on Intelligent Transportation Systems. IEEE, pp 2631–2636.  https://doi.org/10.1109/ITSC.2015.423
  44. Moher D, Liberati A, Tetzlaff J, Altman DG, Group TP (2009) Preferred reporting items for systematic reviews and Meta-Analyses: The PRISMA statement. PLOS Med 6(7):1–6.  https://doi.org/10.1371/journal.pmed.1000097 CrossRefGoogle Scholar
  45. Okoli C, Schabram K (2010) A Guide to Conducting a Systematic Literature Review of Information Systems Research. Sprouts Working Papers on Information Systems.  https://doi.org/10.2139/ssrn.1954824
  46. Orduña-Malea E, Martín-Martín A, Ayllon JM, Delgado Lopez-Cozar E (2014) The Silent Fading of an Academic Search Engine: The Case of Microsoft Academic Search. Online Inf Rev 38(7):936–953.  https://doi.org/10.1108/OIR-07-2014-0169 CrossRefGoogle Scholar
  47. Petersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic Mapping Studies in Software Engineering. In: International Conference on Evaluation and Assessment in Software Engineering. BCS, pp 68–77Google Scholar
  48. Riaz M, Sulayman M, Salleh N, Mendes E (2010) Experiences Conducting Systematic Reviews from Novices’ Perspective. In: International Conference on Evaluation and Assessment in Software Engineering. BCS, pp 44–53Google Scholar
  49. Riedl R (2007) On the Replication of Positivist Case Study Research. In: European Conference on Information Systems, pp 1515–1526Google Scholar
  50. Sackett D L, Straus S E, Richardson W S, Rosenberg W, Haynes RB (1997) Evidence-Based Medicine: How to Practice and Teach EBM. WB Saunders Company, PhiladelphiaGoogle Scholar
  51. Shakeel Y, Krüger J, von Nostitz-Wallwitz I, Lausberger C, Campero Durand G, Saake G, Leich T (2018) (Automated) Literature Analysis - Threats and Experiences. In: International Workshop on Software Engineering for Science. ACM, pp 20–27.  https://doi.org/10.1145/3194747.3194748
  52. Shaw M (2003) Writing Good Software Engineering Research Papers. In: International Conference on Software Engineering. IEEE, pp 726–736.  https://doi.org/10.1109/ICSE.2003.1201262
  53. Soni G, Kodali R (2011) A critical analysis of supply chain management content in empirical research. Bus Process Manag J 17(2):238–266.  https://doi.org/10.1108/14637151111122338 CrossRefGoogle Scholar
  54. Tranfield D, Denyer D, Smart P (2003) Towards a methodology for developing Evidence-Informed management knowledge by means of systematic review. Br J Manag 14(3):207–222.  https://doi.org/10.1111/1467-8551.00375 CrossRefGoogle Scholar
  55. vom Brocke J, Simons A, Riemer K, Niehaves B, Plattfaut R, Cleven A (2015) Standing on the shoulders of giants: Challenges and recommendations of literature search in information systems research. Commun Assoc Inf Syst 37(9):205–224Google Scholar
  56. von Nostitz-Wallwitz I, Krüger J, Siegmund J, Leich T (2018) Knowledge Transfer from Research to Industry: A Survey on Program Comprehension. In: International Conference on Software Engineering. ACM, pp 300–301.  https://doi.org/10.1145/3183440.3194980
  57. Webster J, Watson RT (2002) Analyzing the Past to Prepare for the Future: Writing a Literature Review. Manag Inf Syst Q 26(2):xiii–xxiiiGoogle Scholar
  58. Wohlin C (2014) Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering. In: International Conference on Evaluation and Assessment in Software Engineering. ACM, pp 1–10.  https://doi.org/10.1145/2601248.2601268
  59. Wolfswinkel J F, Furtmueller E, Wilderom C P (2013) Using grounded theory as a method for rigorously reviewing literature. Eur J Inf Syst 22(1):45–55.  https://doi.org/10.1057/ejis.2011.51 CrossRefGoogle Scholar
  60. Yin RK (2018) Case Study Research and Applications: Design and Methods. Sage, Newbury ParkGoogle Scholar
  61. Zhang H, Babar MA (2010) On Searching Relevant Studies in Software Engineering. In: International Conference on Evaluation and Assessment in Software Engineering. BCS Learning & Development Ltd., pp 111–120Google Scholar
  62. Zhang H, Babar M A, Tell P (2011) Identifying relevant studies in software engineering. Inf Softw Technol 53(6):625–637.  https://doi.org/10.1016/j.infsof.2010.12.010 CrossRefGoogle Scholar
  63. Zhang H, Babar M A (2013) Systematic reviews in software engineering: an empirical investigation. Inf Softw Technol 55(7):1341–1354.  https://doi.org/10.1016/j.infsof.2012.09.008 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Otto-von-Guericke UniversityMagdeburgGermany
  2. 2.Harz University of Applied SciencesWernigerodeGermany
  3. 3.METOP GmbHMagdeburgGermany

Personalised recommendations