Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Methodologies, Methods, Techniques and Tools Used on SLR Elaboration: A Mapping Study

  • Conference paper
  • First Online:
Trends and Applications in Software Engineering (CIMPS 2018)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 865))

Included in the following conference series:

Abstract

The aim of this study is to perform a Systematic Literature Mapping (SLM) about methodologies, methods, techniques and tools used on the development of Systematic Literature Reviews (SLR). As a result, on the mapping, we expected to find and classify methodologies, methods, techniques and tools commonly used on SLR. In addition, we have considered other contexts such as, Medicine or Education with the purpose of getting multiple methodologies, methods, techniques and tools that allow performing SLR on efficient ways. It is mainly expected to identify techniques related to research questions formulation and the methods used for building search strings in order to get the higher number of studies associated to the research topic. In our study, we found multiple methodologies, methods, techniques and tools already implemented for performing SLRs On the mapping we describe some of them to highlight the most used and referenced studies.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Kitchenham, B., Brereton, P., Turner, M., Niazi, M., Linkman, S., Pretorius, R., Budgen, D.: Refining the systematic literature review process—two participant-observer case studies. Empir. Softw. Eng. 15(6), 618–653 (2010)

    Article  Google Scholar 

  2. Reed, L.: performing a literature review. In: Frontiers in Education Conference (FIE), pp. 380–383. IEEE (1998)

    Google Scholar 

  3. Brereton, P., Kitchenham, B., Budgen, D., Khalil, M., Turner, M.: Lessons from applying the systematic literature review process within the software engineering domain. J. Syst. Softw. 80(4), 571–583 (2007)

    Google Scholar 

  4. Lavallee, M., Robillard, P., Mirsalari, R.: Performing systematic literature reviews with novices: an iterative approach. Trans. Educ. 57(3), 175–181 (2014)

    Article  Google Scholar 

  5. Levy, Y., Timothy, E.: A systems approach to conduct an effective literature review in support of information systems research. Inf. Sci. J. 9, 171–181 (2006)

    Google Scholar 

  6. Wohlin, C.: Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, p. 38. ACM (2014)

    Google Scholar 

  7. Marshall, C., Brereton, P.: Tools to support systematic literature reviews in software engineering: a mapping study. In: Empirical Software Engineering and Measurement, pp. 296–299. IEEE (2013)

    Google Scholar 

  8. Kitchenham, B., Brereton, P., Budgen, D., Turner, M., Bailey, J., Linkman, S.: Systematic literature reviews in software engineering – a systematic literature review. Inf. Softw. Technol. 51(1), 7–15 (2009)

    Google Scholar 

  9. Kitchenham, B., Brereton, P., Li, Z., Budgen, D., Burn, A.: Repeatability of systematic literature reviews. In: Evaluation & Assessment in Software Engineering (EASE), pp. 46–55 (2011)

    Google Scholar 

  10. Borrego, M., Foster, M., Froyd, J.: Systematic literature reviews in engineering education and other developing interdisciplinary fields. J. Eng. Educ. 103(1), 45–76 (2014)

    Article  Google Scholar 

  11. Kitchenham, B., Charters, S.: Guidelines for Performing Systematic Literature Reviews in Software Engineering. Elsevier, Staffordshire (2007)

    Google Scholar 

  12. Ghafari, M., Saleh, M., Ebrahimi, T.: A federated search approach to facilitate systematic literature review in software engineering. Int. J. Softw. Eng. & Appl. 3(2), 13 (2012)

    Google Scholar 

  13. Riaz, M., Sulayman, M., Salleh, N., Mendes, E.: Experiences conducting systematic reviews from novices’ perspective. In: Proceedings of the 14th International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 44–53 (2010)

    Google Scholar 

  14. Carver, J., Hassler, E., Hernandes, E., Kraft, N.: Identifying barriers to the systematic literature review process. In: Empirical Software Engineering and Measurement, pp. 203–212. IEEE (2013)

    Google Scholar 

  15. Zhang, H., Babar, M.: Systematic reviews in software engineering: an empirical investigation. Inf. Softw. Technol. 55(7), 1341–1354 (2013)

    Article  Google Scholar 

  16. Petersen, K., Mujtaba, S., Feldt, R., Mattsson, M.: Systematic mapping studies in software engineering. 12th International Conference on Evaluation and Assessment in Software Engineering, pp. 1–10 (2008)

    Google Scholar 

  17. Barbara, K., Budgen, D., Brereton, P.: Using mapping studies as the basis for further research – a participant–observer case study. Inf. Softw. Technol. 53(6), 638–651 (2011)

    Article  Google Scholar 

  18. Dybå, T., Dingsøyr, T.: Empirical studies of agile software development: a systematic review. Inf. Softw. Technol. 50(9–10), 833–859 (2008)

    Google Scholar 

  19. Shea, B., Grimshaw, J., Wells, G., Boers, M., Andersson, N., Hamel, C., Bouther, L.: Development of AMSTAR: a measurement tool to assess the methodological. BMC Med. Res. Methodol. (2007)

    Google Scholar 

  20. Popay, J., Arai, L., Rodgers, M., Birtten, N.: Guidance on the conduct of narrative synthesis in systematic reviews: a product from the ESRC methods programme (2006)

    Google Scholar 

  21. Jalali, S., Wohlin, C.: Systematic literature studies: database searches vs. backward snowballing. In: Proceedings of the ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 29–38. ACM (2012)

    Google Scholar 

Download references

Acknowledgements

This work is framed within the “Soporte a RSL” project funded by “III Fondo Concursable para la Innovación en la Docencia Universitaria” by Dirección Académica del Profesorado and partially supported by the Grupo de Investigación y Desarrollo de Ingeniería de Software (GIDIS) from the Pontificia Universidad Católica del Perú.

Author information

Authors and Affiliations

  1. Departamento de Ingeniería, Pontificia Universidad Católica del Perú, Lima, Peru

    Marco Palomino, Abraham Dávila & Karin Melendez

Authors
  1. Marco Palomino
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abraham Dávila
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Karin Melendez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abraham Dávila .

Editor information

Editors and Affiliations

  1. Unidad Zacatecas, Centro de Investigación en Matemáticas A. C., Zacatecas, Zacatecas, Mexico

    Jezreel Mejia

  2. Unidad Zacatecas, Centro de Investigación en Matemáticas A. C., Zacatecas, Zacatecas, Mexico

    Mirna Muñoz

  3. Departamento de Engenharia Informática, Universidade de Coimbra, Coimbra, Portugal

    Álvaro Rocha

  4. Departamento de Ciencias Computacionales, Universidad de Guadalajara, Guadalajara, Jalisco, Mexico

    Adriana Peña

  5. Centro Universitario de Ciencias Exactas, Universidad de Guadalajara, Guadalajara, Jalisco, Mexico

    Marco Pérez-Cisneros

Appendix A. Studies Included in the Review

Appendix A. Studies Included in the Review

  • [D1] Cobo, M., López‐Herrera, A., Herrera‐Viedma, E., & Herrera, F. Science Mapping Software Tools: Review, Analysis, and Cooperative Study Among Tools. Journal of the Association for Information Science and Technology, 67(7), 1382–1402. (2011).

  • [D2] Borrego, M., Foster, M., & Froyd, J. Systematic Literature Reviews in Engineering Education and other Developing Interdisciplinary Fields. Journal of Engineering Education, 103(1), 45–76. (2014).

  • [D3] Godwin, A. Visualizing Systematic Literature Reviews to Identify New Areas of Research. Frontiers in Education Conference (FIE) (págs. 1–8). IEEE. (2016).

  • [D4] Marshall, C., & Brereton, P. Tools to Support Systematic Literature Reviews in Software Engineering: A Mapping Study. Empirical Software Engineering and Measurement (págs. 296–299). IEEE. (2013).

  • [D5] Marshall, C., Brereton, P., & Kitchenham, B. Tools to Support Systematic Reviews in Software Engineering: A Feature Analysis. Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (pág. 13). ACM. (2014).

  • [D6] Bailey, J., Zhang, C., Budgen, D., Turner, M., & Charters, S. Search Engine Overlaps: Do they agree or disagree? Realising Evidence-Based Software Engineering (pág. 2). IEEE. (2007).

  • [D7] Sherman, S., Bohler, J., & Shehane, R. Improving Student Literature Research Skills. Global Education Journal, 2016(1), 18–31. (2016).

  • [D8] Adams, J., Hillier-Brown, F., Moore, H., Lake, A., Araujo-Soares, V., White, M., & Summerbell, C. Searching and Synthesising ‘grey literature’and ‘grey information’ in Public Health: Critical Reflections on Three Case Studies. Systematic Reviews, 5(1), 164. (2016).

  • [D9] Levy, Y., & Timothy, E. A Systems Approach to Conduct an Effective Literature Review in Support of Information Systems Research. Informing Science Journal, 9, 171–181. (2006).

  • [D10] Kastner, M., Tricco, A., Soobiah, C., Lillie, E., Perrier, L., Horsley, T., & Straus, S. What is the Most Appropriate Knowledge Synthesis method to Conduct a Review? Protocol for a Scoping Review. BMC Medical Research Methodology, 12(1), 114. (2012).

  • [D11] Gauch, S., & Smith, J. Query Reformulation Strategies for an Intelligent Search Intermediary. AI Systems in Government Conference (págs. 65–71). IEEE. (1989).

  • [D12] Torraco, R. Writing Integrative Literature Reviews: Guidelines and Examples. Human resource development review, 4(3), 353–367. (2005).

  • [D13] Kitchenham, B., Brereton, P., Budgen, D., Turner, M., Bailey, J., & Linkman, S. Systematic Literature Reviews in Software Engineering – A Systematic Literature Review. Information and Software Technology, 7–15. (2009).

  • [D14] Krainovich-Miller, B., Haber, J., Yost, J., & Kaplan, S. Evidence-based Practice Challenge: Teaching Critical Appraisal of Systematic Reviews and Clinical Practice Guidelines to Graduate Students. Journal of Nursing Education, 48(4), 186–195. (2009).

  • [D15] Rainer, A., & Beecham, S. Supplementary Guidelines, Assessment Scheme and evidence-based evaluations of the use of Evidence Based Software Engineering. University of Hertfordshire. (2008).

  • [D16] Biolchini, J., Mian, P., Natali, A., & Travassos, G. Systematic Review in Software Engineering. System Engineering and Computer Science Department COPPE/UFRJ. (2005).

  • [D17] Carliner, S. Workshop in Conducting Integrative Literature Reviews. Professional Communication Conference (IPCC) (págs. 1–3). IEEE. (2011).

  • [D18] Boaz, A., Ashby, D., & Young, K. Systematic Rreviews: What have they got to Offer Evidence Based Policy and Practice? London: ESRC UK Centre for Evidence Based Policy and Practice. (2002).

  • [D19] Saad, F., Mathiak, B., & Mutschke, P. Supporting Literature Review by Searching, Visualizing and Navigating Related Papers. Cloud and Green Computing (CGC) (págs. 363–368). IEEE. (2013).

  • [D20] Kitchenham, B. Procedures for Performing Systematic Reviews. Keele University, UK. (2004).

  • [D21] Reed, L. Performing a Literature Review. Frontiers in Education Conference (FIE) (págs. 380–383). IEEE. (1998).

  • [D22] Coppola, N., & Carliner, S. Is our peer-reviewed literature sustainable? Professional Communication Conference (IPCC) (págs. 1–7). IEEE. (2011).

  • [D23] Okoli, C., & Schabram, K. A Guide to Conducting a Systematic Literature Review of Information Systems Research. (2010).

  • [D24] Rúbio, T., & Gulo, C. Enhancing Academic Literature Review through Relevance Recommendation. 11th Iberian Conference on Information Systems and Technologies, (págs. 15–18). Gran Canaria, Spain. (2016).

  • [D25] Hart, C. Doing a Literature Review: Releasing the Social Science Research Imagination. London: SAGE. (1998).

  • [D26] Dieste, O., & Padua, A. Developing Search Strategies for Detecting Relevant Experiments for Systematic Reviews. Empirical Software Engineering and Measurement (págs. 215–224). IEEE. (2007).

  • [D27] Laghrabli, S., Benabbou, L., & Berrado, A. A New Methodology for Literature Review Analysis Using Association Rules Mining. Intelligent Systems: Theories and Applications (SITA) (págs. 1–6). IEEE. (2015).

  • [D28] Singh, P., & Singh, K. Exploring Automatic Search in Digital Libraries–A Caution Guide for Systematic Reviewers. Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering (págs. 236–241). ACM. (2017).

  • [D29] Zhang, H., & Babar, M. Systematic Reviews in Software Engineering: An Empirical Investigation. Information and Software Technology, 55(7), 1341–1354. (2013).

  • [D30] Kitchenham, B., Brereton, P., Li, Z., Budgen, D., & Burn, A. Repeatability of Systematic Literature Reviews. In Evaluation & Assessment in Software Engineering. Evaluation & Assessment in Software Engineering (EASE), (págs. 46–55). (2011).

  • [D31] Lavallee, M., Robillard, P., & Mirsalari, R. Performing Systematic Literature Reviews with Novices: An Iterative Approach. Transactions on Education, 57(3), 175–181. (2014).

  • [D32] Felizardo, K., Salleh, N., Martins, R., Mendes, E., MacDonell, S., & Maldonado, J. Using Visual Text Mining to Support the Study Selection Activity in Systematic Literature Reviews. Empirical Software Engineering and Measurement (ESEM), (págs. 77–86). (2011).

  • [D33] Zhou, X., Jin, Y., Zhang, H., Li, S., & Huang, X. A Map of Threats to Validity of Systematic Literature Reviews in Software Engineering. Software Engineering Conference (APSEC), (págs. 153–160). (2016).

  • [D34] Jalali, S., & Wohlin, C. Systematic Literature Studies: Database Searches vs. Backward Snowballing. Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement (págs. 29–38). ACM. (2012).

  • [D35] Carver, J., Hassler, E., Hernandes, E., & Kraft, N. Identifying Barriers to the Systematic Literature Review Process. Empirical Software Engineering and Measurement (págs. 203–212). IEEE. (2013).

  • [D36] Aveyard, H. Doing a Literature Review in Health and Social Care: A Practical Guide. McGraw-Hill Education (UK). (2014).

  • [D37] Reutzel, T. Outpatient Drug Insurance: a Framework to Guide Literature Review, Research, and Teaching. American Journal of Pharmaceutical Education, 62(1). (1998).

  • [D38] Wohlin, C. Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering. Proceedings of the 18th international conference on evaluation and assessment in software engineering (pág. 38). ACM. (2014).

  • [D39] Wang, Y., Liu, D., Qu, H., Luo, Q., & Ma, X. A Guided Tour of Literature Review: Facilitating Academic Paper Reading with Narrative Visualization. Proceedings of the 9th International Symposium on Visual Information Communication and Interaction, (págs. 17–24). (2016).

  • [D40] Singh, P., & Singh, K. Exploring Automatic Search in Digital Libraries–A Caution Guide for Systematic Reviewers. Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering (págs. 236–241). ACM. (2017).

  • [D41] Conn, V., Isaramalai, S., Rath, S., Jantarakupt, P., Wadhawan, R., & Dash, Y. Beyond MEDLINE for literature searches. Journal of Nursing Scholarship, 35(2), 117–182. (2003).

  • [D42] Yoshii, A., Plaut, D., McGraw, K., Anderson, M., & Wellik, K. Analysis of the Reporting of Search Strategies in Cochrane Systematic Reviews. Journal of the Medical Library Association: JMLA, 97(1), 21. (2009).

  • [D43] Im, E., & Chang, S. A Systematic Integrated Literature Review of Systematic Integrated Literature Reviews in Nursing. Journal of Nursing Education, 51(11), 632–636. (2012).

  • [D44] Wong, S., Wilczynski, N., & Haynes, B. Comparison of top-performing Search Strategies for Detecting Clinically Sound Treatment Studies and Systematic Reviews in MEDLINE and EMBASE. Journal of the Medical Library Association, 94(4), 451. (2006).

  • [D45] Sampson, M., McGowan, J., Cogo, E., & Horsley, T. Managing database overlap in systematic reviews using Batch Citation Matcher: case studies using Scopus. Journal of the Medical Library Association, 94(4), 461. (2006).

  • [D46] Riaz, M., Sulayman, M., Salleh, N., & Mendes, E. Experiences Conducting Systematic Reviews from Novices’ Perspective. Proceedings of the 14th international conference on Evaluation and Assessment in Software Engineering (EASE), (págs. 44–53). (2010).

  • [D47] Boell, S., & Cecez-Kecmanovic, D. On being ‘Systematic’ in Literature Reviews in IS. Journal of Information Technology, 30(2), 161–173. (2015).

  • [D48] Montori, V., Wilczynski, N., Morgan, D., & Haynes, B. Optimal Search Strategies for Retrieving Systematic Reviews from Medline: Analytical Survey. BMJ. (2005).

  • [D49] Octaviano, F., Felizardo, K., Maldonado, J., & Fabbri, S. Semi-Automatic Selection of Primary Studies in Systematic Literature Reviews: is it Reasonable? Empirical Software Engineering and Measurement (ESEM), 20(6), 1898–1917. (2015).

  • [D50] Kitchenham, B., Brereton, P., Turner, M., Niazi, M., Linkman, S., Pretorius, R., & Budgen, D. Refining the Systematic Literature Review Process—two participant-Observer Case Studies. Empirical Software Engineering, 15(6), 618–653. (2010).

  • [D51] Felizardo, K., Riaz, M., Sulayman, M., Mendes, E., MacDonell, S., & Maldonado, J. Analysing the Use of Graphs to Represent the Results of Systematic Reviews in Software Engineering. Software Engineering (SBES) (págs. 174–183). IEEE. (2011).

  • [D52] Santos, R., & Da Silva, F. Motivation to Perform Systematic Reviews and their Impact on Software Engineering Practice. Empirical Software Engineering and Measurement (págs. 292–295). IEEE. (2013).

  • [D53] Dieste, O., Anna, G., Juristo, N., & Saxena, H. Quantitative Determination of the Relationship Between Internal Validity and Bias in Software Engineering Experiments: Consequences for Systematic Literature Reviews. Empirical Software Engineering and Measurement (ESEM) (págs. 285–294). IEEE. (2011).

  • [D54] Babar, M., & Zhang, H. Systematic Literature Reviews in Software Engineering: Preliminary Results from Interviews With Researchers. Empirical Software Engineering and Measurement (págs. 346–355). IEEE. (2009).

  • [D55] Pagani, R., Kovaleski, J., & Resende, L. Methodi Ordinatio: A Proposed Methodology to Select and Rank Relevant Scientific Papers Encompassing the Impact Factor, Number of Citation, and Year of Publication. Scientometrics, 105(3), 2109–2135. (2015).

  • [D56] Ghafari, M., Saleh, M., & Ebrahimi, T. A Federated Search Approach to Facilitate Systematic Literature Review in Software Engineering. International Journal of Software Engineering & Applications, 3(2), 13. (2012).

  • [D57] Meline, T. Selecting Studies for Systematic Review: Inclusion and Exclusion Criteria. (2006).

  • [D58] Methley, A., Campbell, S., Chew-Graham, C., McNally, R., & Cheraghi-Sohi, S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC health services research, 14(1), 579. (2014).

  • [D59] Oates, B., & Capper, G. Using Systematic Reviews and Evidence-Based Software Engineering with Masters Students. Evaluation & Assessment in Software Engineering (EASE), (págs. 20–21). (2009).

  • [D60] Miranda, J., Muñoz, M., Uribe, E., Márquez, J., Uribe, G., & Valtierra, C. Systematic Review Tool to Support the Establishment of a Literature Review. New Perspectives in Information Systems and Technologies, (págs. 171–181). (2014).

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Palomino, M., Dávila, A., Melendez, K. (2019). Methodologies, Methods, Techniques and Tools Used on SLR Elaboration: A Mapping Study. In: Mejia, J., Muñoz, M., Rocha, Á., Peña, A., Pérez-Cisneros, M. (eds) Trends and Applications in Software Engineering. CIMPS 2018. Advances in Intelligent Systems and Computing, vol 865. Springer, Cham. https://doi.org/10.1007/978-3-030-01171-0_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-01171-0_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-01170-3

  • Online ISBN: 978-3-030-01171-0

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation