Quality & Quantity

, Volume 47, Issue 1, pp 383–410 | Cite as

Empirical research consolidation: a generic overview and a classification scheme for methods



I define research consolidation as comparing and combining (amalgamating or, at least, juxtaposing) results or other output from multiple previously conducted research activities, relevant to a goal or topic of interest. The concept, not limited to empirical research or any specific method, is similar to and subsumes what is often referred to as research synthesis or research integration; I explain the concept and my rationale for the new term. After introducing the broader concept, I focus on consolidation of empirical research. As background for this, I offer a brief introduction to empirical research. Then I provide a generic overview of empirical research consolidation, abstracted from several of its specific methods, such as quantitative meta-analysis and qualitative meta-synthesis; this sort of overview—method agnostic and capturing the commonality—is not readily available in the literature. At the core of the paper, I propose a scheme to classify methods for empirical research consolidation followed by a review and classification of selected methods, illustrating the scheme. I also discuss and clarify related terminology. The classification scheme differentiates between two major attributes of methods—consolidation technique and type of content to be consolidated—that may reflect the positivist or constructivist research paradigm. Consolidation techniques could be aggregative, interpretive, or a combination thereof. Content consolidated, from empirical research reports, could be empirical or not and could be from quantitative research, qualitative research, other kinds of empirical research, or a combination thereof.


Research synthesis Research integration Aggregative vs. interpretive consolidation Meta-analysis Meta-synthesis Empirical research 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Alford, M.L., Mendes, E.: Scholarly research process: investigating the effects of link type and directionality. In: Proceedings of the 20th ACM Conference on Hypertext and Hypermedia, HT’09, June 29–July 1, 2009, Torino, Italy, pp. 99–108 (2009)Google Scholar
  2. Bartneck, C.: Using the metaphysics of quality to define design science. In: Proceedings of the 4th International Conference on Design Science Research in Information Systems and Technology (DESRIST 2009), May 7–8, 2009, Malvern, PA, USA (2009)Google Scholar
  3. Baskerville, R., Pries-Heje, J., Venable, J.: Soft design science methodology. Design science research in information systems and technologies. In: Proceedings of the 4th International Conference on Design Science Research in Information Systems and Technology [DESRIST ‘09], Malvern, Pennsylvania, USA, May 7–8, 2009. SESSION: Design research: methods. Article No. 9 (2009)Google Scholar
  4. Bassey M.: Creating education through research. Brit. Educ. Res. J. 18(1), 3–16 (1992)CrossRefGoogle Scholar
  5. Bayazit N.: Investigating design: a review of forty years of design research. Des. Issues 20(1), 16–29 (2004)CrossRefGoogle Scholar
  6. Biolchini, J., Mian, P.G., Natali, A.C.C., Travassos, G.H.: Systematic review in software engineering. Technical Report ES 679/05, Systems Engineering and Computer Science Department, COPPE/UFRJ. Rio de Janeiro, Brazil, May 2005. http://www.cin.ufpe.br/~in1037/leitura/systematicReviewSE-COPPE.pdf. (2005). Accessed September 2010
  7. Blagosklonny, M.V., Pardee, A.B.: Conceptual biology: unearthing the gems. Nature 416, 373 (28 March 2002) http://www.nature.com/nature/journal/v416/n6879/full/416373a.html (2002). Accessed September 2010
  8. Bondas T., Hall E.O.C.: Challenges in approaching metasynthesis research. Qual. Health Res. 17(1), 113–121 (2007)CrossRefGoogle Scholar
  9. Bryman A.: The debate about quantitative and qualitative research: a question of method or epistemology?. Brit. J. Sociol. 35(1), 75–92 (1984)CrossRefGoogle Scholar
  10. Burger, M.J.C.: Towards a framework for the elicitation of dilemmas. Qual. Quant. 42, 541–562 (2008). Published online: 8 August 2007 (2008)Google Scholar
  11. Bushman B.J., Cooper H.M.: Effects of alcohol on human aggression: an integrative research review. Psychol. Bull. 107(3), 341–354 (1990)CrossRefGoogle Scholar
  12. Chalmers I., Haynes B.: Systematic reviews: reporting, updating, and correcting systematic reviews of the effects of health care. BMJ 309, 862–865 (1994)CrossRefGoogle Scholar
  13. Chen C., Rada R.: Interacting with hypertext: a meta-analysis of experimental studies. Hum.-Comput. Interact. 11(2), 125–156 (1996)CrossRefGoogle Scholar
  14. Creswell J.W.: Research Design. Qualitative & Quantitative Approaches. Sage Publications, Thousand Oaks, CA (1994)Google Scholar
  15. Creswell J.W., Garrett A.L.: The ‘movement’ of mixed methods research and the role of educators. South Afr. J. Educ. 28, 321–333 (2008)Google Scholar
  16. Cross N.: Designerly ways of knowing: design discipline versus design science. Des. Issues 17(3), 49–55 (2001)CrossRefGoogle Scholar
  17. Davison R.M., Martinsons M.G., Kock N.: Principles of canonical action research. Inform. Syst. J. 14, 65–86 (2004)CrossRefGoogle Scholar
  18. Dixon-Woods, M., Agarwal, S., Young, B., Jones, D., Sutton, A.: Integrative approaches to qualitative and quantitative evidence. Health Development Agency, London. Published on http://www.hda-online.org.uk/documents/integrative_approaches.pdf. (2004). Accessed August 2010
  19. Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A.: Synthesising qualitative and quantitative evidence: a review of possible methods. J. Health Serv. Res. Policy 10(1), 45–53 (2005)CrossRefGoogle Scholar
  20. Dowell J., Hudson H.: A qualitative study of medication-taking behaviour in primary care. Fam. Pract. 14(5), 369–375 (1997)CrossRefGoogle Scholar
  21. Ellis M.V.: Conducting and reporting integrative research reviews: accumulating scientific knowledge. Couns. Educ. Superv. 30(3), 0011–0035 (1991)Google Scholar
  22. Finfgeld D.L.: Metasynthesis: the state of the art—so far. Qual. Health Res. 13(7), 893–904 (2003)CrossRefGoogle Scholar
  23. Firestone W.A.: Meaning in method: the rhetoric of quantitative and qualitative research. Educ. Res. 16(7), 16–21 (1987)Google Scholar
  24. Galliers R.D., Land F.F.: Choosing appropriate information systems research methodologies. Commun. ACM 30(11), 900–902 (1987)CrossRefGoogle Scholar
  25. Gallupe R.B., Tan F.B.: A research manifesto for global information management. J. Glob. Inform. Manage. 7(3), 5–18 (1999)Google Scholar
  26. Gambling T., Brown P., Hogg P.: Research in our practice—a requirement not an option: discussion paper. Radiography 9, 71–76 (2003)CrossRefGoogle Scholar
  27. Gemperle, F., DiSalvo, C., Forlizzi, J., Yonkers, W.: The Hug: a new form for communication. Designing For User Experiences. In: Proceedings of the 2003 conference on Designing for user experiences, San Francisco, California, SESSION: DUX in practice II, pp. 1–4 (2003)Google Scholar
  28. Glahn D.C., Ragland J.D., Abramoff A., Barrett J., Laird A.R., Bearden C.E., Velligan D.I.: Beyond hypofrontality: a quantitative meta-analysis of functional neuroimaging studies of working memory in schizophrenia. Hum. Brain Mapp. 25(1), 60–69 (2005)CrossRefGoogle Scholar
  29. Glasmeier A.K., Farrigan T.: Understanding community forestry: a qualitative meta-study of the concept, the process, and its potential for poverty alleviation in the united states case. Geogr. J. 171(1), 56–69 (2005)CrossRefGoogle Scholar
  30. Glass G.V.: Meta-analysis: the quantitative synthesis of research findings. In: Green, J.L., Camilli, G., Elmore, P.B. (eds) Handbook of Complementary Methods in Education Research, Chapter 25, Lawrence Erlbaum Associates, Mahwah, NJ (2006)Google Scholar
  31. Glass R.L., Vessey I., Ramesh V.: Research in software engineering: an analysis of the literature. Inform. Software Technol. 44(8), 491–506 (2002)CrossRefGoogle Scholar
  32. Guba E.G., Lincoln Y.S.: Competing paradigms in qualitative research. In: Denzin, N.K., Lincoln, Y.S. (eds) Handbook of Qualitative Research, pp. 105–117. Sage, Thousand Oaks, CA (1994)Google Scholar
  33. Hara, N., Bonk, C.J., Angeli, C.: Content analysis of online discussion in an applied educational psychology course. CRLT Technical Report No. 2-98, The Center for Research on Learning and Technology, Indiana University, Bloomington, IN, November 20, 1998. http://users.cdli.ca/bmann/0_ARTICLES/Web_Bonk.pdf (1998). Accessed December 2010
  34. Harrison L.L.: Pulling it all together: the importance of integrative research reviews and meta-analyses in nursing. Guest editorial. J. Adv. Nurs 24(2), 224–225 (1996)CrossRefGoogle Scholar
  35. Head, A.J.: Beyond Google: how do students conduct academic research? First Monday 12(8), August 2007, http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/1998/1873 (2007). Accessed November 1, 2010
  36. Held I.M.: The gap between simulation and understanding in climate modeling. Bull. Am. Meteorol. Soc 86(11), 1609–1614 (2005)CrossRefGoogle Scholar
  37. Higgins, J., Green, S. (eds.): Cochrane Handbook for Systematic Reviews of Interventions, Version 5.0.2 [updated September 2009]. The Cochrane Collaboration (2009). Available from http://www.cochrane-handbook.org.
  38. Holz, H.J., Applin, A., Haberman, B., Joyce, D., Purchase, H., Reed, C.: Research methods in computing: what are they, and how should we teach them? Annual Joint Conference Integrating Technology into Computer Science Education, Working group reports on Innovation and technology in computer science education (ITiCSE), ITiCSE-2006 working group reports, Bologna, Italy, 2006, pp. 96–114 (2006)Google Scholar
  39. Howe K., Eisenhart M.: Standards for qualitative (and quantitative) research: a prolegomenon. Educ. Res. 19(4), 2–9 (1990)Google Scholar
  40. Inui T.S.: The virtue of qualitative and quantitative research. Ann. Intern. Med. 125(9), 770–771 (1996)Google Scholar
  41. Jackson G.B.: Methods for integrative reviews. Rev. Educ. Res. 50(3), 438–460 (1980)Google Scholar
  42. Jager, J., Kooy, A., Lehert, P., Wulffelé, M., Kolk, J., Bets, D., Verburg, J., Donker, A., Stehouwer, C.: Long term treatment with metformin in patients with type 2 diabetes and risk of vitamin B-12 deficiency: randomised placebo controlled trial. BMJ 2010 340, c2181 (2010)Google Scholar
  43. Jarvinen, P.H.: Research questions guiding selection of an appropriate research method. ECIS [European Conference on Information Systems] 2000 Proceedings, 3–5 July 2000, paper 26, pp. 124–131. http://aisel.aisnet.org/ecis2000/26. (2000)
  44. Järvinen P.: Action research is similar to design science. Qual. Quant. 41(1), 37–54 (2007)CrossRefGoogle Scholar
  45. Jensen L.A., Allen M.N.: Meta-synthesis of qualitative findings. Qual. Health Res. 6(4), 553–560 (1996)CrossRefGoogle Scholar
  46. Kassarjian H.H.: Content analysis in consumer research. J. Consum. Res. 4(1), 8–18 (1977)CrossRefGoogle Scholar
  47. Kitchenham B.A., Pfleeger S.L., Pickard L.M., Jones P.W., Hoaglin D.C., El Emam K., Rosenberg J.: Preliminary guidelines for empirical research in software engineering. IEEE Trans. Software Eng. 28(8), 721–734 (2002)CrossRefGoogle Scholar
  48. Knafl K.A., Howard M.J.: Interpreting and reporting qualitative research. Res. Nurs. Health 7(1), 17–24 (1984)CrossRefGoogle Scholar
  49. Kulik, J.A.: An analysis of the research on ability grouping: historical and contemporary perspectives. Ability Grouping: Research-Based Decision Making Series. National Research Center on the Gifted and Talented. Report No. NRC-G/T-9204, February 1992 (1992)Google Scholar
  50. Leuzinger-Bohleber, M., Fischmann, T.: What is conceptual research in psychoanalysis? Int. J. Psycho-Anal. 87(Part 5), 1355–1386 (2006)Google Scholar
  51. March S.T., Smith G.F.: Design and natural science research on information technology. Decis. Support Syst. 15(4), 251–266 (1995)CrossRefGoogle Scholar
  52. McCormick J., Rodney P., Varcoe C.: Reinterpretations across studies: an approach to meta-analysis. Qual. Health Res. 13(7), 933–944 (2003)CrossRefGoogle Scholar
  53. Merriam-Webster’s 11th Collegiate Dictionary and Thesaurus. (Electronic edition, version 4.0.) Merriam-Webster Inc. (2008)Google Scholar
  54. Morrison J., George J.F.: Exploring the software engineering component in MIS research. Commun. ACM 38(7), 80–91 (1995)CrossRefGoogle Scholar
  55. Noblit, G.W., Hare, R.D.: Meta-ethnography: Synthesizing qualitative studies. Qualitative Research Methods Series, vol. 11. Sage Publications, London (1988)Google Scholar
  56. Onwuegbuzie, A.J.: Effect sizes in qualitative research. Annual Meeting of the Association for the Advancement of Educational Research (AAER), Ponte Vedra, FL, November 2000 (2000)Google Scholar
  57. Paterson B.L., Thorne S.E., Canam C., Jillings C.: Meta-study of Qualitative Health Research: A Practical Guide to Meta-analysis and Metasynthesis. Sage, Thousand Oaks, CA (1995)Google Scholar
  58. Paterson, B.L., Thorne, S.E., Canam, C., Jillings, C. (eds.): Meta-Study of Qualitative Health Research: A Practical Guide to Meta-analysis and Meta-Synthesis (Methods in Nursing Research). Sage Publications, Thousand Oaks, CA (2001)Google Scholar
  59. Ponterotto J.G.: Qualitative research in counseling psychology: a primer on research paradigms and philosophy of science. J. Counsel. Psychol. 52(2), 126–136 (2005)CrossRefGoogle Scholar
  60. Roux G., Dingley C., Bush H.: Inner strength in women: Metasynthesis of qualitative findings in theory development. J. Theory Constr. Testing Lisle 6(1), 86–93 (2002)Google Scholar
  61. Rudner, L.M., Schafer, W.D.: How to write a scholarly research report. Pract. Assess. Res. Evaluation 6(13), http://pareonline.net/getvn.asp?v=6&n=13 (1999). Accessed August 2010
  62. Russell, C.L.: An overview of the integrative research review. Progress in Transplantation, March 2005, http://findarticles.com/p/articles/mi_qa4117/is_200503/ai_n13476203/. (2005). Accessed December 22, 2010
  63. Sale J.E.M., Lohfeld L.H., Brazil K.: Revisiting the quantitative-qualitative debate: implications for mixed-methods research. Qual. Quant. 36(1), 43–53 (2002)CrossRefGoogle Scholar
  64. Sambunjak D., Straus S., Marusic A.: A systematic review of qualitative research on the meaning and characteristics of mentoring in academic medicine. J. Gen. Intern. Med. 25(1), 72–78 (2010)CrossRefGoogle Scholar
  65. Sandelowski M.: Whatever happened to qualitative description?. Res. Nurs. Health 23, 334–340 (2000)CrossRefGoogle Scholar
  66. Sandelowski M., Barroso J.: Writing the proposal for a qualitative research methodology project. Qual. Health Res. 13(6), 781–820 (2003a)CrossRefGoogle Scholar
  67. Sandelowski M., Barroso J.: Classifying the findings in qualitative studies. Qual. Health Res. 13(7), 905–923 (2003b)CrossRefGoogle Scholar
  68. Sandelowski M., Docherty S., Emden C.: Qualitative metasynthesis: issues and techniques. Res. Nurs. Health 20, 365–371 (1997)CrossRefGoogle Scholar
  69. Sandelowski M., Voils C.I., Barroso J.: Defining and designing mixed research synthesis studies. Res. Schools 13(1), 29–40 (2006)Google Scholar
  70. Segal, J., Grinyer, A., Sharp, H.: The type of evidence produced by empirical software engineers. In: Proceedings of the 2005 International Conference on Software Engineering, Session: Realising Evidence-Based Software Engineering (REBSE), St. Louis, Missouri, pp. 1–4 (2005)Google Scholar
  71. Shelton J.H.: Handbook for Technical Writing. NTC Business Books, Chicago (1994)Google Scholar
  72. Sjøberg D., Dyboa T., Jørgensen M.: The future of empirical methods in software engineering research. In: Briand, L., Wolf, A. (eds) Future of Software Engineering, IEEE-CS Press, Los Alamitos, CA (2007)Google Scholar
  73. Suri, H.: A critique of contemporary methods of research synthesis. Post-Script, vol. 1, 1, September 2000, Faculty of Education, University of Melbourne, Australia. Available at http://www.edfac.unimelb.edu.au/research/resources/student_res/postscriptfiles/vol1/vol1_1_suri.pdf. (2000). Accessed September 2010
  74. Sutton R.I., Staw B.M.: What theory is Not. Admin. Sci. Quart. 40(3), 371–384 (1995)CrossRefGoogle Scholar
  75. Thorne S.: Data analysis in qualitative research. Evidence Based Nurs. 3, 68–70 (2000)CrossRefGoogle Scholar
  76. Thorne S., Jensen L., Kearney M., Noblit G., Sandelowski M.: Qualitative metasynthesis: reflections on methodological orientation and ideological agenda. Qual. Health Res. 14(10), 1342–1365 (2004)CrossRefGoogle Scholar
  77. Venable, J.R.: The role of theory and theorising in design science research. In: Proceedings of the 1st International Conference on Design Science in Information Systems and Technology (DESRIST 2006), February 24–25, 2006, Claremont, CA (2006)Google Scholar
  78. Voils C., Sandelowski M., Barroso J., Hasselblad V.: Making sense of qualitative and quantitative findings in mixed research synthesis studies. Field Methods 20(1), 3–25 (2008)CrossRefGoogle Scholar
  79. Whittemore R., Knafl K.: The integrative review: updated methodology. J. Adv. Nurs. 52(5), 546–553 (2005)CrossRefGoogle Scholar
  80. Williams A.: Perspectives on spirituality at the end of life: a meta-summary. Palliative Supportive Care 4(4), 407–417 (2006)Google Scholar
  81. Wolf F.M.: Meta-Analysis: Quantitative Methods for Research Synthesis. Sage Publications, Inc., Newbury Park, CA (1986)Google Scholar
  82. Zuber-Skerritt O.: Action learning and action research: paradigm, praxis and programs. In: Sankara, S., Dick, B., Passfield, R. (eds) Effective Change Management through Action Research and Action Learning: Concepts, Perspectives, Processes and Applications, Chapter 1., pp. 1–20. Southern Cross University Press, Lismore (2001)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2011

Authors and Affiliations

  1. 1.Lockheed Martin CorporationEaganUSA

Personalised recommendations