Empirical Software Engineering

, Volume 16, Issue 4, pp 430–459 | Cite as

Qualitative research on software development: a longitudinal case study methodology

  • Laurie McLeod
  • Stephen G. MacDonell
  • Bill Doolin


This paper reports the use of a qualitative methodology for conducting longitudinal case study research on software development. We provide a detailed description and explanation of appropriate methods of qualitative data collection and analysis that can be utilized by other researchers in the software engineering field. Our aim is to illustrate the utility of longitudinal case study research, as a complement to existing methodologies for studying software development, so as to enable the community to develop a fuller and richer understanding of this complex, multi-dimensional phenomenon. We discuss the insights gained and lessons learned from applying a longitudinal qualitative approach to an empirical case study of a software development project in a large multi-national organization. We evaluate the methodology used to emphasize its strengths and to address the criticisms traditionally made of qualitative research.


Qualitative methods Longitudinal case study Software development Empirical research 



This research was funded through a Top Achiever Doctoral Scholarship by the Tertiary Education Commission of New Zealand. We would like to acknowledge the support of AlphaCo and SoftCo.


  1. Allison I, Merali Y (2007) Software process improvement as emergent change: a structurational analysis. Inf Softw Technol 49(6):668–681CrossRefGoogle Scholar
  2. Benbasat I, Goldstein DK, Mead M (1987) The case research strategy in studies of information systems. Manage Inf Syst Q 11(3):369–386Google Scholar
  3. Boehm B (2006) A view of 20th and 21st century software engineering. In: Rombach D, Soffa ML (eds) Proceedings of the 28th International Conference on Software Engineering. ACM, New York, pp 12–29Google Scholar
  4. Bratthall L, Jørgensen M (2002) Can you trust a single data source exploratory software engineering case study? Empire Software Eng 7(1):9–26MATHCrossRefGoogle Scholar
  5. Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3(2):77–101CrossRefGoogle Scholar
  6. Bryman A, Bell E (2007) Business research methods, 2nd edn. Oxford University Press, OxfordGoogle Scholar
  7. Cavaye ALM (1996) Case study research: a multi-faceted research approach for IS. Inf Syst J 6(3):227–242CrossRefGoogle Scholar
  8. Charette RN (2005) Why software fails. IEEE Spectr 42(9):42–49CrossRefGoogle Scholar
  9. Ciborra CU, Lanzara GF (1994) Formative contexts and information technology: understanding the dynamics of innovation in organizations. Acc Manag Inf Technol 4(2):61–86Google Scholar
  10. Coleman G, O’Connor R (2007) Using grounded theory to understand software process improvement: a study of Irish software product companies. Inf Softw Technol 49(6):654–667CrossRefGoogle Scholar
  11. Constantinides P, Barrett M (2006) Negotiating ICT development and use: the case of a telemedicine system in the healthcare region of Crete. Inf Organ 16(1):27–55CrossRefGoogle Scholar
  12. Darke P, Shanks G, Broadbent M (1998) Successfully completing case study research: Combining rigour, relevance and pragmatism. Inf Syst J 8(4):273–289CrossRefGoogle Scholar
  13. Dittrich Y, John M, Singer J, Tessem B (2007) Editorial for the special issue on qualitative software engineering research. Inf Softw Technol 49(6):531–539CrossRefGoogle Scholar
  14. Dittrich Y, Rönkkö K, Eriksson J, Hansson C, Lindeberg O (2008) Cooperative method development: combining qualitative empirical research with method, technique and process improvement. Empire Software Eng 13(3):231–260CrossRefGoogle Scholar
  15. Doolin B (1996) Alternative views of case research in information systems. Aust J Inf Syst 3(2):21–29Google Scholar
  16. Easterbrook SM, Singer J, Storey M-A, Damian D (2008) Selecting empirical methods for software engineering research. In: Shull F, Singer J, Sjøberg DIK (eds) Guide to advanced empirical software engineering. Springer, London, pp 285–311CrossRefGoogle Scholar
  17. El Emam K, Koru AG (2008) A replicated survey of IT software project failures. IEEE Softw 25(5):84–90CrossRefGoogle Scholar
  18. Flyvbjerg B (2006) Five misunderstandings about case-study research. Qual Inq 12(2):219–245CrossRefGoogle Scholar
  19. Galliers RD, Swan JA (2000) There’s more to information systems development than structured approaches: information requirements analysis as a socially mediated process. Requir Eng 5(2):74–82CrossRefGoogle Scholar
  20. Gasson S (1999) A social action model of situated information systems design. Data Base Adv Inf Syst 30(2):82–97Google Scholar
  21. Geertz C (1973) Thick description: toward an interpretative theory of culture. In: The interpretation of cultures: selected essays. Basic Books, New YorkGoogle Scholar
  22. Giddens A (1984) The constitution of society: outline of the theory of structuration. University of California Press, BerkeleyGoogle Scholar
  23. Glaser BG, Strauss A (1967) The discovery of grounded theory: strategies for qualitative research. Aldine, ChicagoGoogle Scholar
  24. Glass RL, Vessey I, Ramesh V (2002) Research in software engineering: an analysis of the literature. Inf Softw Technol 44(8):491–506CrossRefGoogle Scholar
  25. Golden-Biddle K, Locke K (1993) Appealing work: an investigation of how ethnographic texts convince. Organ Sci 4(4):595–616CrossRefGoogle Scholar
  26. Guba EG (1990) The alternative paradigm dialog. In: Guba EG (ed) The paradigm dialog. Sage, London, pp 17–27Google Scholar
  27. Harrison R, Badoo N, Barry E, Biffl S, Parra A, Winter B, Wuest J (1999) Directions and methodologies for empirical software engineering research. Empire Software Eng 4(4):405–410CrossRefGoogle Scholar
  28. Höst M, Runeson P (2007) Checklists for software engineering case study research. In: Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (ESEM’07). IEEE Computer Society, Washington DC, pp 479–481Google Scholar
  29. Klein HK, Myers MD (1999) A set of principles for conducting and evaluating interpretive field studies in information systems. Manage Inf Syst Q 23(1):67–94Google Scholar
  30. Langley A (1999) Strategies for theorizing from process data. Acad Manage Rev 24(4):691–710MathSciNetGoogle Scholar
  31. Langley A, Truax J (1994) A process study of new technology adoption in smaller manufacturing firms. J Manag Stud 31(5):619–652CrossRefGoogle Scholar
  32. Lanzara GF (1999) Between transient constructs and persistent structures: designing systems in action. J Strateg Inf Syst 8(4):331–349CrossRefGoogle Scholar
  33. Latour B (1987) Science in action: how to follow scientists and engineers through society. Harvard University Press, CambridgeGoogle Scholar
  34. Leonard-Barton D (1990) A dual methodology for case studies: synergistic use of a longitudinal single site with replicated multiple sites. Organ Sci 1(3):248–266CrossRefGoogle Scholar
  35. Lethbridge TC, Singer J, Sim SE (2005) Studying software engineers: data collection techniques for software field studies. Empire Software Eng 10(3):311–341CrossRefGoogle Scholar
  36. Madsen S, Kautz K, Vidgen R (2006) A framework for understanding how a unique and local IS development method emerges in practice. Eur J Inf Syst 15(2):225–238CrossRefGoogle Scholar
  37. Markus ML, Lee AS (1999) Special issue on intensive research in information systems: using qualitative, interpretive, and case methods to study information technology. Manage Inf Syst Q 23(1):35–38Google Scholar
  38. Markus ML, Robey D (1988) Information technology and organizational change: causal structure in theory and research. Manag Sci 34(5):583–598CrossRefGoogle Scholar
  39. McLeod L (2008) Understanding IS development and acquisition: a process approach. PhD thesis (http://hdl.handle.net/10292/644), Auckland University of Technology
  40. McLeod L, MacDonell S (in press) Factors that affect software systems development project outcomes: a survey of research. ACM Comput SurvGoogle Scholar
  41. Miller J (2008) Triangulation as a basis for knowledge discovery in software engineering. Empire Software Eng 13(2):223–228CrossRefGoogle Scholar
  42. Mingers J (2001) Combining IS research methods: towards a pluralist methodology. Inf Syst Res 12(3):240–259CrossRefGoogle Scholar
  43. Myers MD, Newman M (2007) The qualitative interview in IS research: examining the craft. Inf Organ 17(1):2–26CrossRefGoogle Scholar
  44. Nandhakumar J, Avison DE (1999) The fiction of methodical development: a field study of information systems development. Inf Technol People 12(2):176–191CrossRefGoogle Scholar
  45. Nandhakumar J, Jones M (1997) Too close for comfort? Distance and engagement in interpretive information systems research. Inf Syst J 7(2):109–131CrossRefGoogle Scholar
  46. Pentland BT (1999) Building process theory with narrative: from description to explanation. Acad Manage Rev 24(4):711–724Google Scholar
  47. Perry DE, Porter AA, Votta LG (2000) Empirical studies of software engineering: a roadmap. In: Proceedings of the Conference on the Future of Software Engineering. ACM, New York, NY, pp 345–355Google Scholar
  48. Perry DE, Sim SE, Easterbrook SM (2004) Case studies for software engineers. In: Proceedings of the 26th International Conference on Software Engineering (ICSE’04). IEEE Computer Society, Washington DC, pp 736–738Google Scholar
  49. Perry DE, Sim SE, Easterbrook SM (2005) Case studies for software engineers. In: Proceedings of the 29th Annual IEEE/NASA Software Engineering Workshop—Tutorial Notes. IEEE Computer Society, Los Alamitos CA, pp 96–159Google Scholar
  50. Pettigrew AM (1990) Longitudinal field research on change: theory and practice. Organ Sci 1(3):267–292CrossRefGoogle Scholar
  51. Pettigrew AM (1997) What is processual analysis? Scand J Manag 13(4):337–348CrossRefGoogle Scholar
  52. Robinson H, Segal J, Sharp H (2007) Ethnographically-informed empirical studies of software practice. Inf Softw Technol 49(6):540–551CrossRefGoogle Scholar
  53. Rönkkö K, Lindeberg O, Dittrich Y (2002) ‘Bad practice’ or ‘bad methods’: are software engineering and ethnographic discourses incompatible? In: Proceedings of the 2002 International Symposium on Empirical Software Engineering (ISESE’02). IEEE Computer Society, Washington DC, pp 204–210Google Scholar
  54. Royal Academy of Engineering (2004) The challenges of complex IT projects. Royal Academy of Engineering, LondonGoogle Scholar
  55. Runeson P, Höst M (2009a) Guidelines for conducting and reporting case study research in software engineering. Empire Software Eng 14(2):131–164CrossRefGoogle Scholar
  56. Runeson P, Höst M (2009b) Tutorial: case studies in software engineering. In: Bomarius F, Oivo M, Jaring P, Abrahamsson P (eds) Product-focused software process improvement (Proceedings of the 10th International Conference, PROFES 2009, Oulu, Finland, June 15–17). Springer, Berlin, pp 441–442Google Scholar
  57. Schultze U, Orlikowski WJ (2004) A practice perspective on technology-mediated network relations: the use of Internet-based self-serve technologies. Inf Syst Res 15(1):87–106CrossRefGoogle Scholar
  58. Seaman C (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans Softw Eng 25(4):557–572CrossRefGoogle Scholar
  59. Segal J, Grinyer A, Sharp H (2005) The type of evidence produced by empirical software engineers. In: Proceedings of the Workshop on Realising Evidence-Based Software Engineering (REBSE’05). ACM, New York, NYGoogle Scholar
  60. Sieber JE (2001) Protecting research subjects, employees and researchers: implications for software engineering. Empire Software Eng 6(4):329–341MATHCrossRefGoogle Scholar
  61. Sim SE, Singer J, Storey M-A (2001) Beg, borrow, or steal: using multidisciplinary approaches in empirical software engineering research. Empire Software Eng 6(1):85–93MATHCrossRefGoogle Scholar
  62. Singer J, Vinson NG (2002) Ethical issues in empirical studies of software engineering. IEEE Trans Softw Eng 28(12):1171–1180CrossRefGoogle Scholar
  63. Sjøberg DIK, Dybå T, Jørgensen M (2007) The future of empirical methods in software engineering research. In: Briand LC, Wolf AL (eds) Future of software engineering (FOSE’07). IEEE Computer Society, Los Alamitos, pp 358–378CrossRefGoogle Scholar
  64. Star SL (1989) The structure of ill-structured solutions: boundary objects and heterogeneous distributed problem solving. In: Gasser L, Huhns MN (eds) Distributed artificial intelligence (vol. 2). Pitman Publishing, London, pp 37–54Google Scholar
  65. Subrahmanian E, Monarch I, Konda S, Granger H, Milliken R, Westerberg A, The N-Dim Group (2003) Boundary objects and prototypes at the interfaces of engineering design. Comput Support Coop Work 12(2):185–203CrossRefGoogle Scholar
  66. Tolich M, Davidson C (1999) Starting fieldwork: an introduction to qualitative research in New Zealand. Oxford University Press, AucklandGoogle Scholar
  67. Voss C, Tsikriktsis N, Frohlich M (2002) Case research in operations management. Int J Oper Prod Manage 22(2):195–219CrossRefGoogle Scholar
  68. Wainer J, Barsottini C (2007) Empirical research in CSCW—a review of the ACM/CSCW conferences from 1998 to 2004. J Braz Comput Soc 13(3):27–36CrossRefGoogle Scholar
  69. Wallace L, Keil M (2004) Software project risks and their effect on outcomes. Commun ACM 47(4):68–73CrossRefGoogle Scholar
  70. Walsham G (1993) Interpreting information systems in organizations. Wiley, ChichesterGoogle Scholar
  71. Walsham G (1995) Interpretive case studies in IS research: nature and method. Eur J Inf Syst 4(2):74–81CrossRefGoogle Scholar
  72. Walsham G (2006) Doing interpretive research. Eur J Inf Syst 15:320–330CrossRefGoogle Scholar
  73. Walsham G, Sahay S (1999) GIS for district-level administration in India: problems and opportunities. Manage Inf Syst Q 23(1):39–66Google Scholar
  74. Westrup C (1993) Information systems methodologies in use. J Inf Technol 8(4):267–275CrossRefGoogle Scholar
  75. Whittle A (2005) Preaching and practising ‘flexibility’: Implications for theories of subjectivity at work. Hum Relat 58(10):1301–1322CrossRefGoogle Scholar
  76. Wynekoop JL, Russo NL (1997) Studying system development methodologies: an examination of research methods. Inf Syst J 7(1):47–65CrossRefGoogle Scholar
  77. Yin RK (2003) Case study research: design and methods, 3rd edn. Sage, Thousand OaksGoogle Scholar
  78. Zuboff S (1988) In the age of the smart machine. Basic Books, New YorkGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Laurie McLeod
    • 1
  • Stephen G. MacDonell
    • 1
  • Bill Doolin
    • 1
  1. 1.Auckland University of TechnologyAucklandNew Zealand

Personalised recommendations