Empirical Evaluation of Agile Software Development: The Controlled Case Study Approach

  • Outi Salo
  • Pekka Abrahamsson
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3009)


Agile software development, despite its novelty, is an important domain of research within software engineering discipline. Agile proponents have put forward a great deal of anecdotal evidence to support the application of agile methods in various application domains and industry sectors. Scientifically grounded empirical evidence is, however, still very limited. Most scientific research to date has been conducted on focused practices performed in university settings. In order to generate impact on both the scientific and practical software engineering community, new approaches are needed for performing empirically validated agile software development studies. To meet these needs, this paper presents a controlled case study approach, which has been applied in a study of extreme programming methodology performed in close-to-industry settings. The approach considers the generation of both quantitative and qualitative data. Quantitative data is grounded on three data points (time, size, and defect) and qualitative data on developers’ research diaries and post-mortem sessions.


Software Development Case Study Approach Case Study Research Software Development Process Agile Method 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Boehm, B., Turner, R.: Balancing Agility and Discipline: A Guide for the Perplexed. Addison-Wesley, Reading (2003)Google Scholar
  2. 2.
    Beck, K.: Extreme Programming Explained: Embrace Change. Addison Wesley Longman, Inc., Amsterdam (2000)Google Scholar
  3. 3.
    Cockburn, A.: Agile Software Development. Addison-Wesley, Boston (2002)Google Scholar
  4. 4.
    Schuh, P.: Recovery, Redemption, and Extreme Programming. IEEE Software 18, 34–41 (2001)CrossRefGoogle Scholar
  5. 5.
    Rasmusson, J.: Introducing XP into Greenfield Projects: Lessons Learned. IEEE Software, 21–28 (2003)Google Scholar
  6. 6.
    Murru, O., Deias, R., Mugheddu, G.: Assessing XP at a European Internet Company. IEEE Software, 37–43 (2003)Google Scholar
  7. 7.
    Lindvall, M., Basili, V.R., Boehm, B., Costa, P., Dangle, K., Shull, F., Tesoriero, R., Williams, L., Zelkowitz, M.V.: Empirical Findings in Agile Methods. Presented at XP/Agile Universe 2002, Chicago, IL, USA (2002)Google Scholar
  8. 8.
    Zelkowitz, M.V., Wallace, D.R.: Experimental Models for Validating Technology. Computer, 23–31 (1998)Google Scholar
  9. 9.
    Noll, J., Atkinson, D.C.: Comparing Extreme Programming to Traditional Development for Student Projects: A Case Study. In: Marchesi, M., Succi, G. (eds.) XP 2003. LNCS, vol. 2675, Springer, Heidelberg (2003)CrossRefGoogle Scholar
  10. 10.
    Rumpe, B.: Quantitative Survey on Extreme Programming Projects. Presented at XP 2002, Alghero, Sardinia, Italy (2002)Google Scholar
  11. 11.
    Tessem, B.: Experiences in Learning XP Practices: A Qualitative Study. Presented at XP 2003, Genova, Italy (2003)Google Scholar
  12. 12.
    Abrahamsson, P.: Extreme Programming: First Results from a Controlled Case Study. Presented at 29th Euromicro Conference, Belek-Antalya, Turkey (2003)Google Scholar
  13. 13.
    Karlström, D.: Introducing Extreme Programming - An Experience Report. Presented at XP 2002, Alghero, Sardinia, Italy (2002)Google Scholar
  14. 14.
    Maurer, F., Martel, S.: On the Productivity of Agile Software Practices: An Industrial Case Study (2002)Google Scholar
  15. 15.
    Müller, M.M., Tichy, W.F.: Case Study: Extreme Programming in a University Environment. Presented at 23rd International Conference on Software Engineering, Toronto (2001)Google Scholar
  16. 16.
    Wood, W.A., Kleb, W.L.: Exploring XP for Scientific Research. IEEE Software, 30–36 (2003)Google Scholar
  17. 17.
    Janes, B., Russo, P.: An Empirical Analysis of the Discontinuous Use of Pair Programming. Presented at XP 2003, Genova, Italy (2003)Google Scholar
  18. 18.
    Heiberg, S., Puus, U., Salumaa, P., Seeba, A.: Pair-Programming Effect on Developers Productivity. Presented at XP 2003, Genova, Italy (2003)Google Scholar
  19. 19.
    Lui, K.M., Chan, K.C.C.: When Does a Pair Outperform Two Individuals? Presented at XP 2003, Genova, Italy (2003)Google Scholar
  20. 20.
    Müller, M.M., Hagner, O.: Experiment about Test-first programming. Software 149, 131–135 (2002)Google Scholar
  21. 21.
    Williams, L., Kessler, R.R., Cunningham, W., Jeffries, R.: Strengthening the Case for Pair Programming. IEEE Software 17, 19–25 (2000)CrossRefGoogle Scholar
  22. 22.
    Rostaher, M., Hericko, M.: Tracking Test First Pair Programming - An Experiment. In: Presented at XP/Agile Universe 2002, Chicago, IL, USA (2002)Google Scholar
  23. 23.
    Basili, V.R., Selby, R.W., Hutchens, D.H.: Experimentation in Software Engineering. IEEE Transactions on Software Engineering SE-12, 733–742 (1986)Google Scholar
  24. 24.
    Tichy, W.F.: Should Computer Scientists Experiment More? Computer, 32–40 (1998)Google Scholar
  25. 25.
    Basili, V.R., Lanuble, F.: Building Knowledge through Families of Experiments. IEEE Transactions on Software Engineering 25, 456–473 (1999)CrossRefGoogle Scholar
  26. 26.
    Fenton, N.: Viewpoint Article: Conducting and Presenting Empirical Software Engineering. Empirical Software Engineering 6, 195–200 (2001)CrossRefGoogle Scholar
  27. 27.
    Wohlin, C., Höst, M., Henningsson, K.: Empirical Research Methods in Software Engineering. In: Conradi, R., Wang, A.I. (eds.) ESERNET 2001. LNCS, vol. 2765, pp. 7–23. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  28. 28.
    Birk, A., Dingsøyr, T., Stålhane, T.: Postmortem: Never Leave a Project without It. IEEE Software 19, 43–45 (2002)CrossRefGoogle Scholar
  29. 29.
    Dingsøyr, T., Hanssen, G.K.: Extending Agile Methods: Postmortem Reviews as Extended Feedback. Presented at 4th International Workshop on Learning Software Organizations (LSO 2002), Chicago, Illinois, USA (2002)Google Scholar
  30. 30.
    Collier, B., DeMarco, T., Fearey, P.: A defined process for project post mortem review. IEEE Software 13 (1996)Google Scholar
  31. 31.
    Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering. Kluwer Academic Publishers, Boston (2000)zbMATHGoogle Scholar
  32. 32.
    Bratthall, L., Jørgensen, M.: Can you Trust a Single Data Source Exploratory Software Engineering Case Study? Empirical Software Engineering 7, 9–26 (2002)zbMATHCrossRefGoogle Scholar
  33. 33.
    Sjøberg, D.I.K., Anda, B., Arisholm, E., Dybå, T., Jørgensen, M., Karahasanovic, A., Vokác: Challenges and Recommendations When Increasing the Realism of Controlled Software Engineering Experiments. In: Conradi, R., Wang, A.I. (eds.) ESERNET 2001. LNCS, vol. 2765, pp. 24–38. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  34. 34.
    Yin, R.K.: Case Study Research Design and Methods, 2nd edn. Sage Publications, Thousand Oaks (1994)Google Scholar
  35. 35.
    Avison, D., Lau, F., Myers, M., Nielsen, P.A.: Action Research. Communications of the ACM 42, 94–97 (1999)CrossRefGoogle Scholar
  36. 36.
    Cunningham, J.B.: Case study principles for different types of cases. Quality and quantity 31, 401–423 (1997)CrossRefGoogle Scholar
  37. 37.
    Oquist, P.: The epistemology of action research. Acta Sociologica 21, 143–163 (1978)Google Scholar
  38. 38.
    Susman, G.I., Evered, R.D.: An Assessment of the Scientific Merits of Action Research. Administrative Science Quarterly 23, 582–603 (1978)CrossRefGoogle Scholar
  39. 39.
    Potts, C.: Software-Engineering Research Revisited. IEEE Software 10, 19–28 (1993)CrossRefGoogle Scholar
  40. 40.
    Kaplan, A.: The conduct of inquiry: Methodology for behavioral science. Chandler, New York (1964)Google Scholar
  41. 41.
    Basili, V.R.: Software Development: A Paradigm for the Future. Presented at COMPSAC 1989, Orlando, Florida (1989)Google Scholar
  42. 42.
    Sandelin, T., Vierimaa, M.: Empirical Studies in ESERNET. In: Conradi, R., Wang, A.I. (eds.) ESERNET 2001. LNCS, vol. 2765, pp. 39–54. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  43. 43.
    Eisenhardt, K.: Building Theories from Case Study Research. Academy of Management Review 14, 532–550 (1989)CrossRefGoogle Scholar
  44. 44.
    Eden, C., Huxham, C.: Action Research for the Study of Organizations. In: Clegg, S.R., Hardy, S. (eds.) Studying Organization: Theory & Method, pp. 272–288. SAGE Publications Ltd, London (1999)Google Scholar
  45. 45.
    Abrahamsson, P., Salo, O., Ronkainen, J., Warsta, J.: Agile Software Development Methods: Review and Analysis. VTT Electronics, Espoo VTT Publications 478 (2002)Google Scholar
  46. 46.
    Höst, M., Regnell, B., Wohlin, C.: Using Students as Subjects - A Comparative Study of Students and Professionals in Lead-Time Impact Assessment. Empirical Software 5, 201–214 (2000)zbMATHCrossRefGoogle Scholar
  47. 47.
    Humphrey, W.S.: A discipline for software engineering. Addison Wesley Longman, Inc., Amsterdam (1995)Google Scholar
  48. 48.
    Zahran, S.: Software Process Improvement: Practical Guidelines for Business Success. Addison-Wesley, Reading (1998)Google Scholar
  49. 49.
    Johnson, P.M.: The personal software process: A cautionary case study. IEEE Software 15, 85–88 (1998)CrossRefGoogle Scholar
  50. 50.
    Dingsøyr, T., Moe, N.B., Nytrø, Ø.: Augmenting Experience Reports with Lightweight Postmortem Reviews. Presented at 3rd Int’l Conference on Product Focused Software Improvement (Profes 2001), Kaiserslautern, Germany (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Outi Salo
    • 1
  • Pekka Abrahamsson
    • 1
  1. 1.VTT Technical Research Centre of FinlandOuluFinland

Personalised recommendations