A Framework for Evaluating System and Software Requirements Specification Approaches

  • Erik Kamsties
  • H. Dieter Rombach
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1526)


Numerous requirements specification approaches have been proposed to improve the quality of requirements documents as well as the developed software and to increase user satisfaction with the final product. However, no or only anecdotal evidence exists about which approach is appropriate in a particular context. This paper discusses the value of experimentation in requirements engineering to gain sound empirical evidence. Subsequently, a framework is suggested, which helps to structure one’s research agenda for experimentation, and supports the development of experiments driven by this agenda. As an example for the application of the framework, our research agenda is outlined, focusing on requirements specification approaches for embedded systems. The experiments found in the literature regarding this topic are analyzed, issues that warrant more experimentation are identified, and finally, an additional set of experiments is proposed.


Software Engineer Requirement Engineering Requirement Engineer User Participation Single Project 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Fenton, N.: How effective are software engineering methods? Journal of Systems and Software 22(2), 141–146 (1993)CrossRefGoogle Scholar
  2. 2.
    Dieter Rombach, H.: Experimentation as a vehicle for software technology transfer: A family of software reading techniques. In: Proceedings of the 1stt International Conference on Empirical Assessment and Evaluation in Software Engineering, Keele (UK) (March 1997); Keynote talkGoogle Scholar
  3. 3.
    Glass, R.L.: Do measuring advocates measure up? In: Proceedings of the 3rd International Conference on Applications of Software Measurement , pp. 1.02–1.12 (1992)Google Scholar
  4. 4.
    Lott, C.M., Rombach, H.D.: Repeatable software engineering experiments for comparing defect-detection techniques. Journal of Empirical Software Engineering (1996)Google Scholar
  5. 5.
    Basili, V.R., Selby, R.W., Hutchens, D.H.: Experimentation in software engineering. IEEE Transactions on Software Engineering  SE-12(7), 733–743 (1986)Google Scholar
  6. 6.
    Kitchenham, B.A.: Evaluating software engineering methods and tools. Parts 1 to 8. ACM SIGSoft Software Engineering Notes (1996/1997)Google Scholar
  7. 7.
    Pfleeger, S.L.: Experimental design and analysis in software engineering. Parts 1 to 5. ACM SIGSoft Software Engineering Notes (1994/1995)Google Scholar
  8. 8.
    Curtis, B.: A methodological and empirical basis for research in software engineering. Technical report, Microelectronics and Computer Technology Corporation (1988)Google Scholar
  9. 9.
    Bowen, J.P., Butler, R.W., Dill, D.L., Glass, R.L., Gries, D., Hall, A., Hinchey, M.G., Holloway, C.M., Jackson, D., Jones, C.B., Lutz, M.J., Parnas, D.L., Rushby, J., Saiedian, H., Wing, J., Zave, P.: An invitation to formal methods. IEEE Computer 29(4), 16–30 (1996)Google Scholar
  10. 10.
    Sommerville, I., Rodden, T., Sawyer, P., Bentley, R., Twidale, M.: Integrating ethnography into the requirements engineering process. In: Proceedings of the IEEE International Symposium on Requirements Engineering (RE 1993), San Diego, California, USA, January 1993, pp. 165–173 (1993)Google Scholar
  11. 11.
    Tichy, W.F., Lukowicz, P., Prechelt, L., Heinz, E.A.: Experimental evaluation in computer science: A quantitative study. Journal of Systems and Software 28(1), 9–18 (1995)CrossRefGoogle Scholar
  12. 12.
    Basili, V.R.: The experimental paradigm in software engineering. In: Rombach, H.D., Selby, R.W., Basili, V.R. (eds.) Experimental Software Engineering Issues: LNCS, vol. 706, pp. 3–12. Springer, Heidelberg (1993)Google Scholar
  13. 13.
    Zave, P.: Classification of research efforts in requirements engineering. ACM Computing Surveys 29(4), 315–321 (1997)CrossRefGoogle Scholar
  14. 14.
    Mohamed, W.-E.A., Sadler, C.J., Law, D.: Experimentation in software engineering: A new framework. In: Proceedings of Software Quality Management 1993, pp. 417–430. Elsevier Science, Essex U.K. (1993)Google Scholar
  15. 15.
    Basili, V.R., Rombach, H.D.: The TAME Project: Towards improvement-oriented software environments. IEEE Transactions on Software Engineering SE-14(6), 758–773 (1988)CrossRefGoogle Scholar
  16. 16.
    Basili, V.R.: Software development: A paradigm for the future. In: Proceedings of the 13th Annual International Computer Software and Application Conference (COMPSAC), Orlando, Florida, September 1989, pp. 471–485 (1989)Google Scholar
  17. 17.
    Basili, V.R., Caldiera, G., Rombach, H.D.: Goal Question Metric Paradigm. In: Marciniak, J.J. (ed.) Encyclopedia of Software Engineering, vol. 1, pp. 528–532. John Wiley & Sons, Chichester (1994)Google Scholar
  18. 18.
    Briand, L., Differding, C., Rombach, D.: Practical guidelines for measurement-based process improvement. In: Proceedings of the International Software Consulting Network Conference (ISCN 1996) (1996)Google Scholar
  19. 19.
    Basili, V.R., Caldiera, G., Rombach, H.D.: Experience Factory. In: Marciniak, J.J. (ed.) Encyclopedia of Software Engineering, vol. 1, pp. 469–476. John Wiley and Sons, Chichester (1994)Google Scholar
  20. 20.
    Ryan, K.: Let’s have more experimentation in requirements engineering. In: Proceedings of the 2nd IEEE International Symposium on Requirements Engineering, York, U.K., March 1995, p. 66 (1995) Panel SessionGoogle Scholar
  21. 21.
    Jackson, M.: Concluding statement at the panel session on Let’s have more Experimentation in Requirements Engineering. In: The International Symposium on Requirements Engineering, RE 1995 (1995)Google Scholar
  22. 22.
    Elmam, K.E., Quintin, S., Madhavji, N.H.: User participation in the requirements engineering process: An empirical study. Requirements Engineering Journal 1(1), 4–26 (1996)CrossRefGoogle Scholar
  23. 23.
    Wing, J.M.: A study of 12 speciications of the library problem. IEEE Software, 66–76 (July 1988)Google Scholar
  24. 24.
    Lewerenz, C., Lindner, T. (eds.): Case study ”production cell”. FZI-Publication 1/94, Forschungszentrum Informatik (FZI), Universität Karlsruhe, Germany (1994)Google Scholar
  25. 25.
    Abrial, J.-R., Börger, E., Langmaack, H. (eds.): Formal Methods for Industrial Applications: Specifying and Programming the Steam Boiler Control. LNCS. Springer, Heidelberg (1996)Google Scholar
  26. 26.
    Faulk, S.R., Finneran, L., Kirby, J., Sutton, J.: Experience applying the CoRE method to the Lookheed C-130J. In: Proceedings of the 9th Annual Conf. on Computer Assurance, Gaithersburg, MD, USA, June 1994, pp. 3–8 (1994)Google Scholar
  27. 27.
    Weidenhaupt, K., Pohl, K., Jarke, M., Haumer, P.: Scenarios in system development: Current practice. IEEE Software, 34–45 (March 1998)Google Scholar
  28. 28.
    Feather, M.S., Fickas, S., Finkelstein, A., van Lamsweerde, A.: Requirements and speciication exemplars. Automated Software Engineering 4(4), 419–438 (1997)Google Scholar
  29. 29.
    Elmam, K.E., Madhavji, N.H.: Measuring the success of requirements engineering processes. In: Proceedings of the 2nd IEEE International Symposium on Requirements Engineering, York, UK, March 1995, pp. 204–211 (1995)Google Scholar
  30. 30.
    Laitenberger, O., DeBaud, J.-M.: Perspective-based reading of code documentsat Robert Bosch GmbH. Information and Software Technology 39, 781–791 (1997)CrossRefGoogle Scholar
  31. 31.
    Feldmann, R.L., Münch, J., Vorwieger, S.: Experiences with systematic reuse: Applying the EF/QIP approach. In: Proceedings of the European Reuse Workshop, Brussels, Belgium (November 1997)Google Scholar
  32. 32.
    Feldmann, R.L., Vorwieger, S.: The web-based Interface to the SFB 501 Experience Base. SFB-501-TR- 01/1998, Sonderforschungsbereich 501, Dept. of Computer Science, University of Kaiserslautern, 67653 Kaiserslautern, Germany (1998)Google Scholar
  33. 33.
    Basili, V., Caldiera, G., McGarry, F., Pajersky, R., Page, G., Waligora, S.: The Software Engineering Laboratory – an operational Software Experience Factory. In: Proceedings oft he 14th International Conference on Software Engineering, May 1992, pp. 370–381 (1992)Google Scholar
  34. 34.
    Laitenberger, O.: Perspective-based reading: Technique, validation, and research in future. ISERN- 95-01, University of Kaiserslautern, Kaiserslautern, Germany (1995)Google Scholar
  35. 35.
    Lutz, R.R., Woodhouse, R.M.: Requirements analysis using forward and backward search. In: Mead, N.R. (ed.) Software Requirements Engineering, September 1997. Annals of Software Engineering, vol. 3, pp. 459–475. Baltzer Science Publishers (1997)Google Scholar
  36. 36.
    Takahashi, K., Oka, A., Yamamoto, S., Isoda, S.: A comparative study of structured and text-oriented analysis and design methodologies. Journal of Systems and Software 28(1), 69–75 (1995)CrossRefGoogle Scholar
  37. 37.
    Mills, K.L.: An experimental evaluation of specification techniques for improving functional testing. Journal of Systems and Software 32(1), 83–95 (1996)CrossRefGoogle Scholar
  38. 38.
    Yadav, S.B., Bravoco, R.R., Chatfield, A.T., Rajkumar, T.M.: Comparison of analysis techniques for information requirements determination. Communications of the ACM 31(9), 1090–1097 (1988)CrossRefGoogle Scholar
  39. 39.
    Porter, A.A., Votta, L.G., Basili, V.R.: Comparing detection methods for software requirements inspections: A replicated experiment. IEEE Transactions on Software Engineering 21(6), 563–575 (1995)CrossRefGoogle Scholar
  40. 40.
    Basili, V.R., Green, S., Laitenberger, O., Lanubile, F., Shull, F., Sorumgard, S., Zelkowitz, M.V.: The empirical investigation of perspective-based reading. Journal of Empirical Software Engineering 1(2), 133–164 (1996)CrossRefGoogle Scholar
  41. 41.
    Vessey, I., Conger, S.A.: Requirements specification: Learning object, process, and data methodologies. Communications of the ACM 37(5), 102–113 (1994)CrossRefGoogle Scholar
  42. 42.
    Jankowski, D.: Computer-aided systems engineering methodology support and its effects on the output of structured analysis. Journal of Empirical Software Engineering, 2, 11–38 (1997)CrossRefGoogle Scholar
  43. 43.
    Gowen, L.D., Collofello, J.S.: Assessing traditional verification’s effectiveness on safety-critical software systems. Journal of Systems and Software 26(2), 103–115 (1994)CrossRefGoogle Scholar
  44. 44.
    Laitenberger, O., Münch, J.: Ein Prozefimodell zur experimentellen Er-probung von Software-Entwicklungsprozessen. SFB-501-TR- 04/1996, University of Kaiserslautern, Special Research Project 501 (1996) (in German)Google Scholar
  45. 45.
    Moen, R., Nolan, T., Provost, L.: Improving Quality Through Planned Experimentation, 1st edn. McGraw-Hill, Inc., New York (1991)Google Scholar
  46. 46.
    Rational Software Corporation. Unified Modeling Language, Version 1 (1997)Google Scholar
  47. 47.
    Rumbaugh, J., Blaha, M., Premerlani, W., Eddy, F., Lorensen, W.: Object-Oriented Modeling and Design. Prentice Hall, Englewood Cliffs (1991)Google Scholar
  48. 48.
    van John Schouwen, A.: The A-7 requirements model: Re-examination for real-time systems and an application to monitoring systems. CRL Report No. 242, McMas-ter University, CRL, Telecommunications Research Institute of Ontario (TRIO), Hamilton, Ontario, Canada (February 1992)Google Scholar
  49. 49.
    Kamsties, E., von Knethen, A., Reussner, R.: An empirical evaluation of two requirements specification techniques. Technical report, University of Kaiser-slautern, Kaiserslautern, Germany (1998) (in preparation)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1998

Authors and Affiliations

  • Erik Kamsties
    • 1
  • H. Dieter Rombach
    • 1
  1. 1.Fraunhofer Institute for Experimental Software EngineeringKaiserslauternGermany

Personalised recommendations