Abstract
Experimentation in software engineering is important but difficult. One reason it is so difficult is that there are a large number of context variables, and so creating a cohesive understanding of experimental results requires a mechanism for motivating studies and integrating results. This paper argues for the necessity of a framework for organizing sets of related studies. With such a framework, experiments can be viewed as part of common families of studies, rather than being isolated events. Common families of studies can contribute to important and relevant hypotheses that may not be suggested by individual experiments. A framework also facilitates building knowledge in an incremental manner through the replication of experiments within families of studies. Building knowledge in this way requires a community of researchers that can replicate studies, vary context variables, and build abstract models that represent the common observations about the discipline. This paper also presents guidelines for lab packages, meant to encourage and support replications, that encapsulate materials, methods, and experiences concerning software engineering experiments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
V.R. Basili, “The experimental paradigm in software engineering”, Experimental Software Engineering Issues: Critical Assessment and Future Directions, International Workshop, Dagstuhl, Germany, 1992. Appeared in Springer-Verlag, Lecture Notes in Computer Science, Number 706, 1993.
V. R. Basili, “Evolving and packaging reading technologies”, Journal of Systems and Software, vol. 38, no. 1, pp.3–12, July 1997.
V. Basili, G. Caldiera, F. Lanubile, and F. Shull, “Studies on reading techniques”, Proc. of the Twenty-First Annual Software Engineering Workshop, SEL-96-002, Goddard Space Flight Center, Greenbelt, Maryland, pp.59–65, December 1996.
V. R. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Soerumgaard, M. Zelkowitz, “The empirical investigation of perspective-based reading”; Empirical Software Engineering Journal, vol. 1, no. 2, 1996.
V. R. Basili, S. Green, O. Laitenburger, F. Lanubile, F. Shull, S. Sorumgard, and M. Zelkowitz, “Packaging researcher experience to assist replication of experiments”, Proc. of the ISERN meeting 1996, Sydney, Australia, 1996.
V. R. Basili, and D. H. Hutchens, “An empirical study of a syntactic metric family”, IEEE Transactions on Software Engineering, vol. SE-9, pp.664–672, November 1983.
V. R. Basili, and R. W. Reiter, “A controlled experiment quantitatively comparing software development approaches”, IEEE Transactions on Software Engineering, vol. SE-7, no. 3, pp.299–320, May 1981.
V. R. Basili, and H. D. Rombach, “The TAME project: Towards improvement-oriented software environments”, IEEE Transactions on Software Engineering, vol. SE-14, no. 6, June 1988.
V. R. Basili, R. W. Selby, and D. H. Hutchens, “Experimentation in software engineering”, IEEE Transactions on Software Engineering, vol. SE-12, no. 7, pp. 733–743, July 1986.
V. Basili, F. Lanubile, F. Shull, “Investigating maintenance processes in a framework-based environment”, Proc. of the Int. Conf. on Software Maintenance, Bethesda, Maryland, pp.256–264, 1998.
D. T. Campbell, and J. C. Stanley, Experimental and Quasi-Experimental Designs for Research, Boston: Houghton Miffilin Co, 1963.
Composable Systems Group, “Model Problems”, http://www.cs.cmu.edu/Compose/html/ModProb/, 1995.
P. Fusaro, F. Lanubile, and G. Visaggio, “A replicated experiment to assess requirements inspections techniques”, Empirical Software Engineering Journal, vol.2, no.1, pp.39–57, 1997.
IEEE. Software Engineering Standards. IEEE Computer Society Press, 1987.
C. M. Judd, E. R. Smith, and L. H. Kidder, Research Methods in Social Relations, sixth edition, Orlando: Harcourt Brace Jovanovich, Inc., 1991.
O. Laitenberger, and J. M. DeBaud, “Perspective-based reading of code documents at Robert Bosch GmbH”, Journal of Information and Software Technology, 39, pp.781–791, 1997.
F. Lanubile, “Empirical evaluation of software maintenance technologies”, Empirical Software Engineering Journal, vol.2, no.2, pp.95–106, 1997.
F. Lanubile, “Report on the results of the parallel project meeting reading techniques”, http://seldi2.uniba.it:1025/isern97/readwg/index.htm, October 1997.
F. Lanubile, F. Shull, V. Basili, “Experimenting with error abstraction in requirements documents”, Proc. of the 5th Int. Symposium on Software Metrics, Bethesda, Maryland, pp.114–121, 1998.
C. M. Lott, and H. D. Rombach, “Repeatable software engineering experiments for comparing defect-detection techniques”, Empirical Software Engineering Journal, vol.1, no.3, pp.241–277, 1996.
K. Popper, The Logic of Scientific Discovery, Harper Torchbooks, New York, NY, 1968.
A. Porter, L. Votta, V. Basili, “Comparing detection methods for software requirements inspections: a replicated experiment”, IEEE Transactions on Software Engineering, vol. 21, no. 6, pp. 563–575, 1995.
F. Shull, F. Lanubile, and V. R. Basili, “Investigating Reading Techniques for Framework Learning”, Technical Report CS-TR-3896, UMCP Dept. of Computer Science, UMIACS-TR-98-26, UMCP Institute for Advanced Computer Studies, ISERN-98-16, International Software Engineering Research Network, May 1998.
F. Shull. Developing Techniques for Using Software Documents: A Series of Empirical Studies. Ph.D. thesis, University of Maryland, College Park, December 1998.
F. Shull, “Reading Techniques for Object-Oriented Frameworks”, http://www.cs.umd.edu/projects/SoftEng/ESEG/manual/sbr_package/manual.html
F. Shull, “Lab Package for the Empirical Investigation of Perspective-Based Reading”, http://www.cs.umd.edu/projects/SoftEng/ESEG/manual/pbr_package/manual.html
Z. Zhang, V. Basili, and B. Shneiderman, “An Empirical Study of Perspective-based Usability Inspection”, Human Factors and Ergonomics Society Annual Meeting, Chicago, Oct. 1998.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2000 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Basili, V., Shull, F., Lanubile, F. (2000). Using Experiments to Build a Body of Knowledge. In: Bjøner, D., Broy, M., Zamulin, A.V. (eds) Perspectives of System Informatics. PSI 1999. Lecture Notes in Computer Science, vol 1755. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-46562-6_24
Download citation
DOI: https://doi.org/10.1007/3-540-46562-6_24
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-67102-2
Online ISBN: 978-3-540-46562-1
eBook Packages: Springer Book Archive