Using Experimental Material Management Tools in Experimental Replication: A Systematic Mapping Study
Context: In experimental software engineering (ESE), experimental replication is applied to validate results of an experiment. Much information and one version of the experimental materials are required for replication. Prior to the replication execution, all or part of the materials may require changes, producing new or modified versions of these, which should be incorporated into the material of the original experiment. There is a direct relationship between the increase in the number of replications with the increase of versions of the experimental material, which commonly causes confusion and disorder for experimental administrators. Objective: The aim of this paper is to conduct a mapping study to locate articles about the use of experimental material management tools in experimental replication in ESE. Method: We applied the mapping study to search, analyze and select published papers from reported replications. Results: We analyzed a total of 592 articles published from 1998 to 2014, 24 of them have been pre-selected and 4 have finally been selected. Conclusion: The results show the limited existence of articles on this subject. In addition, our analysis has allowed to identify that most of them suffer from problems in versions management for both replication and experimental material. These data provide information of interest to start a research about adoption of the paradigm of software configuration management inside the management of the experimental material in ESE.
KeywordsExperimental software engineering Experimental replication Experimental material Experimental material management
The authors thank the University of the Armed Forces (ESPE-L), the Polytechnic School - National University of Asuncion (FPUNA), the Ministry of Higher Education on Science, Technology and Innovation (SENESCYT) and Empirical Software Engineering Research Group (GrISE) for supporting the development of this work.
- 1.Daly, J., Brooks, A., Miller, J., Roper, M., Wood, M.: Verification of results in software maintenance through external replication. In: 1994 Proceedings of International Conference on Software Maintenance, pp. 50–57 (1994)Google Scholar
- 3.Mendonca, M.G., Maldonado, J.C., de Oliveira, M.C.F., Carver, J., Fabbri, S.C.P.F., Shull, F., Travassos, G.H., Hohn, E.N., Basili, V.R.: A framework for software engineering experimental replications. In: 13th IEEE International Conference on Engineering of Complex Computer Systems 2008 ICECCS 2008, pp. 203–212 (2008)Google Scholar
- 4.Shull, F., Basili, V., Carver, J., Maldonado, J.C., Travassos, G.H., Mendonça, M., Fabbri, S.: Replicating software engineering experiments: addressing the tacit knowledge problem. In: Proceedings of 2002 International Symposium on Empirical Software Engineering 2002, pp. 7–16 (2002)Google Scholar
- 7.Vegas, S., Juristo, N., Moreno, A., Solari, M., Letelier, P.: Analysis of the influence of communication between researchers on experiment replication. In: International Symposium on Empiricial Software Engineering, p. 28 (2006)Google Scholar
- 8.Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: Systematic mapping studies in software engineering. In: 12th International Conference on Evaluation and Assessment in Software Engineering (2008)Google Scholar
- 9.Beecham, S., Baddoo, N., Hall, T., Robinson, H., Sharp, H.: Protocol for a systematic literature review of motivation in software engineering (2006)Google Scholar