Using Experimental Material Management Tools in Experimental Replication: A Systematic Mapping Study

  • Edison Espinosa
  • Juan M. Ferreira
  • Henry Chanatasig
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 721)


Context: In experimental software engineering (ESE), experimental replication is applied to validate results of an experiment. Much information and one version of the experimental materials are required for replication. Prior to the replication execution, all or part of the materials may require changes, producing new or modified versions of these, which should be incorporated into the material of the original experiment. There is a direct relationship between the increase in the number of replications with the increase of versions of the experimental material, which commonly causes confusion and disorder for experimental administrators. Objective: The aim of this paper is to conduct a mapping study to locate articles about the use of experimental material management tools in experimental replication in ESE. Method: We applied the mapping study to search, analyze and select published papers from reported replications. Results: We analyzed a total of 592 articles published from 1998 to 2014, 24 of them have been pre-selected and 4 have finally been selected. Conclusion: The results show the limited existence of articles on this subject. In addition, our analysis has allowed to identify that most of them suffer from problems in versions management for both replication and experimental material. These data provide information of interest to start a research about adoption of the paradigm of software configuration management inside the management of the experimental material in ESE.


Experimental software engineering Experimental replication Experimental material Experimental material management 



The authors thank the University of the Armed Forces (ESPE-L), the Polytechnic School - National University of Asuncion (FPUNA), the Ministry of Higher Education on Science, Technology and Innovation (SENESCYT) and Empirical Software Engineering Research Group (GrISE) for supporting the development of this work.


  1. 1.
    Daly, J., Brooks, A., Miller, J., Roper, M., Wood, M.: Verification of results in software maintenance through external replication. In: 1994 Proceedings of International Conference on Software Maintenance, pp. 50–57 (1994)Google Scholar
  2. 2.
    Schmidt, S.: Shall we really do it again? the powerful concept of replication is neglected in the social sciences. Rev. Gen. Psychol. 13, 90–100 (2009)CrossRefGoogle Scholar
  3. 3.
    Mendonca, M.G., Maldonado, J.C., de Oliveira, M.C.F., Carver, J., Fabbri, S.C.P.F., Shull, F., Travassos, G.H., Hohn, E.N., Basili, V.R.: A framework for software engineering experimental replications. In: 13th IEEE International Conference on Engineering of Complex Computer Systems 2008 ICECCS 2008, pp. 203–212 (2008)Google Scholar
  4. 4.
    Shull, F., Basili, V., Carver, J., Maldonado, J.C., Travassos, G.H., Mendonça, M., Fabbri, S.: Replicating software engineering experiments: addressing the tacit knowledge problem. In: Proceedings of 2002 International Symposium on Empirical Software Engineering 2002, pp. 7–16 (2002)Google Scholar
  5. 5.
    Do, H., Elbaum, S., Rothermel, G.: Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact. Empir. Softw. Eng. 10, 405–435 (2005)CrossRefGoogle Scholar
  6. 6.
    Shull, F., Mendoncça, M.G., Basili, V., Carver, J., Maldonado, J.C., Fabbri, S., Travassos, G.H., Ferreira, M.C.: Knowledge-sharing issues in experimental software engineering. Empir. Softw. Eng. 9, 111–137 (2004)CrossRefGoogle Scholar
  7. 7.
    Vegas, S., Juristo, N., Moreno, A., Solari, M., Letelier, P.: Analysis of the influence of communication between researchers on experiment replication. In: International Symposium on Empiricial Software Engineering, p. 28 (2006)Google Scholar
  8. 8.
    Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: Systematic mapping studies in software engineering. In: 12th International Conference on Evaluation and Assessment in Software Engineering (2008)Google Scholar
  9. 9.
    Beecham, S., Baddoo, N., Hall, T., Robinson, H., Sharp, H.: Protocol for a systematic literature review of motivation in software engineering (2006)Google Scholar
  10. 10.
    Basili, V.R., Shull, F., Lanubile, F.: Building knowledge through families of experiments. IEEE Trans. Softw. Eng. 25, 456–473 (1999)CrossRefGoogle Scholar
  11. 11.
    Maldonado, J.C., Carver, J., Shull, F., Fabbri, S., Dória, E., Martimiano, L., Mendonça, M., Basili, V.: Perspective-based reading: a replicated experiment focused on individual reviewer effectiveness. Empir. Softw. Eng. 11, 119–142 (2006)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Edison Espinosa
    • 1
  • Juan M. Ferreira
    • 2
  • Henry Chanatasig
    • 3
  1. 1.Universidad de las Fuerzas Armadas, ESPEQuitoEcuador
  2. 2.Universidad Nacional de Asunción, FPUNAAsunciónParaguay
  3. 3.Universidad Técnica de CotopaxiCotopaxiEcuador

Personalised recommendations