Optimal experiments in the presence of a learning effect: a problem suggested by software production

Abstract.

In software engineering empirical comparisons of different ways of writing computer code are often made. This leads to the need for planned experimentation and has recently established a new area of application of DoE. This paper is motivated by an experiment on the production of multimedia services on the web, performed at the Telecom Research Centre in Turin, where two different ways of developing code, with or without a framework, were compared. As the experiment progresses, the programmer’s performance improves as he/she undergoes a learning process; this must be taken into account as it may affect the outcome of the trial. In this paper we discuss statistical models and D-optimal plans for such experiments and indicate some heuristics which allow a much speedier search for the optimum. Solutions differ according to whether we assume that the learning process depends or not on the treatments.