Providing the Option to Skip Feedback – A Reproducibility Study
Would providing choice lead to improved learning with a tutor? We had conducted and reported a controlled study earlier, wherein, introductory programing students were given the choice of skipping the line-by-line feedback provided after each incorrect answer in a tutor on if/if-else statements. Contrary to expectations, the study found that the choice to skip feedback did not lead to greater learning. We tried to reproduce these results using two tutors on if/if-else and switch statements, and with a larger subject pool. We found that whereas choice did not lead to greater learning on if/if-else tutor in this reproducibility study either, it resulted in decreased learning on switch tutor. We hypothesize that skipping feedback is indeed detrimental to learning. But, inter-relationships among the concepts covered by a tutor and the transfer of learning facilitated by these relationships compensate for the negative effect of skipping line-by-line feedback. We also found contradictory results between the two studies which highlight the need for reproducibility studies in empirical research.
KeywordsSkipping feedback Worked example tutor Reproducibility study
Partial support for this work was provided by the National Science Foundation under grant DUE-1432190.
- 1.Drummond, C.: Replicability is not reproducibility: nor is it good science. In: Proceedings of Evaluation Methods for Machine Learning Workshop, 26th ICML, Montreal, Canada (2009)Google Scholar
- 3.Kumar, A.N.: Explanation of step-by-step execution as feedback for problems on program analysis, and its generation in model-based problem-solving tutors. Technol. Instr. Cogn. Learn. (TICL) 4(1), 65–107 (2006). Special Issue on Problem Solving Support in Intelligent Tutoring SystemsGoogle Scholar