Providing the Option to Skip Feedback – A Reproducibility Study

  • Amruth N. KumarEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11528)


Would providing choice lead to improved learning with a tutor? We had conducted and reported a controlled study earlier, wherein, introductory programing students were given the choice of skipping the line-by-line feedback provided after each incorrect answer in a tutor on if/if-else statements. Contrary to expectations, the study found that the choice to skip feedback did not lead to greater learning. We tried to reproduce these results using two tutors on if/if-else and switch statements, and with a larger subject pool. We found that whereas choice did not lead to greater learning on if/if-else tutor in this reproducibility study either, it resulted in decreased learning on switch tutor. We hypothesize that skipping feedback is indeed detrimental to learning. But, inter-relationships among the concepts covered by a tutor and the transfer of learning facilitated by these relationships compensate for the negative effect of skipping line-by-line feedback. We also found contradictory results between the two studies which highlight the need for reproducibility studies in empirical research.


Skipping feedback Worked example tutor Reproducibility study 



Partial support for this work was provided by the National Science Foundation under grant DUE-1432190.


  1. 1.
    Drummond, C.: Replicability is not reproducibility: nor is it good science. In: Proceedings of Evaluation Methods for Machine Learning Workshop, 26th ICML, Montreal, Canada (2009)Google Scholar
  2. 2.
    Kumar, A.N.: Providing the option to skip feedback in a worked example tutor. In: Micarelli, A., Stamper, J., Panourgia, K. (eds.) ITS 2016. LNCS, vol. 9684, pp. 101–110. Springer, Cham (2016). Scholar
  3. 3.
    Kumar, A.N.: Explanation of step-by-step execution as feedback for problems on program analysis, and its generation in model-based problem-solving tutors. Technol. Instr. Cogn. Learn. (TICL) 4(1), 65–107 (2006). Special Issue on Problem Solving Support in Intelligent Tutoring SystemsGoogle Scholar
  4. 4.
    Meyer, B.J.F., et al.: Web-based tutoring of the structure strategy with or without elaborated feedback or choice for fifth- and seventh-grade readers. Read. Res. Q. 45(1), 62–92 (2010)CrossRefGoogle Scholar
  5. 5.
    Ostrow, K.S., Heffernan, N.T.: The role of student choice within adaptive tutoring. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (eds.) AIED 2015. LNCS (LNAI), vol. 9112, pp. 752–755. Springer, Cham (2015). Scholar
  6. 6.
    Sweller, J., Cooper, G.A.: The use of worked examples as a substitute for problem solving in learning algebra. Cogn. Instr. 2, 59–89 (1985)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Ramapo College of New JerseyMahwahUSA

Personalised recommendations