Classroom Evaluation of a Scaffolding Intervention for Improving Peer Review Localization

  • Huy Nguyen
  • Wenting Xiong
  • Diane Litman
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8474)


A peer review system that automatically evaluates student feedback comments was deployed in a university research methods course. The course required students to create an argument diagram to justify a hypothesis, then use this diagram to write a paper introduction. Diagram and paper first drafts were both reviewed by peers. During peer review, the system automatically analyzed the quality of student comments with respect to localization (i.e. pinpointing the source of the comment in the diagram or paper). Two localization models (one for diagram and one for paper reviews) triggered a system scaffolding intervention to improve review quality whenever the review was predicted to have a ratio of localized comments less than a threshold. Reviewers could then choose to revise their comments or ignore the scaffolding. Our analysis of data from system logs demonstrates that diagram and paper localization models have high prediction accuracy, and that a larger portion of student feedback comments are successfully localized after scaffolded revision.


Peer review review localization scaffolding evaluation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Gielen, S., Peeters, E., Dochy, F., Onghena, P., Struyven, K.: Improving the effectiveness of peer feedback for learning. Learning and Instruction 20(4), 304–315 (2010)CrossRefGoogle Scholar
  2. 2.
    Cho, K.: Machine Classification of Peer Comments in Physics. In: Proceedings of 1st international conference on Educational Data Mining (EDM), pp. 192–196 (2008)Google Scholar
  3. 3.
    Cho, K., Schunn, C.D.: Scaffolded writing and rewriting in the discipline: A web-based reciprocal peer review system. Computers & Education 48(3), 409–426 (2007)CrossRefGoogle Scholar
  4. 4.
    Kumar, A.N.: Error-Flagging support for testing and its effect on adaptation. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010, Part I. LNCS, vol. 6094, pp. 359–368. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  5. 5.
    Lippman, J., Elfenbein, M., Diabes, M., Luchau, C., Lynch, C., Ashley, K.D., Schunn, C.D.: To Revise or Not To Revise: What Influences Undergrad Authors to Implement Peer Critiques of Their Argument Diagrams? In: International Society for the Psychology of Science and Technology 2012 Conference, Poster (2012)Google Scholar
  6. 6.
    Nelson, M.M., Schunn, C.D.: The nature of feedback: How different types of peer feedback affect writing performance. Instructional Science 37(4), 375–401 (2009)CrossRefGoogle Scholar
  7. 7.
    Nguyen, H.V., Litman, D.J.: Identifying Localization in Peer Reviews of Argument Diagrams. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS, vol. 7926, pp. 91–100. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  8. 8.
    Ramachandran, L., Gehringer, E.F.: Automated assessment of review quality using latent semantic analysis. In: 11th IEEE International Conference on Advanced Learning Technologies (ICALT), pp. 136–138 (2011)Google Scholar
  9. 9.
    Razzaq, L., Heffernan, N.T.: Hints: is it better to give or wait to be asked? In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010, Part I. LNCS, vol. 6094, pp. 349–358. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  10. 10.
    Scheuer, O., Loll, F., Pinkwart, N., McLaren, B.M.: Computer-supported argumentation: A review of the state of the art. International Journal of Computer-Supported Collaborative Learning 5(1), 43–102 (2010)CrossRefGoogle Scholar
  11. 11.
    Xiong, W., Litman, D.: Identifying problem localization in peer-review feedback. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010, Part II. LNCS, vol. 6095, pp. 429–431. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  12. 12.
    Xiong, W., Litman, D.: Automatically Predicting Peer-Review Helpfulness. In: Proceedings of 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT), pp. 502–507 (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Huy Nguyen
    • 1
  • Wenting Xiong
    • 1
  • Diane Litman
    • 1
  1. 1.Department of Computer ScienceUniversity of PittsburghPittsburghUSA

Personalised recommendations