Abstract
Peer review is only effective if reviews are of high quality. In a large class, it is unrealistic for the course staff to evaluate all reviews, so a scalable assessment mechanism is needed. In an automated system, several metrics can be calculated for each review. One of these metrics is volume, which is simply the number of distinct words used in the review. Another is tone, which can be positive (e.g., praise), negative (e.g., disapproval), or neutral. A third is content, which we divide into three subtypes: summative, advisory, and problem detection. These metrics can be used to rate reviews, either singly or in combination. This paper compares the automated metrics for hundreds of reviews from the Expertiza system with scores manually assigned by the course staff. Almost all of the automatic metrics are positively correlated with manually assigned scores, but many of the correlations are weak. Another issue is how the review rubric influences review content. A more detailed rubric draws the reviewer’s attention to more characteristics of an author’s work. But ultimately, the author will benefit most from advisory or problem detection review text. And filling out a long rubric may distract the reviewer from providing textual feedback to the author. The data fail to show clear evidence that this effect occurs.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Gehringer, E. F. (2009). Expertiza: Information management for collaborative learning. In: A. A. Juan Perez (Ed.), Monitoring and assessment in online collaborative environments: Emergent computational technologies for e-learning support. Hershey: IGI Global Press.
Ramachandran, L. (2013). Automated assessment of reviews. Ph.D. dissertation, North Carolina State University, May 2013.
Ramachandran, L., & Gehringer, E. F. Automated assessment of the quality of peer reviews using natural language processing techniques, submitted to International Journal of Artificial Intelligence in Education.
Gehringer, E. F., & Peddycord, B. W. (2013). Grading by experience points: An example from computer ethics. In: Proceedings of Frontiers in Education 2013, Oklahoma City, OK, October 23–26.
Xiong, W., & Litman, D. (2011). Automatically predicting peer-review helpfulness. Short paper presented at The 49th Annual Meeting of the Association for Computational Linguistics. Portland, Oregon: Human Language Technologies (ACL-HLT).
Xiong, W. & Litman, D. (2014). Empirical analysis of exploiting review helpfulness for extractive summarization of online reviews. In: The 25th International Conference on Computational Linguistics (COLING 2014), Dublin, Ireland.
Liu, Y., Huang, X., An, A., & Yu, X. (2008). Modeling and predicting the helpfulness of online reviews. In Eighth IEEE International Conference on Data Mining, 2008. ICDM’08, pp. 443–452. New York: IEEE.
Zhuang, L., Jing, F., & Zhu, X. (2006). Movie review mining and summarization. In: Proceedings of the 15th ACM International Conference on Information and Knowledge Management, pp 43–50. New York City: ACM.
Lu, Y., Tsaparas, P., Ntoulas, A., & Polanyi, L. (2010). Exploiting social context for review quality prediction. In: Proceedings of the 19th International conference on World wide web (WWW’10), pp. 691–700. New York, NY, USA: ACM. doi:10.1145/1772690.1772761
Palanski, M., Babik, D., & Ford, E. (2014). Mobius SLIP: Anonymous, peer-reviewed student writing. OBTC 2014 at Vanderbilt University.
de Alfaro, L., & Shavlovsky, M. (2014). CrowdGrader: A tool for crowdsourcing the evaluation of homework assignments. In: Proceedings of the 45th ACM technical symposium on Computer science education (SIGCSE’14), pp. 415–420. New York, NY, USA: ACM. doi:10.1145/2538862.2538900, http://doi.acm.org/10.1145/2538862.2538900
Acknowledgments
This work has been supported by the U.S. National Science Foundation under grant 1432347.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer Science+Business Media Singapore
About this paper
Cite this paper
Yadav, R.K., Gehringer, E.F. (2016). Metrics for Automated Review Classification: What Review Data Show. In: Li, Y., et al. State-of-the-Art and Future Directions of Smart Learning. Lecture Notes in Educational Technology. Springer, Singapore. https://doi.org/10.1007/978-981-287-868-7_41
Download citation
DOI: https://doi.org/10.1007/978-981-287-868-7_41
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-287-866-3
Online ISBN: 978-981-287-868-7
eBook Packages: EducationEducation (R0)