Advice for Action with Automatic Feedback Systems
This Chapter reviews the role of feedback in supporting student learning. It highlights some of the problems that persist with providing meaningful feedback, which should preferably take the form of providing advice, that can be actioned by the student. It then discusses the progress made with automatic feedback through a number of case studies which include the OpenEssayist, Open Comment and OpenMentor computer assisted feedback systems. Findings suggest feedback that provides socio-emotive support to students, together with recognising their effort, in turn encourages the student to continue working on a problem. The use of automatic hints also moves the feedback closer to “Advice for Action ”. Building tools with automatic feedback to support both students and tutors can relieve some of the continual pressure on staff resources and three case studies are presented below that address this issue.
KeywordsAutomatic feedback Advice for action OpenMentor Essay feedback
The OpenEssayist Research was supported by the Engineering and Physical Sciences Research Council (EPSRC, grant numbers EP/J005959/1 & EP/J005231/1). Thanks are also due to the SAFeSEA team who produced OpenEssayist, namely John Richardson, Alison Twiner, Debora Field and Stephen Pulman. I would also like to thank Stuart Watt and the JISC for their support in the development of OpenMentor . Stuart Watt deserves special thanks as he also developed the Open Comment system.
- Alden, B., Van Labeke, N., Field, D., Pulman, S., Richardson, J. T. E., & Whitelock, D. (2014). Using student experience as a model for designing an automatic feedback system for short essays. International Journal of e-Assessment, 4(1), article no. 68.Google Scholar
- Alden Rivers, B., Whitelock, D., Richardson, J. T. E., Field, D., & Pulman, S. (2014). Functional, frustrating and full of potential: Learners’ experiences of a prototype for automated essay feedback. In Proceedings of Computer Assisted Assessment international conference: Research into eAssessment, Zeist, the Netherlands, 30 June–1 July.Google Scholar
- Aleven, V., Roll, I., McLaren, B. M., & Koedinger, K. R. (2010). Automated, unobtrusive, action-by-action assessment of self-regulation during learning with an intelligent tutoring system. Educational Psychologist, 45, 224–233. https://doi.org/10.1080/00461520.2010.517740.
- Anderson, L. W., & Krathwohl, D. R. (Eds.), (2000). A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives. Allyn & Bacon.Google Scholar
- Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: Using learning analytics to increase student success. Paper presented at the 2nd International Conference on Learning Analytics and Knowledge, April 29th–May 2nd, Vancouver, BC, Canada. ACM 978-1-4503-1111-3/12/04.Google Scholar
- Beaumont, C., O’Doherty, M., & Shannon, L. (2011). Reconceptualising assessment feedback: A key to improving student learning? Studies in Higher Education, 36, pp. 671–687. https://doi.org/10.1080/03075071003731135.
- Brown, E., & Glover, C. (2006). Written feedback for students: too much, too detailed or too incomprehensible to be effective? Bioscience Education e-Journal, 7(3).Google Scholar
- Buhagiar, M. A. (2012). Mathematics student teachers’ views on tutor feedback during teaching practice. European Journal of Teacher Education, iFirst Article, 1-13.Google Scholar
- Burstein, J., Chodorow, M., & Leacock, C. (2003). CriterionSM online essay evaluation: An application for automated evaluation of student essays. In J. Riedl & R. Hill (Eds.), Proceedings of the Fifteenth Conference on Innovative Applications of Artificial Intelligence (pp. 3–10). Cambridge, MA: MIT Press.Google Scholar
- Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. American Association of Higher Education Bulletin, 39(7), 3–7.Google Scholar
- Dai, J., Raine, R. B., Roscoe, R. D., Cai, Z., & McNamara, D. S. (2011). The Writing-Pal tutoring system: Development and design. Journal of Engineering and Computer Innovations, 2(1), 1–11. ISSN 2141-6508 2011 Academic Journals.Google Scholar
- DiBattista, D., Mitterer, J. O., & Leanne, G. (2004) Acceptance by undergraduates of the immediate feedback assessment technique for multiple-choice testing. Teaching in Higher Education, 9(1), 17–28. ISSN 1356-2517.Google Scholar
- Dweck, C. (2008). Mindset: The new psychology of success. New York: Ballantine Books.Google Scholar
- Evans, C. (2013). Making sense of assessment feedback in higher education. Review of Educational Research, 83, 70–120. https://doi.org/10.3102/0034654312474350.
- Field, D., Pulman, S., Van Labeke, N., Whitelock, D., & Richardson, J. T. E. (2013). Did I really mean that? Applying automatic summarisation techniques to formative feedback. In: G. Angelova, K. Bontcheva & R. Mitkov (Eds.), Proceedings International Conference Recent Advances in Natural Language Processing (pp. 277–284), 7–13 September, 2013. Hissar, Bulgaria: Association for Computational Linguistics.Google Scholar
- Franzke, M., & Streeter, L. A. (2006). Building student summarization, writing and reading comprehension skills with guided practice and automated feedback. Pearson Knowledge Technologies: White paper.Google Scholar
- Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77, 81–112. https://doi.org/10.3102/003465430298487.
- Holmberg, B. (1983). Guided didactic conversation in distance education. In D. Sewart, D. Keegan, & B. Holmberg (Eds.), Distance education: International perspectives (pp. 114–122). London: Croom Helm.Google Scholar
- Landauer, T. K., Laham, D., & Foltz, P. W. (2003b). Automated scoring and annotation of essays with the Intelligent Essay Assessor. In Automated essay scoring: A cross-disciplinary perspective (pp. 87–112).Google Scholar
- Laurillard, D. (1993). Rethinking University Teaching: A framework for the effective use of educational technology. London: Routledge.Google Scholar
- Mayfield, E., & Rosé, C. P. (2013). LightSIDE: Open Source Machine Learning for Text. In M. D. Shermis & J. C. Burstein (Eds.), Handbook of automated essay evaluation: Current applications and new directions (pp. 124–135). Oxon: Routledge.Google Scholar
- McNamara, D. S., Raine, R., Roscoe, R., Crossley, S., Jackson, G. T., Dai, J., Cai, Z., Renner, A., Brandon, R., Weston, J., Dempsey, K., Lam, D., Sullivan, S., Kim, L., Rus, V., Floyd, R., McCarthy, P., & Graesser, A. (2011). The Writing- Pal: Natural language algorithms to support intelligent tutoring on writing strategies. In P. M. Mc-Carthy & Chutima Boonthum-Denecke (Eds.), Applied Natural Language Processing and Content Analysis: Advances in Identification, Investigation and Resolution (pp. 298–311). Hershey, PA: IGI Global.Google Scholar
- Merry, S., Price, M., Carless, D., & Taras, M. (Eds.). (2013). Reconceptualising feedback in higher education: Developing dialogue with students. London and New York: Routledge.Google Scholar
- Narciss, S. (2013). Designing and evaluating tutoring feedback strategies for digital learning environments on the basis of the interactive tutoring feedback model. Digital Education Review, 23, 7–26.Google Scholar
- Norman, D. (1988). The psychology of everyday things. New York: Basic Books.Google Scholar
- Pask, G. (1976). Conversation theory: Applications in education and epistemology. Amsterdam: Elsevier.Google Scholar
- Rivers, B. A., Whitelock, D., Richardson, J. T., Field, D., & Pulman, S. (2014). Functional, frustrating and full of potential: Learners’ experiences of a prototype for automated essay feedback. In: Kalz, Marco & Ras, Eric (Eds.), Computer Assisted Assessment: Research into E-Assessment. Communications in Computer and Information Science (439) (pp. 40–52). Cham, Switzerland: Springer.Google Scholar
- Rudner, L. M., Garcia, V., & Catherine Welch. (2006). An evaluation of the IntelliMetricSM essay scoring system. The Journal of Technology, Learning, and Assessment, 4(4).Google Scholar
- Scott, D., Evans, C., Hughes, G., Burke, P. J., Watson, D., Walter, C., & Huttly, S. (2011). Facilitating transitions to masters-level learning: Improving formative assessment and feedback processes. Executive summary. Final extended report. London, UK: Institute of Education.Google Scholar
- Simpson, O. (2012). Supporting students for success in online and distance education (3rd ed.). London: Routledge.Google Scholar
- Steffens, K. (2006). Self-regulated learning in technology-enhanced learning environments: Lessons from a European review. European Journal of Education, 41, 353–380. https://doi.org/10.1111/j.1465-3435.2006.00271.x.
- Taras, M. (2003). To feedback or not to feedback in student self-assessment. Assessment and Evaluation in Higher Education, 28, 549–565. https://doi.org/10.1080/0260293032000120415.
- Whitelock, D. (2011). Activating assessment for learning: Are we on the way with Web 2.0. In M. J. W. Lee & C. McLoughlin (Eds.), Web 2.0-Based-E-Learning: Applying Social Informatics for Tertiary Teaching (Vol. 2, pp. 319–342). IGI Global.Google Scholar
- Whitelock, D., & Raw, Y. (2003). Taking an electronic mathematics examination from home: What the students think. In C. P. Constantinou & Z. C. Zacharia (Eds.), Computer Based Learning in Science, New Technologies and their Applications in Education, (Vol. 1, pp. 701–713). Nicosia, Cyprus: Department of Educational Sciences, University of Cyprus. ISBN 9963-8525-1-3.Google Scholar
- Whitelock, D., Watt, S. N. K., Raw, Y., & Moreale, E. (2004). Analysing tutor feedback to students: first steps towards constructing an electronic monitoring system. ALT-J, 1(3), 31–42.Google Scholar
- Whitelock, D., & Brasher, A. (2006). Developing a roadmap for e-Assessment: Which way now? In 10th International Computer Assisted Assessment Conference (pp. 487–501), Loughborough University, 4/5 July 2006. ISBN: 0-9539572-5-X.Google Scholar
- Whitelock, D., & Watt, S. (2008). Putting pedagogy in the driving seat with Open Comment: An open source formative assessment feedback and guidance tool for history students. In Farzana Khandia (Ed.), CAA Conference 2008 (pp. 347–356), Loughborough University, 8/9 July 2008.Google Scholar
- Whitelock, D. M., Gilbert, L., Hatzipanagos, S., Watt, S., Zhang, P., Gillary, P., & Recio, A. (2012a). Addressing the challenges of assessment and feedback in higher education: A collaborative effort across three UK Universities. In Proceedings INTED 2012, Valencia, Spain. ISBN: 978-84-615-5563-5.Google Scholar
- Whitelock, D., Gilbert, L., Hatzipanagos, S., Watt, S., Zhang, P., Gillary, P. Recio, A. (2012b). Assessment for learning: Supporting tutors with their feedback using an electronic system that can be used across the higher education sector. In Proceedings 10th International Conference on Computer Based Learning in Science, CBLIS, 26–29 June 2012, Barcelona, Spain.Google Scholar
- Whitelock, D., Twiner, A., Richardson, J. T., Field, D., & Pulman, S. (2015). OpenEssayist: A supply and demand learning analytics tool for drafting academic essays. Paper presented at the Proceedings of the Fifth International Conference on Learning Analytics and Knowledge, Poughkeepsie, NY, USA.Google Scholar
- Yeager, D. S., Paunesku, D., Walton, G. M., & Dweck, C. (2013). How can we instill productive mindsets at scale? A review of the evidence and an initial R&D agenda. A White Paper prepared for the White House meeting on Excellence in Education: The Importance of Academic Mindsets.Google Scholar