Advice for Action with Automatic Feedback Systems

Chapter
Part of the Lecture Notes on Data Engineering and Communications Technologies book series (LNDECT, volume 11)

Abstract

This Chapter reviews the role of feedback in supporting student learning. It highlights some of the problems that persist with providing meaningful feedback, which should preferably take the form of providing advice, that can be actioned by the student. It then discusses the progress made with automatic feedback through a number of case studies which include the OpenEssayist, Open Comment and OpenMentor computer assisted feedback systems. Findings suggest feedback that provides socio-emotive support to students, together with recognising their effort, in turn encourages the student to continue working on a problem. The use of automatic hints also moves the feedback closer to “Advice for Action ”. Building tools with automatic feedback to support both students and tutors can relieve some of the continual pressure on staff resources and three case studies are presented below that address this issue.

Keywords

Automatic feedback Advice for action OpenMentor Essay feedback 

Notes

Acknowledgements

The OpenEssayist Research was supported by the Engineering and Physical Sciences Research Council (EPSRC, grant numbers EP/J005959/1 & EP/J005231/1). Thanks are also due to the SAFeSEA team who produced OpenEssayist, namely John Richardson, Alison Twiner, Debora Field and Stephen Pulman. I would also like to thank Stuart Watt and the JISC for their support in the development of OpenMentor . Stuart Watt deserves special thanks as he also developed the Open Comment system.

References

  1. Alden, B., Van Labeke, N., Field, D., Pulman, S., Richardson, J. T. E., & Whitelock, D. (2014). Using student experience as a model for designing an automatic feedback system for short essays. International Journal of e-Assessment, 4(1), article no. 68.Google Scholar
  2. Alden Rivers, B., Whitelock, D., Richardson, J. T. E., Field, D., & Pulman, S. (2014). Functional, frustrating and full of potential: Learners’ experiences of a prototype for automated essay feedback. In Proceedings of Computer Assisted Assessment international conference: Research into eAssessment, Zeist, the Netherlands, 30 June–1 July.Google Scholar
  3. Aleven, V., Roll, I., McLaren, B. M., & Koedinger, K. R. (2010). Automated, unobtrusive, action-by-action assessment of self-regulation during learning with an intelligent tutoring system. Educational Psychologist, 45, 224–233.  https://doi.org/10.1080/00461520.2010.517740.
  4. Anderson, L. W., & Krathwohl, D. R. (Eds.), (2000). A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives. Allyn & Bacon.Google Scholar
  5. Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: Using learning analytics to increase student success. Paper presented at the 2nd International Conference on Learning Analytics and Knowledge, April 29th–May 2nd, Vancouver, BC, Canada. ACM 978-1-4503-1111-3/12/04.Google Scholar
  6. Bales, R. F. (1950). A set of categories for the analysis of small group interaction. American Sociological Review, 15, 257–263.CrossRefGoogle Scholar
  7. Beaumont, C., O’Doherty, M., & Shannon, L. (2011). Reconceptualising assessment feedback: A key to improving student learning? Studies in Higher Education, 36, pp. 671–687.  https://doi.org/10.1080/03075071003731135.
  8. Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education, 5(1), 7–74.CrossRefGoogle Scholar
  9. Brown, E., & Glover, C. (2006). Written feedback for students: too much, too detailed or too incomprehensible to be effective? Bioscience Education e-Journal, 7(3).Google Scholar
  10. Buhagiar, M. A. (2012). Mathematics student teachers’ views on tutor feedback during teaching practice. European Journal of Teacher Education, iFirst Article, 1-13.Google Scholar
  11. Burstein, J., Chodorow, M., & Leacock, C. (2003). CriterionSM online essay evaluation: An application for automated evaluation of student essays. In J. Riedl & R. Hill (Eds.), Proceedings of the Fifteenth Conference on Innovative Applications of Artificial Intelligence (pp. 3–10). Cambridge, MA: MIT Press.Google Scholar
  12. Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. American Association of Higher Education Bulletin, 39(7), 3–7.Google Scholar
  13. Dai, J., Raine, R. B., Roscoe, R. D., Cai, Z., & McNamara, D. S. (2011). The Writing-Pal tutoring system: Development and design. Journal of Engineering and Computer Innovations, 2(1), 1–11. ISSN 2141-6508 2011 Academic Journals.Google Scholar
  14. DiBattista, D., Mitterer, J. O., & Leanne, G. (2004) Acceptance by undergraduates of the immediate feedback assessment technique for multiple-choice testing. Teaching in Higher Education, 9(1), 17–28. ISSN 1356-2517.Google Scholar
  15. Dweck, C. (2008). Mindset: The new psychology of success. New York: Ballantine Books.Google Scholar
  16. Evans, C. (2013). Making sense of assessment feedback in higher education. Review of Educational Research, 83, 70–120.  https://doi.org/10.3102/0034654312474350.
  17. Field, D., Pulman, S., Van Labeke, N., Whitelock, D., & Richardson, J. T. E. (2013). Did I really mean that? Applying automatic summarisation techniques to formative feedback. In: G. Angelova, K. Bontcheva & R. Mitkov (Eds.), Proceedings International Conference Recent Advances in Natural Language Processing (pp. 277–284), 7–13 September, 2013. Hissar, Bulgaria: Association for Computational Linguistics.Google Scholar
  18. Franzke, M., & Streeter, L. A. (2006). Building student summarization, writing and reading comprehension skills with guided practice and automated feedback. Pearson Knowledge Technologies: White paper.Google Scholar
  19. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77, 81–112.  https://doi.org/10.3102/003465430298487.
  20. Holmberg, B. (1983). Guided didactic conversation in distance education. In D. Sewart, D. Keegan, & B. Holmberg (Eds.), Distance education: International perspectives (pp. 114–122). London: Croom Helm.Google Scholar
  21. Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25(2–3), 259–284.CrossRefGoogle Scholar
  22. Landauer, T. K., Laham, D., & Foltz, P. W. (2003a). Automatic essay assessment. Assessment in Education: Principles, Policy and Practice, 10(3), 295–308.CrossRefGoogle Scholar
  23. Landauer, T. K., Laham, D., & Foltz, P. W. (2003b). Automated scoring and annotation of essays with the Intelligent Essay Assessor. In Automated essay scoring: A cross-disciplinary perspective (pp. 87–112).Google Scholar
  24. Laurillard, D. (1993). Rethinking University Teaching: A framework for the effective use of educational technology. London: Routledge.Google Scholar
  25. Mayfield, E., & Rosé, C. P. (2013). LightSIDE: Open Source Machine Learning for Text. In M. D. Shermis & J. C. Burstein (Eds.), Handbook of automated essay evaluation: Current applications and new directions (pp. 124–135). Oxon: Routledge.Google Scholar
  26. McNamara, D. S., Raine, R., Roscoe, R., Crossley, S., Jackson, G. T., Dai, J., Cai, Z., Renner, A., Brandon, R., Weston, J., Dempsey, K., Lam, D., Sullivan, S., Kim, L., Rus, V., Floyd, R., McCarthy, P., & Graesser, A. (2011). The Writing- Pal: Natural language algorithms to support intelligent tutoring on writing strategies. In P. M. Mc-Carthy & Chutima Boonthum-Denecke (Eds.), Applied Natural Language Processing and Content Analysis: Advances in Identification, Investigation and Resolution (pp. 298–311). Hershey, PA: IGI Global.Google Scholar
  27. Merry, S., Price, M., Carless, D., & Taras, M. (Eds.). (2013). Reconceptualising feedback in higher education: Developing dialogue with students. London and New York: Routledge.Google Scholar
  28. Mueller, C. M., & Dweck, C. S. (1998). Praise for intelligence can undermine children’s motivation and performance. Journal of Personality and Social Psychology, 75(1), 33–52.  https://doi.org/10.1037/0022-3514.75.1.33.CrossRefGoogle Scholar
  29. Narciss, S. (2013). Designing and evaluating tutoring feedback strategies for digital learning environments on the basis of the interactive tutoring feedback model. Digital Education Review, 23, 7–26.Google Scholar
  30. Nelson, M. M., & Schunn, C. D. (2009). The nature of feedback: How different types of peer feedback affect writing performance. Instructional Science, 37(4), 375–401.CrossRefGoogle Scholar
  31. Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31, 199–218.  https://doi.org/10.1080/03075070600572090.CrossRefGoogle Scholar
  32. Norman, D. (1988). The psychology of everyday things. New York: Basic Books.Google Scholar
  33. Pask, G. (1976). Conversation theory: Applications in education and epistemology. Amsterdam: Elsevier.Google Scholar
  34. Pitcher, N., Goldfinch, J., & Beevers, C. (2002). Aspects of computer bases assessment in mathematics. Active Learning in Higher Education, 3(2), 19–25.CrossRefGoogle Scholar
  35. Rivers, B. A., Whitelock, D., Richardson, J. T., Field, D., & Pulman, S. (2014). Functional, frustrating and full of potential: Learners’ experiences of a prototype for automated essay feedback. In: Kalz, Marco & Ras, Eric (Eds.), Computer Assisted Assessment: Research into E-Assessment. Communications in Computer and Information Science (439) (pp. 40–52). Cham, Switzerland: Springer.Google Scholar
  36. Rudner, L. M., Garcia, V., & Catherine Welch. (2006). An evaluation of the IntelliMetricSM essay scoring system. The Journal of Technology, Learning, and Assessment, 4(4).Google Scholar
  37. Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18, 119–144.CrossRefGoogle Scholar
  38. Scott, D., Evans, C., Hughes, G., Burke, P. J., Watson, D., Walter, C., & Huttly, S. (2011). Facilitating transitions to masters-level learning: Improving formative assessment and feedback processes. Executive summary. Final extended report. London, UK: Institute of Education.Google Scholar
  39. Simpson, O. (2012). Supporting students for success in online and distance education (3rd ed.). London: Routledge.Google Scholar
  40. Steffens, K. (2006). Self-regulated learning in technology-enhanced learning environments: Lessons from a European review. European Journal of Education, 41, 353–380.  https://doi.org/10.1111/j.1465-3435.2006.00271.x.
  41. Taras, M. (2003). To feedback or not to feedback in student self-assessment. Assessment and Evaluation in Higher Education, 28, 549–565.  https://doi.org/10.1080/0260293032000120415.
  42. Weizenbaum, J. (1966). ELIZA: A computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36–45.CrossRefGoogle Scholar
  43. Whitelock, D. (2011). Activating assessment for learning: Are we on the way with Web 2.0. In M. J. W. Lee & C. McLoughlin (Eds.), Web 2.0-Based-E-Learning: Applying Social Informatics for Tertiary Teaching (Vol. 2, pp. 319–342). IGI Global.Google Scholar
  44. Whitelock, D., & Raw, Y. (2003). Taking an electronic mathematics examination from home: What the students think. In C. P. Constantinou & Z. C. Zacharia (Eds.), Computer Based Learning in Science, New Technologies and their Applications in Education, (Vol. 1, pp. 701–713). Nicosia, Cyprus: Department of Educational Sciences, University of Cyprus. ISBN 9963-8525-1-3.Google Scholar
  45. Whitelock, D., Watt, S. N. K., Raw, Y., & Moreale, E. (2004). Analysing tutor feedback to students: first steps towards constructing an electronic monitoring system. ALT-J, 1(3), 31–42.Google Scholar
  46. Whitelock, D., & Brasher, A. (2006). Developing a roadmap for e-Assessment: Which way now? In 10th International Computer Assisted Assessment Conference (pp. 487–501), Loughborough University, 4/5 July 2006. ISBN: 0-9539572-5-X.Google Scholar
  47. Whitelock, D., & Watt, S. (2008). Putting pedagogy in the driving seat with Open Comment: An open source formative assessment feedback and guidance tool for history students. In Farzana Khandia (Ed.), CAA Conference 2008 (pp. 347–356), Loughborough University, 8/9 July 2008.Google Scholar
  48. Whitelock, D. M., Gilbert, L., Hatzipanagos, S., Watt, S., Zhang, P., Gillary, P., & Recio, A. (2012a). Addressing the challenges of assessment and feedback in higher education: A collaborative effort across three UK Universities. In Proceedings INTED 2012, Valencia, Spain. ISBN: 978-84-615-5563-5.Google Scholar
  49. Whitelock, D., Gilbert, L., Hatzipanagos, S., Watt, S., Zhang, P., Gillary, P. Recio, A. (2012b). Assessment for learning: Supporting tutors with their feedback using an electronic system that can be used across the higher education sector. In Proceedings 10th International Conference on Computer Based Learning in Science, CBLIS, 26–29 June 2012, Barcelona, Spain.Google Scholar
  50. Whitelock, D., Twiner, A., Richardson, J. T., Field, D., & Pulman, S. (2015). OpenEssayist: A supply and demand learning analytics tool for drafting academic essays. Paper presented at the Proceedings of the Fifth International Conference on Learning Analytics and Knowledge, Poughkeepsie, NY, USA.Google Scholar
  51. Yeager, D. S., Paunesku, D., Walton, G. M., & Dweck, C. (2013). How can we instill productive mindsets at scale? A review of the evidence and an initial R&D agenda. A White Paper prepared for the White House meeting on Excellence in Education: The Importance of Academic Mindsets.Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.The Open UniversityMilton KeynesUK

Personalised recommendations