Journal of Science Education and Technology

, Volume 24, Issue 6, pp 861–874 | Cite as

Automated Guidance for Thermodynamics Essays: Critiquing Versus Revisiting

  • Dermot F. Donnelly
  • Jonathan M. Vitale
  • Marcia C. Linn


Middle school students struggle to explain thermodynamics concepts. In this study, to help students succeed, we use a natural language processing program to analyze their essays explaining the aspects of thermodynamics and provide guidance based on the automated score. The 346 sixth-grade students were assigned to either the critique condition where they criticized an explanation or the revisit condition where they reviewed visualizations. Within each condition, the student was assigned one of two types of tailored guidance based on the sophistication of their original essay. Both forms of guidance led to significant improvement in student understanding on the posttest. Guidance was more effective for students with low prior knowledge than for those with high prior knowledge (consistent with regression toward the mean). However, analysis of student responses to the guidance illustrates the value of aligning guidance with prior knowledge. All students were required to revise their essay as an embedded assessment. While effective, teachers involved in this study reported that revising is resisted by students and does not align with typical, vocabulary-focused classroom writing activities.


Inquiry Automated scoring Guidance Critique Revisit Thermodynamics 



This material is based upon work supported by National Science Foundation (NSF) Grant Number 1119670 (CLASS: Continuous Learning and Automated Scoring in Science). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF. The authors gratefully acknowledge the teachers and students who participated in this study, and the reviewers who gave feedback on this paper.


  1. Azevedo R, Bernard RM (1995) A meta-analysis of the effects of feedback in computer-based instruction. J Educ Comput Res 13(2):111–127CrossRefGoogle Scholar
  2. Blanchard M, Southerland S, Osborne J, Sampson V, Annetta L, Granger E (2010) Is inquiry possible in light of accountability? A quantitative comparison of the relative effectiveness of guided inquiry and verification laboratory instruction. Sci Educ 94(4):577–616CrossRefGoogle Scholar
  3. Butcher PG, Jordan SE (2010) A comparison of human and computer marking of short free-text student responses. Comput Educ 55(2):489–499. doi: 10.1016/j.compedu.2010.02.012 CrossRefGoogle Scholar
  4. Chang H-Y, Chang H-C (2013) Scaffolding students’ online critiquing of expert- and peer-generated molecular models of chemical reactions. Int J Sci Educ 35(12):2028–2056CrossRefGoogle Scholar
  5. Chang H-Y, Linn MC (2013) Scaffolding learning from molecular visualizations. J Res Sci Teach 50(7):858–886. doi: 10.1002/tea.21089 CrossRefGoogle Scholar
  6. Chang H-Y, Quintana C, Krajcik JS (2010) The impact of designing and evaluating molecular animations on how well middle school students understand the particulate nature of matter. Sci Educ 94(1):73–94Google Scholar
  7. Chase C, Chin D, Oppezzo M, Schwartz D (2009) Teachable agents and the protégé effect: increasing the effort towards learning. J Sci Educ Technol 18(4):334–352CrossRefGoogle Scholar
  8. Chin C (2006) Classroom interaction in science: teacher questioning and feedback to students’ responses. Int J Sci Educ 28(11):1315–1346CrossRefGoogle Scholar
  9. Chinn CA, Brewer WF (2001) Models of data: a theory of how people evaluate data. Cognit Instr 19(3):323–393CrossRefGoogle Scholar
  10. Clark D, Jorde D (2004) Helping students revise disruptive experientially supported ideas about thermodynamics: computer visualizations and tactile models. J Res Sci Teach 41(1):1–23CrossRefGoogle Scholar
  11. Clauser BE, Kane MT, Swanson DB (2002) Validity issues for performance-based tests scored with computer-automated scoring systems. Appl Meas Educ 15(4):413–432CrossRefGoogle Scholar
  12. Cuthbert AF, Slotta JD (2004) Designing a web-based design curriculum for middle school science: the WISE ‘houses in the desert’ project. Int J Sci Educ 26(7):821–844CrossRefGoogle Scholar
  13. Davis EA (2003) Prompting middle school science students for productive reflection: generic and directed prompts. J Learn Sci 12(1):91–142CrossRefGoogle Scholar
  14. de Jong T, Linn MC, Zacharia ZC (2013) Physical and virtual laboratories in science and engineering education. Science 340(6130):305–308. doi: 10.1126/science.1230579 CrossRefGoogle Scholar
  15. diSessa AA (1988) Knowledge in pieces. In: Forman G, Pufall P (eds) Constructivism in the computer age. Hillsdale, Lawrence Erlbaum Associates, New JerseyGoogle Scholar
  16. diSessa AA (1996) What do ‘‘just plain folk’’ know about physics? In: Olson DR, Torrance N (eds) The handbook of education and human development: new models of learning, teaching, and schooling. Blackwell, Cambridge, pp 709–730Google Scholar
  17. diSessa AA (2006) A history of conceptual change research: threads and fault lines. In: Sawyer K (ed) Cambridge handbook of the learning sciences. Cambridge University Press, Cambridge, pp 265–281Google Scholar
  18. Donnelly DF, Linn MC, Ludvigsen S (2014) Impacts and characteristics of computer-based science inquiry learning environments for precollege students. Rev Educ Res 84(4):572–608. doi: 10.3102/0034654314546954 CrossRefGoogle Scholar
  19. Erickson GL (1985) Heat and temperature: part A. An overview of pupils’ ideas. In: Driver R, Guesne E, Tiberghien A (eds) Children’s ideas in science. Open University Press, Milton Keynes, pp 55–66Google Scholar
  20. Gerard LF, Linn MC (2013) What kind of feedback improves understanding? A comparison of knowledge integration and generic feedback on students’ scientific explanations. Paper presented at the annual meeting of the American Education Research Association. San Francisco, CAGoogle Scholar
  21. Gerard L, Matuk C, McElhaney K, Linn MC (in press) Automated, adaptive guidance for K-12 education. Educ Res Rev. doi: 10.1016/j.edurev.2015.04.001
  22. Gerard L, Ryoo K, McElhaney K, Liu L, Raferty A, Linn MC (in press) Automated guidance for student inquiry. J Educ PsycholGoogle Scholar
  23. Haertel GD, Means B, Penuel W (2008) Technology tools for collecting, managing, and using assessment data to inform instruction and improve achievement. In: Smolin L, Lawless K, Burbules N (Eds), NSSE yearbook 2007, Information and communication technologies: considerations for teachers and teacher education. Chicago, National Society for the Study of EducationGoogle Scholar
  24. Hamp-Lyons L (2001) Fourth generation writing assessment. In: Silva T, Matsuda PK (eds) On second language writing. Lawrence Erlbaum Associates Inc, Mahwah, NJ, pp 117–125Google Scholar
  25. Havnes A, Smith K, Dysthe O, Ludvigsen K (2012) Formative assessment and feedback: making learning visible. Stud Educ Eval 38(1):21–27CrossRefGoogle Scholar
  26. Hmelo-Silver C, Duncan R, Chinn C (2007) Scaffolding and achievement in problem-based and inquiry learning: a response to Kirschner, Sweller, and Clark (2006). Educ Psychol 42(2):99–107CrossRefGoogle Scholar
  27. Höffler TN, Leutner D (2007) Instructional animation versus static pictures: a meta-analysis. Learn Instr 17(6):722–738CrossRefGoogle Scholar
  28. Hovardas T, Tsivitanidou OE, Zacharia ZC (2014) Peer versus expert feedback: an investigation of the quality of peer feedback among secondary school students. Comput Educ 71(1):133–152CrossRefGoogle Scholar
  29. Hutchison P, Hammer D (2010) Attending to student epistemological framing in a science classroom. Sci Educ 94(3):506–524Google Scholar
  30. Hyland F (1998) The impact of teacher written feedback on individual writers. J Second Lang Writ 7(3):255–286CrossRefGoogle Scholar
  31. Jordan S (2012) Student engagement with assessment and feedback: some lessons from short-answer free-text e-assessment questions. Comput Educ 58(2):818–834CrossRefGoogle Scholar
  32. Kali Y (2006) Collaborative knowledge building using the Design Principles Database. Int J Comput-Support Collab Learn 1(2):187–201CrossRefGoogle Scholar
  33. Kegel CAT, Bus AG (2013) Links between DRD4, executive attention, and alphabetic skills in a nonclinical sample. J Child Psychol Psychiatry 54(3):305–312CrossRefGoogle Scholar
  34. Kluger AN, DeNisi A (1996) The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 119(2):254–284. doi: 10.1037/0033-2909.119.2.254 CrossRefGoogle Scholar
  35. Landauer TK, Laham D, Foltz P (2003) Automated scoring and annotation of essays with intelligent essay assessor™. In: Shermis MD, Burstein J (eds) Automated essay scoring: a cross-disciplinary perspective. Lawrence Erlbaum, Mahwah, NJGoogle Scholar
  36. Leacock C, Chodorow M (2003) c-rater: automated scoring of short-answer questions. Comput Humanit 37(4):389–405CrossRefGoogle Scholar
  37. Lee H-S, Liu OL (2010) Assessing learning progression of energy concepts across middle school grades: the knowledge integration perspective. Sci Educ 94(4):665–688CrossRefGoogle Scholar
  38. Lewis E (1996) Conceptual change among middle school students studying elementary thermodynamics. J Sci Educ Technol 5(1):3–31CrossRefGoogle Scholar
  39. Lewis EL, Linn MC (1994) Heat energy and temperature concepts of adolescents, adults, and experts: implications for curricular improvements. J Res Sci Teach 31(6):657–677. doi: 10.1002/tea.3660310607 CrossRefGoogle Scholar
  40. Linn M, Eylon B-S (2011) Science learning and instruction: taking advantage of technology to promote knowledge integration. Routledge, New YorkGoogle Scholar
  41. Linn MC, Songer NB (1991) Teaching thermodynamics to middle school students: what are appropriate cognitive demands? J Res Sci Teach 28(10):885–918. doi: 10.1002/tea.3660281003 Google Scholar
  42. Linn M, Songer N, Lewis E, Stern J (1993) Using technology to teach thermodynamics: achieving integrated understanding. In: Ferguson D (Ed), Advanced educational technologies for mathematics and science SE—1 (Vol 107, pp 5–60), Springer, Berlin. doi: 10.1007/978-3-662-02938-1_1
  43. Lipnevich A, Smith J (2009) Effects of differential feedback on students’ examination performance. J Exp Psychol Appl 15(4):319–333CrossRefGoogle Scholar
  44. McNeill KL, Krajcik J (2012) Supporting grade 5–8 students in constructing explanations in science: the claim, evidence and reasoning framework for talk and writing. Pearson Allyn & Bacon, New York, NYGoogle Scholar
  45. Mortimer E, Scott P (2000) Analysing discourse in the science classroom. In: Millar R, Leach J, Osborne J (eds) Improving science education: the contribution of research. Open University Press, BuckinghamGoogle Scholar
  46. National Research Council (2012) A framework for K-12 science education: practices, crosscutting concepts, and core ideas. National Academy Press, Washington, DCGoogle Scholar
  47. Nuthall G (2005) The cultural myths and realities of classroom teaching and learning: a personal journey. Teach Coll Rec 107(5):895–934CrossRefGoogle Scholar
  48. Osborne J (2010) Arguing to learn in science. The role of collaborative, critical discourse. Science 328:463CrossRefGoogle Scholar
  49. Page EB (2003) Project essay grade: PEG. In: Shermis MD, Burstein J (eds) Automated essay scoring: a cross-disciplinary perspective. Lawrence Erlbaum Associates, Mahwah, NJ, pp 43–54Google Scholar
  50. Ruiz-Primo MA (2011) Informal formative assessment: the role of instructional dialogues in assessing students’ learning. Stud Educ Eval 37(1):15–24CrossRefGoogle Scholar
  51. Ryoo K, Linn MC (2012) Can dynamic visualizations improve middle school students’ understanding of energy in photosynthesis? J Res Sci Teach 49(2):218–243. doi: 10.1002/tea.21003 CrossRefGoogle Scholar
  52. Ryoo K, Linn MC (2014) Comparison of specific and knowledge integration automated guidance for concept diagrams in inquiry instruction. In: Paper presented at the International Conference of the Learning Sciences (ICLS) meeting, Boulder, COGoogle Scholar
  53. Sandoval WA, Millwood KA (2005) The quality of students’ use of evidence in written scientific explanations. Cognit Instr 23(1):23–55CrossRefGoogle Scholar
  54. Sato E, Linn MC (2014) Designing critique to improve conceptual understanding. In: Penuel B, Jurow S, O’Connor K (eds), Proceedings of learning and becoming in practice, Vol 1, p 471–478 International Society of the Learning SciencesGoogle Scholar
  55. Scalise K, Timms M, Moorjani A, Clark L, Holtermann K, Irvin PS (2011) Student learning in science simulations: design features that promote learning gains. J Res Sci Teach 48(9):1050–1078CrossRefGoogle Scholar
  56. Segedy JR, Kinnebrew JS, Biswas G (2013) The effect of contextualized conversational feedback in a complex open-ended learning environment. Educ Tech Res Dev 61(1):71–89CrossRefGoogle Scholar
  57. Shen J (2010) Nurturing students’ critical knowledge using technology-enhanced scaffolding strategies in science education. J Sci Educ Technol 19(1):1–12CrossRefGoogle Scholar
  58. Sukkarieh JZ, Blackmore J (2009). C-rater: automatic content scoring for short constructed responses. In: Proceedings of the 22nd-International FLAIRS Conference. Sanibel Island, FL: Association for the Advancement of Artificial IntelligenceGoogle Scholar
  59. Swift JN, Gooding CT (1983) Interaction of wait time feedback and questioning instruction on middle school science teaching. J Res Sci Teach 20(8):721–730CrossRefGoogle Scholar
  60. Tiberghien A (1980) Models and conditions of learning. An example: the learning of some aspects of the concepts of heat. University of Leeds Printing Services, LeedsGoogle Scholar
  61. Tsai C-C, Lin SSJ, Yuan S-M (2002) Developing science activities through a networked peer assessment system. Comput Educ 38(1–3):241–252CrossRefGoogle Scholar
  62. Van der Kleij FM, Eggen TJHM, Timmers CF, Veldkamp BP (2012) Effects of feedback in a computer-based assessment for learning. Comput Educ 58(1):263–272CrossRefGoogle Scholar
  63. Vygotsky L (1978) Mind in society: the development of higher psychological processes. Harvard University Press, Cambridge, MAGoogle Scholar
  64. Wang H-C, Chang C-Y, Li T-Y (2008) Assessing creative problem-solving with automated text grading. Comput Educ 51(4):1450–1466CrossRefGoogle Scholar
  65. White BY, Frederiksen JR (2000) Technological tools and instructional approaches for making scientific inquiry accessible to all. In: Jacobson MJ, Kozma RB (eds) Innovations in science and mathematics education: advanced designs for technologies of learning. Lawrence Erlbaum Associates, Mahwah, NJ, pp 321–359Google Scholar
  66. Williams M, DeBarger AH, Montgomery BL, Zhou X, Tate E (2012) Exploring middle school students’ conceptions of the relationship between genetic inheritance and cell division. Sci Educ 96(1):78–103CrossRefGoogle Scholar
  67. Zhang ZH, Linn MC (2013) Learning from chemical visualizations: comparing generation and selection. Int J Sci Educ 35(13):2174–2197. doi: 10.1080/09500693.2013.792971 CrossRefGoogle Scholar
  68. Zwiep S, Straits W (2013) Inquiry science: the gateway to English language proficiency. J Sci Teach Educ 24(8):1315–1331CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Graduate School of EducationUniversity of CaliforniaBerkeleyUSA

Personalised recommendations