Keypresses and Mouse Clicks: Analysis of the First National Computer-Based Writing Assessment
The quality of students’ writing skills continues to concern educators. Because writing is essential to success in both college and career, poor writing can have lifelong consequences. Writing is now primarily done digitally, but students receive limited explicit instruction in digital writing. This lack of instruction means that students fail to take advantage of the affordances of digital tools. The writing process is shaped by the tools used, which makes digital writing, to an extent, different from writing with pen and paper. To better understand students’ digital writing skills, we take advantage of the information provided by computer-based assessments—keyboard and mouse activity data. We examine the relationship between students’ use of the keyboard and mouse during the assessment and students’ writing achievement. Our data comes from the first national computer-based writing assessment in the United States, the 2011 National Assessment of Educational Progress (NAEP) assessment. Using data from over 24,100 eighth-grade students, we found that the number of keypresses had a distinct and direct effect on writing achievement scores, controlling for word count. We also identified several different patterns of keyboard and mouse activity on the computer-based NAEP assessment.
KeywordsNAEP Writing Secondary students Digital writing Log data Clickstream data Trace data
This work has been supported by funding from the Spencer Foundation, Grant No. 201500153.
- Allen, N. L., & Donoghue, J. R. (1996). Applying the Mantel–Haenszel procedure to complex samples of items. Journal of Educational Measurement, 33(2), 231–251. https://doi.org/10.1111/j.1745-3984.1996.tb00491.x.CrossRefGoogle Scholar
- Allen, N. L., Donoghue, J. R., & Schoeps, T. L. (2001). The NAEP 1998 Technical Report. NCES 2001-509. Washington, DC: National Center for Education Statistics.Google Scholar
- Almond, R., Deane, P., Quinlan, T., Wagner, M., & Sydorenko, T. (2012). A preliminary analysis of keystroke log data from a timed writing task. ETS Research Report Series, 2012(2), i–61.Google Scholar
- Applebee, A. N. (2007). Issues in large-scale writing assessment perspectives from the National Assessment of Educational Progress. Journal of Writing Assessment, 3(2), 81–98. Retrieved from http://www.journalofwritingassessment.org/archives/3-2.2pdf.
- Applebee, A. N., & Langer, J. A. (2011). A snapshot of writing instruction in middle schools and high schools. English Journal, 100(6), 14–27.Google Scholar
- Bazerman, C., Applebee, A., Berninger, V., Brandt, D., Graham, S., Jeffery, J., et al. (Eds.). (2018). The lifespan development of writing. Urbana, IL: National Council of English.Google Scholar
- Beaton, A., Rogers, A., Gonzalez, E., Hanly, M., Kolstad, A., Rust, K., et al. (2011). The NAEP Primer. Washington, DC: National Center for Educational Statistics.Google Scholar
- Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300.Google Scholar
- Bergman, L. R., Magnusson, D., & El-Khouri, B. M. (2002). Studying individual development in an interindividual context. A person-oriented approach. Mahwah, New Jersey, London: Lawrence-Erlbaum Associates.Google Scholar
- Chen, E., Niemi, D., Wang, J., Wang, H., & Mirocha, J. (2007). Examining the generalizability of direct writing assessment tasks. CSE technical report 718. Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). https://doi.org/10.1037/e643812011-001.
- Common Core State Standards Initiative (CCSSI). (2017). Language progressive skills. Retrieved June 07, 2017, from http://www.corestandards.org/ELA-Literacy/L/language-progressive-skills/.
- Daiute, C. (1986). Physical and cognitive factors in revising: Insights from studies with computers. Research in the Teaching of English, 20(2), 141–159.Google Scholar
- Deane, P. (2011). Writing assessment and cognition. ETS Research Report Series, 1, i–60. https://doi.org/10.1002/j.2333-8504.2011.tb02250.x.CrossRefGoogle Scholar
- DeVoss, D. N., Eidman-Aadahl, E., & Hicks, T. (2010). Because digital writing matters: Improving student writing in online and multimedia environments. Hoboken: Wiley.Google Scholar
- Eccles, J., & Midgley, C. (1990). Changes in academic motivation and self-perception during early adolescence. In R. Montemayor (Ed.), Early adolescence as a time of transition (pp. 1–29). Beverly Hills, CA: Sage Publishing.Google Scholar
- Engeström, Y. (1987). Learning by expanding: An activity-theoretical approach to developmental research. Helsinki: Orienta-Konsultit Oy.Google Scholar
- Goldfine, R. (2001). Making word processing more effective in the composition classroom. Teaching English in the Two-Year College, 28(3), 307–315.Google Scholar
- Graham, S. (2018). A writer(s) within community model of writing. In C. Bazerman, A. Applebee, V. Berninger, D. Brandt, S. Graham, J. Jeffery, P. Matsuda, S. Murphy, D. Rowe, M. Schleppegrell, & K. Wilcox (Eds.), The lifespan development of writing (pp. 271–325). Urbana, IL: National Council of English. https://doi.org/10.1080/00461520.2018.1481406.CrossRefGoogle Scholar
- Horkay, N., Bennett, R. E., Allen, N., Kaplan, B., Yan, F. (2006). Does it matter if I take my writing test on computer? An empirical study of mode effects in NAEP. Journal of Technology, Learning, and Assessment, 5(2), 4–48.Google Scholar
- Johnson, E. G. (1992). The design of the National Assessment of Educational Progress. Journal of Educational Measurement, 29(2), 95–110. https://doi.org/10.1111/j.1745-3984.1992.tb00369.x.CrossRefGoogle Scholar
- KewalRamani, A., Zhang, J., Wang, X., Rathbun, A., Corcoran, L., Diliberti, M., et al. (2018). Student access to digital learning resources outside of the classroom (NCES 2017-098). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved April 6, 2018 from https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2017098.
- Leontyev, A. N. (1978). Activity, consciousness, and personality. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
- Leontyev, A. N. (1981). Problems of the development of mind. Moscow: Progress.Google Scholar
- Mo, Y., & Troia, G. (2017). Similarities and differences in constructs represented by U.S. States’ middle school writing tests and the 2007 National Assessment of Educational Progress writing assessment. Assessing Writing, 33(July 17), 48–67. https://doi.org/10.1016/j.asw.2017.06.001.CrossRefGoogle Scholar
- Murphy, S., & Smith, M. A. (2018). The faraway stick cannot kill the nearby snake. In C. Bazerman, A. Applebee, V. Berninger, D. Brandt, S. Graham, J. Jeffery, P. Matsuda, S. Murphy, D. Rowe, M. Schleppegrell, & K. Wilcox (Eds.), The lifespan development of writing (pp. 210–243). Urbana, IL: National Council of English.Google Scholar
- National Assessment Governing Board (NAGB). (2010). Writing framework for the 2011 National Assessment of Educational Progress. Washington, DC: U.S. Department of Education.Google Scholar
- National Center for Education Statistics (NCES). (2012). The Nation’s report card: Writing 2011. Washington, DC: Institute of Education Sciences, U.S. Department of Education.Google Scholar
- National Center for Education Statistics (NCES). (2014a). Lessons learned from the 2012 grade 4 writing computer-based assessment study. Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved February 28, 2019 from, http://nces.ed.gov/nationsreportcard/writing/lessons/default.aspx.
- National Center for Education Statistics (NCES). (2014b). NAEP 2012 pilot writing computer-based assessment study-grade 4: Technical memorandum: Institute of Education Sciences. Institute of Education Sciences, U.S. Department of Education: Washington, DC.Google Scholar
- National Center for Education Statistics (NCES). (n.d.). NAEP technical documentation: Calculation and use of replicate weights. Washington, D.C.: Institute of Education Sciences, U.S. Department of Education. Retrieved April 19, 2018 from https://nces.ed.gov/nationsreportcard/tdw/weighting/2000_2001/2000state_repweights.aspx.
- Olinghouse, N., Santangelo, T., & Wilson, J. (2012). Examining the validity of single-occasion, single-genre, holistically scored writing assessments. In E. V. Steendam, M. Tillema, G. Rijlaarsdam, & H. V. D. Bergh (Eds.), Measuring writing: Recent insights into theory, methodology and practices (pp. 55–82). New York: Guilford.Google Scholar
- Purcell, K., Buchanan, J., & Friedrich, L. (2013a). The impact of digital tools on student writing and how writing is taught in schools. Washington, DC: Pew Research Center’s Interent & American Life Project. Retrieved February 28, 2019 from http://pewinternet.org/Reports/2013/Teachers-technology-and-writing.
- Purcell, K., Heaps, A., Buchanan, J., & Friedrich, L. (2013b). How teachers are using technology at home and in their classrooms. Washington, DC: Pew Research Center’s Interent & American Life Project. Retrieved February 28, 2019 from http://pewinternet.org/Reports/2013/Teachers-and-technology.
- Rogers, A., Stoeckel, J., & Sikali, E. (2013). National Assessment of Educational Progress (NAEP) 2011 writing assessment restricted-use data files data companion. Washington, DC: National Center for Education Statistics.Google Scholar
- Russell, M., & Plati, T. (2002). Does it matter with what I write? Comparing performance on paper, computer and portable writing devices. Current Issues in Education, 5(4), 1–15.Google Scholar
- Sandene, B., Horkay, N., Bennett, R. E., Allen, N., Braswell, J., Kaplan, B., et al. (2005). Online assessment in mathematics and writing: Reports from the NAEP Technology-Based Assessment Project, Research and Development Series. NCES 2005-457. Washington, DC: Institute of Education Sciences, U.S. Department of Education.Google Scholar
- Vygotsky, L. S. (1978). Mind and society. Cambridge, MA: MIT Press.Google Scholar
- Vygotsky, L. S. (1981). The genesis of higher mental functions. In J. V. Wertsch (Ed.), The concept of activity in Soviet psychology (pp. 144–188). Armonk, NY: M.E. Sharpe.Google Scholar
- Warschauer, M. (2006). Laptops and literacy: Learning in the wireless classroom. New York: Teachers College Press.Google Scholar
- Way, W. D., Davis, L. L., & Strain-Seymour, E. (2008). The validity case for assessing direct writing by computer. A Pearson Assessments & Information White Paper.Google Scholar
- Wenglinsky, H. (2005). Using technology wisely: The keys to success in schools. New York, NY: Teachers College Press.Google Scholar
- Wertsch, J. V. (1991). Voices of the mind: A sociocultural approach to mediated action. Cambridge, MA: Harvard University Press.Google Scholar
- White, S., Kim, Y. Y., Chen, J., & Liu, F. (2015). Performance of fourth-grade students in the 2012 NAEP computer-based writing pilot assessment: Scores, text length, and use of editing tools. NCES 2015-119. Washington, DC: Institute of Education Sciences, US Department of Education.Google Scholar