Advertisement

Identifying and Comparing Writing Process Patterns Using Keystroke Logs

  • Mo ZhangEmail author
  • Mengxiao Zhu
  • Paul Deane
  • Hongwen Guo
Conference paper
Part of the Springer Proceedings in Mathematics & Statistics book series (PROMS, volume 265)

Abstract

There is a growing literature on the use of process data in digitally delivered assessments. In this study, we analyzed students’ essay writing processes using keystroke logs. Using four basic writing performance indicators, writers were grouped into four clusters, representing groups from fluent to struggling. The clusters differed significantly on the mean essay score, mean total time spent on task, and mean total number of words in the final submissions. Two of the four clusters were significantly different on the aforementioned three dimensions but not on typing skill. The higher scoring group even showed signs of less fluency than the lower scoring group, suggesting that task engagement and writing efforts might play an important role in generating better quality text. The four identified clusters further showed distinct sequential patterns over the course of the writing session on three process characteristics and, as well, differed on their editing behaviors during the writing process.

Keywords

Writing process Keystroke logs CBAL Sequential pattern Editing behavior 

References

  1. Alves, R. A., Castro, S. L., de Sousa, L., & Stromqvist, S. (2007). Influence of typing skill on pauseexecution cycles in written composition. In M. Torrance, L. van Waes, & D. Galbraith (Eds.), Writing and Cognition: Research and Applications (pp. 55–65). Amsterdam: Elsevier.Google Scholar
  2. Bennett, R. E., Deane, P., & van Rijn, P. W. (2016). From cognitive domain theory to assessment practice. Educational Psychologist, 51, 82–107.CrossRefGoogle Scholar
  3. Cleveland, William S. (1979). Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association., 74, 829–836. https://pdfs.semanticscholar.org/414e/5d1f5a75e2327d99b5bbb93f2e4e241c5acc.pdf.MathSciNetCrossRefGoogle Scholar
  4. Cooley, W. W., & Lohnes, P. R. (1971). Multivariate data analysis. John Wiley and Sons.Google Scholar
  5. Deane, P., Feng, G., Zhang, M., Hao, J., Bergner, Y., Flor, M., Wagner, M., Lederer. N.: Generating scores and feedback for writing assessment and instruction using electronic process logs. US Patent and Trademark Office. Application No. 14/937,164 (2016).Google Scholar
  6. Ercikan, K., & Pellegrino, J. W. (2017). Validation of score meaning for the next generation of assessments: The use of response processes. Taylor & Francis.Google Scholar
  7. Guo, H., Deane, P., van Rijn, P., Zhang, M., & Bennett, R. (2018). Exploring the heavy-tailed key-stroke data in writing processes. Journal of Educational Measurement, 194–216,Google Scholar
  8. Murtagh, F., Legendre, P.: Ward’s hierarchical clustering method: Clustering criterion and agglomerative algorithm. Accessed in October 2018: http://arxiv.org/abs/1111.6285.pdf (2011)
  9. Murtagh, F., & Legendre, P. (2014). Ward’s hierarchical agglomerative clustering method: Which algorithm implement Ward’s criterion. Journal of Classification, 31, 274–295.MathSciNetCrossRefGoogle Scholar
  10. Rencher, A. C. (1992). Interpretation of canonical discriminant functions, canonical variates, and principal components. The American Statistician, 46, 217–225.Google Scholar
  11. Sinharay, S., Zhang, M., Deane, P.: Application of data mining for predicting essay scores from writing process and product features. Applied Measurement in Education (2019). https://www.tandfonline.com/doi/full/10.1080/08957347.2019.1577245.
  12. Stevenson, M., Schoonen, R., & de Glopper, K. (2006). Revising in two languages: A multi-dimensional comparison of online writing revisions in L1 and FL. Journal of Second Language Writing, 201–233,Google Scholar
  13. van Rijn, P., Yan-Koo, Y.: Statistical results from the 2013 CBAL English Language Arts multistate study: Parallel forms for argumentative writing. RM-16-15. Princeton, NJ: Educational Testing Service (2016).Google Scholar
  14. van Rijn, P., Chen, J., Yan-Koo, Y.: Statistical results from the 2013 CBALTM English Language Arts multistate study: Parallel forms for policy recommendation writing. RR-16-01. Princeton, NJ: Educational Testing Service (2016).Google Scholar
  15. Ward, J. H, Jr. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58, 236–244.MathSciNetCrossRefGoogle Scholar
  16. Zhang, M., Bennett, R., Deane, P., & van Rijn, P. (2019). Are there gender differences in how students write their essays? An analysis of writing processes. Educational Measurement: Issues and Practice, Online First.Google Scholar
  17. Zhang M., Deane, P.: Process features in writing: Internal structure and incremental value over product features. RR-15-27. Princeton, NJ: Educational Testing Service (2015).CrossRefGoogle Scholar
  18. Zhang, M., Feng, G., Deane, P., H, Guo.: Investigating an approach to evaluating keyboarding fluency. To be submitted for publication (2018).Google Scholar
  19. Zhang, M., Hao, J., Li, C., & Deane, P. (2016). Classification of writing patterns using keystroke logs. In L. A. van der Ark, D. M. Bolt, W.-C. Wang, J. A. Douglas, & M. Wiberg (Eds.), Quantitative Psychology Research. New York: Springer.Google Scholar
  20. Zhang, M., Hao, J., Li, C., Deane, P.: Defining personalized writing burst measures of translation using keystroke logs. In proceedings of the 2018 Educational Data Mining Conference, 549 - 552 (2018).Google Scholar
  21. Zhu, M., Zhang, M., Deane, P.: Analysis of keystroke sequences in writing logs. RR-xx-xx. Princeton, NJ: Educational Testing Service (2019). https://onlinelibrary.wiley.com/doi/10.1002/ets2.12247.

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Mo Zhang
    • 1
    Email author
  • Mengxiao Zhu
    • 1
  • Paul Deane
    • 1
  • Hongwen Guo
    • 1
  1. 1.Educational Testing ServicePrincetonUSA

Personalised recommendations