Advertisement

Behavior Research Methods

, Volume 51, Issue 1, pp 14–27 | Cite as

The Tool for the Automatic Analysis of Cohesion 2.0: Integrating semantic similarity and text overlap

  • Scott A. CrossleyEmail author
  • Kristopher Kyle
  • Mihai Dascalu
Article

Abstract

This article introduces the second version of the Tool for the Automatic Analysis of Cohesion (TAACO 2.0). Like its predecessor, TAACO 2.0 is a freely available text analysis tool that works on the Windows, Mac, and Linux operating systems; is housed on a user’s hard drive; is easy to use; and allows for batch processing of text files. TAACO 2.0 includes all the original indices reported for TAACO 1.0, but it adds a number of new indices related to local and global cohesion at the semantic level, reported by latent semantic analysis, latent Dirichlet allocation, and word2vec. The tool also includes a source overlap feature, which calculates lexical and semantic overlap between a source and a response text (i.e., cohesion between the two texts based measures of text relatedness). In the first study in this article, we examined the effects that cohesion features, prompt, essay elaboration, and enhanced cohesion had on expert ratings of text coherence, finding that global semantic similarity as reported by word2vec was an important predictor of coherence ratings. A second study was conducted to examine the source and response indices. In this study we examined whether source overlap between the speaking samples found in the TOEFL-iBT integrated speaking tasks and the responses produced by test-takers was predictive of human ratings of speaking proficiency. The results indicated that the percentage of keywords found in both the source and response and the similarity between the source document and the response, as reported by word2vec, were significant predictors of speaking quality. Combined, these findings help validate the new indices reported for TAACO 2.0.

Keywords

Cohesion Coherence Natural language processing Essay quality Speaking proficiency 

References

  1. Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48.  https://doi.org/10.18637/jss.v067.i01 CrossRefGoogle Scholar
  2. Bestgen, Y., & Vincze, N. (2012). Checking and bootstrapping lexical norms by means of word similarity indexes. Behavior Research Methods, 44, 998–1006.CrossRefGoogle Scholar
  3. Bird, S., Klein, K., & Loper, E. (2009). Natural language processing with Python. Beijing, China: O’Reilly.Google Scholar
  4. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.Google Scholar
  5. Broadway, J. M., & Engle, R. W. (2010). Validating running memory span: Measurement of working memory capacity and links with fluid intelligence. Behavior Research Methods, 42, 563–570.CrossRefGoogle Scholar
  6. Chen, D., & Manning, C. (2014). A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 740–750). Stroudsburg, PA: Association for Computational Linguistics.Google Scholar
  7. Chiang, S. (2003). The importance of cohesive conditions to perceptions of writing quality at the early stages of foreign language learning. System, 31, 471–484.  https://doi.org/10.1016/j.system.2003.02.002 CrossRefGoogle Scholar
  8. Cree, G. S., & Armstrong, B. C. (2012). Computational models of semantic memory. In M. Spivey, K. McRae, & M. Joanisse (Eds.), The Cambridge handbook of psycholinguistics (pp. 259–282). New York, NY: Cambridge University Press.CrossRefGoogle Scholar
  9. Crossley, S. A., Clevinger, A., & Kim, Y. (2014). The role of lexical properties and cohesive devices in text integration and their effect on human ratings of speaking proficiency. Language Assessment Quarterly, 11, 250–270.CrossRefGoogle Scholar
  10. Crossley, S. A., Kyle, K., & McNamara, D. S. (2016). The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion. Behavior Research Methods, 48, 1227–1237.  https://doi.org/10.3758/s13428-015-0651-7 CrossRefGoogle Scholar
  11. Crossley, S. A., Liu, R., & McNamara, D. S. (2017a). Predicting math performance using natural language processing tools. In LAK ’17: Proceedings of the 7th International Learning Analytics and Knowledge Conference: Understanding, informing and improving learning with data (pp. 339–347). New York, NY: ACM Press.  https://doi.org/10.1145/3027385.3027399
  12. Crossley, S. A., & McNamara, D. S. (2010). Cohesion, coherence, and expert evaluations of writing proficiency. In S. Ohlsson & R. Catrambone (Eds.), Cognition in flux: Proceedings of the 32nd Annual Meeting of the Cognitive Science Society (pp. 984–989). Austin, TX: Cognitive Science Society.Google Scholar
  13. Crossley, S. A., & McNamara, D. S. (2011). Text coherence and judgments of essay quality: Models of quality and coherence. In L. Carlson, C. Hölscher, & T. F. Shipley (Eds.), Expanding the space of cognitive science: Proceedings of the 33rd Annual Conference of the Cognitive Science Society (pp. 1236–1241). Austin, TX: Cognitive Science Society.Google Scholar
  14. Crossley, S. A., & McNamara, D. S. (2012). Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35, 115–135.  https://doi.org/10.1111/j.1467-9817.2010.01449.x CrossRefGoogle Scholar
  15. Crossley, S. A., & McNamara, D. (2013). Applications of text analysis tools for spoken response grading. Language Learning & Technology, 17, 171–192.Google Scholar
  16. Crossley, S. A., & McNamara, D. S. (2016). Say more and be more coherent: How text elaboration and cohesion can increase writing quality. Grantee Submission, 7, 351–370.Google Scholar
  17. Crossley, S. A., Roscoe, R., & McNamara, D. S. (2013). Using automatic scoring models to detect changes in student writing in an intelligent tutoring system. In FLAIRS 2013—Proceedings of the 26th International Florida Artificial Intelligence Research Society Conference (pp. 208–213). Association for the Advancement of Artificial Intelligence.Google Scholar
  18. Crossley, S. A., Russell, D., Kyle, K., & Römer, U. (2017b). Applying natural language processing tools to a student academic writing corpus: How large are disciplinary differences across science and engineering fields? Journal of Writing Analytics, 1, 48–81.Google Scholar
  19. Crossley, S. A., Weston, J. L., Sullivan, S. T. M., & McNamara, D. S. (2011). The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication, 28, 282–311.  https://doi.org/10.1177/0741088311410188 CrossRefGoogle Scholar
  20. Crowhurst, M. (1987). Cohesion in argument and narration at three grade levels. Research in the Teaching of English, 21, 185–201.Google Scholar
  21. Cumming, A., Grant, L., Mulcahy-Ernt, P., & Powers, D. E. (2005a). A teacher-verification study of speaking and writing prototype tasks for a new TOEFL test (TOEFL Monograph No. MS-26). Princeton, NJ: Educational Testing Service.Google Scholar
  22. Cumming, A., Kantor, R., Baba, K., Eouanzoui, K., Erdosy, U., & Jamse, M. (2005b). Analysis of discourse features and verification of scoring levels for independent and integrated prototype written tasks for the new TOEFL® [ETS Research Report Series, 2005(1)]. Princeton, NJ: Educational Testing Service.Google Scholar
  23. Cumming, A., Kantor, R., Powers, D., Santos, T., & Taylor, C. (2000). TOEFL 2000 writing framework. In TOEFL-MS-18. Princeton, NJ: Educational Testing Service.Google Scholar
  24. Dascalu, M., McNamara, D. S., Trausan-Matu, S., & Allen, L. K. (2018). Cohesion network analysis of CSCL participation. Behavior Research Methods, 50, 604–619.  https://doi.org/10.3758/s13428-017-0888-4 CrossRefGoogle Scholar
  25. Davies, M. (2008). The corpus of contemporary American English. Provo, UT: Brigham Young University.Google Scholar
  26. Douglas, D. (1997). Testing speaking ability in academic contexts: Theoretical considerations. Princeton, NJ: Educational Testing Service.Google Scholar
  27. Evola, J., Mamer, E., & Lentz, B. (1980). Discrete point versus global scoring of cohesive devices. In J. W. Oller & K. Perkins (Eds.), Research in language testing (pp. 177–181). Rowley, MA: Newbury House.Google Scholar
  28. Faigley, L., & Witte, S. (1981). Analyzing Revision. College Composition and Communication, 32, 400–414.  https://doi.org/10.2307/356602 CrossRefGoogle Scholar
  29. Fitzgerald, J., & Spiegel, D. L. (1983). Enhancing children’s reading comprehension through instruction in narrative structure. Journal of Reading Behavior, 15, 1–17.  https://doi.org/10.1080/10862968309547480 CrossRefGoogle Scholar
  30. Gernsbacher, M. A. (1990). Language comprehension as structure building. Hillsdale, NJ: Erlbaum.CrossRefGoogle Scholar
  31. Givón, T. (1995). Coherence in the text and coherence in the mind. In M. A. Gernsbacher & T. Givón, Coherence in spontaneous text (pp. 59–115). Amsterdam, The Netherlands: Benjamins.CrossRefGoogle Scholar
  32. Golub, G. H., & Reinsch, C. (1970). Singular value decomposition and least squares solutions. Numerische Mathematik, 14, 403–420.CrossRefGoogle Scholar
  33. Graesser, A. C., McNamara, D. S., & Louwerse, M. M. (2003). What do readers need to learn in order to process coherence relations in narrative and expository text. In A. P. Sweet & C. E. Snow (Eds.), Rethinking reading comprehension (pp. 82–98). New York, NY: Guilford Press.Google Scholar
  34. Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36, 193–202.  https://doi.org/10.3758/BF03195564 CrossRefGoogle Scholar
  35. Guo, L., Crossley, S. A., & McNamara, D. S. (2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18, 218–238.  https://doi.org/10.1016/j.asw.2013.05.002 CrossRefGoogle Scholar
  36. Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English. London, UK: Longman.Google Scholar
  37. Haswell, R. H. (2000). Documenting improvement in college writing: A longitudinal approach. Written Communication, 17, 307–352.  https://doi.org/10.1177/0741088300017003001 CrossRefGoogle Scholar
  38. Hothorn, T., Bretz, F., & Westfall, P. (2008). Simultaneous inference in general parametric models. Biometrical Journal, 50, 346–363.CrossRefGoogle Scholar
  39. Jacobs, A. M., Schuster, S., Xue, S., & Lüdtke, J. (2017). What’s in the brain that ink may character . . . : A quantitative narrative analysis of Shakespeare’s 154 sonnets for use in neurocognitive poetics. Scientific Study of Literature, 7, 4–51.  https://doi.org/10.13140/RG.2.2.27126.40004 CrossRefGoogle Scholar
  40. Jafarpur, A. (1991). Cohesiveness as a basis for evaluating compositions. System, 19, 459–465.  https://doi.org/10.1016/0346-251X(91)90026-L CrossRefGoogle Scholar
  41. King, M. L., & Rentel, V. (1979). Toward a theory of early writing development. Research in the Teaching of English, 13, 243–253.Google Scholar
  42. Kintsch, W. (1995). How readers construct situation models for stories: the role of syntactic cues and causal inferences. In M. A. Gernsbacher & T. Givón, Coherence in spontaneous text (pp. 139–160). Amsterdam, The Netherlands: Benjamins.CrossRefGoogle Scholar
  43. Kotz, S., Balakrishnan, N., & Johnson, N. L. (2004). Continuous multivariate distributions: Vol. 1. Models and applications. Hoboken, NJ: Wiley.CrossRefGoogle Scholar
  44. Kuznetsova, A., Brockhoff, B., & Christensen, H. B. (2016). lmerTest: Tests in linear mixed effects models (R package version 2.0-32). Retrieved from https://CRAN.R-project.org/package=lmerTest
  45. Kyle, K., Crossley, S. A., & McNamara, D. S. (2016). Construct validity in TOEFL iBT speaking tasks: Insights from natural language processing. Language Testing, 33, 319–340.CrossRefGoogle Scholar
  46. Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to latent semantic analysis. Discourse Processes, 25, 259–284.  https://doi.org/10.1080/01638539809545028 CrossRefGoogle Scholar
  47. Landauer, T., McNamara, D. S., Dennis, S., & Kintsch, W. (2007). Latent semantic analysis: A road to meaning. Mahwah, NJ: Erlbaum.Google Scholar
  48. Levy, O., & Goldberg, Y. (2014). Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Vol. 2, pp. 302–308). Stroudsburg, PA: Association for Computational Linguistics.Google Scholar
  49. Liu, M., & Braine, G. (2005). Cohesive features in argumentative writing produced by Chinese undergraduates. System, 33, 623–636.  https://doi.org/10.1016/j.system.2005.02.002 CrossRefGoogle Scholar
  50. Loxterman, J. A., Beck, I. L., & McKeown, M. G. (1994). The effects of thinking aloud during reading on students’ comprehension of more or less coherent text. Reading Research Quarterly, 29, 353–367.  https://doi.org/10.2307/747784 CrossRefGoogle Scholar
  51. Mackey, A., Adams, R., Stafford, C., & Winke, P. (2010). Exploring the relationship between modified output and working memory capacity. Language Learning, 60, 501–533.CrossRefGoogle Scholar
  52. McCallum, A. K. (2002). Mallet: A machine learning for language toolkit (User’s manual). Retrieved from http://mallet.cs.umass.edu/
  53. McCulley, G. A. (1985). Writing quality, coherence, and cohesion. Research in the Teaching of English, 19, 269–282.Google Scholar
  54. McCutchen, D. (1986). Domain knowledge and linguistic knowledge in the development of writing ability. Journal of Memory and Language, 25, 431–444.  https://doi.org/10.1016/0749-596X(86)90036-7 CrossRefGoogle Scholar
  55. McCutchen, D., & Perfetti, C. A. (1982). Coherence and connectedness in the development of discourse production. Text—Interdisciplinary Journal for the Study of Discourse, 2, 113–140.  https://doi.org/10.1515/text.1.1982.2.1-3.113 CrossRefGoogle Scholar
  56. McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010). Linguistic features of writing quality. Written Communication, 27, 57–86.  https://doi.org/10.1177/0741088309351547 CrossRefGoogle Scholar
  57. McNamara, D. S., Crossley, S. A., & Roscoe, R. (2013). Natural language processing in an intelligent writing strategy tutoring system. Behavior Research Methods, 45, 499–515.  https://doi.org/10.3758/s13428-012-0258-1 CrossRefGoogle Scholar
  58. McNamara, D. S., & Kintsch, W. (1996). Learning from texts: Effects of prior knowledge and text coherence. Discourse Processes, 22, 247–288.  https://doi.org/10.1080/01638539609544975 CrossRefGoogle Scholar
  59. McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 14, 1–43.CrossRefGoogle Scholar
  60. Medimorec, S., Young, T. P., & Risko, E. F. (2017). Disfluency effects on lexical selection. Cognition, 158, 28–32.CrossRefGoogle Scholar
  61. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. ArXiv preprint. ArXiv:1310.4546Google Scholar
  62. Miller, G. A. (1995). WordNet: A lexical database for English. Communications of the ACM, 38, 39–41.CrossRefGoogle Scholar
  63. Nakagawa, S., & Schielzeth, H. (2013). A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4, 133–142.CrossRefGoogle Scholar
  64. Neuner, J. L. (1987). Cohesive ties and chains in good and poor freshman essays. Research in the Teaching of English, 21, 92–105.Google Scholar
  65. O’Reilly, T., & McNamara, D. S. (2007). The impact of science knowledge, reading skill, and reading strategy knowledge on more traditional “High-Stakes” measures of high school students’ science achievement. American Educational Research Journal, 44, 161–196.CrossRefGoogle Scholar
  66. Owen, S., Anil, R., Dunning, T., & Friedman, E. (2011). Mahout in action. Greenwich, CT, USA: Manning.Google Scholar
  67. Piotrkowicz, A., Dimitrova, V., Treasure-Jones, T., Smithies, A., Harkin, P., Kirby, J., & Roberts, T. (2017). Quantified self analytics tools for self-regulated learning with myPAL. In Proceedings of the 7th Workshop on Awareness and Reflection in Technology Enhanced Learning Co-located With the 12th European Conference on Technology Enhanced Learning (EC-TEL 2017). CEUR Workshop Proceedings. Retrieved from http://eprints.whiterose.ac.uk/121100/
  68. R Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/ Google Scholar
  69. Řehůřek, R., & Sojka, P. (2010). Software framework for topic modelling with large corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks (pp. 45–50).  https://doi.org/10.13140/2.1.2393.1847
  70. Sanders, T., & Maat, H. P. (2006). Cohesion and coherence: Linguistic approaches. Reading, 99, 440–466.Google Scholar
  71. Schillinger, D., McNamara, D., Crossley, S., Lyles, C., Moffet, H. H., Sarkar, U., . . . Karter, A. J. (2017). The next frontier in communication and the ECLIPPSE study: Bridging the linguistic divide in secure messaging. Journal of Diabetes Research, 2017, 1348242.  https://doi.org/10.1155/2017/1348242
  72. Skalicky, S., Crossley, S. A., McNamara, D. S., & Muldner, K. (2017). Identifying creativity during problem solving using linguistic features. Creativity Research Journal, 29, 343–353.CrossRefGoogle Scholar
  73. Wilson, J., Roscoe, R., & Ahmed, Y. (2017). Automated formative writing assessment using a levels of language framework. Assessing Writing, 34, 16–36.CrossRefGoogle Scholar
  74. Yde, P., & Spoelders, M. (1985). Text cohesion: An exploratory study with beginning writers. Applied Psycholinguistics, 6, 407–415.  https://doi.org/10.1017/S0142716400006330 CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  • Scott A. Crossley
    • 1
    Email author
  • Kristopher Kyle
    • 2
  • Mihai Dascalu
    • 3
  1. 1.Department of Applied Linguistics/ESLGeorgia State UniversityAtlantaUSA
  2. 2.Department of Second Language StudiesUniversity of Hawai’i at ManoaHonoluluUSA
  3. 3.Department of Computer SciencesPolitehnica University of BucharestBucharestRomania

Personalised recommendations