Behavior Research Methods

, Volume 51, Issue 1, pp 342–360 | Cite as

Beyond frequency counts: Novel conceptual recurrence analysis metrics to index semantic coordination in team communications

  • Michael T. TolstonEmail author
  • Michael A. Riley
  • Vincent Mancuso
  • Victor Finomore
  • Gregory J. Funke


Semantic alignment is a key process underlying interpersonal and team communication. However, semantic similarity is difficult to quantify, and statistical approaches designed to measure it often rely on methods that make the identification of the relative importance of key words difficult. This study outlines how conceptual recurrence analysis (CRA) can address these issues and can be used to detect conceptual structure in interpersonal communication. We developed several novel CRA metrics to analyze communication data reported previously by Mancuso, Finomore, Rahill, Blair, and Funke (Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58, 405–409, 2014), gathered from teams who worked cooperatively on a logic puzzle under different cognitive biasing contexts. CRA, like other measures of semantic coordination, relies on parameters whose values affect estimates of semantic alignment. We evaluated how the dimensionality of semantic spaces affects metrics quantifying the conceptual similarity of communicative exchanges, and whether metrics calculated from top-down, a priori semantic spaces or bottom-up semantic spaces empirically derived from each data set were more sensitive to biasing context. We found that the novel CRA measures were sensitive to manipulations of cognitive bias, and that higher-dimensional, bottom-up semantic spaces generally yielded more sensitivity to the experimental manipulations, though when the communication was evaluated with respect to specific key concepts, lower-dimensional, top-down spaces performed nearly as well. We conclude that CRA is sensitive to experimental manipulations in ways consistent with prior findings and that it presents a customizable framework for testing predictions about interpersonal communication patterns and other linguistic exchanges.


Conceptual recurrence analysis Text analysis Semantic similarity Semantic coordination Team communication Interpersonal communication 


  1. Angus, D., Smith, A., & Wiles, J. (2012b). Conceptual recurrence plots: Revealing patterns in human discourse. IEEE Transactions on Visualization and Computer Graphics, 18, 988–997. CrossRefGoogle Scholar
  2. Angus, D., Smith, A. E., & Wiles, J. (2012a). Human communication as coupled time series: Quantifying multi-participant recurrence. IEEE Transactions on Audio, Speech, and Language Processing, 20, 1795–1807. CrossRefGoogle Scholar
  3. Angus, D., Watson, B., Smith, A., Gallois, C., & Wiles, J. (2012). Visualising conversation structure across time: Insights into effective doctor-patient consultations. PLoS ONE, 7, e38014. CrossRefGoogle Scholar
  4. Atay, C., Conway, E. R., Angus, D., Wiles, J., Baker, R., & Chenery, H. J. (2015). An automated approach to examining conversational dynamics between people with dementia and their carers. PLoS ONE, 10, e0144327. CrossRefGoogle Scholar
  5. Babcock, M. J., Ta, V. P., & Ickes, W. (2014). Latent semantic similarity and language style matching in initial dyadic interactions. Journal of Language and Social Psychology, 33, 78–88. CrossRefGoogle Scholar
  6. Broniatowski, D. A., & Magee, C. L. (2012). Studying group behaviors: A tutorial on text and network analysis methods. IEEE Signal Processing Magazine, 29, 22–32. CrossRefGoogle Scholar
  7. Bunnin, N., & Yu, J. (2004), The Blackwell dictionary of Western philosophy. Malden: Blackwell.CrossRefGoogle Scholar
  8. Cabana, Á., Valle-Lisboa, J. C., Elvevåg, B., & Mizraji, E. (2011). Detecting order-disorder transitions in discourse: Implications for schizophrenia. Schizophrenia Research, 131, 157–164. CrossRefGoogle Scholar
  9. Carletta, J., Isard, S., Doherty-Sneddon, G., Isard, A., Kowtko, J. C., & Anderson, A. H. (1997). The reliability of a dialogue structure coding scheme. Computational Linguistics, 23, 13–31.Google Scholar
  10. Clark, H. H. (1996). Using language. Cambridge: Cambridge University Press. CrossRefGoogle Scholar
  11. Coco, M. I., & Dale, R. (2014). Cross-recurrence quantification analysis of categorical and continuous time series: An R package. Frontiers in Quantitative Psychology and Measurement, 5, 510.Google Scholar
  12. Cooke, N. J., Gorman, J. C., Myers, C. W., & Duran, J. L. (2013). Interactive team cognition. Cognitive Science, 37, pp. 255–285. CrossRefGoogle Scholar
  13. Cooke, N. J., Salas, E., Kiekel, P. A., & Bell, B. (2004). Advances in measuring team cognition. In E. Salas & S. M. Fiore (Eds.), Team cognition: Understanding the factors that drive process and performance (pp. 83–106). Washington, DC: American Psychological Association. CrossRefGoogle Scholar
  14. Dale, R., Duran, N. D., & Coco, M. (2018). Dynamic natural language processing with recurrence quantification analysis. arXiv Preprint arXiv:1803.07136.Google Scholar
  15. Dale, R., & Spivey, M. J. (2005). Categorical recurrence analysis of child language. In B. G. Bara, L. Barsalou & M. Bucciarelli (Eds.), Proceedings of the 27th Annual Meeting of the Cognitive Science Society (pp. 530–535). Mahwah: Lawrence Erlbaum.Google Scholar
  16. Dale, R., & Spivey, M. J. (2006). Unraveling the dyad: Using recurrence analysis to explore patterns of syntactic coordination between children and caregivers in conversation. Language Learning, 56, 391–430. CrossRefGoogle Scholar
  17. Entin, E. E., & Entin, E. B. (2001). Measures for evaluation of team processes and performance in experiments and exercises. Paper presented at the 6th International Command and Control Research and Technology Symposium, Annapolis. Retrieved from Google Scholar
  18. Finomore, V., Sitz, A., Blair, E., Rahill, K., Champion, M., Funke, G., … Knott, B. (2013). Effects of cyber disruption in a distributed team decision making task. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 57, pp. 394–398).
  19. Foltz, P. W. (1996). Latent semantic analysis for text-based research. Behavior Research Methods, Instruments, & Computers, 28, 197–202. CrossRefGoogle Scholar
  20. Foltz, P. W., Laham, D., & Landauer, T. K. (1999). Automated essay scoring: Applications to educational technology. Paper presented at the ED-MEDIA 99 World Conference on Educational Multimedia, Hypermedia & Telecommunications, Seattle. Retrieved from Scholar
  21. Foltz, P. W., & Martin, M. J. (2009). Automated communication analysis of teams. In E. Salas, G. F. Goodwin, & C. S. Burke (Eds.), Team effectiveness in complex organizations: Cross-disciplinary perspectives and approaches (pp. 411–432). New York: Taylor & Francis.Google Scholar
  22. Fusaroli, R., Bahrami, B., Olsen, K., Roepstorff, A., Rees, G., Frith, C., & Tylén, K. (2012). Coming to terms: Quantifying the benefits of linguistic coordination. Psychological Science, 23, 937–939. CrossRefGoogle Scholar
  23. Fusaroli, R., & Tylén, K. (2016). Investigating conversational dynamics: Interactive alignment, interpersonal synergy, and collective task performance. Cognitive Science, 40, 145–171. CrossRefGoogle Scholar
  24. Garrod, S., & Pickering, M. J. (2004). Why is conversation so easy? Trends in Cognitive Sciences, 8, 8–11. CrossRefGoogle Scholar
  25. Gefen, D., Endicott, J. E., Fresneda, J. E., Miller, J., & Larsen, K. R. (2017). A guide to text analysis with latent semantic analysis in R with annotated code: Studying online reviews and the stack exchange community. Communications of the Association for Information Systems, 41, 450–496. CrossRefGoogle Scholar
  26. Gorman, J. C., Foltz, P. W., Kiekel, P. A., Martin, M. J., & Cooke, N. J. (2003). Evaluation of Latent Semantic Analysis-based measures of team communications content. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 47, 424–428. CrossRefGoogle Scholar
  27. Gorman, J. C., Martin, M. J., Dunbar, T. A., Stevens, R. H., Galloway, T., Amazeen, P. G., & Likens, A. D. (2016). Cross-level effects between neurophysiology and communication during team training. Human Factors: The Journal of the Human Factors and Ergonomics Society, 58(1), 181–199.
  28. Healey, P. G., Purver, M., & Howes, C. (2014). Divergence in dialogue. PLoS ONE, 9, e98598. CrossRefGoogle Scholar
  29. Hilbert, M. (2012). Toward a synthesis of cognitive biases: How noisy information processing can bias human decision making. Psychological Bulletin, 138, 211. CrossRefGoogle Scholar
  30. Hodges, B. H., & Fowler, C. A. (2015). Fields, waves, and particles: Finding common ground in understanding language as a public activity. Ecological Psychology, 27, 175–189. CrossRefGoogle Scholar
  31. Iliev, R., Dehghani, M., & Sagi, E. (2014). Automated text analysis in psychology: Methods, applications, and future developments. Language and Cognition, 7, 1–26. Google Scholar
  32. Indulska, M., Hovorka, D. S., & Recker, J. (2012). Quantitative approaches to content analysis: Identifying conceptual drift across publication outlets. European Journal of Information Systems, 21, 49–69. CrossRefGoogle Scholar
  33. Kello, C. T., Brown, G. D. A., Ferrer-i-Cancho, R., Holden, J. G., Linkenkaer-Hansen, K., Rhodes, T., & Van Orden, G. C. (2010). Scaling laws in cognitive sciences. Trends in Cognitive Sciences, 14, 223–232.
  34. Luo, H., Liu, Z., Luan, H., & Sun, M. (2015). Online learning of interpretable word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (pp. 1687–1692). 10.18653/v1/d15-1196Google Scholar
  35. Mancuso, V. F., Finomore, V. S., Rahill, K. M., Blair, E. A., & Funke, G. J. (2014). Effects of cognitive biases on distributed team decision making. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58, 405–409. CrossRefGoogle Scholar
  36. Marwan, N., Romano, M.C., Thiel, M. & Kurths, J. (2007). Recurrence plots for the analysis of complex systems. Physics Reports, 438(5), 237–329.
  37. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, & K. Q. Weinberger (Eds.), Proceedings of the 26th International Conference on Neural Information Processing Systems (Vol. 2, pp. 3111–3119). New York: Curran Associates.Google Scholar
  38. Mills, G. J. (2014). Dialogue in joint activity: Complementarity, convergence and conventionalization. New Ideas in Psychology, 32, 158–173. CrossRefGoogle Scholar
  39. Mota, N. B., Vasconcelos, N. A., Lemos, N., Pieretti, A. C., Kinouchi, O., Cecchi, G. A., . . . Ribeiro, S. (2012). Speech graphs provide a quantitative measure of thought disorder in psychosis. PLoS ONE, 7, e34928. CrossRefGoogle Scholar
  40. Orsucci, F., Giuliani, A., Webber, C., Zbilut, J., Fonagy, P., & Mazza, M. (2006). Combinatorics and synchronization in natural semiotics. Physica A, 361, 665–676. CrossRefGoogle Scholar
  41. Paice, C. D. (1996). Method for evaluation of stemming algorithms based on error counting. Journal of the American Society for Information Science, 47, 632–649.<632::aid-asi8>;2-u CrossRefGoogle Scholar
  42. Park, S., Bak, J., & Oh, A. (2017). Rotated word vector representations and their interpretability. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, (pp. 401–411). 10.18653/v1/d17-1041Google Scholar
  43. Paxton, A., Dale, R., & Richardson, D. C. (2016). Social coordination of verbal and nonverbal behaviours. In P. Passos, K. Davids, & J. Y. Chow (Eds.), Interpersonal coordination and performance in social systems. New York: Routledge.Google Scholar
  44. Pickering, M. J., & Garrod, S. (2004). Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences, 27, 169–190. Google Scholar
  45. Quesada, J. (2007). Creating your own LSA spaces. In T. K. Landauer, D. S. McNamara, S. Dennis, & W. Kintsch (Eds.), Handbook of latent semantic analysis (pp. 71–85). Mahwah: Erlbaum. Google Scholar
  46. Rehurek, R., & Sojka, P. (2010). Software framework for topic modelling with large corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks (pp. 46–50).Google Scholar
  47. Ruddy, M., & Nissen, M. (2008). New software platform capabilities and experimentation campaign for ELICIT. Paper presented at the 13th International Command and Control Research and Technology Symposium, Seattle. Retrieved from Google Scholar
  48. Runge, J., Petoukhov, V., Donges, J. F., Hlinka, J., Jajcay, N., Vejmelka, M., … Kurths, J. (2015). Identifying causal gateways and mediators in complex spatio-temporal systems. Nature Communications, 6, 8502.
  49. Salton, G. (1989). Automatic text processing: The transformation, analysis, and retrieval of information by computer. Boston: Addison-Wesley Longman.Google Scholar
  50. Schippers, M. C., Edmondson, A. C., & West, M. A. (2014). Team reflexivity as an antidote to team information-processing failures. Small Group Research, 45, 731–769. CrossRefGoogle Scholar
  51. Shockley, K., Sabadini, L., & Fowler, C. A. (2004). Imitation in shadowing words. Perception & Psychophysics, 66, 422–429. CrossRefGoogle Scholar
  52. Shockley, K., Santana, M.-V., & Fowler, C. A. (2003). Mutual interpersonal postural constraints are involved in cooperative conversation. Journal of Experimental Psychology: Human Perception and Performance, 29, 326–332. Google Scholar
  53. Smalheiser, N. R., & Bonifield, G. (2018). Unsupervised low-dimensional vector representations for words, phrases and text that are transparent, scalable, and produce similarity metrics that are complementary to neural embeddings. arXiv preprint. arXiv:1801.01884Google Scholar
  54. Smith, A. E. (2000). Machine mapping of document collections: The leximancer system. Paper presented at the 5th Australasian Document Computing Symposium, Sunshine Coast. Retrieved from Google Scholar
  55. Smith, A. E., & Humphreys, M. S. (2006). Evaluation of unsupervised semantic mapping of natural language with Leximancer concept mapping. Behavior Research Methods, 38, 262–279. CrossRefGoogle Scholar
  56. Spevack, S. C., Falandays, J. B., Batzloff, B., & Spivey, M. J. (2018). Interactivity of language. Language and Linguistics Compass, e12282.
  57. Stasser, G., & Titus, W. (1985). Pooling of unshared information in group decision making: Biased information sampling during discussion. Journal of Personality and Social Psychology, 48(6), 1467–1478.
  58. Tolston, M., Strang, A. J., Funke, G. J., Miller, B., Brown, R., & Menke, L. (2017). Identifying physio-behavioral drivers of performance using multivariate causal networks. Paper presented at the 19th International Symposium on Aviation Psychology, Dayton.Google Scholar
  59. Tolston, M. T., Finomore, V., Funke, G. J., Mancuso, V., Brown, R., Menke, L, & Riley, M. A. (2016). Effects of biasing information on the conceptual structure of team communications. In K. S. Hale & K. M. Stanney (Eds.), Advances in neuroergonomics and cognitive engineering (pp. 433–445). Cham: Springer.Google Scholar
  60. Turney, P. D., & Pantel, P. (2010). From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37, 141–188. CrossRefGoogle Scholar
  61. Tylén, K., Weed, E., Wallentin, M., Roepstorff, A., & Frith, C. D. (2010). Language as a tool for interacting minds. Mind & Language, 25, 3–29. CrossRefGoogle Scholar
  62. Volpe, C. E., Cannon-Bowers, J. A., Salas, E., & Spector, P. E. (1996). The impact of cross-training on team functioning: An empirical investigation. Human Factors, 38, 87–100. CrossRefGoogle Scholar
  63. Watson, B. M., Angus, D., Gore, L., & Farmer, J. (2015). Communication in open disclosure conversations about adverse events in hospitals. Language & Communication, 41, 57–70. CrossRefGoogle Scholar
  64. Weber, R. P. (1990). Basic content analysis. Newbury Park: Sage.CrossRefGoogle Scholar
  65. Wildman, J. L., Salas, E., & Scott, C. P. (2013). Measuring cognition in teams: A cross-domain review. Human Factors, 56, 911–941. CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. (This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply) 2018

Authors and Affiliations

  • Michael T. Tolston
    • 1
    Email author
  • Michael A. Riley
    • 2
  • Vincent Mancuso
    • 3
  • Victor Finomore
    • 4
  • Gregory J. Funke
    • 4
  1. 1.Ball Aerospace and Technologies CorporationDaytonUSA
  2. 2.Center for Cognition, Action, & PerceptionUniversity of CincinnatiCincinnatiUSA
  3. 3.MIT Lincoln LaboratoryLexingtonUSA
  4. 4.Air Force Research LaboratoryWright Patterson Air Force BaseWright-Patterson AFBUSA

Personalised recommendations