Analyzing Collaborative Interactions Across Domains and Settings: An Adaptable Rating Scheme

  • Nikol Rummel
  • Anne Deiglmayr
  • Hans Spada
  • George Kahrimanis
  • Nikolaos Avouris
Part of the Computer-Supported Collaborative Learning Series book series (CULS, volume 12)


In this chapter we report on the development of a rating scheme for the analysis of collaborative process data, and on its implementation in diverse CSCL settings. The rating scheme is composed of nine dimensions measuring ­different aspects of collaboration quality: sustaining mutual understanding, dialogue ­management, information pooling, reaching consensus, task division, time management, technical coordination, reciprocal interaction, and individual task orientation. It can be applied to recordings (video, audio, screen recordings, or log data) of student interaction and does not necessarily require transcripts or written records. While the rating scheme was originally developed in the context of a specific CSCL setting (video-based interdisciplinary problem-solving in the medical domain; Meier et al. (2007)), we demonstrate in our chapter that it can successfully be adapted to other CSCL settings. First, we introduce the initial rating scheme and its dimensions. Next we describe the process of adapting it to data from a very different CSCL setting (chat-based interaction in computer science classes). We briefly report on a study that used the ratings of collaboration quality as basis for adaptive feedback to students on how to improve their collaboration. Finally, we describe how we have integrated our rating scheme with ActivityLens (Avouris et al. 2007), a software tool which allows for a combined analysis of multiple sources of data (e.g., logfiles, audio and video recordings). Several tool modifications were made to permit analysis of collaborative process data from yet another CSCL study in which high-school students collaborated face-to-face on solving algebra problems with support from an intelligent tutoring system. We conclude our chapter with a discussion of practical implications for practitioners who may wish to adapt and apply our rating scheme.


Rating Scheme Collaborative Process Collaboration Process Successful Collaboration Adaptive Feedback 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Anderson, A. H., O’Malley, C., Doherty-Sneddon, G., Langton, S., Newlands, A., Mullin, J., et al. (1997). The impact of VMC on collaborative problem solving: An analysis of task performance, communicative process, and user satisfaction. In K. E. Finn, A. J. Sellen, & S. B. Wilbur (Eds.), Video-mediated communication (pp. 133–156). Mahwah: Lawrence Erlbaum Associates.Google Scholar
  2. Angiolillo, J. S., Blanchard, H. E., Israelski, E. W., & Mané, A. (1997). Technology constraints of videomediated communication. In K. E. Finn, A. J. Sellen, & S. B. Wilbur (Eds.), Video-mediated communication (pp. 51–74). Mahwah: Lawrence Erlbaum Associates.Google Scholar
  3. Avouris, N., Fiotakis, G., Kahrimanis, G., Margaritis, M., & Komis, V. (2007). Beyond logging of fingertip actions: Analysis of collaborative learning using multiple sources of data. Journal of Interactive Learning Research JILR, 18(2), 231–250.Google Scholar
  4. Avouris, N., Margaritis, M., & Komis, V. (2004). Modelling interaction during small-group synchronous problem solving activities: The Synergo approach. In 2nd International Workshop on Designing Computational Models of Collaborative Learning Interaction, ITS 2004, Brazil.Google Scholar
  5. Barron, B. (2000). Achieving coordination in collaborative problem-solving groups. Journal of the Learning Sciences, 9, 403–436.CrossRefGoogle Scholar
  6. Clark, H. H. (1996). Using language. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  7. Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 127–148). Washington: American Psychological Association.CrossRefGoogle Scholar
  8. Deutsch, M. (2003). Cooperation and conflict: A personal perspective on the history of the social psychological study of conflict resolution. In M. A. West, D. Tjosvold, & K. G. Smith (Eds.), International handbook of organizational teamwork and cooperative working (pp. 9–43). Chichester: Wiley.Google Scholar
  9. Dillenbourg, P. (1999). Introduction: What do you mean by “collaborative learning”? In P. Dillenbourg (Ed.), Collaborative learning: Cognitive and computational approaches (pp. 1–19). Amsterdam: Pergamon.Google Scholar
  10. Diziol, D., Rummel, N., Spada, H., & McLaren, B. (2007). Promoting learning in mathematics: Script support for collaborative problem solving with the Cognitive Tutor Algebra. In Chinn, C. A., Erkens, G., & Puntambekar, S. (Eds.), Mice, minds, and society. Proceedings of the Computer Supported Collaborative Learning (CSCL) Conference 2007 I (Vol. 8, pp. 39–41). International Society of the Learning Sciences, Inc.Google Scholar
  11. Erkens, G., Jaspers, J., Prangsma, M., & Kanselaar, G. (2005). Coordination processes in computer supported collaborative writing. Computers in Human Behavior, 21, 463–486.CrossRefGoogle Scholar
  12. Fischer, F., & Mandl, H. (2003). Being there or being where? Videoconferencing and cooperative learning. In H. van Oostendorp (Ed.), Cognition in a digital world (pp. 205–223). Mahwah: Lawrence Erlbaum Associates.Google Scholar
  13. Heckhausen, H. (1989). Motivation und Handeln. [Motivation and behavior]. Berlin, Heidelberg, New York: Springer.Google Scholar
  14. Hermann, F., Rummel, N., & Spada, H. (2001). Solving the case together: The challenge of net-based interdisciplinary collaboration. In P. Dillenbourg, A. Eurelings, & K. Hakkarainen (Eds.), Proceedings of the first European conference on Computer-Supported Collaborative Learning (pp. 293–300). Maastricht: McLuhan Institute.Google Scholar
  15. Hinsz, V. B., Tindale, R. S., & Vollrath, D. A. (1997). The emerging conceptualization of groups as information processors. Psychological Bulletin, 121(1), 43–64.CrossRefGoogle Scholar
  16. Janis, I. L. (1982). Groupthink. Boston: Houghton Mifflin.Google Scholar
  17. Johnson, D. W., & Johnson, R. T. (2003). Training for cooperative group work. In M. A. West, D. Tjosvold, & K. G. Smith (Eds.), International handbook of organizational teamwork and cooperative working (pp. 167–183). Chichester: Wiley.Google Scholar
  18. Jucks, R., Bromme, R., & Runde, A. (2003). Audience Design von Experten in der netzgestützten Kommunikation: Die Rolle von Heuristiken über das geteilte Vorwissen. [Audience design of experts in net-based communication: The role of heuristics about shared knowledge]. Zeitschrift für Psychologie, 211(2), 60–74.CrossRefGoogle Scholar
  19. Kerlinger, F. N., & Lee, H. B. (2000). Foundations of behavioral research. Fort Worth: Harcourt College.Google Scholar
  20. Kneser, C., & Ploetzner, R. (2001). Collaboration on the basis of complementary domain knowledge: Observed dialogue structures and their relation to learning success. Learning and Instruction, 11(1), 53–83.CrossRefGoogle Scholar
  21. Larson, J. R., & Christensen, C. (1993). Groups as problem-solving units: Toward a new meaning of social cognition. The British Journal of Social Psychology, 32, 5–30.CrossRefGoogle Scholar
  22. Mayring, P. (2003). Qualitative Inhaltsanalyse. Grundlagen und Techniken [Qualitative content analysis. Foundations and techniques. Weinheim: Beltz.Google Scholar
  23. Malone, T. W., & Crowston, K. (1990). What is coordination theory and how can it help design cooperative work systems? In Proceedings of the conference on computer-supported cooperative work, Los Angeles (pp. 357–370). New York: ACM.Google Scholar
  24. Malone, T. W., & Crowston, K. (1994). The interdisciplinary study of coordination. ACM Computing Surveys, 26(1), 87–119.CrossRefGoogle Scholar
  25. Meier, A., Spada, H., & Rummel, N. (2007). A rating scheme for assessing the quality of computer-supported collaboration processes. International Journal of Computer-Supported Collaborative Learning, 2(1), 63–86.CrossRefGoogle Scholar
  26. Meier, A., Voyiatzaki, E., Kahrimanis, G., Rummel, N., Spada, H., & Avouris, N. (2008). Teaching students how to improve their collaboration: Assessing collaboration quality and providing adaptive feedback in a CSCL setting. Paper presented as part of the symposium by Rummel, N., & Weinberger, A. New challenges in CSCL: Towards adaptive script support. Proceedings of the 2008 International Conference of the Learning Sciences, Utrecht, The Netherlands.Google Scholar
  27. Moreland, R. L., & Myaskovsky, L. (2000). Exploring the performance benefits of group training: Transactive memory or improved communication? Organizational Behavior and Human Decision Processes, 82(1), 117–133.CrossRefGoogle Scholar
  28. Nickerson, R. S. (1999). How we know—and sometimes misjudge—what others know: Imputing one’s own knowledge to others. Psychological Bulletin, 125(6), 737–759.CrossRefGoogle Scholar
  29. O’Conaill, B., & Whittaker, S. (1997). Characterizing, predicting, and measuring video-mediated communication: A conversational approach. In K. E. Finn, A. J. Sellen, & S. B. Wilbur (Eds.), Videomediated communication (pp. 107–132). Mahwah: Lawrence Erlbaum Associates.Google Scholar
  30. Postmes, T., Spears, R., & Cihangir, S. (2001). Quality of decision making and group norms. Journal of Personality and Social Psychology, 80(6), 918–930.CrossRefGoogle Scholar
  31. Rummel, N., Diziol, D., & Spada, H. (2010). Scripted collaborative learning with the Cognitive Tutor Algebra: An experimental classroom study. Manuscript under revision.Google Scholar
  32. Rummel, N., Hauser, S., & Spada, H. (2007). How does net-based interdisciplinary collaboration change with growing domain expertise? In Chinn, C. A., Erkens, G., & Puntambekar, S. (Eds.), Mice, minds, and sociecty. Proceedings of the Computer Supported Collaborative Learning (CSCL) Conference 2007 (pp. 611–620). International Society of the Learning Sciences, Inc. ISSN 1819-0146.Google Scholar
  33. Rummel, N., & Spada, H. (2005a). Instructional support for collaboration in desktop videoconferencing settings: How it can be achieved and assessed. In R. Bromme, F. W. Hesse, & H. Spada (Eds.), Barriers and biases in computer-mediated knowledge communication—and how they may be overcome (pp. 59–88). Berlin, Heidelberg, New York: Springer.CrossRefGoogle Scholar
  34. Rummel, N., & Spada, H. (2005b). Learning to collaborate: An instructional approach to ­promoting collaborative problem-solving in computer-mediated settings. Journal of the Learning Sciences, 14(2), 201–241.CrossRefGoogle Scholar
  35. Rummel, N., Spada, H., & Hauser, S. (2009). Learning to collaborate from being scripted or from observing a model. International Journal of Computer-Supported Collaborative Learning, 4(1), 69–92.CrossRefGoogle Scholar
  36. Sacks, H., Schegloff, E., & Jefferson, G. (1974). A simplest systematic for the organization of turn-taking in conversation. Language, 50, 696–753.CrossRefGoogle Scholar
  37. Sosa y Fink, S. (2003). Merkmale gelungener Kooperation. Eine qualitative Analyse ­netzgestützter Zusammenarbeit. [Characteristics of successful cooperation. A qualitative analysis of net-based collaboration.] Diploma thesis, Albert-Ludwigs-Universität, Freiburg.Google Scholar
  38. Stasser, G., Stewart, D., & Wittenbaum, G. (1995). Expert roles and information exchange during discussion: The importance of knowing who knows what. Journal of Experimental Social Psychology, 31, 244–265.CrossRefGoogle Scholar
  39. Stasser, G., & Titus, W. (1985). Pooling of unshared information in group decision making: Biased information sampling during group discussion. Journal of Personality and Social Psychology, 48, 1467–1478.CrossRefGoogle Scholar
  40. Steiner, I. D. (1972). Group process and productivity. New York: Academic.Google Scholar
  41. Strijbos, J.-W., Martens, R. L., Prins, F. J., & Jochems, W. M. G. (2006). Content analysis: What are they talking about? Computers and Education, 46, 29–48.CrossRefGoogle Scholar
  42. Tindale, R. S., Kameda, T., & Hinsz, V. B. (2003). Group decision making. In M. A. Hogg & J. Cooper (Eds.), Sage handbook of social psychology (pp. 381–403). London: Sage.Google Scholar
  43. Voyiatzaki, E., Meier, A., Kahrimanis, G., Rummel, N., Spada, H., & Avouris, N. (2008a). Rating the quality of collaboration during networked problem solving activities. In Proceedings of the 6th International Conference on Networked Learning (pp. 409–416).Google Scholar
  44. Voyiatzaki, E., Papadakis, S., Rossiou, E., Avouris, N., Paparizzos, K., & Hadzilacos, T. (2008b). One size does not fit all: A case study of combining networked learning methods and tools. In Proceedings of the 6th International Conference on Networked Learning, 5–6 May 2008, Halkidiki, Greece. Retrieved 5 November 2008.
  45. Webb, N. M. (1989). Peer interaction and learning in small groups. Int J Educ Res, 13, 21–39.CrossRefGoogle Scholar
  46. Wegner, D. M. (1987). Transactive memory: A contemporary analysis of the group mind. In B. Mullen & G. R. Goethals (Eds.), Theories of group behavior (pp. 185–208). Berlin, Heidelberg, New York: Springer.CrossRefGoogle Scholar
  47. Wirtz, M., & Caspar, F. (2002). Beurteilerübereinstimmung und Beurteilerreliabilität. [Inter-rater agreement and inter-rater reliability]. Göttingen: Verlag für Psychologie.Google Scholar
  48. Wittenbaum, G. M., Vaughan, S. I., & Stasser, G. (1998). Coordination in task performing groups. In R. S. Tindale et al. (Eds.), Theory and research on small groups (pp. 177–204). New York: Plenum.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Nikol Rummel
    • 1
  • Anne Deiglmayr
  • Hans Spada
  • George Kahrimanis
  • Nikolaos Avouris
  1. 1.Institute of EducationRuhr-Universität BochumBochumGermany

Personalised recommendations