Multilevel Analysis in CSCL Research

  • Jeroen Janssen
  • Gijsbert Erkens
  • Paul A. Kirschner
  • Gellof Kanselaar
Chapter
Part of the Computer-Supported Collaborative Learning Series book series (CULS, volume 12)

Abstract

The aim of this chapter is to explain why multilevel analysis (MLA) is often necessary to correctly answer the questions CSCL researchers address. Although CSCL researchers continue to use statistical techniques such as ­analysis of variance or regression analysis, their datasets are often not suited for these ­techniques. The first reason is that CSCL research deals with individuals ­collaborating in groups, often creating hierarchically nested datasets. This means that such datasets for example contain variables measured at the level of the individual (e.g., learning performance) and variables measured at the level of the group (e.g., group composition or group performance). The number of unique observations at the lowest level, the individual, is higher than at the highest level, the group. Related to this, CSCL datasets often contain differing units of analysis. Some variables that CSCL researchers are interested in are measured at the individual level (e.g., gender, interactive behavior, familiarity with other group members), whereas other variables are ­measured at the group level (e.g., gender group composition, group performance, group consensus). Finally, because group members interact with each other in CSCL environments, this leads to nonindependence of the dependent variable(s) in the dataset. Because of their common experience during the collaboration, students’ scores on the dependent variables will likely correlate (e.g., in a group with a relatively long history of successful collaboration, group members will report similar, high levels of trust, while in groups with a negative collaboration history, group members will likely report low levels of trust). Whether nonindependence is ­present in a dataset can be established by calculating the intraclass correlation coefficient. Whenever researchers encounter datasets with hierarchically nested data, differing units of analysis, and nonindependence, MLA is needed to appropriately model this data structure since it can appropriately disentangle the effects of the different ­levels on the dependent variable(s) of interest. Researchers however also employ other strategies to deal with nonindependence and hierarchy in their ­datasets (e.g., ignoring nonindependence and hierarchy, or aggregating or disaggregating their data). We will highlight the dangers of these strategies using examples from our own research (e.g., increasing the chance of committing a Type I error). The chapter ends with a discussion of the advantages and disadvantages of using MLA for CSCL research. For example, although MLA is a powerful technique to address the data analytical problems CSCL researchers encounter, relatively large sample sizes are necessary.

Keywords

Covariance Autocorrelation CSCL 

References

  1. Baker, M. (2003). Computer-mediated argumentative interactions for the co-elaboration of scientific notions. In J. Andriessen, M. Baker, & D. Suthers (Eds.), Arguing to learn: Confronting cognitions in computer-supported collaborative learning environments (pp. 47–78). Dordrecht: Kluwer Academic.Google Scholar
  2. Bonito, J. A. (2002). The analysis of participation in small groups: Methodological and conceptual issues related to interdependence. Small Group Research, 33, 412–438.CrossRefGoogle Scholar
  3. Bonito, J. A., & Lambert, B. L. (2005). Information similarity as a moderator of the effect of gender on participation in small groups: A multilevel analysis. Small Group Research, 36, 139–165.CrossRefGoogle Scholar
  4. Bosker, R. J., & Snijders, T. A. (1990). Statistische aspecten van multi-niveau onderzoek (Statistical aspects of multilevel research). Tijdschrift voor Onderwijsresearch, 15, 317–329.Google Scholar
  5. Campbell, L., & Kashy, D. A. (2002). Estimating actor, partner, and interaction effects for dyadic data using PROC MIXED and HLM: A user-friendly guide. Personal Relationships, 9, 327–342.CrossRefGoogle Scholar
  6. Chiu, M. M., & Khoo, L. (2003). Rudeness and status effects during group problem solving: Do they bias evaluations and reduce the likelihood of correct solutions? Journal of Education and Psychology, 95, 506–523.CrossRefGoogle Scholar
  7. Chiu, M. M., & Khoo, L. (2005). A new method for analyzing sequential processes: Dynamic multilevel analysis. Small Group Research, 36(5), 600–631.CrossRefGoogle Scholar
  8. Cress, U. (2008). The need for considering multilevel analysis in CSCL research: An appeal for the use of more advanced statistical methods. International Journal of Computer-Supported Collaborative Learning, 3, 69–84.CrossRefGoogle Scholar
  9. De Leeuw, J., & Kreft, I. (1986). Random coefficient models for multilevel analysis. Journal of Educational Statistics, 11, 57–85.CrossRefGoogle Scholar
  10. De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to ­analyze transcripts of online asynchronous discussion groups: A review. Computers and Education, 46, 6–28.CrossRefGoogle Scholar
  11. De Wever, B., Van Keer, H., Schellens, T., & Valcke, M. (2007). Applying multilevel modelling to content analysis data: Methodological issues in the study of role assignment in asynchronous discussion groups. Learning and Instruction, 17, 436–447.CrossRefGoogle Scholar
  12. Dewiyanti, S., Brand-Gruwel, S., Jochems, W., & Broers, N. J. (2007). Students’ experiences with collaborative learning in asynchronous computer-supported collaborative learning environments. Computers in Human Behavior, 23, 496–514.CrossRefGoogle Scholar
  13. Engelmann, T., Dehler, J., Bodemer, D., & Buder, J. (2009). Knowledge awareness in CSCL: A psychological perspective. Computers in Human Behavior, 25(4), 949–960.CrossRefGoogle Scholar
  14. Francescato, D., Porcelli, R., Mebane, M., Cuddetta, M., Klobas, J., & Renzi, P. (2006). Evaluation of the efficacy of collaborative learning in face-to-face and computer-supported university contexts. Computers in Human Behavior, 22, 163.CrossRefGoogle Scholar
  15. Guiller, J., & Durndell, A. (2007). Students’ linguistic behaviour in online discussion groups: Does gender matter? Computers in Human Behavior, 23, 2240–2255.CrossRefGoogle Scholar
  16. Hmelo-Silver, C. E., & Bromme, R. (2007). Coding discussions and discussing coding: Research on collaborative learning in computer-supported environments. Learning and Instruction, 17, 460–464.CrossRefGoogle Scholar
  17. Hox, J. (2003). Multilevel analysis: Techniques and applications. Mahwah: Lawrence Erlbaum.Google Scholar
  18. Janssen, J., Erkens, G., & Kanselaar, G. (2007). Visualization of agreement and discussion processes during computer-supported collaborative learning. Computers in Human Behavior, 23, 1105–1125.CrossRefGoogle Scholar
  19. Janssen, J., Erkens, G., Kirschner, P. A., & Kanselaar, G. (2010). Effects of representational guidance during computer-supported collaborative learning. Instructional Science, 38, 59–88.Google Scholar
  20. Kashy, D. A., & Kenny, D. A. (2000). The analysis of data from dyads and groups. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 451–477). Cambridge: Cambridge University Press.Google Scholar
  21. Kenny, D. A. (1995). The effect of nonindependence on significance testing in dyadic research. Personal Relationships, 2, 67–75.CrossRefGoogle Scholar
  22. Kenny, D. A. (1996). Models of non-independence in dyadic research. Journal of Social and Personal Relationships, 13, 279–294.CrossRefGoogle Scholar
  23. Kenny, D. A., & Judd, C. M. (1986). Consequences of violating the independence assumption in analysis of variance. Psychological Bulletin, 99, 422.CrossRefGoogle Scholar
  24. Kenny, D. A., & Judd, C. M. (1996). A general procedure for the estimation of interdependence. Psychological Bulletin, 119, 138–148.CrossRefGoogle Scholar
  25. Kenny, D. A., Kashy, D. A., & Cook, W. L. (2006). Dyadic data analysis. New York/London: Guilford.Google Scholar
  26. Kenny, D. A., Mannetti, L., Pierro, A., Livi, S., & Kashy, D. A. (2002). The statistical analysis of data from small groups. Journal of Personality and Social Psychology, 83, 126–137.CrossRefGoogle Scholar
  27. Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: A review of the research. Computers in Human Behavior, 19, 335–353.CrossRefGoogle Scholar
  28. Leech, N. L., & Onwuegbuzie, A. J. (2009). A typology of mixed methods research designs. Quality and Quantity, 43, 265–275.CrossRefGoogle Scholar
  29. Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modelling. European Journal of Research Methods for the Behavioral and Social Sciences, 1, 85–91.CrossRefGoogle Scholar
  30. O’Donnell, A. M., & O’Kelly, J. (1994). Learning from peers: Beyond the rhetoric of positive results. Educational Psychology Review, 6, 321–349.CrossRefGoogle Scholar
  31. Savicki, V., & Kelley, M. (2000). Computer mediated communication: Gender and group composition. Cyberpsychology and Behavior, 3, 817–826.CrossRefGoogle Scholar
  32. Savicki, V., Kelley, M., & Lingenfelter, D. (1996). Gender and group composition in small task groups using computer-mediated communication. Computers in Human Behavior, 12, 209–224.CrossRefGoogle Scholar
  33. Schellens, T., Van Keer, H., & Valcke, M. (2005). The impact of role assignment on knowledge construction in asynchronous discussion groups: A multilevel analysis. Small Group Research, 36, 704–745.CrossRefGoogle Scholar
  34. Schellens, T., Van Keer, H., Valcke, M., & De Wever, B. (2007). Learning in asynchronous ­discussion groups: A multilevel approach to study the influence of student, group and task characteristics. Behaviour and Information Technology, 26, 55–71.CrossRefGoogle Scholar
  35. Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.Google Scholar
  36. Stahl, G. (2006). Group cognition: Computer support for building collaborative knowledge. Cambridge: MIT Press.Google Scholar
  37. Strijbos, J.-W., & Fischer, F. (2007). Methodological challenges for collaborative learning research. Learning and Instruction, 17, 389–393.CrossRefGoogle Scholar
  38. Strijbos, J. W., Martens, R. L., Jochems, W. M. G., & Broers, N. J. (2004). The effect of functional roles on group efficiency: Using multilevel modeling and content analysis to investigate computer-supported collaboration in small groups. Small Group Research, 35, 195–229.CrossRefGoogle Scholar
  39. Strijbos, J. W., Martens, R. L., Jochems, W. M. G., & Broers, N. J. (2007). The effect of functional roles on perceived group efficiency during computer-supported collaborative learning: A matter of triangulation. Computers in Human Behavior, 23, 353–380.CrossRefGoogle Scholar
  40. Stylianou-Georgiou, A., Papanastasiou, E., & Puntambekar, S. (this volume). Analyzing collaborative processes and learning from hypertext through hierarchical linear modelling. In S. Puntambekar, G. Erkens, & C. Hmelo-Silver (Eds.), Analyzing interactions in CSCL: Methods, approaches and issues.Google Scholar
  41. Suthers, D. D. (2001). Towards a systematic study of representational guidance for collaborative learning discourse. Journal of Universal Computer Science, 7, 254–277.Google Scholar
  42. Suthers, D. D. (2006). Technology affordances for intersubjective meaning making. International Journal of Computer Supported Collaborative Learning, 1, 315–337.CrossRefGoogle Scholar
  43. Suthers, D. D., & Hundhausen, C. D. (2003). An experimental study of the effects of representational guidance on collaborative learning processes. Journal of the Learning Sciences, 12, 183–218.CrossRefGoogle Scholar
  44. Van der Meijden, H., & Veenman, S. (2005). Face-to-face versus computer-mediated communication in a primary school setting. Computers in Human Behavior, 21, 831–859.CrossRefGoogle Scholar
  45. Webb, N. M., & Palincsar, A. S. (1996). Group processes in the classroom. In D. C. Berliner (Ed.), Handbook of educational psychology (pp. 841–873). New York: Simon & Schuster Macmillan.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Jeroen Janssen
    • 1
  • Gijsbert Erkens
  • Paul A. Kirschner
  • Gellof Kanselaar
  1. 1.Research Centre Learning in InteractionUtrecht UniversityUtrechtThe Netherlands

Personalised recommendations