Analysis of Group Conversations: Modeling Social Verticality

  • Oya Aran
  • Daniel Gatica-Perez


This chapter presents computational methods for the analysis of social interaction. We focus on nonverbal behavior of social interaction, in particular social verticality, such as dominance, leadership, and roles. We describe processing, feature extraction, and inference methods that are widely used in the computational social interaction analysis literature. In the last section of the chapter, we present four case studies on dominance estimation, identifying emergent leadership, role recognition, and analysis of leadership styles.


Latent Dirichlet Allocation Visual Activity Influence Model Microphone Array Prosodic Feature 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work is supported by the EU FP7 Marie Curie Intra-European Fellowship project “Automatic Analysis of Group Conversations via Visual Cues in nonverbal Communication” (NOVICOM), and by the Swiss National Science Foundation under the National Centre of Competence in Research (NCCR) on “Interactive Multimodal Information Management” (IM2) and by the Sinergia project on “Sensing and Analysing Organizational Nonverbal Behaviour” (SONVB). The authors would like to thank Dinesh Jayagopi, Dairazalia Sanchez-Cortes, and Gokul Chittaranjan for their contributions to several studies presented in this chapter.


  1. 1.
    Ambady, N., Rosenthal, R.: Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. Psychol. Bull. 111, 256–274 (1992) CrossRefGoogle Scholar
  2. 2.
    Aran, O., Gatica-Perez, D.: Fusing audio-visual nonverbal cues to detect dominant people in small group conversations. In: 20th International Conference on Pattern Recognition (ICPR), Istanbul, Turkey (2010) Google Scholar
  3. 3.
    Aran, O., Hung, H., Gatica-Perez, D.: A multimodal corpus for studying dominance in small group conversations. In: LREC Workshop on Multimodal Corpora, Malta (LREC MMC’10) (2010) Google Scholar
  4. 4.
    Asavathiratham, C., Roy, S., Lesieutre, B., Verghese, G.: The influence model. IEEE Control Syst. Mag. 21, 52–64 (2001) CrossRefGoogle Scholar
  5. 5.
    Ba, S.O., Odobez, J.-M.: Multiperson visual focus of attention from head pose and meeting contextual cues. IEEE Trans. Pattern Anal. Mach. Intell. 33, 101–116 (2011) CrossRefGoogle Scholar
  6. 6.
    Barzilay, R., Collins, M., Hirschberg, J., Whittaker, S.: The rules behind roles: Identifying speaker role in radio broadcasts. In: 17th National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, pp. 679–684. AAAI, Washington (2000) Google Scholar
  7. 7.
    Basu, S., Choudhury, T., Clarkson, B., Pentland, A.: Learning human interactions with the influence model. Technical report, MIT Media Lab, Cambridge, MA (2001) Google Scholar
  8. 8.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003) MATHGoogle Scholar
  9. 9.
    Bobick, A.F., Davis, J.W.: The recognition of human movement using temporal templates. IEEE Trans. Pattern Anal. Mach. Intell. 23, 257–267 (2001) CrossRefGoogle Scholar
  10. 10.
    Campbell, N., Douxchamps, D.: Processing image and audio information for recognising discourse participation status through features of face and voice. In: INTERSPEECH 2007, pp. 730–733 (2007) Google Scholar
  11. 11.
    Carletta, J., Ashby, S., Bourban, S., Flynn, M., Guillemot, M., Hain, T., Kadlec, J., Karaiskos, V., Kraaij, W., Kronenthal, M., Lathoud, G., Lincoln, M., Lisowska, A., McCowan, I., Post, W., Reidsma, D., Wellner, P.: The AMI meeting corpus: A pre-announcement. In: Workshop Mach. Learn. for Multimodal Interaction (MLMI’05), Edinburgh, UK, pp. 28–39 (2005) Google Scholar
  12. 12.
    Charfuelan, M., Schröder, M., Steiner, I.: Prosody and voice quality of vocal social signals: the case of dominance in scenario meetings. In: Interspeech 2010, Makuhari, Japan, Sept. (2010) Google Scholar
  13. 13.
    Chippendale, P.: Towards automatic body language annotation. In: 7th International Conference on Automatic Face and Gesture Recognition (FG ’06), Washington, DC (2006) Google Scholar
  14. 14.
    Chittaranjan, G., Aran, O., Gatica-Perez, D.: Exploiting observers’ judgments for multimodal nonverbal group interaction analysis. In: 9th IEEE Conference on Automatic Face and Gesture Recognition, Santa Barbara, CA (2011) Google Scholar
  15. 15.
    Dong, W., Lepri, B., Cappelletti, A., Pentland, A.S., Pianesi, F., Zancanaro, M.: Using the influence model to recognize functional roles in meetings. In: 9th International Conference on Multimodal Interfaces (ICMI’07), pp. 271–278 (2007) CrossRefGoogle Scholar
  16. 16.
    Dovidio, J.F., Ellyson, S.L.: Decoding visual dominance: Attributions of power based on relative percentages of looking while speaking and looking while listening. Soc. Psychol. Q. 45(2), 106–113 (1982) CrossRefGoogle Scholar
  17. 17.
    Dunbar, N.E., Burgoon, J.K.: Perceptions of power and interactional dominance in interpersonal relationships. J. Soc. Pers. Relatsh. 22(2), 207–233 (2005) CrossRefGoogle Scholar
  18. 18.
    Funder, D.C.: Personality. Annu. Rev. Psychol. 52, 197–221 (2001) CrossRefGoogle Scholar
  19. 19.
    Garg, N.P., Favre, S., Salamin, H., Hakkani Tür, D., Vinciarelli, A.: Role recognition for meeting participants: an approach based on lexical information and social network analysis. In: 16th ACM International Conference on Multimedia (MM’08), pp. 693–696 (2008) CrossRefGoogle Scholar
  20. 20.
    Gatica-Perez, D.: Automatic nonverbal analysis of social interaction in small groups: a review. Image Vis. Comput. 27(12), 1775–1787 (2009) CrossRefGoogle Scholar
  21. 21.
    Gorga, S., Otsuka, K.: Conversation scene analysis based on dynamic bayesian network and image-based gaze detection. In: ICMI-MLMI 2010 (2010) Google Scholar
  22. 22.
    Hall, J.A., Coats, E.J., Smith LeBeau, L.: Nonverbal behavior and the vertical dimension of social relations: A meta-analysis. Psychol. Bull. 1313(6), 898–924 (2005) CrossRefGoogle Scholar
  23. 23.
    Hung, H., Jayagopi, D.B., Ba, S., Gatica-Perez, D., Odobez, J.-M.: Investigating automatic dominance estimation in groups from visual attention and speaking activity. In: Int. Conf. on Multimodal Interfaces (ICMI), Chania, Greece (2008) Google Scholar
  24. 24.
    Hung, H., Huang, Y., Friedland, G., Gatica-Perez, D.: Estimating dominance in multi-party meetings using speaker diarization. IEEE Trans. Audio Speech Lang. Process. 19(4), 847–860 (2011) CrossRefGoogle Scholar
  25. 25.
    Jayagopi, D.B., Gatica-Perez, D.: Mining group nonverbal conversational patterns using probabilistic topic models. IEEE Trans. Multimed. (2010) Google Scholar
  26. 26.
    Jayagopi, D.B., Ba, S., Odobez, J.-M., Gatica-Perez, D.: Predicting two facets of social verticality in meetings from five-minute time slices and nonverbal cues. In: Int. Conf. on Multimodal Interfaces (ICMI), Special Session on Social Signal Processing, Chania, Greece (2008) Google Scholar
  27. 27.
    Jayagopi, D.B., Hung, H., Yeo, C., Gatica-Perez, D.: Modeling dominance in group conversations from nonverbal activity cues. IEEE Trans. Audio Speech Lang. Process. 17(3), 501–513 (2009). Special Issue on Multimodal Processing for Speech-based Interactions CrossRefGoogle Scholar
  28. 28.
    Kalma, A.K., Visser, L., Peeters, A.: Sociable and aggressive dominance: Personality differences in leadership style? Leadersh. Q. 4(1), 45–64 (1993) CrossRefGoogle Scholar
  29. 29.
    Kickul, J., Neuman, G.: Emergent leadership behaviours: The function of personality and cognitive ability in determining teamwork performance and KSAs. J. Bus. Psychol. 15(1) (2000) Google Scholar
  30. 30.
    Kim, T., Chang, A., Pentland, A.: Meeting mediator: Enhancing group collaboration with sociometric feedback. In: ACM Conference on Computer Supported Collaborative Work, San Diego, CA, pp. 457–466 (2008) Google Scholar
  31. 31.
    Knapp, M.L., Hall, J.A.: Nonverbal Communication in Human Interaction, 7th edn. Wadsworth, Belmont (2009) Google Scholar
  32. 32.
    Kumano, S., Otsuka, K., Mikami, D., Yamato, J.: Recognizing communicative facial expressions for discovering interpersonal emotions in group meetings. In: 11th International Conference on Multimodal interfaces (ICMI’09), ICMI-MLMI ’09, pp. 99–106 (2009) Google Scholar
  33. 33.
    Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley, New York (2004) MATHCrossRefGoogle Scholar
  34. 34.
    Lane, N.D., Miluzzo, E., Lu, H., Peebles, D., Choudhury, T., Campbell, A.T.: A survey of mobile phone sensing. IEEE Commun. Mag. 48, 140–150 (2010) CrossRefGoogle Scholar
  35. 35.
    Lewin, K., Lippit, R., White, R.K.: Patterns of aggressive behavior in experimentally created social climates. J. Soc. Psychol. 10, 271–301 (1939) CrossRefGoogle Scholar
  36. 36.
    Otsuka, K., Yamato, J., Takemae, Y., Murase, H.: Conversation scene analysis with dynamic Bayesian network based on visual head tracking. In: ICME 2006 (2006) Google Scholar
  37. 37.
    Otsuka, K., Sawada, H., Yamato, J.: Automatic inference of cross-modal nonverbal interactions in multiparty conversations. In: ACM 9th Int. Conf. Multimodal Interfaces (ICMI2007), pp. 255–262 (2007) CrossRefGoogle Scholar
  38. 38.
    Pentland, A.: Socially aware computation and communication. Computer 38(3), 33–40 (2005) CrossRefGoogle Scholar
  39. 39.
    Pentland, A.: Honest Signals: How They Shape Our World. MIT Press, Cambridge (2008) Google Scholar
  40. 40.
    Pianesi, F., Zancanaro, M., Lepri, B., Cappelletti, A.: A multimodal annotated corpus of consensus decision making meetings. Lang. Resour. Eval. 41, 409–429 (2007) CrossRefGoogle Scholar
  41. 41.
    Raducanu, B., Gatica-Perez, D.: Inferring competitive role patterns in reality TV show through nonverbal analysis. Multimed. Tools Appl. 1–20 (2010) Google Scholar
  42. 42.
    Rienks, R.J., Heylen, D.: Automatic dominance detection in meetings using easily detectable features. In: Workshop Mach. Learn. for Multimodal Interaction (MLMI’05), Edinburgh, UK (2005) Google Scholar
  43. 43.
    Salamin, H., Favre, S., Vinciarelli, A.: Automatic role recognition in multiparty recordings: Using social affiliation networks for feature extraction. IEEE Trans. Multimed. 11(7), 1373–1380 (2009) CrossRefGoogle Scholar
  44. 44.
    Sanchez-Cortes, D., Aran, O., Schmid-Mast, M., Gatica-Perez, D.: Identifying emergent leadership in small groups using nonverbal communicative cues. In: 12th International Conference on Multimodal Interfaces (ICMI’10), Beijing, China (2010) Google Scholar
  45. 45.
    Schmid-Mast, M.: Dominance as expressed and inferred through speaking time: A meta-analysis. Hum. Commun. Res. 28(3), 420–450 (2002) Google Scholar
  46. 46.
    Stein, R.T.: Identifying emergent leaders from verbal and nonverbal communications. J. Pers. Soc. Psychol. 32(1), 125–135 (1975) CrossRefGoogle Scholar
  47. 47.
    Steyvers, M., Griffiths, T.: Probabilistic Topic Models. Erlbaum, Hillsdale (2007) Google Scholar
  48. 48.
    Varni, G., Volpe, G., Camurri, A.: A system for real-time multimodal analysis of nonverbal affective social interaction in user-centric media. IEEE Trans. Multimed. 12(6), 576–590 (2010) CrossRefGoogle Scholar
  49. 49.
    Yeo, C., Ahammad, P., Ramchandran, K., Sastry, S.S.: High-speed action recognition and localization in compressed domain videos. IEEE Trans. Circuits Syst. Video Technol. 18(8), 1006–1015 (2008) CrossRefGoogle Scholar
  50. 50.
    Zancanaro, M., Lepri, B., Pianesi, F.: Automatic detection of group functional roles in face to face interactions. In: 8th International Conference on Multimodal Interfaces (ICMI’06), pp. 28–34 (2006) CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  1. 1.Idiap Research InstituteMartignySwitzerland

Personalised recommendations