Skip to main content
SpringerLink
Log in

Computing Technologies for Social Signals

  • Chapter
  • First Online:
Book cover Emotions and Personality in Personalized Services

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

Social signal processing is the domain aimed at modelling, analysis and synthesis of nonverbal communication in human–human and human–machine interactions. The core idea of the field is that common nonverbal behavioural cues—facial expressions, vocalizations, gestures, postures, etc—are the physical, machine-detectable evidence of social phenomena such as empathy, conflict, interest, attitudes, dominance, etc. Therefore, machines that can automatically detect, interpret and synthesize social signals will be capable to make sense of the social landscape they are part of while, possibly, participating in it as full social actors.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.cogitocorp.com.

  2. 2.

    http://www.engagem8.com.

  3. 3.

    http://www.behaviouralinsights.co.uk.

  4. 4.

    http://groupspaces.com/ROCKIT.

  5. 5.

    http://www.eu-robotics.net.

  6. 6.

    http://www.sony-aibo.co.uk.

References

  1. Al Moubayed, N., Vazquez-Alvarez, Y., McKay, A., Vinciarelli, A.: Face-based automatic personality perception. In: Proceedings of the ACM International Conference on Multimedia, pp. 1153–1156 (2014)

    Google Scholar 

  2. André, E.: Exploiting unconscious user signals in multimodal human-computer interaction. ACM Trans. Multimedia Comput. Commun. Appl. 9(1s), 48 (2013)

    Article  Google Scholar 

  3. André, E.: Challenges for social embodiment. In: Proceedings of the Workshop on Roadmapping the Future of Multimodal Interaction Research Including Business Opportunities and Challenges, pp. 35–37 (2014)

    Google Scholar 

  4. Argamon, S., Dhawle, S., Koppel, M., Pennbaker, J.: Lexical predictors of personality type. In: Proceedings of Interface and the Classification Society of North America (2005)

    Google Scholar 

  5. Avril, M., Leclere, C., Viaux-Savelon, S., Michelet, S., Achard, C., Missonnier, S., Keren, M., Cohen, D., Chetouani, M.: Social signal processing for studying parent-infant interaction. Frontiers Psychol. 5, 1437 (2014)

    Article  Google Scholar 

  6. Bakeman, R., Gottman, J.: Observing Interaction: An Introduction to Sequential Analysis. Cambridge University Press (1997)

    Google Scholar 

  7. Benford, S., Greenhalgh, C., Giannachi, G., Walker, B., Marshall, J., Rodden, T.: Uncomfortable interactions. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2005–2014. ACM (2012)

    Google Scholar 

  8. Bonin, F., Gilmartin, E., Vogel, C., Campbell, N.: Topics for the future: genre differentiation, annotation, and linguistic content integration in interaction analysis. In: Proceedings of the Workshop on Roadmapping the Future of Multimodal Interaction Research Including Business Opportunities and Challenges, pp. 5–8 (2014)

    Google Scholar 

  9. Bousmalis, K., Méhu, M., Pantic, M.: Spotting agreement and disagreement: a survey of nonverbal audiovisual cues and tools. In: Proceedings of Social Signal Processing Workshop, pp. 1–9 (2009)

    Google Scholar 

  10. Brunet, P., Cowie, R.: Towards a conceptual framework of research on social signal processing. J. Multimodal User Interfaces 6(3–4), 101–115 (2012)

    Article  Google Scholar 

  11. Brunet, P.M., Cowie, R., Heylen, D., Nijholt, A., Schroeder, M.: Conceptual frameworks for multimodal social signal processing. J. Multimodal User Interfaces 6(3–4), 1–5 (2012)

    Google Scholar 

  12. Brunswik, E.: Perception and the Representative Design of Psychological Experiments. University of California Press (1956)

    Google Scholar 

  13. Cristani, M., Ferrario, R.: Statistical pattern recognition meets formal ontologies: towards a semantic visual understanding. In: Proceedings of the Workshop on Roadmapping the Future of Multimodal Interaction Research Including Business Opportunities and Challenges, pp. 23–25 (2014)

    Google Scholar 

  14. Cristani, M., Paggetti, G., Vinciarelli, A., Bazzani, L., Menegaz, G., Murino, V.: Towards computational proxemics: inferring social relations from interpersonal distances. In: Proceedings of IEEE International Conference on Social Computing, pp. 290–297 (2011)

    Google Scholar 

  15. Cristani, M., Raghavendra, R., Del Bue, A., Murino, V.: Human behavior analysis in video surveillance: a social signal processing perspective. Neurocomputing 100, 86–97 (2013)

    Article  Google Scholar 

  16. Cristani, M., Vinciarelli, A., Segalin, C., Perina, A.: Unveiling the multimedia unconscious: implicit cognitive processes and multimedia content analysis. In: Proceedings of the ACM International Conference on Multimedia, pp. 213–222. ACM (2013)

    Google Scholar 

  17. Curhan, J.R., Pentland, A.: Thin slices of negotiation: predicting outcomes from conversational dynamics within the first 5 minutes. J. Appl. Psychol. 92(3), 802 (2007)

    Article  Google Scholar 

  18. Dautenhahn, K., Werry, I.: Towards interactive robots in autism therapy: background, motivation and challenges. Pragmat. Cogn. 12(1), 1–35 (2004)

    Article  Google Scholar 

  19. Delaherche, E., Chetouani, M., Mahdhaoui, A., Saint-Georges, C., Viaux, S., Cohen, D.: Interpersonal synchrony: a survey of evaluation methods across disciplines. IEEE Trans. Affect. Comput. 3(3), 349–365 (2012)

    Article  Google Scholar 

  20. Fong, T., Nourbakhsh, I., Dautenhahn, K.: A survey of socially interactive robots. Rob. Auton. Syst. 42(3), 143–166 (2003)

    Article  MATH  Google Scholar 

  21. Garg, N.P., Favre, S., Salamin, H., Hakkani Tür, D., Vinciarelli, A.: Role recognition for meeting participants: an approach based on lexical information and social network analysis. In: Proceedings of the ACM International Conference on Multimedia, pp. 693–696 (2008)

    Google Scholar 

  22. Gosling, S.D., Ko, S.J., Mannarelli, T., Morris, M.E.: A room with a cue: personality judgments based on offices and bedrooms. J. Pers. Soc. Psychol. 82(3), 379 (2002)

    Article  Google Scholar 

  23. Hammal, Z., Cohn, J.F.: Intra- and interpersonal functions of head motion in emotion communication. In: Proceedings of the Workshop on Roadmapping the Future of Multimodal Interaction Research Including Business Opportunities and Challenges, pp. 19–22 (2014)

    Google Scholar 

  24. Hung, H., Kroese, B.: Detecting F-formations as dominant sets. In: Proceedings of the International Conference on Multimodal Interfaces, pp. 231–238 (2011)

    Google Scholar 

  25. Jayagopi, D.B., Hung, H., Yeo, C., Gatica-Perez, D.: Modeling dominance in group conversations using nonverbal activity cues. IEEE Trans. Audio Speech Lang. Process. 17(3), 501–513 (2009)

    Article  Google Scholar 

  26. Koutsombogera, M., Papageorgiou, H.: Multimodal analytics and its data ecosystem. In: Proceedings of the Workshop on Roadmapping the Future of Multimodal Interaction Research Including Business Opportunities and Challenges, pp. 1–4 (2014)

    Google Scholar 

  27. Lu, L., Liu, D., Zhang, H.-J.: Automatic mood detection and tracking of music audio signals. IEEE Trans. Audio Speech Lang. Process. 14(1), 5–18 (2006)

    Article  MathSciNet  Google Scholar 

  28. Martin, P., Bateson, P.: Measuring Behaviour. Cambridge University Press (2007)

    Google Scholar 

  29. McRorie, Margaret, Sneddon, Ian, McKeown, Gary, Bevacqua, Elisabetta, de Sevin, Etienne, Pelachaud, Catherine: Evaluation of four designed virtual agent personalities. IEEE Trans. Affect. Comput. 3(3), 311–322 (2011)

    Article  Google Scholar 

  30. Mehu, M., Scherer, K.: A psycho-ethological approach to social signal processing. Cogn. Process. 13(2), 397–414 (2012)

    Article  Google Scholar 

  31. Nass, C., Brave, S.: Wired for Speech: How Voice Activates and Advances the Human-Computer relationship. The MIT Press (2005)

    Google Scholar 

  32. Nass, C., Steuer, J., Tauber, E.R.: Computers are social actors. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 72–78 (1994)

    Google Scholar 

  33. Nie, J., Cui, P., Yan, Y., Huang, L., Li, Z., Wei, Z.: How your portrait impresses people?: inferring personality impressions from portrait contents. In: Proceedings of the ACM International Conference on Multimedia, pp. 905–908 (2014)

    Google Scholar 

  34. Pantic, M., Vinciarelli, A.: Implicit human-centered tagging [social sciences]. IEEE Sign. Process. Mag. 26(6), 173–180 (2009)

    Article  Google Scholar 

  35. Pentland, A.: Social signal processing. IEEE Sign. Proces. Mag. 24(4), 108–111 (2007)

    Article  Google Scholar 

  36. Pentland, A.: Honest Signals. MIT Press (2010)

    Google Scholar 

  37. Pianesi, F., Zancanaro, M., Not, E., Leonardi, C., Falcon, V., Lepri, B.: Multimodal support to group dynamics. Pers. Ubiquit. Comput. 12(3), 181–195 (2008)

    Article  Google Scholar 

  38. Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. 23(10), 1175–1191 (2001)

    Article  Google Scholar 

  39. Poggi, I., D’Errico, F.: Social signals: a framework in terms of goals and beliefs. Cogn. Process. 13(2), 427–445 (2012)

    Article  Google Scholar 

  40. Poppe, R.: A survey on vision-based human action recognition. Image Vis. Comput. 28(6), 976–990 (2010)

    Article  Google Scholar 

  41. Potamianos, A.: Cognitive multimodal processing: from signal to behavior. In: Proceedings of the Workshop on Roadmapping the Future of Multimodal Interaction Research Including Business Opportunities and Challenges, pp. 27–34 (2014)

    Google Scholar 

  42. Quercia, D., Kosinski, M., Stillwell, D., Crowcroft, J.: Our twitter profiles, our selves: predicting personality with twitter. In: Proceedings of the IEEE International Conference on Social Computing, pp. 180–185 (2011)

    Google Scholar 

  43. Quercia, D., Lambiotte, R., Stillwell, D., Kosinski, M., Crowcroft, J.: The personality of popular facebook users. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work, pp. 955–964 (2012)

    Google Scholar 

  44. Raento, M., Oulasvirta, A., Eagle, N.: Smartphones an emerging tool for social scientists. Sociol. Methods Res. 37(3), 426–454 (2009)

    Article  MathSciNet  Google Scholar 

  45. Ramanathan, S., Katti, H., Sebe, N., Kankanhalli, M., Chua, T.-S.: An eye fixation database for saliency detection in images. In: Proceedings of European Conference on Computer Vision, pp. 30–43 (2010)

    Google Scholar 

  46. Reeves, B., Nass, C.: The media equation: how people treat computers, television, and new media like real people and places. Cambridge University Press (1996)

    Google Scholar 

  47. Renals, S., Carletta, J., Edwards, K., Bourlard, H., Garner, P., Popescu-Belis, A., Klakow, D., Girenko, A., Petukova, V., Wacker, P., Joscelyne, A., Kompis, C., Aliwell, S., Stevens, W., Sabbah, Y.: ROCKIT: roadmap for conversational interaction technologies. In: Proceedings of the Workshop on Roadmapping the Future of Multimodal Interaction Research Including Business Opportunities and Challenges, pp. 39–42 (2014)

    Google Scholar 

  48. Riccardi, G.: Towards healthcare personal agents. In: Proceedings of the Workshop on Roadmapping the Future of Multimodal Interaction Research Including Business Opportunities and Challenges, pp. 53–56 (2014)

    Google Scholar 

  49. Sabourin, J.L., Lester, J.C.: Affect and engagement in game-based learning environments. IEEE Trans. Affect. Comput. 5(1), 45–56 (2014)

    Article  Google Scholar 

  50. Salah, A.A.: Natural multimodal interaction with a social robot: what are the premises? In: Proceedings of the Workshop on Roadmapping the Future of Multimodal Interaction Research Including Business Opportunities and Challenges, pp. 43–45 (2014)

    Google Scholar 

  51. Salamin, H., Polychroniou, A., Vinciarelli, A.: Automatic recognition of personality and conflict handling style in mobile phone conversations. In: Proceedings of International Workshop on Image Analysis for Multimedia Interactive Services, pp. 1–4 (2013)

    Google Scholar 

  52. Sanchez-Cortes, D., Aran, O., Schmid-Mast, M., Gatica-Perez, D.: A nonverbal behavior approach to identify emergent leaders in small groups. IEEE Trans. Multimedia 14(3), 816–832 (2012)

    Article  Google Scholar 

  53. Schegloff, E.A.: Analyzing single episodes of interaction: an exercise in conversation analysis. Soc. Psychol. Q. 101–114 (1987)

    Google Scholar 

  54. Schroeder, M.: Expressive speech synthesis: past, present, and possible futures. In: Affective information processing, pp. 111–126. Springer (2009)

    Google Scholar 

  55. Schuller, B., Batliner, A.: Computational Paralinguistics: Emotion, Affect and Personality in Speech and Language Processing. Wiley (2013)

    Google Scholar 

  56. Schuller, B., Steidl, S., Batliner, A., Noeth, E., Vinciarelli, A., Burkhardt, F., van Son, R., Weninger, F., Eyben, F., Bocklet, T., Mohammadi, G., Weiss, B.: A survey on perceived speaker traits: personality, likability, pathology, and the first challenge. Comput. Speech Lang. 29(1), 100–131 (2015)

    Article  Google Scholar 

  57. Staiano, J., Pianesi, F., Lepri, B., Sebe, N., Aharony, N., Pentland, A.: Friends don’t lie—inferring personality traits from social network structure. In: Proceedings of the ACM International Conference on Ubiquitous Computing, pp. 321–330 (2012)

    Google Scholar 

  58. Tapus, A., Mataric, M.: Socially assistive robots: the link between personality, empathy, physiological signals, and task performance. In: Proceedings of AAAI Spring Symposium (2008)

    Google Scholar 

  59. Tapus, A., Ţăpuş, C., Mataric, M.J.: User–robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy. Intell. Serv. Robot. 1(2), 169–183 (2008)

    Article  Google Scholar 

  60. Tausczik, Y.R., Pennebaker, J.W.: The psychological meaning of words: LIWC and computerized text analysis methods. J. Lang. Soc. Psychol. 29(1), 24–54 (2010)

    Article  Google Scholar 

  61. Tkalčič, M., Burnik, U., Košir, A.: Using affective parameters in a content-based recommender system for images. User Model. User Adap. Inter. 20(4), 279–311 (2010)

    Article  Google Scholar 

  62. Uleman, J.S., Newman, L.S., Moskowitz, G.B.: People as flexible interpreters: evidence and issues from spontaneous trait inference. Adv. Exp. Soc. Psychol. 28, 211–279 (1996)

    Article  Google Scholar 

  63. Valstar, M.: Automatic behaviour understanding in medicine. In: Proceedings of the Workshop on Roadmapping the Future of Multimodal Interaction Research Including Business Opportunities and Challenges, pp. 57–60 (2014)

    Google Scholar 

  64. Vinciarelli, A., Mohammadi, G.: A survey of personality computing. IEEE Trans. Affect. Comput. 5(3), 273–291 (2014)

    Article  Google Scholar 

  65. Vinciarelli, A., Pantic, M., Bourlard, H., Pentland, A.: Social signal processing: state-of-the-art and future perspectives of an emerging domain. In: Proceedings of the ACM International Conference on Multimedia, pp. 1061–1070 (2008)

    Google Scholar 

  66. Vinciarelli, A., Pantic, M., Bourlard, H.: Social signal processing: survey of an emerging domain. Image Vis. Comput. J. 27(12), 1743–1759 (2009)

    Article  Google Scholar 

  67. Vinciarelli, A., Kim, S., Valente, F., Salamin, H.: Collecting data for socially intelligent surveillance and monitoring approaches: the case of conflict in competitive conversations. In: Proceedings of the International Symposium on Communications, Control and Signal Processing, pp. 1–4 (2012)

    Google Scholar 

  68. Vinciarelli, A., Pantic, M., Heylen, D., Pelachaud, C., Poggi, I., D’Errico, F., Schroeder, M.: Bridging the gap between social animal and unsocial machine: a survey of social signal processing. IEEE Trans. Affect. Comput. 3(1), 69–87 (2012)

    Article  Google Scholar 

  69. Vinciarelli, A., Chatziioannou, P., Esposito, A.: When the words are not everything: the use of laughter, fillers, back-channel, silence and overlapping speech in phone calls. Frontiers Hum. Media Interact. (2015)

    Google Scholar 

  70. Vodlan, T., Tkalčič, M., Košir, A.: The impact of hesitation, a social signal, on a user’s quality of experience in multimedia content retrieval. Multimedia Tools Appl. 1–26 (2014)

    Google Scholar 

  71. Weisman, O., Delaherche, E., Rondeau, M., Chetouani, M., Cohen, D., Feldman, R.: Oxytocin shapes parental motion during father-infant interaction. Biol. Lett. 9(6), 20130828 (2013)

    Article  Google Scholar 

  72. Yaakub, C.Y., Sulaiman, N., Kim, C.W.: A study on personality identification using game based theory. In: Proceedings of the International Conference on Computer Technology and Development, pp. 732–734 (2010)

    Google Scholar 

  73. Yanulevskaya, V., Uijlings, J., Bruni, E., Sartori, A., Zamboni, E., Bacci, F., Melcher, D., Sebe, N.: In the eye of the beholder: employing statistical analysis and eye tracking for analyzing abstract paintings. In: Proceedings of ACM International Conference on Multimedia, pp. 349–358 (2012)

    Google Scholar 

  74. Yee, N., Ducheneaut, N., Nelson, L., Likarish, P.: Introverted elves and conscientious gnomes: the expression of personality in world of warcraft. In: Proceedings of the Annual Conference on Human Factors in Computing Systems, pp. 753–762, Vancouver (2011)

    Google Scholar 

  75. Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A.: Face recognition: a literature survey. ACM Comput. Surv. 35(4), 399–458 (2003)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Glasgow, Sir A.Williams Building, Glasgow, G12 8QQ, UK

    Alessandro Vinciarelli

Authors
  1. Alessandro Vinciarelli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alessandro Vinciarelli .

Editor information

Editors and Affiliations

  1. Department of Computational Perception, Johannes Kepler University, Linz, Austria

    Marko Tkalčič

  2. Department of Computer Science, University of Bari Aldo Moro, Bari, Italy

    Berardina De Carolis

  3. Department of Computer Science, University of Bari Aldo Moro, Bari, Italy

    Marco de Gemmis

  4. Preserje, Ljubljana, Slovenia

    Ante Odić

  5. Faculty of electrical engineering, University of Ljubljana, Ljubljana, Slovenia

    Andrej Košir

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Vinciarelli, A. (2016). Computing Technologies for Social Signals. In: Tkalčič, M., De Carolis, B., de Gemmis, M., Odić, A., Košir, A. (eds) Emotions and Personality in Personalized Services. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-319-31413-6_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-31413-6_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-31411-2

  • Online ISBN: 978-3-319-31413-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation