Mobile Phones and Social Signal Processing for Analysis and Understanding of Dyadic Conversations

  • Alessandro Vinciarelli
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8045)


Social Signal Processing is the domain aimed at bridging the social intelligence gap betweeen humans and machines via modeling, analysis and synthesis of nonverbal behavior in social interactions. One of the main challenges of the domain is to sense unobtrusively the behavior of social interaction participants, one of the key conditions to preserve the spontaneity and naturalness of the interactions under exam. In this respect, mobile devices offer a major opportunity because they are equipped with a wide array of sensors that, while capturing the behavior of their users with an unprecedented depth, are still invisible. This is particularly important because mobile devices are part of the everyday life of a large number of individuals and, hence, they can be used to investigate and sense natural and spontaneous scenarios.


Mobile Phone Mobile Device Ubiquitous Computing Nonverbal Behavior Nonverbal Communication 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ambady, N., Bernieri, F., Richeson, J.: Towards a histology of social behavior: judgmental accuracy from thin slices of behavior. In: Zanna, M.P. (ed.) Advances in Experimental Social Psychology, pp. 201–272 (2000)Google Scholar
  2. 2.
    Ambady, N., Rosenthal, R.: Thin slices of expressive behavior as predictors of interpersonal consequences: a meta-analysis. Psychological Bulletin 111(2), 256–274 (1992)CrossRefGoogle Scholar
  3. 3.
    Arminen, I., Weilenmann, A.: Mobile presence and intimacy - reshaping social actions in mobile contextual configuration. Journal of Pragmatics 41(10), 1905–1923 (2009)CrossRefGoogle Scholar
  4. 4.
    Bachorowski, J.-A.: Vocal Expression and Perception of Emotion. Current Directions in Psychological Science 8(2), 53–57 (1999)CrossRefGoogle Scholar
  5. 5.
    Bauman, Z., Lyon, D.: Liquid Surveillance. Polity Press (2013)Google Scholar
  6. 6.
    Curhan, J.R., Pentland, A.: Thin slices of negotiation: predicting outcomes from conversational dynamics within the first 5 minutes. Journal of Applied Psychology 92(3), 802–811 (2007)CrossRefGoogle Scholar
  7. 7.
    Dourish, P.: What we talk about when we talk about context. Personal and Ubiquitous Computing 8(1), 19–30 (2004)CrossRefGoogle Scholar
  8. 8.
    Dourish, P., Bell, G.: Divining a digital future: mess and mythology in ubiquitous computing. MIT Press (2011)Google Scholar
  9. 9.
    Eagle, N., Pentland, A.: Reality mining: sensing complex social systems. Personal and Ubiquitous Computing 10(4), 255–268 (2005)CrossRefGoogle Scholar
  10. 10.
    Figo, D., Diniz, P.C., Ferreira, D.R., Cardoso, J.M.P.: Preprocessing techniques for context recognition from accelerometers data. Personal and Ubiquitous Computing 14(7), 645–662 (2010)CrossRefGoogle Scholar
  11. 11.
    Fortunati, L., Manganelli, A.M.: The social representation of communications. Personal and Ubiquitous Computing 12(6), 421–431 (2008)CrossRefGoogle Scholar
  12. 12.
    ITU. The World in 2010: ICT Facts and Figures. Technical report, International Telecommunication Union (2010)Google Scholar
  13. 13.
    Kalba, K.: The Global Adoption and Diffusion of Mobile Phones. Technical Report December, Center for Information Policy Research Harvard University (2008)Google Scholar
  14. 14.
    Kela, J., Korpipää, P., Mäntyjärvi, J., Kallio, S., Savino, G., Jozzo, L., Di Marca, S.: Accelerometer-based gesture control for a design environment. Personal and Ubiquitous Computing 10(5), 285–299 (2006)CrossRefGoogle Scholar
  15. 15.
    Knapp, M.L., Hall, J.A.: Nonverbal Communication in Human Interaction. Harcourt Brace College Publishers (1972)Google Scholar
  16. 16.
    Knight, J.F., Bristow, H.W., Anastopoulou, S., Baber, C., Schwirtz, A., Arvanitis, T.N.: Uses of accelerometer data collected from a wearable system. Personal and Ubiquitous Computing 11(2), 117–132 (2007)CrossRefGoogle Scholar
  17. 17.
    Kosinski, M., Stillwell, D., Graepel, T.: Private traits and attributes are predictable from digital records of human behavior. Proceeedings of the National Academy of Sciences 110(15), 5802–5805 (2013)CrossRefGoogle Scholar
  18. 18.
    Licoppe, C.: Recognizing mutual ‘proximity’ at a distance: Weaving together mobility, sociality and technology. Journal of Pragmatics 41(10), 1924–1937 (2009)CrossRefGoogle Scholar
  19. 19.
    Ling, R.: New tech, new ties. MIT Press (2008)Google Scholar
  20. 20.
    Mehl, M.R., Pennebaker, J.W., Crow, D.M., Dabbs, J., Price, J.H.: The Electronically Activated Recorder (EAR): a device for sampling naturalistic daily activities and conversations. Behavior Research Methods, Instruments and Computers 33(4), 517–523 (2001)CrossRefGoogle Scholar
  21. 21.
    Mehl, M.R., Holleran, S.E.: An Empirical Analysis of the Obtrusiveness of and Participants’ Compliance with the Electronically Activated Recorder (EAR). European Journal of Psychological Assessment 23(4), 248–257 (2007)CrossRefGoogle Scholar
  22. 22.
    Olguin Olguin, D., Waber, B.N., Kim, T., Mohan, A., Ara, K., Pentland, A.: Sensible organizations: technology and methodology for automatically measuring organizational behavior. IEEE Transactions on Systems, Man and Cybernetics Part B 39(1), 43–55 (2009)CrossRefGoogle Scholar
  23. 23.
    Raento, M., Oulasvirta, A., Eagle, N.: Smartphones: An emerging tool for social scientists. Sociological Methods & Research 37(3), 426–454 (2009)CrossRefMathSciNetGoogle Scholar
  24. 24.
    Richmond, V.P., McCroskey, J.C.: Nonverbal Behavior in Interpersonal Relations. Allyn and Bacon (2000)Google Scholar
  25. 25.
    Robinson, S., Eslambolchilar, P., Jones, M.: Exploring casual point-and-tilt interactions for mobile geo-blogging. Personal and Ubiquitous Computing 14(4), 363–379 (2010)CrossRefGoogle Scholar
  26. 26.
    Russell, J.A., Bachorowski, J.A., Fernandez-Dols, J.M.: Facial and Vocal Expressions of Emotion. Annual Reviews in Psychology 54(1), 329–349 (2003)CrossRefGoogle Scholar
  27. 27.
    Schegloff, E.A.: Analyzing Single Episodes of Interaction: An Exercise in Conversation Analysis. Social Psychology Quarterly 50(2), 101–114 (1987)CrossRefGoogle Scholar
  28. 28.
    Scherer, K.R.: Vocal communication of emotion: A review of research paradigms. Speech Communication 40(1-2), 227–256 (2003)CrossRefzbMATHGoogle Scholar
  29. 29.
    Schuller, B.: Voice and Speech Analysis in Search of States and Traits, pp. 233–258. Springer (2011)Google Scholar
  30. 30.
    Uleman, J.S., Newman, L.S., Moskowitz, G.B.: People as flexible interpreters: Evidence and issues from spontaneous trait inference, vol. 28, pp. 211–279. Elsevier (1996)Google Scholar
  31. 31.
    Uleman, J.S., Saribay, S.A., Gonzalez, C.M.: Spontaneous inferences, implicit impressions, and implicit theories. Annual Reviews of Psychology 59, 329–360 (2008)CrossRefGoogle Scholar
  32. 32.
    Ververidis, D., Kotropoulos, C.: Emotional speech recognition: Resources, features, and methods 48(9), 1162–1181 (2006)Google Scholar
  33. 33.
    Vinciarelli, A.: Capturing Order in Social Interactions. IEEE Signal Processing Magazine 26(5), 133–137 (2009)CrossRefGoogle Scholar
  34. 34.
    Vinciarelli, A., Pantic, M., Bourlard, H.: Social Signal Processing: Survey of an emerging domain. Image and Vision Computing Journal 27(12), 1743–1759 (2009)CrossRefGoogle Scholar
  35. 35.
    Vinciarelli, A., Pantic, M., Heylen, D., Pelachaud, C., Poggi, I., D’Errico, F., Schroeder, M.: Bridging the Gap Between Social Animal and Unsocial Machine: A Survey of Social Signal Processing. IEEE Transactions on Affective Computing 3(1), 69–87 (2012)CrossRefGoogle Scholar
  36. 36.
    Wharton, T.: The Pragmatics of Non-Verbal Communication. Cambridge University Press (2009)Google Scholar
  37. 37.
    World Economic Forum. Personal data: the emergence of a new asset class. Technical report, World Economic Forum (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Alessandro Vinciarelli
    • 1
    • 2
  1. 1.University of GlasgowGlasgowUK
  2. 2.Idiap Research InstituteMartignySwitzerland

Personalised recommendations