Skip to main content

Advertisement

SpringerLink
Log in

Towards pervasive geospatial affect perception

  • Published:
GeoInformatica Aims and scope Submit manuscript

Abstract

Due to the enormous penetration of connected computing devices with diverse sensing and localization capabilities, a good fraction of an individual’s activities, locations, and social connections can be sensed and spatially pinpointed. We see significant potential to advance the field of personal activity sensing and tracking beyond its current state of simple activities, at the same time linking activities geospatially. We investigate the detection of sentiment from environmental, on-body and smartphone sensors and propose an affect map as an interface to accumulate and interpret data about emotion and mood from diverse set of sensing sources. In this paper, we first survey existing work on affect sensing and geospatial systems, before presenting a taxonomy of large-scale affect sensing. We discuss model relationships among human emotions and geo-spaces using networks, apply clustering algorithms to the networks and visualize clusters on a map considering space, time and mobility. For the recognition of emotion and mood, we report from two studies exploiting environmental and on-body sensors. Thereafter, we propose a framework for large-scale affect sensing and discuss challenges and open issues for future work.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. http://csea.phhp.ufl.edu/media.html.

  2. Already a naive calculation of, for instance, 10 sensors with a typical 40Hz sampling rate (simple estimation with 1Byte/sample) will result in about 1.5M B/hour/individual which is not manageable when aggregated from multiple subjects on mobile and on-body devices.

  3. https://www.youtube.com/watch?v=lyVctz5BAro.

  4. http://www.cs.waikato.ac.nz/ml/weka/.

References

  1. Abdelnasser H, Youssef M, Harras KA (2015) WiGest: A Ubiquitous WiFi-based Gesture Recognition System. ArXiv e-prints

  2. Adib F, Katabi D (2013) See through Walls with Wifi!. SIGCOMM Comput Commun Rev 43(4):75–86

    Article  Google Scholar 

  3. Nguyen A-T, Chen W, Rauterberg M (2012) The role of human body expression in affect detection: A review. In: 10th Asia Pacific Conference on Computer Human Interaction (APCHI 2012)

  4. Ark WS, Dryer DC, Lu DJ (1999) The emotion mouse. In: HCI (1), pp 818–823

  5. Bachmann A, Klebsattel C, Budde M, Riedel T, Beigl M, Reichert M, Santangelo P, Ebner-Priemer U (2015) How to use smartphones for less obtrusive ambulatory mood assessment and mood recognition. In: Adjunct Proceedings of Ubicomp’15. ACM, pp 693–702

  6. Bachmann A, Klebsattel C, Schankin A, Riedel T, Beigl M, Reichert M, Santangelo P, Ebner-Priemer U (2015) Leveraging smartwatches for unobtrusive mobile ambulatory mood assessment. In: Adjunct Proceedings of Ubicomp’15. ACM, pp 1057–1062

  7. Bao X, Fan S, Varshavsky A, Li K, Roy Choudhury R (2013) Your reactions suggest you liked the movie: Automatic content rating via reaction sensing. In: UbiComp

  8. Jackson B, Brennis L-W (2000) The pupillary system. Cambridge University Press, pp 142–162

  9. Beedie CJ, Terry PC, Lane AM, Devonport TJ (2011) Differential assessment of emotions and moods: Development and validation of the emotion and mood components of anxiety questionnaire. Personal Individ Differ 50(2):228–233

    Article  Google Scholar 

  10. Bideaux A, Anastasopoulou P, Hey S, Canadas A, Fernandez A (2014) Mobile monitoring of epileptic patients using a reconfigurable cyberphysical system that handles multi-parametric data acquisition and analysis. In: 2014 EAI 4th International Conference on Wireless Mobile Communication and Healthcare (Mobihealth). IEEE, pp 377–380

  11. Bradley MM, Lang PJ (1994) Measuring Emotion: the Self-Assessment Manikin and the Semantic Differential. J Behav Ther Exp Psychiatry 25(1):49–59

    Article  Google Scholar 

  12. Burgoon JK, Guerrero LK, Floyd K (2010) Nonverbal communication. Allyn & Bacon, Boston, MA

    Google Scholar 

  13. Calvo MG, Lang PJ (2004) Gaze patterns when looking at emotional pictures: Motivationally biased attention. Motiv Emot 28(3):221–243

    Article  Google Scholar 

  14. Cambria E, Schuller B, Xia Y, Havasi C (2013) New avenues in opinion mining and sentiment analysis. IEEE Intelligent Systems, pp. 1

  15. Canzian L, Musolesi M (2015) Trajectories of depression: Unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis. International Joint Conference on Pervasive and Ubiquitous Computing, pp 1293–1304

  16. Cardone G, Cirri A, Corradi A, Foschini L, Ianniello R, Montanari R (2014) Crowdsensing in urban areas for city-scale mass gathering management: Geofencing and activity recognition. Sensors pp. 4185–4195

  17. Chen M, Gonzalez S, Vasilakos A, Cao H, Leung VC (2011) Body area networks: A survey. Mob Netw Appl 16(2):171–193

    Article  Google Scholar 

  18. Coan JA, Allen JJ (2007) Handbook of emotion elicitation and assessment. Oxford University Press

  19. Collet C, Vernet-Maury E, Delhomme G, Dittmar A (1997) Autonomic nervous system response patterns specificity to basic emotions. J Autonom Nerv Syst 62(1-2):45–57

    Article  Google Scholar 

  20. Coviello L, Sohn Y, Kramer ADI, Marlow C, Franceschetti M, Christakis NA, Fowler JH Detecting emotional contagion in massive social networks. PLoS ONE (2014). doi:10.1371/journal.pone.0090315

  21. Crane E, Gross M (2007) Motion capture and emotion: Affect detection in whole body movement. In: Affective computing and intelligent interaction. Springer, pp 95–101

  22. Dalgleish T, Power MJ (1999) Handbook of cognition and emotion. Wiley Online

  23. Dawson ME, Schell AM, Bohmelt AH (2008) Startle modification: Implications for neuroscience, cognitive science, and clinical science. Cambridge University Press

  24. DePasquale JP, Geller E, Clarke SW, Littleton LC (2001) Measuring road rage: development of the propensity for angry driving scale. J Saf Res 32(1):1–16. doi:10.1016/S0022-4375(00)00050-5

    Article  Google Scholar 

  25. Dichter GS, Tomarken AJ, Baucom BR (2002) Startle modulation before, during and after exposure to emotional stimuli. Int J Psychophysiol 43(2):191–196

    Article  Google Scholar 

  26. Van Dongen S (2000) Graph clustering by flow simulation. Ph.D. thesis, University of Utrecht

  27. Ekman P (1992) Are there basic emotions?

  28. El Mawass N, Kanjo E (2013) A supermarket stress map. In: Ubicomp’13 adjunct Proceedings. ACM, pp 1043–1046

  29. Exler A, Klebsattel C, Schankin A, Beigl M (2016) A wearable system for mood assessment considering smartphone features and mobile ecg measurements. In: Adjunct Proceedings of Ubicomp’16. ACM. To appear

  30. Fisher CD, To ML (2012) Using experience sampling methodology in organizational behavior. J Organ Behav 33(7):865–877

    Article  Google Scholar 

  31. Eyben F, W?llmer M, Poitschke T (2010) Emotion on the road?necessity, acceptance, and feasibility of affective computing in the car. Advances in Human-Computer Interaction p 17. doi:10.1155/2010/263593

  32. De Gelder B, Hortensius R (2009) Why bodies? Twelve reasons for including bodily expressions in affective neuroscience. Philos Trans Royal Soc 364:3475–3484

    Article  Google Scholar 

  33. Geller T (2014) How do you feel?: Your computer knows. Commun ACM 57 (1):24–26

    Article  Google Scholar 

  34. Giaccardi E, Fogli D (2008) Affective Geographies: Toward Richer Cartographic Semantics for the Geospatial Web. In: Proceedings of the working conference on Advanced visual interfaces (AVI ’08), pp 173–180

  35. Granholm E, Steinhauer SR (2004) Int J Psychophysiol 52(1):1–6. Pupillometric Measures of Cognitive and Emotional Processes

    Article  Google Scholar 

  36. Grunerbl A, Muaremi A, Osmani V, Bahle G, Ohler S, Troster G, Mayora O, Haring C, Lukowicz P (2014) Smartphone-based recognition of states and state changes in bipolar disorder patients. IEEE J Biomed Health Inf 19(1):140–148

    Article  Google Scholar 

  37. Hatfield E, Cacioppo J, Rapson R (1993) Emotional contagion. Current directions. Psychol Sci 2:96–99. doi:10.1111/1467-8721.ep10770953

    Google Scholar 

  38. Healey J, Picard R (2000) Smartcar: Detecting driver stress. In: Pattern Recognition. 15th International Conference on 2000. Proceedings, vol 4, pp 218–221. doi:10.1109/ICPR.2000.902898

  39. Hemminki S, Zhao K, Ding AY, Rannanjarvi M, Tarkoma S, Nurmi P (2013) Cosense: A collaborative sensing platform for mobile devices. Embedded Networked Sensor Systems, p 34

  40. Herdem KC (2012) Reactions: Twitter based mobile application for awareness of friends’ emotions. In: UbiComp’12. ACM

  41. Hernandez J, McDuff DJ, Picard RW (2015) Biophone: Physiology monitoring from peripheral smartphone motions. Engineering in Medicine and Biology Society

  42. Jercic P, Astor PJ, Adam MTP, Hilborn O, Schaaff K, Lindley C, Sennersten C, Eriksson J (2012) A serious game using physiological interfaces for emotion regulation training in the context of financial decision-making. In: ECIS, p 207

  43. Joint M (1995) Road rage

  44. Knighten J, McMillan S, Chambers T, Payton J (2015) Recognizing social gestures with a Wrist-Worn Smartband. Pervasive Computing and Communication, pp 544–549

  45. Konomi S (2011) Colocation networks: exploring the use of social andgeographical patterns in context-aware services. In: UbiComp’11 adjunct proceedings. ACM, pp 565–566

  46. Koolagudi S, Rao K (2012) Emotion recognition from speech: a Review. Int. J. Speech Technol. 15(2):99–117

    Article  Google Scholar 

  47. Kramer AD, Guillory JE, Hancock JT (2014) Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences, p 201320040

  48. Kramer ADI, Guillory JE, Hancock JT (2014) Experimental evidence of massive-scale emotional contagion through social networks. Proc Natl Acad Sci 111(24)

  49. Lane N, Miluzzo E, Lu H, Peebles D, Choudhury T, Campbell A (2010) A survey of mobile phone sensin. Commun Mag 48:140–150

    Article  Google Scholar 

  50. Lang PJ, Greenwald MK, Bradley MM, Hamm AO (1993) Looking at pictures: Affective, Facial, Visceral, and behavioral reactions. Psychophysiology, pp 161–273

  51. Lee H, Choi YS, Lee S, Park I (2012) Towards unobtrusive emotion recognition for affective social communication. In: CCNC’12

  52. Leng H, Lin Y, Zanzi LA (2007) An experimental study on physiological parameters toward driver emotion recognition. Springer Berlin Heidelberg, Berlin, Heidelberg, pp 237–246. doi:10.1007/978-3-540-73333-1_30

    Google Scholar 

  53. LiKamWa R, Liu Y, Lane ND, Zhong L (2013) Moodscope: Building a mood sensor from smartphone usage patterns. In: MobiSys 2013, MobiSys ’13. ACM

  54. Mackerron G, Mourato S (2013) Happiness is greater in natural environments. Glob Environ Chang 23(5):992–1000

    Article  Google Scholar 

  55. Mehrabian A (1996) Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Curr Psychol 14(4):261–292

    Article  Google Scholar 

  56. Mody RN, Willis KS, Kerstein R (2009) Wimo: location-based emotion tagging. Mobile and Ubiquitous Multimedia

  57. Myrtek M, Aschenbrenner E, Brügner G. (2005) Emotions in everyday life: An ambulatory monitoring study with female students. Biol Psychol 68(3):237–55

    Article  Google Scholar 

  58. Nardelli M, Valenza G, Greco A, Lanata A, Scilingo E (2015) Recognizing emotions induced by affective sounds through heart rate variability. IEEE Trans Affect Comput PP(99):1–1

    Google Scholar 

  59. Nold C (2004) Bio mapping

  60. Nurmi P, Koolwaaij J (2006) Identifying meaningful locations. Mobile and Ubiquitous Systems: Networks and Services, pp 1–8

  61. De Oliveira THM (2015) The emotion-aware city: using ambient geographic information (agi) in order to understand emotion and stress within smart cities. In: AGILE PhD School

  62. Paltoglou G, Thelwall M (2012) Twitter, myspace, digg: Unsupervised sentiment analysis in social media. ACM Trans Intell Syst Technol 3(4):66:1–66:19

    Article  Google Scholar 

  63. Pánek J., Pászto V., Marek L (2017) Mapping emotions: Spatial distribution of safety perception in the city of Olomouc. Springer International Publishing, Cham, pp 211–224

  64. Picard RW (2015) Recognizing stress, engagement, and positive emotion. In: 20th International Conference on Intelligent User Interfaces. ACM, pp 3–4

  65. Prusa JD, Khoshgoftaar TM, Dittmann DJ (2015) Impact of feature selection techniques for tweet sentiment classification. In: Twenty-Eighth International Florida Artificial Intelligence Research Society Conference

  66. Pu Q, Gupta S, Gollakota S, Patel S (2013) Whole-home gesture recognition using wireless signals. In: 19th Annual International Conference on Mobile Computing and Networking, MobiCom ’13, pp 27–38

  67. Purves D, Augustine GJ, Fitzpatrick D, Katz LC, LaMantia AS, McNamara JO, Williams SM (2001) Physiological changes associated with emotion

  68. Raja M, Sigg S (2016) Applicability of rf-based methods for emotion recognition: A survey. In: IEEE PerCom’16 adjunct proceedings

  69. Russell JA (1980) A circumplex model of affect. J Pers Soc Psychol 39(6):1161

    Article  Google Scholar 

  70. Samuli H, Petteri N, Sasu T (2013) Accelerometer-based transportation mode detection on smartphones. Embedded Networked Sensor Systems (SenSys)

  71. Scherer KR (2005) What are emotions? And how can they be measured? Soc Sci Inf 44(4):695–729

    Article  Google Scholar 

  72. Schimmack U, Grob A (2000) Dimensional models of core affect: A quantitative comparison by means of structural equation modeling. Eur J Personal 14(4):325–345

    Article  Google Scholar 

  73. Schroeder M, Cowie R (2006) Issues in emotion-oriented computing - towards a shared understanding. In: Proceedings Workshop on Emotion and Computing at KI 2006

  74. Sekimoto Y, Shibasaki R, Kanasugi H, Usui T, Shimazaki Y (2011) PFlow: Reconstructing people flow recycling large-scale social survey data. IEEE Pervasive Comput 10(4):27–35

    Article  Google Scholar 

  75. Sigg S, Blanke U, Troster G (2014) The telepathic phone: Frictionless activity recognition from wifi-rssi. In: 2014 IEEE International Conference on Pervasive Computing and Communications (PerCom), pp 148–155

  76. Sigg S, Scholz M, Shi S, Ji Y, Beigl M (2014) Rf-sensing of activities from non-cooperative subjects in device-free recognition systems using ambient and local signals. IEEE Trans Mob Comput 13(4):907–920

    Article  Google Scholar 

  77. Stanners RF, Coulter M, Sweet AW, Murphy P (1979) The pupillary response as an indicator of arousal and cognition. Motiv Emot 3(4):319–340

    Article  Google Scholar 

  78. Steyer R, Schwenkmezger P, Notz P, Eid M (1997) Mdbf-mehrdimensionaler befindlichkeitsfragebogen [multidimensional mood questionnaire]

  79. Sun FT, Kuo C, Cheng HT, Buthpitiya S, Collins P, Griss M (2012) Activity-aware mental stress detection using physiological sensors. Lect Notes Inst Comput Sci Soc Inf 72:282–301

    Google Scholar 

  80. Tan CSS, Schöning J, Luyten K, Coninx K (2013) Informing intelligent user interfaces by inferring affective states from body postures in ubiquitous computing environments. In: International Conference on Intelligent user interfaces. ACM, pp 235–246

  81. Valenza G, Citi L, Lanatá A, Scilingo EP, Barbieri R (2014) Revealing real-time emotional responses: A personalized assessment based on heartbeat dynamics. Scientific reports, p 4

  82. Venture G, Kadone H, Zhang T, Grèzes J., Berthoz A, Hicheur H (2014) Recognizing emotions conveyed by Human Gait. International Journal of Social Robotics

  83. Verheyen C, Göritz A.S. (2009) Plain texts as an online mood-induction procedure. Soc Psychol 40(1):6–15

    Article  Google Scholar 

  84. Wang H, Zhang D, Ma J, Wang Y, Wang Y, Wu D, Gu T (2016) Human respiration detection with commodity wifi devices: Do user location and body orientation matter? In: Ubicomp 2016. ACM

  85. Wang W, Chen L, Thirunarayan K, Sheth A Harnessing twitter ‘big data’ for automatic emotion identification

  86. Wang W, Liu A, Shahzad M (2016) Gait recognition using wifi signals. In: Ubicomp 2016. ACM

  87. Wilhelm P, Schoebi D (2007) Assessing mood in daily life. Eur J Psychol Assess 23(4):258–267

    Article  Google Scholar 

  88. Wilhelm P, Schoebi D (2007) Assessing mood in daily life: Structural validity, Sensitivity to change, and reliability of a short-scale to measure three basic dimensions of mood. Eur J Psychol Assess 23(4):258

    Article  Google Scholar 

  89. Wu D, Zhang D, Xu C, Wang Y, Wang H (2016) Widir: Walking direction estimation using wireless signals. In: Ubicomp 2016. ACM

  90. Zheng BS, Murugappan M, Yaacob S (2013) Fcm clustering of emotional stress using ecg features. In: 2013 International Conference on Communications and Signal Processing (ICCSP). IEEE, pp 305–309

Download references

Acknowledgments

We are thankful to Christoph Klebsattel for his support in conducting the experiments on the mood induction and recognition. We would further like to thank Dr. Andrea Schankin for the fruitful discussions about affective states, their characteristics and assessment methods. We are also thankful to Syed Safi Ali Shah for sharing his knowledge in DFAR research.

Author information

Authors and Affiliations

  1. Aalto University, Espoo, Finland

    Muneeba Raja & Stephan Sigg

  2. Karlsruhe Insitute of Technology, Karlsruhe, Germany

    Anja Exler

  3. Helsinki University, Helsinki, Finland

    Samuli Hemminki

  4. The University of Tokyo, Tokyo, Japan

    Shin’ichi Konomi

  5. Kyushu Institute of Technology, Fukuoka, Japan

    Sozo Inoue

Authors
  1. Muneeba Raja
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anja Exler
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Samuli Hemminki
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shin’ichi Konomi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stephan Sigg
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sozo Inoue
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Samuli Hemminki.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Raja, M., Exler, A., Hemminki, S. et al. Towards pervasive geospatial affect perception. Geoinformatica 22, 143–169 (2018). https://doi.org/10.1007/s10707-017-0294-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10707-017-0294-1

Keywords

Search

Navigation