Multimedia Tools and Applications

, Volume 77, Issue 17, pp 21861–21877 | Cite as

Cloud-oriented emotion feedback-based Exergames framework

  • M. Shamim HossainEmail author
  • Ghulam Muhammad
  • Muhammad Al-Qurishi
  • Mehedi Masud
  • Ahmad Almogren
  • Wadood Abdul
  • Atif Alamri


Exergames platform can be more appealing to the users if they can interact with emotion. Therefore, in this paper, we propose an automatic emotion recognition system from speech to be embedded in the Exergames platform. While playing and doing exercise, the user expresses his or her feeling by uttering some phrases. The speech is recorded by an omnidirectional mic and transmitted to an emotion recognition server in a cloud environment, where the emotion (e.g. happy, sad or neutral) is recognized. For the recognition, we use MPEG-7 low-level audio features and a Gaussian mixture model based classifier. A tactile vibration is generated based on the emotion and feedback to the user for a real feeling. The user can thus have an instantaneous vibrational feeling based on his or her satisfaction. The recognized emotion can also be used as the user’s satisfaction of the framework on the fly without the need of a survey session. The experimental study and performance comparison show that the proposed framework has positive effects on the perception of physical activities.


Emotion recognition Obesity monitoring Exergames 



This work is financially supported by the King Saud University, Deanship of Scientific Research, Research Chair of Pervasive and Mobile Computing.


  1. 1.
    Alamri, A Hassan, MM Hossain, MA Al-Qurishi, M Aldukhayyil, Y Hossain, M.S. (2014) “Evaluating the impact of a cloud-based serious game on obese people,” Computers in Human Behavior, Vol. 30, pp. 468–475Google Scholar
  2. 2.
    Basori, AH Daman, D Bade, A Sunar, MS and Saari N (2008) “The feasibility of human haptic emotion as a feature to enhance interactivity and immersiveness on virtual reality game,” In Proc. VRCAI '08, SingaporeGoogle Scholar
  3. 3.
    Burkhardt, F Paeschke, A Rolfes, M Sendlmeier W and Weiss B (2005) “A database of German emotional speech”, Proc. Interspeech, 1517-1520, Lisbon, PortugalGoogle Scholar
  4. 4.
    Chen M, Zhang Y, Li Y, Mao S, Leung V (2015) EMC: emotion-aware mobile cloud computing in 5G. IEEE Netw 29(2):32–38CrossRefGoogle Scholar
  5. 5.
    Chen M, Ma Y, Hao Y, Li Y, Wu D, Zhang Y, Song E (2016) “CP-robot: cloud-assisted pillow robot for emotion sensing and interaction”, Industrialiot 2016. Guangzhou, ChinaGoogle Scholar
  6. 6.
    Ciota Z (2007), “Audio-haptic feedback in speech processing”, IEEE Intl. Workshop on Haptic Audio Visual Environments and their Applications (HAVE), pp. 67–70.Google Scholar
  7. 7.
    Ginsburg KR (2007) The importance of play in promoting healthy child development and maintaining strong parent-child bonds. Pediatrics 119(1):182–191CrossRefGoogle Scholar
  8. 8.
    Hassan MM, Hossain MA, Alamri A, Hossain MS, Yaldukhayyil Y, Al-Qurishi M (2013) Health Monitoring of Obese People Through a Cloud-based Serious Game Framework. in Proc. IEEE ICME, San JoseCrossRefGoogle Scholar
  9. 9.
    Hossain MS, Muhammad G (2016) Audio-visual emotion recognition using multi-directional regression and Ridgelet transform. Springer J Multimodal User Interfaces 10(4):325–333CrossRefGoogle Scholar
  10. 10.
    Hossain; M. S. Muhammad, G. "An emotion recognition system for mobile applications," IEEE Access, vol.PP, no.99, pp.1–1, doi: 10.1109/ACCESS.2017.2672829
  11. 11.
    Hossain MS, Hassan MM, Alamri A (2013) An Exergames Framework for Obesity Monitoring and Management, in Proc. IEEE HAVE’2013, IstanbulGoogle Scholar
  12. 12.
    Hossain MS, Muhammad G, Song B, Hassan MM, Alelaiwi A, Alamri A (2015) Audio–visual emotion-aware cloud gaming framework. IEEE Trans Circuits Syst Video Technol 25(12):2105–2118CrossRefGoogle Scholar
  13. 13.
    Hossain MS, Hardy S, Alamri A, Alelawi A, Hardy V, Wilhelm C (2016) AR-based serious games framework for post-stroke rehabilitation. ACM/Springer Multimed Syst 22(6):659–674CrossRefGoogle Scholar
  14. 14.
    Hossain MS, Muhammad G, Al Hamid MF, Song B (2016) Audio-visual emotion-aware big data recognition towards 5G. Mob Networks Appl 21(5):753–763CrossRefGoogle Scholar
  15. 15.
    Information Technology-Multimedia Content Description Interface-Part 4: Audio, ISO/IEC CD 15938–4, 2001Google Scholar
  16. 16.
    Karime A et al (2012) Tele-Wobble: A Telerehabilitation Wobble Board for Lower Extremity Therapy. IEEE Trans Instrum Meas 61(7):1816–1824CrossRefGoogle Scholar
  17. 17.
    Koolagudi SG, Rao KS (June 2012) Emotion recognition from speech: a review. Intern J Speech Technol 15(2):99–117CrossRefGoogle Scholar
  18. 18.
    Mateas, M et al. (2007) “presence and engagement in an interactive Drama”, in Proc. CHI’07 conference on computer human interaction (CHI), San Jose, CA, USA.Google Scholar
  19. 19.
    Muhammad G and Alghathbar K (2009), “Environment recognition from audio using MPEG-7 features,” Proc. The 4th International Conference on Embedded and Multimedia Computing (EM-Com09), S. Korea.Google Scholar
  20. 20.
    Muhammad G, Alghathbar K (2011) Environment recognition for digital audio forensics using MPEG–7 and Mel cepstral features. J Electr Eng 62(4):199–205Google Scholar
  21. 21.
    Parker JR, Baradoy G, Katz L Using virtual reality technology and biometric interfaces in obesity reduction. Can J Diabetes 35(2):187Google Scholar
  22. 22.
    Patsi C. et al. (2012) “Exergamess and their Effect on Emotional State in People with Schizophrenia “ Balkan Military Medical Review 15(4):275–281Google Scholar
  23. 23.
    Schuller, B Rigoll, G and Lang M (2004) “Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine - belief network architecture”, International Conf. Acoustics, Speech, and Signal Processing (ICASSP), pp. I-577-580.Google Scholar
  24. 24.
    Sinclair, J Hingston, P and Masek M (2007) “Considerations for the design of Exergamess,” In Proc. ACM GRAPHITE’07, Perth, AustraliaGoogle Scholar
  25. 25.
    Tadeusz Stach and T.C. Nicholas Graham (2011) “Exploring Haptic Feedback in Exergamess,” P. Campos et al. (Eds.): INTERACT 2011, Part II, LNCS 6947, pp. 18–35Google Scholar
  26. 26.
    Szczuko, P Dalka, P Dabrowski, M and Kostek B (2004) “MPEG-7-based Low-Level Descriptor Effectiveness in the Automatic Musical Sound Classification”, Proc. AES 116th Convention.Google Scholar
  27. 27.
    TU-Berlin MPEG-7 Audio Analyzer.
  28. 28.
    Ververidis D, Kotropoulos C (2006) Emotional speech recognition: resources, features, and methods. Speech Comm 48(9):1162–1181CrossRefGoogle Scholar
  29. 29.
    Warburton D, Bredin S, Horita L, Zbogar D, Scott J, Esch B et al (2007) The health benefits of interactive video game exercise. Appl Physiol Nutr Metab 32(4):655–663CrossRefGoogle Scholar
  30. 30.
    WHO Obesity and overweight fact sheet. Retrieved Oct 2016
  31. 31.
    Yim J, Graham TCN (2007) Using games to increase exercise motivation. Proc. Future Play, In, pp 166–173Google Scholar
  32. 32.
    Zacharatos H (2013) Emotion recognition for Exergamess using Laban movement analysis. In Proc. ACM SIGGRAPH conference on Motion in Games, Dublin, UKGoogle Scholar
  33. 33.
    Zhang Y (2016) Grorec: a group-centric intelligent recommender system integrating social, mobile and big data technologies. IEEE Trans Serv Comput 9(5):786–795CrossRefGoogle Scholar
  34. 34.
    Zhang Y et al (2017) iDoctor: personalized and professionalized medical recommendations based on hybrid matrix factorization. Futur Gener Comput Syst 66:30–35CrossRefGoogle Scholar
  35. 35.
    Y Zhou, Y Sun, J Zhang, and Y Yan (2009) "Speech Emotion Recognition Using Both Spectral and Prosodic Features", International Conf. Information Engineering and Computer Science (ICIECS), 1–4.Google Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  • M. Shamim Hossain
    • 1
    Email author
  • Ghulam Muhammad
    • 2
  • Muhammad Al-Qurishi
    • 3
  • Mehedi Masud
    • 4
  • Ahmad Almogren
    • 3
  • Wadood Abdul
    • 2
    • 3
  • Atif Alamri
    • 1
    • 3
  1. 1.Department of Software Engineering, College of Computer and Information Sciences (CCIS)King Saud UniversityRiyadhSaudi Arabia
  2. 2.Department of Computer Engineering, College of Computer and Information Sciences (CCIS)King Saud UniversityRiyadhSaudi Arabia
  3. 3.Research Chair of Pervasive and Mobile ComputingKing Saud UniversityRiyadhSaudi Arabia
  4. 4.Department of Computer ScienceTaif UniversityTaifSaudi Arabia

Personalised recommendations