Identification of an Individual’s Frustration in the Work Environment Through a Multi-sensor Computer Mouse

  • David Portugal
  • Marios BelkEmail author
  • João Quintas
  • Eleni Christodoulou
  • George Samaras
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9755)


Older adults traditionally face major challenges at work when it comes to dealing with new technological tools. A sense of overwhelm and frustration can quickly arise under these circumstances. Continuous negative feelings in the work environment may lead to the increase of the risks for cognitive decline and threaten independence and quality of life. In this work, we focus on the seamless identification of frustration of older adults at work via physiological sensors embedded in an in-house developed computer mouse, denoted as CogniMouse. For the purpose of this research, we have developed a probabilistic classification algorithm that receives real-time signals and physiological measurement streams as input, and accordingly identifies frustration events. Ultimately, such classification can be leveraged to deliver user interventions and personalized solutions to help reduce user frustration.


Active assisted living Intelligent mouse Physiological sensors Cognitive support 



This work was partially carried out in the frame of the CogniWin project (, funded by the EU Ambient Assisted Living Joint Program (AAL 2013-6-114).


  1. 1.
    Pantic, M., Pentland, A., Nijholt, A., Huang, T.S.: Human computing and machine understanding of human behavior: a survey. In: Huang, T.S., Nijholt, A., Pantic, M., Pentland, A. (eds.) ICMI/IJCAI Workshops 2007. LNCS (LNAI), vol. 4451, pp. 47–71. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  2. 2.
    Camarinha-Matos, L.M., Afsarmanesh, H.: Virtual communities and elderly support. In: Advances in Automation, Multimedia and Video Systems, and Modern Computer Science, pp. 279–284 (2001)Google Scholar
  3. 3.
    Hanke, S., et al.: CogniWin – a virtual assistance system for older adults at work. In: Zhou, J., Salvendy, G. (eds.) ITAP 2015. LNCS, vol. 9194, pp. 257–268. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  4. 4.
    Jansen, F., Nielsen, T.: Bayesian Networks and Decision Graphs. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  5. 5.
    Lawson, R.: Frustration: The Development of a Scientific Concept. Macmillan, New York (1965)Google Scholar
  6. 6.
    Grafsgaard, J.F., Wiggins, J.B., Boyer, K.E., Wiebe, E.N., Lester, J.C.: Automatically recognizing facial indicators of frustration: a learning-centric analysis. In: IEEE Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), pp. 159–165, Geneva, Switzerland, 2–5 September 2013Google Scholar
  7. 7.
    Qi, Y., Reynolds, C., Picard, R.W.: The bayes point machine for computer-user frustration detection via pressure mouse. In: Workshop on Perceptive user interfaces (PUI 2001). ACM, New York (2001)Google Scholar
  8. 8.
    Rodrigo, M.M., Baker, R.S.: Coarse-grained detection of student frustration in an introductory programming course. In: 5th International Workshop on Computing Education Research (ICER 2009), pp. 75–80, Berkeley, California, 10–11 August 2009Google Scholar
  9. 9.
    Harrison, L., Dou, W., Lu, A., Ribarsky, W., Wang, X.: Analysts aren’t machines: inferring frustration through visualization interaction. In: IEEE Conference on Visual Analytics Science and Technology (VAST 2011), pp. 279–280 (2001)Google Scholar
  10. 10.
    Boril, H., Sadjadi, S.O., Kleinschmidt, T., Hansen, J.: Analysis and detection of cognitive load and frustration in drivers’ speech. In: International Speech Communication Association (INTERSPEECH), pp. 502–505, Chiba, Makuhari, Japan (2010)Google Scholar
  11. 11.
    Belle, A., Ji, S.Y., Ansari, S., Hakimzadeh, R., Ward, K., Najarian, K.: Frustration detection with electrocardiograph signal using wavelet transform. In: IEEE International Conference on Biosciences (BIOSCIENCESWORLD), pp. 91–94, Cancun, Mexico, 7–13 March 2010Google Scholar
  12. 12.
    Kapoor, A., Burleson, W., Picard, R.W.: Automatic prediction of frustration. Int. J. Hum.-Comput. Stud. 65(8), 724–736 (2007)CrossRefGoogle Scholar
  13. 13.
    Noronha, H., Sol, R., Vourvopoulos, A.: Comparing the levels of frustration between an eye-tracker and a mouse: a pilot study. In: Holzinger, A., Ziefle, M., Hitz, M., Debevc, M. (eds.) SouthCHI 2013. LNCS, vol. 7946, pp. 107–121. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  14. 14.
    Taylor, B., Dey, A., Siewiorek, D., Smailagic, A.: Using physiological sensors to detect levels of user frustration induced by system delays. In: ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2015), pp. 517–528, Osaka, Japan, 7–11 September 2015Google Scholar
  15. 15.
    Gao, Y., Bianchi-Berthouze, N., Meng, H.: What does touch tell us about emotions in touchscreen-based gameplay? ACM Trans. Comput.-Hum. Interact. (TOCHI) 19(4), 31 (2012)Google Scholar
  16. 16.
    Belk, M., Portugal, D., Christodoulou, E., Samaras, G.: Cognimouse: on detecting users’ task completion difficulty through computer mouse interaction. In: Extended Abstracts on Human Factors in Computing Systems (CHI 2015), pp. 1019–1024, Seoul, South Korea, 18–23 April 2015Google Scholar
  17. 17.
    Aliakbarpour, H., Ferreira, J.F., Khoshhal, K., Dias, J.: A novel framework for data registration and data fusion in presence of multi-modal sensors. In: Camarinha-Matos, L.M., Pereira, P., Ribeiro, L. (eds.) DoCEIS 2010. IFIP AICT, vol. 314, pp. 308–315. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  18. 18.
    Rosenzweig, S.: The Rosenzweig Picture Frustration (P-F) Study. Rana House, St. Louis (1978)Google Scholar
  19. 19.
    Hart, S.G.: NASA-Task Load Index (NASA-TLX); 20 years later. In: Proceedings of the Human Factors and Ergonomics Society (HFES), vol. 50, no. 9, pp. 904–908. SAGE Publications, Santa Monica (2006)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • David Portugal
    • 1
  • Marios Belk
    • 1
    • 2
    Email author
  • João Quintas
    • 3
  • Eleni Christodoulou
    • 1
  • George Samaras
    • 1
    • 2
  1. 1.CiTARD Services Ltd.NicosiaCyprus
  2. 2.Department of Computer ScienceUniversity of CyprusNicosiaCyprus
  3. 3.Laboratory of Automatics and SystemsInstituto Pedro NunesCoimbraPortugal

Personalised recommendations