Applying Emotion Recognition to Graphic Design Research

  • Chia-yin YuEmail author
  • Chih-hsiang Ko


We derive useful information about the feelings and inner states of other human beings by looking at the expressions on their faces. In this study, we conducted an experiment on the identification of emotions based on facial expression recognition to assess the participants’ emotional responses to different types of graphics of Chinese dragons. We used a paired sample t-test to analyze valid data collected from 112 participants. The results of our analysis indicate that there were differences between the participants’ expressions for a neutral response, happiness, sadness, and disgust. The effects of generalizing the participants’ facial expressions and associated verbalizations indicated that personalized designs aroused positive emotions, while the participants had more ideas associated with Chinese dragons when presented with realistic drawings. Furthermore, we found that the participants’ facial expressions were consistent with their verbal responses. This research demonstrates the importance and practicality of using facial expression recognition to assess peoples’ emotional responses to different graphic styles. Our results will help designers to build emotional communication with consumers and link their designs to consumers’ interests when promoting marketing activities.


  1. 1.
    Aggarwal P, Mcgill AL (2007) Is that car smiling at me? Schema congruity as a basis for evaluating anthropomorphized products. J Consum Res 34:468–479CrossRefGoogle Scholar
  2. 2.
    Asthana HS (2012) Facial asymmetry in expression of emotion: a neurocultural perspective. Concept Publishing Company, New DelhiGoogle Scholar
  3. 3.
    Brunswik E (1955) Representative design and probabilistic theory in a functional psychology. Psychol Rev 62:193–217CrossRefGoogle Scholar
  4. 4.
    Carretié L, Hinojosa JA, Albert J, Mercado F (2006) Neural response to sustained affective visual stimulation using an indirect task. Exp Brain Res 174:630–637CrossRefGoogle Scholar
  5. 5.
    Carretié L, Martín-Loeches M, Hinojosa JA, Mercado F (2001) Emotion and attention interaction studied through event-related potentials. J Cogn Neurosci 13:1109–1128CrossRefGoogle Scholar
  6. 6.
    Cohen AS, Morrison SC, Callaway DA (2013) Computerized facial analysis for understanding constricted/blunted affect: Initial feasibility, reliability, and validity data. Schizophr Res 148:111–116CrossRefGoogle Scholar
  7. 7.
    Cuthbert BN, Schupp HT, Bradley MM, Birbaumer N, Lang PJ (2000) Brain potentials in affective picture processing: covariation with autonomic arousal and affective report. Biol Psychol 52:95–111CrossRefGoogle Scholar
  8. 8.
    Danner L, Haindl S, Joechl M, Duerrschmid K (2014) Facial expressions and autonomous nervous system responses elicited by tasting different juices. Food Res Int 64:81–90CrossRefGoogle Scholar
  9. 9.
    Delplanque S, Silvert L, Hot P, Rigoulot S, Sequeira H (2006) Arousal and valence effects on event-related P3a and P3b during emotional categorization. Int J Psychophysiol 60:315–322CrossRefGoogle Scholar
  10. 10.
    Dunn SW, Barban AM (1986) Advertising: its role in modern marketing. Dryden Press, Fort WorthGoogle Scholar
  11. 11.
    Ekman P (2003) Emotions revealed—recognizing faces and feelings to improve communication and emotional life. Times Books, New YorkGoogle Scholar
  12. 12.
    Ekman P, Friesen WV (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo AltoGoogle Scholar
  13. 13.
    Ekman P, Friesen WV, Hager JC (2002a) Facial action coding system—investigator’s guide. Research Nexus, Salt Lake CityGoogle Scholar
  14. 14.
    Ekman P, Friesen WV, Hager JC (2002b) Facial action coding system—the manual. Research Nexus (a subsidiary of Network Information Research Corporation), Salt Lake CityGoogle Scholar
  15. 15.
    Essa IA, Pentland AP (1997) Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans Pattern Anal Mach Intell 19:757–763CrossRefGoogle Scholar
  16. 16.
    Fridlund AJ (2014) Human facial expression: an evolutionary view. Academic Press, LondonGoogle Scholar
  17. 17.
    Hazlett R (2006) Measuring emotional valence during interactive experience: boys at video game play. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM Press, New YorkGoogle Scholar
  18. 18.
    Heath R, Brandt D, Nairn A (2006) Brand relationships: strengthened by emotion, weakened by attention. J Advert Res 46:410–419CrossRefGoogle Scholar
  19. 19.
    Heath R, Hyder P (2005) Measuring the hidden power of emotive advertising. J Advert Res 47:467–486Google Scholar
  20. 20.
    Hill D (2010) Emotionomics: leveraging emotions for business success. Kogan Page, LondonGoogle Scholar
  21. 21.
    Hirschman EC, Holbrook MB (1982) Hedonic consumption: emerging concepts, methods and propositions. J Market 46:92–101CrossRefGoogle Scholar
  22. 22.
    Hot P, Saito Y, Mandai O, Kobayashi T, Sequeira H (2006) An ERP investigation of emotional processing in European and Japanese individuals. Brain Res 1122:171–178CrossRefGoogle Scholar
  23. 23.
    I B, Lin YC (2006) Eye movements in image viewing processing. J Design 11:59–80Google Scholar
  24. 24.
    Juslin PN, Persson RS (2002) Emotional communication. In: Parncutt R, McPherson G (eds) The science and psychology of music performance: creative strategies for teaching and learning. Oxford University Press, OxfordGoogle Scholar
  25. 25.
    Keil A, Bradley MM, Hauk O, Rockstroh B, Elbert T, Lang PJ (2002) Large-scale neural correlates of affective picture processing. Psychophysiology 39:641–649CrossRefGoogle Scholar
  26. 26.
    Kemper TD, Ekman P (1981) The face of man: expressions of universal emotions in a New Guinea village. Contemp Sociol 10(4):521CrossRefGoogle Scholar
  27. 27.
    Kirk J, Jahoda A (2009) The role of emotional imagery and somatosensory amplification in atypical chest pain in patients with angina pectoris: a single-case experimental design. J Cardiopulmonary Rehabil Prevent 29:121–125CrossRefGoogle Scholar
  28. 28.
    Lajevardi SM, Wu HR (2012) Facial expression recognition in perceptual color space. IEEE Trans Image Process 21:3721–3733MathSciNetCrossRefGoogle Scholar
  29. 29.
    Lau BT (2009) Portable real time needs expression for people with communication disabilities. In: Versick D (ed) Proceedings of communication in computer and information science: intelligent interactive assistance and mobile multimedia computing. Springer, Berlin, pp 85–95Google Scholar
  30. 30.
    Loijens L, Krips O (2013) FaceReader methodology. Noldus Information Technology, WageningenGoogle Scholar
  31. 31.
    Mahike S, Minge M, Thüring M (2006) Measuring multiple components of emotions in interactive contexts. In: Work in progress CHI 2006. ACM Press, New YorkGoogle Scholar
  32. 32.
    Mahrabian A (1981) Silent messages. Wadsworth, BelmontGoogle Scholar
  33. 33.
    Matsunaga M, Isowa T, Kimura K, Miyakoshi M, Kanayama N, Murakami H, Fukuyama S, Shinoda J, Yamada J, Konagaya T, Kaneko H, Ohira H (2009) Associations among positive mood, brain, and cardiovascular activities in an affectively positive situation. Brain Res 1263:93–103CrossRefGoogle Scholar
  34. 34.
    Mitterschiffthaler MT, Fu CHY, Dalton JA, Andrew CM, Williams SCR (2007) A functional MRI study of happy and sad affective states induced by classical music. Human Brain Mapp 28:1150–1162CrossRefGoogle Scholar
  35. 35.
    Motterlini M (2008) Economia emotiva: Che cosa si nasconde dietro i nostri conti quotidiani. Milano, RizzoliGoogle Scholar
  36. 36.
    Norman DA (2004) Emotional design: why we love (or hate) everyday things. Basic Books, New YorkCrossRefGoogle Scholar
  37. 37.
    Norris JS (1990) Advertising. Prentice Hall, New JerseyGoogle Scholar
  38. 38.
    Olofsson JK, Nordin S, Sequeira H, Polich J (2008) Affective picture processing: an integrative review of ERP findings. Biol Psychol 77:247–265CrossRefGoogle Scholar
  39. 39.
    Ozkaramanli D, Desmet PMA (2012) I knew i shouldn’t, yet i did it again! Emotion-driven Design as a means to motivate subjective well-being. Int J Design 6:27–39Google Scholar
  40. 40.
    Pantic M, Patras I (2006) Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans Syst Man Cybern Part B 36:433–449CrossRefGoogle Scholar
  41. 41.
    Planalp S (1999) Communicating emotion: social, moral, and cultural processes. Cambridge University Press, New YorkCrossRefGoogle Scholar
  42. 42.
    Pollak SD, Messner M, Kistler DJ, Cohn JF (2009) Development of perceptual expertise in emotion recognition. Cognition 110:242–247CrossRefGoogle Scholar
  43. 43.
    Pos OD, Green-Armytage P (2007) Facial expression, colours and basic emotions. Colour Design Creativity 1(2):1–20Google Scholar
  44. 44.
    Ray A, Chakrabarti A (2012) Design and implementation of affective e-learning strategy based on facial emotion recognition. In: Satapathy SC, Avadhani PS, Abraham A (eds) International conference on information systems design and intelligent applications, Visakhapatnam, India, pp 613–622Google Scholar
  45. 45.
    Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychol Bull 124:372–422CrossRefGoogle Scholar
  46. 46.
    Reeve J, Nix G (1997) Expressing intrinsic motivation through acts of exploration and facial displays of interest. Motiv Emot 21:237–250CrossRefGoogle Scholar
  47. 47.
    Ricoeur P (1977) The rule of metaphor: multi-disciplinary studies of the creation of meaning in language. University of Toronto Press, TorontoGoogle Scholar
  48. 48.
    Rinn WE (1984) The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions. Psychol Bull 95:52–77CrossRefGoogle Scholar
  49. 49.
    Schachter S, Singer JE (1962) Cognitive, social, and physiological determinants of emotional state. Psychol Rev 69:379–399CrossRefGoogle Scholar
  50. 50.
    Scherer KR, Clark-Polner E, Mortillaro M (2011) In the eye of the beholder? Universality and cultural specificity in the expression and perception of emotion. Int J Psychol 46:401–435CrossRefGoogle Scholar
  51. 51.
    Song KT, Han MJ, Hong JW (2010) Online learning design of an image-based facial expression recognition system. Intel Serv Robot 3:151–162CrossRefGoogle Scholar
  52. 52.
    Wolf B (2005) Brunswik’s original lens model. Available: Accessed 20 May 2013
  53. 53.
    Yu CY, Ko CH (2015) The application of facial expression recognition to the evaluation of stylized images. J Design Sci 18Google Scholar
  54. 54.
    Yu CY, Ko CH (2017) Applying FaceReader to recognize consumer emotions in graphic styles. Proc CIRP 60:104–109CrossRefGoogle Scholar
  55. 55.
    Zaman B, Shrimpton-Smith T (2006) The FaceReader: measuring instant fun of use. In: Nordic conference on human-computer interaction 2006, Olso, NorwayGoogle Scholar
  56. 56.
    Zhan Y, Zhou G (2007) Facial expression recognition based on hybrid features and fusing discrete HMM. In: Shumaker R (ed) Virtual reality, HCII 2007, (LNCS 4563). Springer, BerlinGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Applied ArtsFu Jen Catholic UniversityXinzhuang District, New Taipei CityTaiwan
  2. 2.Department of DesignNational Taiwan University of Science and TechnologyTaipei CityTaiwan

Personalised recommendations