The Role of User Emotions for Content Personalization in e-Commerce: Literature Review

  • Artem BielozorovEmail author
  • Marija Bezbradica
  • Markus Helfert
Open Access
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11588)


Purchasing decisions do not always come from the rational mental processes but are often being driven by emotions. This insight made researchers think of emotions as of an essential contextual variable capable of enhancing personalized services and providing more precise recommendations within e-Commerce. In this paper we explore the studies made to discover why emotions are an important research domain necessary to understand purchasing behavior of online shoppers. We also explore how user emotions can be captured and recognized by existing technologies to provide enhanced personalization. Specifically, we apply Webster and Watson (2002) literature review approach to create a sample of studies published in scientific journals and conference proceedings. We synthesize the extant studies on the role of user emotions for personalized services within e-Commerce. We also provide a comprehensive concept-matrix which aggregates the range of existing emotions recognition technologies and highlights which specific emotions these technologies are able to recognize as well as in which domains these solutions are applied. Our study extends prior reviews and provides insights into open research areas which will benefit Human-Computer Interactions (HCI) practitioners and researchers in academia and industry.


Emotions recognition technologies Personalization Purchasing behavior Recommender system 

1 Introduction: Content Personalization as a Key Strategic Element

According to the definition given by Osterwalder and Pigneur (2002), business strategy represents an approach based on the profiles of actors, their dynamics and roles within a specific business model that determines certain operational-level steps necessary to be undertaken to implement the entire business strategy. In the context of e-Commerce, information strategy constitutes the vital part of the overall business strategy as it aims at discovering new and profitable insights to excel customer relationship and enhance customer satisfaction (Osterwalder and Pigneur 2002). The deep analysis of the customers’ preferences and behaviors is among the key information strategy goals and the very success of this strategy relies on how thorough and accurate the customers’ data are and how efficiently online vendors can react upon these data in order to provide value propositions based on the data-driven insights (Salonen and Karjaluoto 2016). This approach, however, requires fostering more customer-centric business culture and prioritizing the alignment of the company’s information and analytic strategy, and business capabilities with the customers’ needs (Simon et al. 2016).

To stay competitive, e-Commerce players have been developing and adopting differentiation strategies in order to attract and retain customers. A common differentiation strategic approach among online retailers is a web content personalization aimed at matching customers’ needs and expectations (Tam and Ho 2006).

Various marketing studies adopt different definitions of personalization. The general concept of personalization, however, refers to the process where products and services are adjusted to match individual preferences by learning customer needs and transforming the gathered data into recommendations, offers, and promotions aimed at forming competitive advantage for the focal company by creation of idiosyncratic value (Montgomery and Smith 2009; Vesanen and Raulas 2006; Salonen and Karjaluoto 2016). In the e-Commerce context, personalization means the use of adaptive web content to convey personally relevant data to the consumer (Karat et al. 2003).

Content personalization is generally assumed to be among the most efficient information strategy approaches aimed to maximize immediate and future business opportunities for online merchants by delivering the right content to the right person at the right moment (Cao and Li 2007; Ho and Bodoff 2014; Aguirre et al. 2015). Adolphs and Winkelmann (2008) argue that content personalization is a powerful tool which can help build strong customer relationships as it allows addressing customers personally. In his research Vesanen (2007) makes a conclusion that the ability of online merchants to provide personalized services more accurately, enables their customers to enjoy a better preference match, experience, communication, reduced cognitive overload, and convenience (Ansari and Mela 2003) which eventually increases customer satisfaction and loyalty (Rust and Chung 2006).

However, as the field grows, there is a plethora of important topics around content personalization within e-Commerce field that lacks substantial clarification what creates ambiguity around this domain (Salonen and Karjaluoto 2016). One of the points, which currently remains understudied, focuses on the role of users’ emotions for content personalization in the context of online shopping (Pappas et al. 2017).

Although the concept of personalized services has been known to marketers since 1987 (Surprenant and Solomon 1987), it was not largely embraced until the idea of customer-centric approach within e-Commerce attracted the attention of information systems and human-computer interaction researchers (Li and Karahanna 2015). Today web content personalization is a focus research area for many scientific fields, especially within marketing and information systems, and since web content personalization involves human-computer interaction, it is being largely investigated in relation to technological advancements and applications (Salonen and Karjaluoto 2016). Essentially, implementation of web content personalization implies extensive use of technologies, which makes such topics as recommender systems, user profiling principles, contextual data, and user emotions recognition (Adolphs and Winkelmann 2008) key research areas within multidisciplinary fields, where technological research models are supplemented with approaches from marketing and psychology studies (Salonen and Karjaluoto 2016).

The purpose of this research is to analyze the stream of studies on the role of user emotions for content personalization in e-Commerce. Specifically, the main goal of the paper is to extract the data about why emotions play an important role in online shopping and which technologies and approaches exist to detect and capture user emotions and make a comprehensive overview of these findings. We fill the research gap by synthesizing the extensive studies on the role of user emotions for personalized services within e-Commerce and provide an aggregated comprehensive concept-matrix. The proposed concept-matrix incorporates market-available emotions recognition technologies mapped against the specific emotions these technologies are able to detect and process as well as industry domains where these solutions are currently being applied.

This paper is organized as follows: Sect. 2 presents the methodology we applied in this research. Section 3 introduces the background of this paper regarding the role of user emotions in content personalization within e-Commerce field. In Sect. 4 we present a table which provides a clear snapshot of the strategic approach applied to extract user emotions as well as data acquisition methods and the list of key research contributors. Furthermore, we identify the comprehensive list of technologies which are currently available on the market and used to capture user emotions via facial expressions and voice recognition. We developed a concept matrix based on the methodology proposed by Webster and Watson (2012) to synthesize the technologies we identified with the user emotions these technologies are able to capture as well as the application fields of the selected solutions. In Sect. 5 we discuss the results, conclude the paper, and propose future research directions.

2 Methodology

The state-of-the-art review of prior relevant literature is an essential approach aimed at facilitation of further academic research through closing the gaps of areas where a plethora of studies already exists (Webster and Watson 2002). The goal of the state-of-the-art literature review is to uncover the directions where the research is needed by providing points of reflections on the present location and potential future directions of the subject matter (Salonen and Karjaluoto 2016). The current literature review is aimed at providing a comprehensive overview of the role of emotions within e-Commerce giving special attention to the significance of emotional variable in building content personalization strategies as well as to extensive theoretical background.

2.1 Literature Search

The current study is built upon the stages of the concept-centric approach defined by Webster and Watson (2002) in order to determine the most relevant sources for the literature review. The method suggests commencing the review from identifying a preliminary pool of scientific papers which will be further elaborated to extract the most relevant materials to build the review upon. The resources relevant for building the review were selected with the clear research focus after reviewing a wide variety of the respected and well-established academic and scientific materials. The primary data sources that we leveraged were peer-reviewed academic journals which fall into the set of “AIS Basket of 8”, databases such as Google Scholar, PubsOnLine, EBSCOhost (Academic Search Complete), and ISI Web of Knowledge/ISI Web of Science as well as subject-specific database such as ACM Digital Library.

2.2 Keywords, Inclusion and Exclusion Criteria

The main requirement for the article to be considered for further revision was presence of the certain keywords. We were mostly focused at finding the papers which would contain the following keywords in their titles: “Personali*” (both z- and s-forms of the terms are used in the literature, so the search of the mentioned keywords we did in combination with “AND”), “Recommender system”, “Customer-centric”, “Customer AND user emotions”, “Emotions recognition AND identification AND detection”, “Behavi*” (both options “behavior” and “behaviour” can be found in the literature, so these keywords were also sought in combination with “AND”) which gave us 300 papers at the end.

In order to retrieve the most relevant information, we applied the range of exclusion criteria. The first criteria concerned the review period, which included studies published between 2005 and 2018. This allowed us to elaborate on the broad range of literature, without, however, considering those studies which might have become somewhat obsolete to date. We also made a decision to focus the review solely on the concept of personalization omitting the concept of customization in order to have a clear and grounded literature review at the end. After applying the mentioned filters, we screened through the paper abstracts in order to identify how relevant the paper was to be considered for further elaboration. The scientific articles, focused on technical aspects of recommender systems or recommender algorithms were also elaborated, however, we considered them as less important for the current study. At the end of this step, we ended up with having 80 papers.

The collected papers were read with the clear research focus. At this stage, we were also applying backward referencing to examine the works cited in some of the most relevant papers as well as forward referencing to make a deep search across the papers which cited the papers we considered relevant. We finished this step with having 45 papers which we further read in details.

The remained pool was comprised of 27 papers which represented the most relevant materials as they had a clear focus on the role of user emotions in personalized services within e-Commerce. These papers became the foundation for building the current literature review. Figure 1 demonstrates the process of articles selection we applied in this review.
Fig. 1.

Articles selection process.

3 Background

In their study Aguirre et al. (2015) claim that since personalization is all about adapting to customer preferences and needs, e-Commerce players need to be fully aware of how to collect high-quality primary data about their customers. This is made possible by deploying intelligent personalization algorithms capable of collecting and analyzing web activities to generate highly adaptive, dynamic, and personalized content for individual users (Mobasher et al. 2000). Data collection and processing involves different methods of harvesting data aimed at building customer profiles and adapting it to be used for web personalization mostly by recommender systems (Salonen and Karjaluoto 2016).

3.1 Recommender Systems as a Content Personalization Tool

Today recommender systems are widely implemented solutions and represent fully functional software systems that apply at least one “recommendation approach”. This approach is specifically designed to predict the needs of online customers and recommend them the online purchase of some products based on the collected data (Bǎdicǎ et al. 2011; Beel et al. 2016). According to Beel et al. (2016) a “recommendation approach” is a model which describes how to bring a recommendation concept into the practice. The mostly adopted recommendation concepts within modern recommender systems are collaborative filtering and content-based filtering which have fundamental differences in their underlying ideas (Beel et al. 2016).
  • Collaborative filtering. Collaborative filtering was one of the first personalization technologies which became widely available within e-Commerce domain (Montgomery and Smith 2009). Collaborative filtering does not require the user’s explicit profile (Koren 2010) and, in the online retail context, collaborative filtering generates recommendations by predicting the utility of particular items to a specific user based on user votes retrieved from a user database. In order to generate more specific suggestions of products or services, items or services need to be rated by many customers (Breese et al. 1998).

  • Content-based filtering. Content-based filtering analyses the content of information sources and creates a user profile based on the customer’s interests (past searches, item rating or preferences about specific goods) in terms of regularities of the items and services that have been rated highly (Pazzani 1999). Content-based filtering then relies on a user profile and recommends those items which match with the customer’s needs and preferences reflected in the customer profile (Uçar and Karahoca 2015).

  • Hybrid recommender systems. The previously introduced approaches may be combined in hybrid systems to complement and eliminate drawbacks of each other (Uçar and Karahoca 2015). For instance, collaborative filtering is based upon ratings on items made by other customers, what makes implementation of the new recommender system challenging as these data are not yet available. Leveraging of content-based filtering on top of collaborative filtering will help make a deep analysis of user profiles what will eliminate the generation of immature suggestions by collaborative filtering-based recommender system in the early stages (Tran and Cohen 2000).

All the approaches described focus on recommending items to customers based on rigidly established algorithms and operate only within a user-item matrix when new data are simply being compounded on top of the current user’s profile (Lombardi et al. 2009). Costa and Macedo (2013) argue that although multiple approaches and dimensions exist which make performance of recommender systems more efficient, most of the existing recommender systems do not consider human emotions when recommending content. In his research, Polignano (2015) also concludes that traditional recommender systems mostly do not address emotions in the computational process and only recent papers have some explanation on why emotions are highly important in providing personalized services and how it is possible to recognize emotion with the help of technologies.

3.2 Emotions as a Contextual Variable in Content Personalization

Although the role of user emotions has been widely acknowledged in multiple studies, the amount of studies on the importance of emotions for content personalization within e-Commerce is relatively low. This might be explained by the lack of a commonly accepted definition of what constitutes emotions. However, scholars in e-Commerce domain have elaborated on the range of emotions and prioritized the following categories: social/personality issues (the effect of emotions on consumption judgment and reasoning), cognitive factors (cognitive and social construction in emotions), and the correlation of emotions with other consumption factors (Gaur et al. 2014).

To attain theoretical clarity a precise clarification of what is the difference between emotions and feelings is necessary. In his paper Walla (2017) makes an attempt to distinguish between emotions and feelings and provide clear definitions of both notions. According to Walla (2017) feelings represent “consciously felt bodily responses” which can appear from subcortical affective processing whereas emotions are behavioral output based on affective processing, which can help communicate but not necessarily fully reflect feelings of the person to others.

In the past decade, several context-aware recommender algorithms have been implemented and proved to be useful to enhance recommendation performance in numerous domains by additionally incorporating context information (Gorgoglione et al. 2011). The most widely adopted context factors are the time of the day, the day of the week, and location, which, according to Zheng (2016) can be relatively easy captured from the ubiquitous environment. Recommender systems have traditionally applied data-centric computational approaches for content recommendation and user modeling (Tkalčič et al. 2011). However, relatively few studies have been done about why it is important to take into consideration user emotions when designing personalized services (Zheng et al. 2013).

Research by González et al. (2007) supports that purchasing decisions are always being accompanied by emotions. This makes customer-centric companies put efforts into detecting the existing connections between users’ actions and their emotions to have a better understanding of how to increase retention, loyalty, and satisfaction rates among customers through providing enhanced personalized services (González et al. 2007). In their research Zheng et al. (2013) proved that the increased precision in providing relevant suggestions by recommender systems is achieved when an emotional variable is considered. In their study Tkalčič et al. (2013) conducted a comparative analysis of the generic metadata- and affective metadata-based recommender systems. The authors designed a methodology and conducted an experiment which proved that recommender system which was capable of utilizing emotional context, demonstrated improved performance over recommender system based only on the generic metadata.

4 Results

4.1 Strategies to Identify User Emotions

According to Zheng (2016), emotional states are usually dynamic and tend to change from time to time. Therefore, the users’ profiles associated with different affective reactions can be of high importance to assist recommender systems in making relevant content suggestions (Zheng 2016) what, in turn, can facilitate the decision-making process of users (Picard et al. 2004). In their study González et al. (2007) describe emotional factor as “the relevance that each user gives to differential values of items (i.e. events, services, products), which are demonstrated in the user’s decision-making process by means of his or her actions”. González et al. (2007) also argue that emotional factor particularly influences the rational thinking when users get any recommendations.

The important role of emotions in recommender systems has been discussed by Tkalčič et al. (2011) where the authors demonstrate how to identify emotions during the interaction with recommender system in three data acquisition stages in the consumption chain of an item: entry stage (i.e. before the activity), the consumption stage (i.e. during the activity), and exit stage (i.e. after the activity).

In Table 1 we summarize the key data about each stage and also provide the most prominent studies made to address the peculiarities of data acquisition methods within each stage. The way we compiled and synthesized the data with the help of the table gives the reader a chance to quickly capture the key points of emotions identification strategy proposed by Tkalčič et al. (2011). Furthermore, we are also presenting the list of key contributors who put significant efforts into exploring each strategic step separately to discover new and important insights.
Table 1.

Description of emotions identification stages, data acquisition methods, and key research contributors.



Data acquisition method

Key contributors

Entry stage

When starting using the recommender system, the user is in the entry mood caused by previous activities and actions unknown to the system, which, however, have an impact on the user’s decisions

• Matrix factorization approach with emotion-specific regularization enriched with contextual parameters

• Porayska-Pomsta et al. (2007)

• Baltrunas (2008)

• Koren (2010)

• Shi et al. (2010)

Consumption stage

During the content consumption, user experiences emotional responses activated by the content which, however, do not induce any further actions, but rather is a passive response to the stimuli

• Explicit data can be collected through paper-and-pencil questionnaire and to be categorized according to the six universal emotions: happiness, sadness, surprise, fear, disgust, anger (Ekman and Rosenberg 1993)

• Implicit emotional data could be collected through monitoring the facial or voice expressions of users with the help of specific emotions recognition tools and technologies (Polignano 2015)

• Pantic and Vinciarelli (2009)

• Arapakis et al. (2009)

• Soleymani et al. (2012)

• Joho et al. (2011)

Exit stage

The emotions induced by the content during the exit stage will influence the user’s further actions

• The implicit data on users’ emotions during the exit stage can be collected by using facial or voice expressions recognition technologies as well as heart rate sensors in certain domains such as physical interactive playgrounds

• Arapakis et al. (2009)

• Yannakakis et al. (2008)

• Soleymani et al. (2011)

According to Tkalčič et al. (2011), emotional states of the users can be identified by applying explicit and implicit approaches. Explicit acquisition of data about user emotions is usually carried out by applying the Self-Assessment Manikin developed by Bradley and Lang in 1994. This approach represents a non-verbal pictorial assessment technique designed as a paper-and-pencil questionnaire which is used in groups to evaluate the pleasure, arousal, and dominance associated with a person’s emotional reaction in response to an object or event (Bradley and Lang 1994). The implicit detection of emotions, although being intrusive in nature, proved to be a more accurate emotions detection approach rather than an explicit one since users under surveillance are not aware of their emotions being captured what reduce the likelihood of data distortion (Tkalčič et al. 2011).

The most widely adopted emotion categories include happiness, sadness, fear, anger, disgust, and surprise. The description of these fundamental emotions was thoroughly investigated and categorized into the universal emotions model by the cross-cultural studies conducted by Ekman and Rosenberg (1993), indicating that humans perceive certain basic emotions with respect to facial expressions in the same way, regardless of their culture (Zeng et al. 2009). However, since the universal model provides a limited set of distinct emotional categories, the circumplex multidimensional VAD Model of affect was proposed by Posner et al. (2005). The proposed model describes each emotion as a dynamic concept within multidimensional space (Tkalčič et al. 2011).

The VAD Model describes emotions using 3 dimensions: Valence (V) that evaluates how positive or pleasant an emotion is, ranging from negative to positive; Arousal (A) which assesses the involvement level of the person, ranging from “non-active”, “in calm to agitated”, “ready to act” and Dominance (D) that measures the level of control over the situation by the person, ranging from “submissive”, “non-control” “dominant/in-control” (Kosti et al. 2017).

Despite the fact that most of the emotional expressions occur in a realistic interpersonal or human-computer interactions are non-basic emotions (Cowie et al. 2005) the majority of the existing technologies and approaches developed to date are mostly designed to identify the six basic human emotions (Kosti et al. 2017).

4.2 Emotions Recognition from Facial Expressions

As people use facial expressions as a natural mean of emotional communication, one of the most well-studied channels of emotional expressions is facial expressions recognition. The analysis of facial expressions is a relevant and important topic from both scientific and applied standpoints (Soleymani et al. 2012). However, in his paper, Walla (2017) states that eliciting respective responses in the observer’s brain by observing someone’s facial expressions might be challenging since faces do not necessarily reflect deep internal affective states of the person what might become a confounder for emotions recognition approaches.

The topic of understanding internal decision-making drivers by recognizing facial expressions, however, keeps attracting the interests of information systems and human-computer interactions researchers as well as business-oriented professionals what resulted in rapid development of multiple technology companies (Garcia-Garcia et al. 2017). We briefly present the six technologies capable of recognizing emotions from facial expressions which are available on the market today.
  • nViso. Developed by the Swiss company, this technology can detect six basic emotional groups, described by Ekman and Rosenberg (1993) by using a proprietary deep learning algorithm. According to the data provided on the website, nViso can capture emotions of one person or a group of people in real time with the help of the webcam, which tracks muscle movements of the face (nViso 2018). nViso partnered with IBM to create an emotional intelligence cloud solution capable of analyzing facial expressions to enable financial advisors better understand their clients’ financial needs (IBM 2018). Furthermore, together with ePAT Technologies, nViso actively works on a smartphone-based medical device which is able to assess pain levels in real time by analyzing facial muscle movements of patients (IBM 2018; ePAT Technologies Ltd. 2017).

  • Affectiva. By analyzing emotions from the twenty facial zones retrieved from a database of videos and images (Affectiva 2018a), this emotion recognition software can detect seven emotions (anger, contempt, disgust, fear, happiness, sadness and surprise) as well as measure valence and arousal of the person (Affectiva 2018b). Affectiva has recently partnered with Voxpopme, which is a global emotion recognition software provider, to work on the platform to enable advanced analysis of facial expressions within video feedback (Busines Wire 2017). Furthermore, Affectiva helps clients develop analytics solutions in multiple domains, including healthcare, education, media and advertising, retail, and gaming (Affectiva 2018a).

  • EmoVu by Eyeris. This software solution exploits the deep learning algorithms which retrieve data from large datasets about people of various ages, ethnicities, genders, etc. EmoVu can recognize such emotions as anger, disgust, fear, happiness, neutral, sadness, and surprise and can also measure the degree of arousal and valence (Eyeris 2018a). Eyeris is mostly specialized in the development of facial analytics and emotion recognition technology for the automotive sector and the most prominent Eyeris’s customers include Toyota Motor Corporation and Honda Motor Co. (Eyeris 2017). Furthermore, Eyeris has recently partnered with AvatarMind, the creator of iPal® Robot, a humanoid robot which serves as a social companion, educator, and safety monitor for children and elderly (Eyeris 2018b).

  • Kairos. This technology provides data about persons’ six emotions, level of attention, and sentiment based on the analyzed videos or images. Furthermore, the services provided by Kairos include age, ethnicity, and gender identification as well as group faces recognition and detection (Kairos 2018b). The emotion recognition software, provided by Kairos was implemented by companies such as The Interpublic Group of Companies, Legendary Entertainment, PepsiCo, etc. operating in multiple domains including advertising and media, retail, and banking and insurance (Kairos 2018a).

  • Microsoft Cognitive Services. The Cognitive Services Pack provided by Microsoft can identify the face and emotional expressions of people after processing pictures and videos. This software identifies six basic emotional groups described by Ekman and Rosenberg (1993) as well as contempt and neutrality (Microsoft Microsoft 2018a). Microsoft provides its Cognitive Services to the businesses involved in manufacturing, healthcare, media and telecommunications, education, banking and insurance, retail, etc. The featured clients are ABB Group, Daimler AG, Allergan, Telefonica, etc. (Microsoft 2018b).

  • FaceReader by Noldus. This automatic recognition software can analyze up to 500 facial points to recognize such emotions as neutral, contempt, boredom, interest, and confusion. Furthermore, FaceReader also calculates gaze direction, head orientation, and person characteristics (Noldus 2018a). The Noldus’s clients mostly involved in healthcare, retail, and education services and include such companies as Pfizer, GlaxoSmithKline, Carnegie Mellon University, University of Maryland, Johnson & Johnson, etc. (Noldus 2018b).

4.3 Emotions Recognition from Speech

Human speech is another essential channel for gathering emotional data since accurate and real-time understanding of human-speech can significantly facilitate human-computer interactions (Tao et al. 2018) and therefore improve the work of personalized services and technologies within online shopping. In the last few years, emotions recognition from speech has become of particular scientific interest which resulted in multiple studies conducted by scientists (El Ayadi et al. 2011) which in turn became a solid ground for rapid technological development. We make a brief overview of five technologies which utilize speech to elicit emotions and are available on the market.
  • Vokaturi. The Amsterdam-based company developed a solution, which can measure whether people are happy, sad, afraid, angry, or have a neutral state of mind directly from their voice. Vokaturi has been validated with the multiple existing emotion databases and works in a language-independent manner (Vokaturi 2018). Vokaturi has recently established a partnership with Affectiva with the purpose of joining efforts to work on the emotion-sensing product into the autonomous vehicles sector (Affectiva 2018).

  • Good Vibrations Company B.V. This solution can recognize the emotions of the person by processing recorded voice. Good Vibrations measures acoustic properties of the user’s voice and performs a real-time analysis of the user’s emotions to recognize stress, pleasure, and arousal (Good Vibrations Company B.V. 2018). According to the official website of Good Vibrations Company B.V., the areas where the developed solution has the greatest potential include healthcare, advertising, gaming, sports, business, robotics, safety, and matching (Good Vibrations Company B.V. 2018).

  • audEERING. This Munich-based tech company developed intelligent audio analysis algorithms to help organizations integrate audio analysis technology into their products. The embedded automated paralinguistic speech analysis allows detecting a multitude of attributes from the human voice, such as emotions and affective states (valence, arousal, dominance), age, alertness, or personality (audEERING 2018a). audEERING’c clients represent multiple domains such as manufacturing, telecommunications, education, retail and include BMW, Daimler, T-Mobile, Deutsche Welle, Huawei, etc. (audEERING 2018b).

  • Beyond Verbal. The solution, provided by this Israel-based company, is capable of extracting multiple acoustic features from a speaker’s voice in real time and providing insights into the emotional, health, and wellbeing condition of the user. By utilizing voice-driven emotions analytics, the technology can recognize anger, sadness, neutral, and happiness as well as can measure valence, arousal, and temper in the voice of the speaker. Beyond Verbal’s clients mostly represent companies from retail, and media and marketing sectors and include such companies as Amdoc, FRONTLINE Selling, and Department26 (Beyond Verbal 2018).

  • Nemesysco. The company provides advanced voice analysis technologies for emotion detection, personality, and risk assessment. The technology is based on the proprietary signal-processing algorithms which can extract over 150 acoustic parameters from voice and classify the collected properties into major emotional groups including anger, happiness, satisfaction, and arousal. The key domains currently covered by Nemesysco are retail, banking and insurance, and security. Among the key customers we featured Nestle, Allianz, and Europ Assistance (Nemesysco 2018).

Table 2 aggregates the technologies described and maps them against emotional dimensions of human beings we have described in the Sect. 4.1. Furthermore, the concept-matrix also demonstrates in which domains the listed technologies are currently in use. The data provided in this table are retrieved from the official websites and contains only information which has been officially published by the software vendors.
Table 2.

Emotions recognition technologies, targeted emotions, and technologies application fields.

Emotions & Application fields

Facial expressions recognition technologies

Speech expressions recognition technologies





MS Cognitive Services



Good Vibrations


Beyond Verbal


















































Application fields

Banking & Insurance
















Security & Safety













Media & Telecommunications





Retail & Marketing




5 Discussion

People’s emotions play an essential role in many aspects of life, especially in decision-making processes. When it comes to user-centric approaches in e-Commerce, enriching the capabilities of recommender systems with the emotions recognition functionality is a vital technical and strategic decision which can amplify generation of useful and relevant content suggestions. There is an obvious shift towards embracement of emotional variable into personalized services, however, technical capabilities of recommender systems available today are still quite limited in terms of emotions recognition and utilization for content personalization.

Multiple studies prove that technologies and recommender systems capable of recognizing and interpreting emotions in the same way as other human beings have tremendous potential in human-computer interaction, human-assistive technologies, and e-Commerce. According to Walla (2017), however, the scientific community currently does not have a unified definition of what is an emotion as some of the scholars understand emotions as neural activities, others describe them as felt affective phenomena, and some scientists see emotions as facial expressions. Furthermore, in his paper Walla (2017) argues that affective processing does not always result in observable and measurable emotions. This might have serious implications for marketing and IS experts since emotions to be elicited and measured might not fully reflect the underlying affective processing. This, in turn, might mislead the interpretation of outcomes and result in irrelevant products and services that nobody wants or needs (Walla 2017).

According to Kreibig (2010), collecting reliable and valid data on autonomic responding in emotions has long been and remains a challenge within emotions research. To gain a deeper understanding of the functional role of emotions, future researchers need to thoroughly investigate and verify the particular type of emotions elicited (Kreibig 2010). This needs to be done in order to get a deeper understanding of how emotions, as well as their variations, reflect a specific state of affective processing of a particular individual. A comprehensive approach aimed at retrieving user emotions by deep analysis of affective processing may be created to facilitate understanding of users’ tasks and goals, internal decision drivers, contextual properties and characteristics. This scientific method can be of great significance for providing insights into how to make personalized services within e-Commerce more precise, effective, and efficient.

The deep knowledge of the correlation between user emotions utilization and development of content personalization systems would lead to significant improvement in creation of specific personalization features and technologies. Furthermore, a better understanding of impact of emotional variable on decision-making during online shopping would facilitate designing of recommender systems architectures to enable online merchants to provide a better customer experience. A deep understanding of how user emotions impact and shape decision-making process could give rise to a new generation of personalization technologies providing value both to the users and the organizations.

In the future, we will provide a systematic study, which would consider a higher number of studies, including the most recent state-of-the art research in order to provide even more valuable insights into how user emotions are being captured and processed by emerging technologies. As a further step, our next research will go deeper into the concept of context-aware recommender systems and affective computing to identify which scientific methods and approaches are being created to advance development of personalization-focused technologies.



The present work was conducted within the training network project PERFORM funded by the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No. 76539. This study reflects only the authors’ view, the EU Research Executive Agency is not responsible for any use that may be made of the information it contains.

This work was supported, in part, by Science Foundation Ireland grant 13/RC/2094 and co-funded under the European Regional Development Fund through the Southern & Eastern Regional Operational Programme to Lero - the Irish Software Research Centre (


  1. Adolphs, C., Winkelmann, A.: Personalisation in E-Commerce - a state of the art review (2000–2008). J. Electron. Commer. Res. 11(4), 326–341 (2008)Google Scholar
  2. Affectiva: About Us (2018a).
  3. Affectiva: Determining Accuracy (2018b).
  4. Aguirre, E., Mahr, D., Grewal, D., de Ruyter, K., Wetzels, M.: Unraveling the personalization paradox: the effect of information collection and trust-building strategies on online advertisement effectiveness. J. Retail. 91(1), 34–49 (2015)Google Scholar
  5. Ansari, A., Mela, C.F.: E-Customization. J. Mark. Res. 40(2), 131–145 (2003)Google Scholar
  6. Arapakis, I., Konstas, I., Jose, J.M.: Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance categories and subject descriptors. In: Proceedings of the 17th ACM International Conference on Multimedia, Beijing, China, pp. 461–470 (2009)Google Scholar
  7. Bǎdicǎ, C., Budimac, Z., Burkhard, H.D., Ivanović, M.: Software agents: languages, tools, platforms. Comput. Sci. Inf. Syst. 8(2), 255–296 (2011)Google Scholar
  8. Baltrunas, L.: Exploiting contextual information in recommender systems. In: Proceedings of the 2008 ACM Conference on Recommender Systems, Lausanne, Switzerland, pp. 295–298 (2008)Google Scholar
  9. Beel, J., Gipp, B., Langer, S., Breitinger, C.: Research-paper recommender systems: a literature survey. Int. J. Digit. Libr. 17(4), 305–338 (2016)Google Scholar
  10. Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25(1), 49–59 (1994)Google Scholar
  11. Breese, J.S., Heckerman, D., Kadie, C.: Empirical analysis of predictive algorithms for collaborative filtering. In: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, Madison, Wisconsin, pp. 43–52 (1998)Google Scholar
  12. Business Wire: Voxpopme Partners with Affectiva to Enhance Emotional Analysis of Video (2017).
  13. Cao, Y., Li, Y.: An intelligent fuzzy-based recommendation system for consumer electronic products. Expert Syst. Appl. 33(1), 230–240 (2007)Google Scholar
  14. Cowie, R., Douglas-Cowie, E., Cox, C.: Beyond emotion archetypes: databases for emotion modelling using neural networks. Neural Netw. 18(4), 371–388 (2005)Google Scholar
  15. Ekman, P., Rosenberg, E.: Facial expression and emotion. Am. Psychol. 48(4), 384–392 (1993)Google Scholar
  16. El Ayadi, M., Kamel, M.S., Karray, F.: Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recognit. 44(3), 572–587 (2011)zbMATHGoogle Scholar
  17. Garcia-Garcia, J.M., Penichet, V.M.R., Lozano, M.D.: Emotion detection. In: Proceedings of the XVIII International Conference on Human Computer Interaction, Cancun, Mexico, pp. 1–8 (2017)Google Scholar
  18. Gaur, S.S., Herjanto, H., Makkar, M.: Review of emotions research in marketing, 2002–2013. J. Retail. Consum. Serv. 21(6), 917–923 (2014)Google Scholar
  19. González, G., de la Rosa, J.L., Montaner, M.: Embedding emotional context in recommender systems. In: 23rd International Conference on Data Engineering, Istanbul, Turkey, pp. 50–53 (2007)Google Scholar
  20. Gorgoglione, M., Panniello, U., Tuzhilin, A.: The effect of context-aware recommendations on customer purchasing behavior and trust. In: Proceedings of the Fifth ACM Conference on Recommender Systems, Chicago, Illinois, USA, pp. 85–92 (2011)Google Scholar
  21. Good Vibrations Company B.V. (2018).
  22. Ho, S.Y., Bodoff, D.: The effects of web personalization on user attitude and behavior: an integration of the elaboration likelihood Model and Consumer Search Theory. MIS Q. 38(2), 497–520 (2014)Google Scholar
  23. Joho, H., Staiano, J., Sebe, N., Jose, J.M.: Looking at the viewer: analysing facial activity to detect personal highlights of multimedia contents. Multimed. Tools Appl. 51(2), 505–523 (2011)Google Scholar
  24. Kairos: What is Kairos? (2018).
  25. Karat, C., Blom, J., Karat, J.: Designing personalized user experiences for eCommerce: theory, methods, and research. In: CHI 2003 Extended Abstracts on Human-Computer Interactions, pp. 1040–1041 (2003)Google Scholar
  26. Koren, Y.: Factor in the neighbors. ACM Trans. Knowl. Discov. Data 4(1), 1–24 (2010)MathSciNetGoogle Scholar
  27. Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: Emotion recognition in context. In: Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, pp. 1667–1675 (2017)Google Scholar
  28. Kreibig, S.D.: Autonomic nervous system activity in emotion: a review. Biol. Psychol. 84(3), 394–421 (2010)Google Scholar
  29. Li, S.S., Karahanna, E.: Online recommendation systems in a B2C E-commerce context: a review and future directions. J. Assoc. Inf. Syst. 16(2), 72–107 (2015)Google Scholar
  30. Lombardi, S., Anand, S.S., Gorgoglione, M.: Context and customer behaviour in recommendation. In: Workshop on Context-Aware Recommender Systems, New York, NY, USA (2009)Google Scholar
  31. Mobasher, B., Cooley, R., Srivastava, J.: Web usage mining can help improve the scalability, accuracy, and flexibility of recommender systems. Commun. ACM 43(8), 142–151 (2000)Google Scholar
  32. Montgomery, A.L., Smith, M.D.: Prospects for personalization on the internet. J. Interact. Mark. 23(2), 130–137 (2009)Google Scholar
  33. nViso: Advancing Human Potential (2018).
  34. Osterwalder, A., Pigneur, Y.: An eBusiness model ontology for modeling eBusiness. In: 15th Bled Electronic Commerce Conference, Bled, Slovenia, pp. 75–91 (2002)Google Scholar
  35. Pantic, M., Vinciarelli, A.: Implicit human-centered tagging. IEEE Signal Process. Mag. 26(6), 173–180 (2009)Google Scholar
  36. Costa, H., Macedo, L.: Emotion-based recommender system for overcoming the problem of information overload. In: Corchado, Juan M., Bajo, J., Kozlak, J., Pawlewski, P., Molina, Jose M., Julian, V., Silveira, R.A., Unland, R., Giroux, S. (eds.) PAAMS 2013. CCIS, vol. 365, pp. 178–189. Springer, Heidelberg (2013). Scholar
  37. Pappas, I.O., Kourouthanassis, P.E., Giannakos, M.N., Chrissikopoulos, V.: Sense and sensibility in personalized e-commerce: how emotions rebalance the purchase intentions of persuaded customers. Psychol. Mark. 34(10), 972–986 (2017)Google Scholar
  38. Pazzani, M.J.: A framework for collaborative, content-based and demographic filtering. Artif. Intell. Rev. 13(5), 393–408 (1999)Google Scholar
  39. Picard, R.W., et al.: Affective learning - a manifesto. BT Technol. J. 22(4), 253–268 (2004)Google Scholar
  40. Polignano, M.: A framework for emotion-aware Recommender systems supporting decision making. In: Proceedings of the 6th Symposium on Future Directions in Information Access, Thessaloniki, Greece, pp. 12–15 (2015)Google Scholar
  41. Porayska-Pomsta, K., Mavrikis, M., Pain, H.: Diagnosing and acting on student affect: the tutor’s perspective. User Model. User-Adapt. Interact. 18(1–2), 125–173 (2007)Google Scholar
  42. Posner, J., Russell, J.A., Peterson, B.: The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17(3), 715–734 (2005)Google Scholar
  43. Rust, R.T., Chung, T.S.: Marketing models of service and relationships. Mark. Sci. 25(6), 560–580 (2006)Google Scholar
  44. Salonen, V., Karjaluoto, H.: Web personalization: the state of the art and future avenues for research and practice. Telemat. Inform. 33(4), 1088–1104 (2016)Google Scholar
  45. Shi, Y., Larson, M., Hanjalic, A.: Mining mood-specific movie similarity with matrix factorization for context-aware recommendation. In: Proceedings of the Workshop on Context-Aware Movie Recommendation, Barcelona, Spain, pp. 34–40 (2010)Google Scholar
  46. Simon, M., Van Den Driest, F., Wilms, T.O.M.: Driving customer-centric growth: a practical roadmap. J. Advert. Res. 56(2), 159–168 (2016)Google Scholar
  47. Soleymani, M., Koelstra, S., Patras, I., Pun, T.: Continuous emotion detection in response to music videos. Face Gesture 2011, 803–808 (2011)Google Scholar
  48. Soleymani, M., Lichtenauer, J., Pun, T., Pantic, M.: A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 3(1), 42–55 (2012)Google Scholar
  49. Surprenant, C.F., Solomon, M.R.: Predictability and personalization in the service encounter. J. Mark. 51(2), 86–96 (1987)Google Scholar
  50. Tam, Y.K., Ho, S.Y.: Understanding the impact of web personalization on user information processing and decision outcomes. MIS Q. 30(4), 865–890 (2006)Google Scholar
  51. Tao, F., Liu, G., Zhao, Q.: An ensemble framework of voice-based emotion recognition system for films and TV programs. In: Proceedings of the First Asian Conference on Affective Computing and Intelligent Interaction, Beijing, China, pp. 363–364 (2018)Google Scholar
  52. Tkalčič, M., Košir, A., Tasič, J.: Affective recommender systems: the role of emotions in recommender systems. CEUR Work. Proc. 811(i), 9–13 (2011)Google Scholar
  53. Tkalčič, M., Odić, A., KoTkalšičir, A., Tasič, J.: Affective labeling in a content-based recommender system for images. IEEE Trans. Multimed. 15(2), 391–400 (2013)Google Scholar
  54. Tran, T., Cohen, R.: Hybrid recommender systems for electronic commerce. AAAI Technical report (2000)Google Scholar
  55. Uçar, T., Karahoca, A.: Personalizing trip recommendations: a framework proposal. Glob. J. Comput. Sci. 05(51), 30–35 (2015)Google Scholar
  56. Vesanen, J., Raulas, M.: Building bridges for personalization: a process model for marketing. J. Interact. Mark. 20(1), 5–20 (2006)Google Scholar
  57. Vesanen, J.: What is personalization? A conceptual framework. Eur. J. Mark. 41(5/6), 409–418 (2007)Google Scholar
  58. Vokaturi (2018).
  59. Walla, P.: Affective processing guides behavior and emotions communicate feelings: towards a guideline for the NeuroIS community. Lect. Notes Inf. Syst. Organ. 25, 141–150 (2017)Google Scholar
  60. Webster, J., Watson, R.T.: Analyzing the past to prepare for the future: writing a literature review. MIS Q. 26(2), 13–23 (2002)Google Scholar
  61. Yannakakis, G.N., Hallam, J., Lund, H.H.: Entertainment capture through heart rate activity in physical interactive playgrounds. User Model. User-Adapt. Interact. 18(1–2), 207–243 (2008)Google Scholar
  62. Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2009)Google Scholar
  63. Zheng, Y.: Adapt to emotional reactions in context-aware personalization. In: CEUR Workshop Proceedings, vol. 1680, pp. 1–8, September 2016Google Scholar
  64. Zheng, Y., Burke, R., Mobasher, B.: The role of emotions in context-aware recommendation. In: CEUR Workshop Proceedings, vol. 1050, pp. 21–28 (2013)Google Scholar

Copyright information

© The Author(s) 2019

Open Access This chapter is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, a link is provided to the Creative Commons license and any changes made are indicated.

The images or other third party material in this chapter are included in the work's Creative Commons license, unless indicated otherwise in the credit line; if such material is not included in the work's Creative Commons license and the respective action is not permitted by statutory regulation, users will need to obtain permission from the license holder to duplicate, adapt or reproduce the material.

Authors and Affiliations

  • Artem Bielozorov
    • 1
    Email author
  • Marija Bezbradica
    • 1
  • Markus Helfert
    • 1
  1. 1.Dublin City UniversityDublinIreland

Personalised recommendations