1 Introduction

Based on data from the U.S. Census Bureau, U.S. retail e-commerce sales for the first quarter of 2016 has reached $92.8 billion, which accounts for 7.8% of total retail sales (Denale and Weidenhamer 2016). To boost sales, more and more retailers implement recommender systems to support customers’ decision making process. However, there are still some recommender systems that are poorly designed or ineffectively implemented. This research explores the effects of the display timing (i.e., early, mid, and late) of the recommendation and the sources of recommendation content (i.e., expert review vs. consumer review). The eye-tracking method is employed in an experiment to study users’ attention and interest. We are interested in the differences of pupil dilation and fixation duration in different recommendation timing and recommendation source conditions.

This research contributes to the online product recommendation literature by examining the decision process via eye-tracking method. We expect the outcome of this research to be helpful to online retailers in improving their online recommender systems.

2 Literature Review

2.1 Recommendation Timing

Online product recommender systems are widely used to provide consumers with alternatives that they might be interested in. Online retailers rely on recommender systems as a decision aid to the customers to provide better service and to boost sales. According to research conducted by Forrester Research, product recommender systems accounted for 10% to 30% of total sales by a retailer (Schonfeld 2007). Prior studies on product recommender systems mainly focus on the optimization of algorithms to provide more accurate predictions and suggestions to the customers (Hostler et al. 2012).

Timing is a least studied design feature in human-computer interaction research (Zhang et al. 2002). In online production recommendation literature, Ho et al. (2011) studies the timing of web personalization and suggested that an early recommendation is more likely to be accepted than a late recommendation.

2.2 Online Product Review

Two types of online product review are studied as recommendation support in this paper: one is consumer review which is written by product users, and the other is expert review which is written by domain experts. Prior studies examined the different effect of them. Consumer reviews significantly influence participants’ evaluation while expert reviews do not in Jacobs et al. (2015)’s study of consumers’ evaluation of motion pictures. Consumer recommendations were found to be significantly more trustworthy than expert recommendations (Senecal and Nantel 2004). The results from the study by Utz et al. (2012) also indicated that consumer reviews were the key factor to judge the trustworthiness of online store However, Chiou et al. (2014) found out that online expert reviews have a significantly higher credibility than consumer reviews.

2.3 Eye Tracking Method

The use of eye-tracking devices on information processing tasks has been around for more than a century. With the development of personal computers, researchers start using eye-tracking devices to study and solve problems in human-computer interaction.

Eye fixation is a well-developed predictor of attention. People’s attention only focus on the things they need and ignore others that are presumed to be irrelevant (Triesch et al. 2003). Decision makers direct their attention to goal related stimuli (Orquin and Mueller Loose 2013). Cognitive processing during a fixation affects the fixation duration (Rayner 1978). A longer fixation duration on a certain piece of information implies a higher intensity of cognitive processing and higher preference upon choices (Shimojo et al. 2003). Preferences can reinforce people’s fixations and enhance their perceptions of attractiveness which in turn influence decision making. Krajbich et al. (2010) suggest that visual fixation process could have a causal effect on people’s value comparison process.

Human pupils react not only to the change of environmental luminance, but also to change in cognitive processing (Brisson et al. 2013). Pupil dilation was found to be a consequence of attentional effort (Hoeks and Levelt 1993). According to Laeng et al. (2012), pupil diameter, which is also called “pupillometry”, has been used to estimate the intensity of mental activities, change of emotions, change of mental states, and change of attention for more than 50 years. Pupil diameter is very difficult to control voluntarily, which makes pupil dilation a good objective measure.

3 Theoretical Background and Hypotheses

The first factor of this study is the source of recommendations provided by an e-commerce website: expert vs. consumer recommendations. Due to the limited processing capability, consumers cannot process all available information so that they only put their interest in information that are perceived to be relevant to their goals (Bettman et al. 1998). We propose consumer review based recommendation may win over expert review in terms of consumer attention and interest based on the similarity-attraction paradigm (Byrne and Griffitt 1973). The similarity-attraction paradigm posits that people are attracted to people who are similar to them (Byrne and Griffitt 1973). Attraction was found to be positively affected by people with similarities. Also, economic status, simple behavioral acts, and task performance were also found to positively influence perceived attraction among people.

Consumer reviews were written by former consumers who were previously likely to be in or who were facing similar situations with the current customer. Hence, customers can empathize and relate well with consumers who likely had more similarity in goals, experiences, and/or attitudes. Experts, on the other hand, though considered to be a higher authority in certain fields, may not share similarities with current customer. Consequently, consumer recommendations are expected to attract more interest than expert recommendations. Therefore, we propose:

  • H1: Consumer recommendations will attract greater user interest than expert recommendations.

The second factor refers to timing of recommendations - when the recommendations are offered, i.e., early, middle, or late in the e-commerce shopping process. Galinsky and Mussweiler (2001) found that the first offers served as anchors and were a strong predictor of the final deal in a seller-buyer context. During the buyer’s decision making process, his or her judgements rely heavily on the initial anchor. People’s judgement are severely biased by uncertainty and anchoring bias can occur (Tversky and Kahneman 1974). As consumers work on shopping tasks, their uncertainty about the outcome will be lower as they carry out the evaluation process. Hence, the initial anchor on a specific product that have gone through the evaluation process can deter attention on subsequent product recommendations offered by the online recommender system.

Based on the anchoring effect and bias, decision makers tend to be more interested in the initial anchor (Adomavicius et al. 2013). In an e-commerce context, after a decision maker is attracted by specific products, they are less likely to attend to other recommendations offered by online recommender systems. Hence, the following hypothesis is proposed:

  • H2: The earlier a recommendation is offered by online recommender systems, the greater the user attention toward the recommendation.

4 Methodology

A 2 (source) X 3 (timing) X 2 (product type) mixed experimental design was used for his research and the experiment was conducted in a university. The first and second factors are between-subject factors whereas the third factor (product type, laptop & cell phone) is a within-subject factor. Hence, there are 6 (i.e., 2 × 3) experimental conditions in this study. 76 subjects were recruited. They were given extra credits for their class and were provided with souvenirs after the experiment. All subjects have normal eye-sight before or after adjustment. Subjects were randomly assigned to one of the 6 conditions, and the sequence of the two products was also randomly assigned. There was a training session on the use of the experiment shopping website. The subjects need to carry out two shopping tasks: cell phones and laptops. They were asked to complete two shopping tasks: (i) purchase a laptop, and (ii) purchase a cell phone. Both products were chosen because of their popularity among the pilot test subjects. The task sequence was counterbalanced such that some subjects shopped for a cell phone first while others shopped for a laptop first.

The recommendation source was manipulated in two categories: expert review and consumer review. In the experiment, the recommendation source was highlighted on the recommendation pages. The heading used for the recommendation page was either “other consumers recommend this product to you” or “experts recommend this product to you”. Several product reviews were provided on each product recommendation page and they were extracted from existing e-commerce websites. On each recommendation page, an image of the recommended product along with specifications of the recommended product were displayed. The recommendation timing was manipulated at three level: right after entering the website (i.e., early recommendation), after clicking “add to shopping cart” for the first chosen product (i.e., mid recommendation), and after clicking “purchase” button (i.e., late recommendation). The shopping website allowed subjects to search using various combination of search criteria to browse product details from the search results. The subjects were allowed to conduct search activities within the product database until decisions were made. Single criteria searches and multiple criteria searches were both supported. There was no time limit given to complete each task.

Three Tobii T60 eye-trackers in three separate lab rooms were used during the experiment. The resolution of the display is 1280 * 1024. The use of three eye-trackers allowed us to conduct three concurrent experimental sessions with the subjects. The moderator (or experimenter) at each of the three stations was given a standardized moderator script to following in conducting the experiment to avoid moderator biases. The luminance of all lab rooms were controlled at the same level.

5 Data Analysis and Results

All data were recorded by Tobii Studio software on Tobii T60 eye-trackers. The corneal reflection based devices computed and recorded the data including time, coordinates of eye movement activities, eye movement activities, and pupil diameter at a sample rate of 60 per second. Several variables were computed by using the video recordings of all subjects. Due to eye-tracking recording failure, 5 out of the 76 data points were excluded from the data set. The manipulations were successful based on results of manipulation check questions in the after-experiment questionnaire. Also, by reviewing the recording footages, we observed that all subjects fixated on the recommendation title which indicated their awareness of the recommendation source.

A data reduction procedure was conducted to convert raw data into cleansed fixation data on the recommendation pages. All data were exported from Tobii Studio in the format of xlsx. Five Excel VBAs were implemented to achieve the following goals: calculating pupil diameter baseline, cleansing data by time, cleansing data by gaze type, removing duplicate fixation entries, and calculating targeted pupil diameters. The pupil diameter baseline was calculated based on the first 100 s of recording during which all subjects were going through the instructions for the experiment.

Fixation durations on the recommendation pages for each subject were calculated. As the total browsing time varied across subjects, we calculated fixation duration per second by dividing total fixation duration by total recommendation browsing time. Pupil dilation was calculated as the percentage of pupil diameter change when browsing the product recommendation page versus the baseline condition (i.e., when reading instructions).

Outlier tests were conducted to detect and remove potential outliers for both dependent variables. 4 outliers were detected and removed for data analysis on pupil dilation. 8 outliers were detected and removed for data analysis on fixation duration per second. Order effects were tested for both dependent variables and no order effects for tasks (i.e., order of product types) were found for pupil dilation or fixation duration per second as dependent variables. ANOVA was conducted using SPSS for each of the dependent variables for the two between-subjects factors: recommendation source and recommendation timing, and one within-subjects factor: product type.

5.1 Data Analysis on Pupil Dilation

The pupil diameter for each task was calculated by averaging the left and right pupil diameters. The average of the pupil diameters was then calculated based on the time stamp of product recommendation page to reveal the target pupil diameter (target PD): diameter of the pupil when looking at the product recommendation page. Pupil dilation was then computed relative to the pupil diameter baseline (PDBL) using following equation.

$$ Pupil\,dilation = (target\,PD - PDBL) \div PDBL $$

Pupil dilation reveals the percentage of change on pupil diameter at a given period of time as compared to the baseline. Excluding the outliers, 67 sets of data were used for the analysis. We have an average sample size of 11 for each of the experimental conditions. The descriptive statistics for pupil dilation was shown in Table 1.

Table 1. Descriptive statistics for pupil dilation
Full size table

The results indicate that, recommendation source has a significant effect on pupil dilation there is no significant within-subjects effect (product type) on pupil dilation. However, expert recommendations resulted in an average pupil dilation of −2.5% while consumer recommendations resulted in an average pupil dilation of 0.2%. The difference between them is significant at p value of 0.003 which is less than 0.05. Thus, H1 is supported, indicating that there was higher interest in consumer recommendations than expert recommendations. The negative value of pupil dilation on expert recommendations indicates that participants have lower interest when browsing expert recommendations.

5.2 Data Analysis on Fixation Duration Per Second

The fixation duration for each task was calculated by adding all fixation time based on the timestamp of product recommendation page. We then calculate the fixation duration per second (FDPS) by dividing the total fixation duration by total browsing time of the recommendation page using following equation.

$$ FDPS = Fixation\,Duration \div Total\,browsing\,time $$

We use FDPS to control for different browsing time of the recommendation pages among subjects. For example, a FDPS value of 0.6 indicates that for every 1 s a subject spent on the recommendation page, he/she fixated 0.6 s on the content. This measure revealed the attention levels of the subjects. A higher FDPS indicates a higher level of attention on the recommendation page. 8 sets of data were excluded from the analysis because they were outliers.

Table 2 shows the descriptive statistics for FDPS. The ANOVA results suggest that timing and source do not have significant effects on FDPS. Thus, the second hypothesis is not supported.

Table 2. Descriptive statistics for FDPS
Full size table

6 Discussion and Conclusion

6.1 Discussion

This study used eye-tracking data to explain the effect of recommendation timing and source on user attention and interest during online shopping tasks. The results suggest that pupil dilation varies across sources of recommendations. Trustworthiness of consumer recommendations, which was found to be higher for consumer recommendations by Bettman et al. (1998) and Senecal et al. (2004), may have contributed to the higher interest through larger pupil dilations. This is in line with the similarity-attraction paradigm (Byrne 1973). Former and potential consumers are more similar in terms of experiences, goals, interest, etc. These similarities result in a higher level of attraction between them. The attraction is the foundation of the interest that consumers have on online consumer recommendations. Based on our results, we conclude that using consumer reviews as the source for recommender systems has its advantages in gaining consumers’ interest than using expert.

Contrary to our prediction, fixation intensity is not significantly influenced by recommendation timing. Recommendation timing does not have anchoring effect on consumer’s attention and interest.

6.2 Limitation

There are some limitations in this study which calls for future work. First, for better experiment control, the recommender system algorithm is not included in the recommender system design in this study. Future research can integrate personalized product recommendation in the experiment design. Second, the subjects are undergraduate students from a university in the United States, which may limit the generalizability of the study. It is possible that their attention to the recommendations varies from those with different demographic and cultural backgrounds.

6.3 Contribution

This research contributes to the understanding of the characteristics of online recommender systems via eye tracking approach. Despite the importance of online recommender systems to online retailers, few guideline exists for online recommender systems on which features of online recommender systems that can help to boost sales. The findings from this research can help some online retail business owners to increase the effectiveness of their recommender systems in attracting consumers’ attention.

The eye tracking method can help open the black box of the decision-making process during online shopping. Eye-trackers are used as a source for objective, non-invasive, continuous, and quantitative data which has the potential to help researchers studying human attention, mental load, cognitive processes, etc.