Is everybody an expert? An investigation into the impact of professional versus user reviews on movie revenues

Abstract

This study is the first attempt to examine the effect of electronic word of mouth (user reviews) relative to expert reviews on moviegoing decisions. For the first time, we use time-varying data on expert reviews. We find that expert ratings matter much more for moviegoing decisions than user ratings and volume. Our data also show that experts tend to be more critical but more consistent in their reviews than users. We find that experts, but not eWOM, affect wide release moviegoing, contrary to industry thinking. Finally, we show that experts’ reviews matter most when consumers and critics are in closer agreement about the quality of the film. The study uses OLS as well as instrumental variables analysis to account for possible endogeneity.

Appendices

Appendix 1: Instrumental variables and first-stage estimations

1.1 Instrumental variables

Rossi (2014) in a recent critical overview of instrumental variable analysis highlights the need for researchers to adequately identify the potential sources of omitted variable bias (i.e. endogeneity issues) and discuss why the chosen instruments should be considered exogenous from the estimation equation but related to the independent variables. We address these issues below. Then we discuss additional instruments generated leveraging heteroskedasticity in the first-stage estimations (Lewbel 2012) to further aid in identification.

There are seven potentially endogenous variables in our analysis: $UserVolume$, $UserRating$, $UserVariance$, $CriticRating$, $CriticVariance$, Screen, and Advs. We note that we are not concerned with correlation between the number of critic reviews ($CriticVolume$) and the error term because critics are typically assigned to review a film well in advance of the film’s release date¹²—it is highly unlikely that editors when assigning critics to review films consider predictions of changes in a film’s unobservable characteristics weeks after its release.

While we control for time invariant film characteristics with first differencing, we still need to control for possible time variant endogeneity. For example, both critical and user reviews may be driven by the characteristics of the focal film relative to other films on the market. The focal film’s relative “quality” may change over the 10-week run as the set of other films on the market changes every week. Differential relative “quality” of the same movie may drive different average ratings from both critics and users as well as induce a higher volume of internet reviews which consequently can generate higher box office revenues. This results in endogeneity concerns for variance of critic and user ratings as well since they are a function of their respective averages. Similarly, different relative “quality” may impact studio distribution and advertising strategies. Chintagunta et al. (2010) and Gopinath et al. (2013) express similar concerns in their study and identify viable instruments for three of these variables, namely UserVolume, $UserRating$, and Screen.

We follow Neelamegham and Chintagunta (1999) in identifying an instrument for Screen using the average number of screens that show movies of the same genre as the focal movie i in week t ($CompScreen_{it}$). However, we deviate from Chintagunta et al. (2010) in our choice of instruments for $UserVolume$ and $UserRating$ since our data are national rather than local.

Instruments need to be correlated with the suspected endogenous variable but not with the error term in Eq. (2), $\Delta \varepsilon_{it}$ (Greene 2011a; Rossi 2014). One benefit of first differencing to account for time invariant film-specific effects is that we can use lagged levels of our endogenous variables as instruments (Greene 2011a): “without group effects, there is a simple instrumental variables estimator available. Assuming that the time series is long enough, one could use the lagged differences … or the lagged levels … (p. 308)”. Lagged levels are appropriate as long as they are not correlated with $\Delta \varepsilon_{it}$ and influence the independent variable (Greene 2011a; Rossi 2014). For $UserVolume$ and $UserRating$ we believe users are highly unlikely to consider forecasts of $\Delta \varepsilon_{it}$ when (1) deciding to leave a review and (2) evaluating the film. Also, Moon et al. (2010) show that previous user ratings influence future user ratings. Additionally, in a recent article published in Science, Muchnik et al. (2013) show in a randomized experiment that social influence works on users in that prior ratings of others significantly affected individual ratings. Indeed, one problem of using lagged levels as instruments is that they are typically weak with little correlation to the first difference (Greene 2011a). However, as we show in our first-stage estimations below, this is not a concern in our dataset—the lagged levels of $UserVolume$ and $UserRating$ have significant explanatory power on their counterparts in the first-stage estimations.

We leverage the longitudinal aspect of our data set to obtain additional instruments which can help identify $UserVolume$. For film i observed in week t after release, we find the average number of user reviews in the tth week after release for all movies released before film i from a different genre. For example, the $UserVolume$ of a movie observed in its 3rd week will be instrumented by the average 3rd week $UserVolume$ of all movies from a different genre released before the focal movie. A similar approach is used in Lee’s (2013) study of the video game industry when obtaining instruments for console and game prices. This measure will be uncorrelated with $\Delta \varepsilon_{it}$ by construction since users are very unlikely to consider forecasts of future films’ $\Delta \varepsilon$, especially future films in different genres, when deciding to leave their reviews.

For each film in each week t, we obtain the average UserVolume in the tth week of all previously released movies in a different genre ($PrevUserVolumeDiffGen$). We use the change in average tth from the $t - 1{\text{th}}$ week to the tth week of all previously released movies in a different genre ($\Delta PrevUserVolumeDiffGen$). We expect a positive correlation with the change in UserVolume as the instrument likely captures industry wide changes in the amount of reviews consumers typically leave for films from week to week. Another concern highlighted by Rossi (2014) is the use of instruments that may not vary among groups (e.g. price indices). It is important to note that there is variation in this instrument by film and over time since very seldom we find two films of the same genre released at the same time in our dataset.

$UserVariance$ is likely to be identified by instruments for $UserRating$ because the latter is a nonlinear function of the former. To address this nonlinear relationship, we include the natural log of the lagged level of $UserRating$, $Log\left( {1 + UserRating} \right)$,¹³ as an additional instrument for UserValence.

The valence of professional critic reviews, $CriticRating$ may also be endogenous since critical reviews should be correlated with unobserved relative movie “quality.” We use an instrument for critics’ ratings designed to capture critic experience.¹⁴ More experienced critics can be in a different position vis-a-vis corporate headquarters. There is an entire literature in finance and economics suggesting that people with more experience may require higher incentives to act in accordance with shareholders’ values (Prendergast and Stole 1996; Scharfstein and Stein 1990; Zwiebel 1995). In line with this research, Ravid et al. (2006) argue that professional movie critics with a better reputation/more experience exhibit stronger corporate biases than others. As such, we expect average critics’ experience to be negatively correlated with the change in average critical rating.

Reviewers’ experience is almost by definition completely uncorrelated with the unobserved relative “quality” of the movie. It is difficult to obtain data on critics’ tenure and experience, but as a proxy we are able to obtain previous reviews on RT.¹⁵ We postulate that the greater the number of reviews, the greater the experience. For each critic, we find the number of reviews posted prior to reviewing the focal film. Then we find the level and the natural log of the average for all critics who review the focal film. Though we expect a positive impact of experience on the change in average critical rating, we include the natural log to address a possible nonlinear relationship. These values, $AvgPrevRev_{it}$ and $Log\left( {1 + AvgPrevRev_{it} } \right)$, change as new critical reviews become available throughout a movie’s run.¹⁶ Note that we do not use the lagged level of $CriticRating$ as an instrument because it is unlikely that professional/well-trained critics coordinate their reviews taking into account previous critical assessment of a film (although this remains an empirical question).¹⁷

As with $UserVariance$ and $UserRating$, we expect $CriticVariance$ is in part identified by instruments for $CriticRating$. However, we generate additional internal instruments leveraging heteroskedasticity in the first-stage estimation of $CriticVariance$ (Lewbel 2012) to help separately identify the impact of the variance of critic rating on box office. We discuss this in greater depth below.

Another potential endogenous variable is advertising. We follow Chintagunta et al. (2010) and note that (a) prerelease advertising accounts for the vast majority of advertising spending in the movie industry (e.g., Elberse and Anand 2007 find that 88% of television advertising spending was spent prior to initial release), (b) prerelease advertising budgets are typically a fixed proportion of production budget (see Ravid 1999; Vogel 2007). Finally, any impact of prerelease advertising would be captured by $\mu_{i}$ since this value is unchanged over the run of the film—the estimations control for this through first differencing. However, changes in advertising expenditure over the course of a film’s run may be influenced by the unobserved relative “quality” of competing films. As an instrument we leverage the panel structure of the data and use the level of advertising expenditure in the previous week. Since the endogenous variable in the estimation equation is the lagged first difference of advertising, the instrument is the 2-period lagged level of advertising. This is a valid instrument as it is unlikely that firms set advertising expenditures in a given week considering the change in forecasted future shocks two or more periods out, $\Delta \varepsilon_{it}$. In the movie industry, as is well established in the marketing literature, the key emphasis is on buzz creation and brand awareness (Houston et al. 2018). Advertising is less likely to have such long-term effects in this industry. For example, De Vries et al. (2017) find that empirically only one lag of advertising mattered in their VARX model exploring the influence of several variables (including traditional advertising) on their contemporaneous counterparts.¹⁸

1.2 Additional instruments leveraging heteroskedasticity in the first-stage estimations

We note that the empirical results below are robust to including only the traditional instruments we outline above. However, in our empirical setting it is difficult to strongly identify all endogenous variables using traditional external instruments alone. Identification improves when we leverage the procedure outlined in Lewbel (2012) and augment the traditional instruments with internal instruments that exploit heteroskedasticity in the first-stage estimations.¹⁹

Lewbel (2012) shows that if heteroskedasticity is present in the first-stage estimations,²⁰ then additional instruments can be created by interacting the residuals of the respective first-stage estimation with any (or all) demeaned independent variables included in the first-stage estimation. The key idea is that while the exogenous regressors in the first-stage estimation are uncorrelated with the error term in the first-stage regression by construction, there is no reason to believe that the residuals will be independent of the regressors in the reduced form estimation. If the residuals are heteroskedastic (i.e. dependent on the regressors), then this information can be used to further untangle the endogenous part of the offending variable from the exogenous part. In fact, more heteroskedasticity aids in identifying the endogenous regressor (Lewbel 2012).

In the extreme, Lewbel (2012) shows that models are identified using only heteroskedasticity in first-stage estimations without any additional instruments, though “[t]he resulting identification is based on higher moments and so likely to provide less reliable estimates than identification based on standard exclusion restrictions, but may be useful in applications where traditional instruments are not available or could be used along with traditional instruments to increase efficiency” (p. 67). We follow this advice and use any additional instruments along with the traditional instruments we discuss below.

Additional instruments are created first by estimating the first-stage regression including all exogenous variables and instruments and obtaining residuals. Then the residuals are interacted with demeaned values of the relevant regressors from the first-stage estimation to create new variables. Any new variable created using this method is then included like a standard instrument in instrumental variable analysis.

We do not include extra instruments generated from the residuals of all first-stage estimations interacted with all exogenous variables to avoid instrument proliferation which may weaken results (Roodman 2009). Rather, we focus on generating instruments for endogenous variables that may be difficult to identify using external instruments alone. Our main concern is with identifying $\Delta UserVolume$, $\Delta UserVariance$, $\Delta CriticRating$, $\Delta CriticVariance$, and ∆Screen. We believe the impact of the variance of user opinion will be difficult to separately identify from $UserRating$ and $UserVolume$ (given $UserVariance$ is a function of both) with external instruments only. It also may be difficult to separately identify critic rating and variance of critic rating for similar reasons; this is likely compounded by the fact that we have only two external instruments ($AvgPrevRev_{it}$ and $Log\left( {1 + AvgPrevRev_{it} } \right)$) for both endogenous variables. Lastly, we augment the single instrument for $\Delta Screen$, which is not based on lagged level of the variable as the instrument for advertising is, to aid in identification.

For $\Delta UserVariance$ we create an additional instrument by interacting the demeaned $\Delta PrevUserVolumeDiffGen$ (${\text{DM}}\left( {\Delta PrevUserVolumeDiffGen} \right)$ where ${\text{DM}}\left( \bullet \right)$ is the demeaning operator) with the residuals from the preliminary first-stage estimation of lagged first differenced $UserVariance$ ($Res\Delta UserVariance_{it - 1}$). Our expectation is that the size of the variation in the variance of user opinion changes as more users leave reviews—we expect $\Delta PrevUserVolumeDiffGen$ will capture this since it is a relevant instrument for $UserVolume$. We include the interaction of the first-stage residuals for $\Delta UserVolume$ with its demeaned lag level to help separately identify from $\Delta UserVariance$. For $\Delta CriticRating$ and $\Delta CriticVariance$, we create two additional instruments by interacting the residuals from both preliminary first-stage estimations with demeaned $Log\left( {1 + AvgPrevRev_{it} } \right)$ because it is likely that critic experience influences both critic rating and the size of the variation in critic rating.²¹ Additionally, we create another Lewbel (2012) style instrument for $\Delta CriticRating$ using lagged level user rating. The key idea is that the level of disagreement among critics may be related to the popular appeal of the film. Also as the variance in critical opinion may be related to the popularity of the film, we include another instrument for $\Delta CriticVariance$ using lagged level user volume. Finally, we use lagged level user volume to create another instrument for $\Delta Screen$. This is a relevant instrument if studios alter their distribution intensity in response to online chatter and if the variation in the size of the response varies with the amount of online chatter.

1.3 First-stage estimations

We list the descriptive statistics for the traditional external instruments as well as the Lewbel (2012) style instruments in Table 11. We show the first-stage estimations for our full model in Table 12. The first-stage results indicate the instruments are relevant (all first-stage F-statistics are well above 10) and the signs generally conform to priors: $\Delta UserVolume$, $\Delta UserRating$, and $\Delta Log\left( {1 + Advs} \right)$ are significantly identified by using lagged levels; $\Delta PrevUserVolumeDiffGen$ has a positive and significant impact on $\Delta UserVolume$; critical experience captured in $AvgPrevRev_{it}$ and $Log\left( {1 + AvgPrevRev_{it} } \right)$ is significantly related $\Delta CriticRating$²² and $\Delta CriticVariance$; $\Delta CompScreen$ significantly identifies ∆Screen.

Table 11 Descriptive statistics for instrumental variables

Table 12 First-stage regressions. The dependent variable in each estimation is listed across the first row

Finally, the additional instruments created leveraging heteroskedasticity in the preliminary first-stage estimations aid in identification. They significantly impact their respective endogenous variables, and they are all significant in other first-stage estimations. Note that we do not make predictions of the signs of these additional instruments since we do not have theoretical guidance on the impact of higher moments (Lewbel 2012). However, the significance of these variables suggests they are related to heteroskedasticity in the first-stage estimations (which is what is required for identification).

Appendix 2: Robustness check of main model using imdb.com and metacritic.com data

We obtain data on critical evaluation from Metacritic.com and consumer evaluation from IMDB.com for each movie included in the analysis in a similar fashion as the RT data. We use the Metacritic.com and IMDB.com data to calculate variance in critic and user valence within a week rather than the variance in average evaluation from week to week, something we are not able to do with the RT data. We estimate the main models in the paper (Table 7, Models 3 and 4) using this data, along with the more precise measures of variance. Descriptive statistics for these variables and for the instruments used in this robustness check are in Table 13 (note that we are able to identify the endogenous variables in instrumental variable estimation with fewer Lewbel style instruments); estimation results are in Table 14. The results are qualitatively similar to the findings in the text based on the RT data.

Table 13 Descriptive statistics for IMDB.com and Metacritic.com data

Table 14 OLS and instrument variables robustness check regression results using GMM estimations