Behavior Research Methods

, Volume 49, Issue 6, pp 2061–2082 | Cite as

A film set for the elicitation of emotion in research: A comprehensive catalog derived from four decades of investigation

  • T. Lee Gilman
  • Razan Shaheen
  • K. Maria Nylocks
  • Danielle Halachoff
  • Jessica Chapman
  • Jessica J. Flynn
  • Lindsey M. Matt
  • Karin G. Coifman
Article

Abstract

Emotions are highly influential to many psychological processes. Indeed, research employing emotional stimuli is rapidly escalating across the field of psychology. However, challenges remain regarding discrete evocation of frequently co-elicited emotions such as amusement and happiness, or anger and disgust. Further, as much contemporary work in emotion employs college students, we sought to additionally evaluate the efficacy of film clips to discretely elicit these more challenging emotions in a young adult population using an online medium. The internet is an important tool for investigating responses to emotional stimuli, but validations of emotionally evocative film clips across laboratory and web-based settings are limited in the literature. An additional obstacle is identifying stimuli amidst the numerous film clip validation studies. During our investigation, we recognized the lack of a categorical database to facilitate rapid identification of useful film clips for individual researchers’ unique investigations. Consequently, here we also sought to produce the first compilation of such stimuli into an accessible and comprehensive catalog. We based our catalog upon prior work as well as our own, and identified 24 articles and 295 film clips from four decades of research. We present information on the validation of these clips in addition to our own research validating six clips using online administration settings. The results of our search in the literature and our own study are presented in tables designed to facilitate and improve a selection of highly valid film stimuli for future research.

Keywords

Emotion Film Stimuli Emotion elicitation 

The study of emotion in relation to cognition, behavior, and health has grown exponentially over the last several decades. Indeed, emotion responses and emotion regulatory strategies are increasingly recognized as central to many, if not most, psychological processes. As such, there is a growing reliance on laboratory paradigms employing emotional stimuli to induce, alter, or simulate emotional contexts for investigation across the social sciences and, most notably, in psychology. Although there are currently a variety of methods used including emotional images (e.g., International Affective Picture System: Lang, Bradley, & Cuthbert, 2008), music (Koelsch, 2010; Logeswaran & Bhattacharya, 2009), and personal recollection (e.g., Lench & Levine, 2005; Papa & Bonanno, 2008), there is an increasing reliance on emotional film clips. For example, a search for “emotion elicitation” and “film clips” on Google Scholar and on psychology-specific databases (e.g., APA) yielded over 1,000 results. The use of film clips for emotion elicitation has many advantages. Clips are easily standardized and therefore reliable as compared to idiographic methods (e.g., personal recollection; see Mills & D'Mello 2014; Salas, Radovic, & Turnbull, 2012). Film clips readily engage participants for extended periods and allow for an ecologically valid induction, progression, and assessment of emotional responses (Kring & Gordan, 1998). Clips capture participant attention in a manner that is consistent with contemporary life and allow for the simulation of real world conflicts or distress with relatively few ethical concerns (Rottenberg, Ray, & Gross, 2007). Moreover, clips can be transformed or adapted to fit specific needs, and film clips are accessible to varying populations. Due to these advantages, film clips will continue to be an integral part of emotion elicitation in research.

For the present investigation, we focused on testing the efficacy of clips in a contemporary young adult population, targeting emotions that are less easily differentiated, and using online administration of emotionally evocative film clips that were initially piloted in a laboratory setting (see Supplemental Material for details). There has been a tremendous increase in the application of emotion stimuli to online research with relatively little validation. In particular, there have only been two prior studies evaluating the efficacy of film clips to elicit emotions online (Aldao & Nolen-Hoeksema, 2013; Samson, Kreibig, Soderstrom, Wade, & Gross, 2016). Moreover, previous laboratory studies have noted difficulties encountered by participants in differentiating certain emotional experiences with word labels, particularly for positive emotions (Ellsworth & Smith, 1988; Herring, Burleson, Roberts, & Devine, 2011). Similarly, some negative emotions are frequently co-elicited and have been traditionally more challenging to reliably target, particularly online (Samson et al., 2016). For instance, anger is commonly recognized as especially difficult to elicit with standard stimuli, and there are few highly reliable film clips derived from contemporary films. This may be because anger often co-occurs with disgust, particularly when content is perceived as morally disgusting (Salerno & Peter-Hagene, 2013; Whitton, Henry, Rendell, & Grisham, 2014). As such, in this investigation, we focused on testing the utility of presenting film clips online to elicit specific, discrete emotions that are frequently co-elicited, specifically happiness and amusement, as well as anger and disgust. We selected film clips geared towards college students that we hypothesized would discretely elicit happiness, amusement, anger, or disgust. For these film clips, we evaluated elicitation of the target emotion as well as the paired, potentially co-elicited emotion in young adults, a highly common sample in emotion-related research across all of the behavioral sciences. However, in our efforts to identify useful evocative stimuli for these challenging applications, we recognized the lack of a contemporary catalog of film clips to aid emotion elicitation researchers in identifying validated stimuli for future investigations. Therefore, this investigation’s goal was two-fold: in conjunction with our own validation study on effective film stimuli for the discrete elicitation of frequently co-elicited emotions, we also provide here a database reflecting a compilation of validated, emotionally evocative film clips to facilitate future emotion elicitation research.

Online validation study

We conducted an online study (N = 784 young adults) testing the effectiveness of 15 film clips, identified through in-lab pilot research (see Supplemental Material). We employed a rigorous and conservative analytic strategy that explicitly tested the discreteness of each clip in eliciting the targeted emotion(s). Specifically, we tested the utility for a given clip in discretely eliciting each of two possible, frequently co-elicited, target emotions, as well as their degree of shared activation. Typically, prior research studies (indeed most of the studies in the catalog) have tested the utility of a clip in eliciting one target emotion (e.g., sadness in The Champ: Gross & Levenson, 1995) and thus the effectiveness of the clip is only analyzed in comparison to other clips targeting the same emotion. However, given our clip selection and motivation to identify clips particularly effective at eliciting emotions that are often harder to differentiate, we employed a slightly different strategy. We tested the utility of the clip in discretely eliciting both the target emotion as well as a likely secondary emotion, as in the case of disgust and anger or amusement and happiness. As such, most clips were analyzed twice and compared against a broader range of films. In addition, we generated a mixed-feelings score (Hemenover & Schimmack, 2007) to determine the shared intensity of emotions likely to be co-elicited (anger and disgust, happiness and amusement).

We based our film clip selection on pilot research in the laboratory with groups of college-aged participants (total pilot N = 393), enabling us to evaluate the effectiveness of film clips across study settings (i.e., in the laboratory and online). The pilot research is described in complete detail in the accompanying Supplemental Material. However, a brief summary of the research is as follows: from an initial pool of approximately 40 clips, 13 clips were selected and evaluated by presenting to groups of 2–15 college students. Two separate cohorts (Sample A, n = 91; Sample B, n = 302) of participants were employed for the pilot, and film clips were presented in a fixed order under supervision by laboratory personnel. Participants completed emotion ratings on the same index as in the Online Validation Study. For more information regarding the specifics of film clip presentation and emotion rating assessments, sample demographics, statistical evaluation, and global findings, please refer to the Supplementary Material. Importantly, the validated film clips from the pilot study were consistent with those validated in the Online Validation Study. However, limitations of the pilot study included the fixed-order presentation of film clips in the pilot study, the lack of a film clip eliciting disgust in the absence of any anger confounds, and the absence of a neutral film clip.

As stated above, the goals of this study were to investigate the effectiveness of a set of film clips in eliciting emotions through an online medium, replicating the in-lab pilot work. In addition, we sought to address limitations from our pilot study to strengthen the present investigation. First, to evaluate the clips’ elicitation of discrete emotions in the absence of potential order confounds, we took advantage of an online format – which also allowed for a considerably larger sample due to reduced lab personnel demand – and randomized the order of clip presentation to avoid any order confounds. Second, we included two additional clips in the present study that improved our ability to contrast clip effectiveness. Specifically, we included a previously validated clip eliciting only physical disgust (not anger or moral disgust1) as a comparator against clips that might co-elicit (moral) disgust when anger was the target emotion. Feelings of moral disgust are often reported when content that is intended to elicit anger is perceived as morally upsetting (Salerno & Peter-Hagene, 2013; Whitton et al., 2014). In addition, we included a novel clip for potential use as a neutral reference stimulus. These two clips were included with the 13 preliminarily evaluated in our pilot study (see Table S1).

Procedure

Participants aged 18–44 years were recruited (N = 784, age M = 19.98 SD = 3.09, 76% female) from the undergraduate subject pool of a large public university in the Midwest for a study on reactions to emotion films. This is after removal of non-English speaking participants (N = 8), as well as omission of any participants that completed the survey twice (N = 5). All materials for the Online Validation Study were distributed via Qualtrics secure server to participants’ email addresses. The investigation involved each participant viewing five (randomly selected out of a possible 15) emotion elicitation film clips in randomized order, followed by a sixth mood-lifting clip (to ensure no lasting mood effects). After each of the five emotional film clips, participants were asked to complete affect ratings then answer an accuracy question about the film clip they just viewed. Time between clips was approximately 2 min. Participants were compensated with course credit after completing the study. Refer to Supplemental Appendix 1 for Qualtrics programming details.

Film stimuli

Fifteen clips were assessed, garnered from commercially available films or publically available documentaries, internet programs, and personal videos. Twelve new clips (Alive, Between Two Ferns, College Conspiracy, Crash, D2: The Mighty Ducks, Fahrenheit 911Bin Laden, Fahrenheit 911Recruitment, Funny Cats, The Office, Police Brutality, The Road to Guantanamo, Whose Line is it Anyway; see Table S1 for details) were utilized based on our pilot investigations to elicit specific emotional responses in college students including happiness, amusement, anger, disgust, and sadness.2 A clip from Big Cat Diary (BBC Earth, 2010) was investigated for potential use as a neutral reference film clip. Two of the 15 clips were included based on prior research: The Champ (MGM 1979, c.f. Gross & Levenson, 1995; also included in our pilot studies) to providing a contrasting negative emotion (sadness), and Trainspotting (Miramax, 1996) previously validated to elicit disgust without eliciting anger (see Schaefer, Nils, Sanchez, & Philippot 2010). Each of the 15 film clips evaluated were approximately 5 min in length.

Emotion ratings

Immediately after each film clip, participants were instructed to rate how they were feeling using a Likert scale from 1 (none) to 7 (strong). The following emotions were rated: anger, fear, sadness, guilt, surprise, interest, happiness, amusement, affection, and disgust. These words were selected as they are most consistently used in this type of emotion elicitation research (e.g., Gross & Levenson, 1995; Rottenberg et al., 2007) and were restricted to a total of ten to limit demand on participants.

Verification of participant engagement with online content

Because this study was reliant on participants engaging with the film content on their own from their personal computer (as compared to in lab during the pilot study; see Supplemental Material), we included several additional features to also assess the degree of participant engagement in the task itself. For example, after every film clip was viewed, participants were asked to respond to an accuracy question. This accuracy question was the primary means for determining whether participants had indeed engaged in watching the film clips, as online presentation of the videos inherently lacks the authoritative oversight of experimenters as in the pilot study. Those participants that failed to answer the accuracy question correctly for a particular video, or who indicated that they experienced problems when viewing the film clip, were excluded from analysis for that video (see Table S8 in the Supplemental Material for details about accuracy questions and answers). For example, the accuracy question for the Trainspotting clip was “In what room did most of the action in this film clip take place?” with a fill-in-the-blank response format. This clip involves the main character entering a room with the word “TOILET” on the door and proceeding to reach and crawl into a porcelain toilet bowl. Responses such as “bathroom,” “toilet,” “restroom,” “in a run-down gross bathroom,” or “in a nasty bathroom” were considered correct, and data from those participants were included in analyses. Responses such as “family,” “dining,” “office,” “I forget,” “household room,” “dorm room,” or “the interrogation room” were incorrect, and data from these participants were excluded from analysis of this video. Those participants who responded by saying “I do not know because the video never showed up,” “video was not working,” or “my film didn’t appear” were also excluded. Such open-ended questions allowed for identification of participants who actually were engaged in watching the video to rigorously ensure reliability of affect ratings and eliminated potentially inaccurate data from subsequent analyses (e.g., for Trainspotting 218 participants viewed the film clip, but 51 participants (23.4%) were excluded due to inaccurate answers). Participants could also email a study coordinator (contact information was provided in the instructional email) if they experienced any issues with viewing film clips. See Table S9 in the Supplemental Materials for specifics on the number of participants excluded for each film clip in the Online Validation Study. Across all films, the mean exclusion was 32 individuals, range = 6–64.

In addition to the accuracy question for each film clip, we included several programming features to permit additional preventative and confirmative measures to ensure participant involvement (cf., Crump, McDonnell, & Gureckis, 2013; Ferrer, Grenen, & Taber, 2015; Gureckis et al., 2015; Mason & Suri, 2012; Woods, Velasco, Levitan, Wan, & Spence, 2015;). These supplemental data accuracy checks included pre-experiment software download instructions, limiting the time for responses to ratings and accuracy questions, minimizing the delay between film clip presentation and affect reporting, and quantifying mouse clicks during the video presentation (see Table S10 in the Supplemental Material). These permitted analyses of behaviors indicating lack of engagement with the film clips (e.g., how many mouse clicks occurred during the film clip presentation). We also evaluated whether prior exposure to the film clips would correlate with reduced intensity of the targeted emotional responses. Overall, there were few meaningful associations.3 However, these techniques proved quite useful in reducing the sample to one that had the highest likelihood of having actually engaged with each film clip in question.

Results

Data analytic strategy

Mean ratings across all emotions for all film clips are presented in Table 1. Responses to clips were analyzed in three steps, employing first techniques developed by Gross and Levenson (1995), then following recent examples (e.g., Jenkins & Andrewes, 2012), and finally using tools specifically designed for evaluating co-elicitation (Hemenover & Schimmack, 2007). In particular, we strove to test the distinctness of each clip in eliciting the target emotion as well as a secondary emotion (or common confounds), particularly because clips were targeting emotions that are often hard for participants to reliably differentiate (e.g., anger from disgust). Specifically, following the example of Gross and Levenson (1995), we first identified the most successful film clips (i.e., the “success index”) using a combination of the standardized mean intensity of a given clip for the target emotion(s) (e.g., mean rating of sadness in The Champ) as well as a discreteness value (hit rate), which reflects the percentage of participants rating the target emotion(s) at least one point above all non-target emotions for a given clip. We categorized films by target emotions, and for each target emotion, the mean intensity and mean discreteness across the sample within each film clip were standardized as z-scores. The sum of these intensity and discreteness z-scores determined the success index of each film clip relative to the other film clips assessed for that same target emotion, taking into account the ability of the film clip to both intensely and discretely elicit the targeted emotion. Clips were compared against all others eliciting similar emotions for their success rate for each emotion. As such, all clips potentially eliciting anger or disgust were compared against each other for anger and for disgust, respectively. This permitted identification of the film clip most successful at eliciting the particular target emotion in relation to the other clips potentially eliciting that emotion. Definitive thresholds do not exist for a film clip to be considered successful in eliciting an emotion, particularly in instances when emotions are co-elicited, such as anger and disgust, or happiness and amusement. Traditionally, the highest success rate has been the recommended clip (see Gross & Levenson, 1995).
Table 1

Online Validation Study: Film clips for eliciting discrete emotional states online

Film clip

Sex

Interest

Fear

Sadness

Surprise

Guilt

Happiness

Anger

Amusement

Disgust

Affection

Alive

M (N=55)

3.13 (1.36)

1.73 (1.26)

2.36 (1.38)

2.61 (1.44)

1.38 (0.85)

3.98 (1.92)

1.46 (1.08)

2.80 (1.69)b

1.41 (0.93)

2.66 (1.85)

F (N=138)

2.96 (1.52)

1.44 (0.84)

1.89 (1.25)

1.95 (1.41)

1.13 (0.43)

3.68 (1.91)

1.16 (0.64)

2.13 (1.32)

1.12 (1.44)

1.86 (1.32)

Between Two Ferns

M (N=54)

4.39 (1.77)

1.20 (0.63)

1.31 (1.08)

2.89 (1.89)

1.22 (0.66)

4.37 (1.97)b

1.30 (0.77)

4.76 (2.15)

1.70 (1.46)

1.78 (1.38)

F (N=131)

3.66 (1.76)

1.16 (0.65)

1.35 (0.98)

2.52 (1.72)

1.25 (0.92)

3.45 (1.90)

1.30 (0.93)

4.35 (2.06)

1.87 (1.45)

1.50 (1.04)

Funny Cats

M (N=46)

4.02 (1.73)

1.30 (0.81)

1.15 (0.56)

2.63 (1.69)

1.17 (0.64)

4.76 (1.55)

1.20 (0.62)

4.85 (1.61)

1.33 (1.06)

2.67 (1.92)

F (N=124)

4.59 (1.65)

1.18 (0.59)

1.17 (0.54)

3.10 (1.84)

1.11 (0.37)

5.07 (1.77)

1.10 (0.40)

5.52 (1.66)a

1.12 (0.40)

3.53 (1.94)

D2: The Mighty Ducks

M (N=34)

4.26 (1.54)

1.15 (0.44)

1.21 (0.59)

2.12 (1.39)

1.00 (0.00)

4.21 (1.68)

1.29 (0.80)

3.71 (1.73)

1.21 (0.88)

2.38 (1.99)

F (N=120)

3.82 (1.59)

1.15 (0.51)

1.14 (0.49)

2.16 (1.50)

1.10 (0.51)

4.15 (1.83)

1.18 (0.72)

3.59 (1.74)

1.10 (0.48)

2.16 (1.53)

The Office

M (N=46)

4.89 (1.84)

1.57 (0.94)

1.43 (0.89)

2.17 (1.27)

1.24 (0.71)

4.22 (2.16)a

1.46 (1.13)

4.72 (2.13)

1.52 (1.07)

2.02 (1.56)

F (N=136)

4.16 (1.89)

1.63 (1.19)

1.29 (0.80)

2.40 (1.71)

1.24 (0.70)

3.42 (2.04)

1.49 (1.06)

4.62 (2.11)

1.58 (1.19)

1.63 (1.31)

Whose Line is it Anyway

M (N=44)

4.89 (1.72)

1.18 (0.58)

1.91 (1.58)

2.80 (1.75)

1.27 (0.54)

4.80 (1.85)

1.43 (1.00)

5.41 (1.80)

1.23 (0.83)

2.84 (1.93)

F (N=137)

4.42 (1.81)

1.12 (0.49)

1.66 (1.09)

2.44 (1.63)

1.20 (0.62)

4.58 (1.94)

1.09 (0.44)

5.01 (1.90)

1.14 (0.44)

2.11 (1.62)

College Conspiracy

M (N=58)

4.79 (1.51)

3.69 (1.80)

3.26 (1.69)

3.69 (1.82)

1.90 (1.36)

1.47 (0.80)

4.66 (1.73)

1.83 (1.13)

4.74 (1.79)

1.55 (1.25)

F (N=166)

4.78 (1.56)

3.80 (2.02)

3.55 (1.88)

3.37 (1.82)

1.95 (1.44)

1.36 (0.75)

4.42 (1.82)

1.70 (1.15)

4.64 (1.99)

1.31 (0.80)

Crash

M (N=56)

4.21 (1.75)

2.96 (1.63)

3.48 (1.67)

3.39 (1.66)

2.21 (1.44)

1.46 (0.85)

5.00 (1.51)

1.91 (1.23)

5.36 (1.87)

1.68 (1.19)

F (N=175)

4.12 (1.81)

3.19 (1.78)

3.77 (1.98)

3.70 (1.82)

2.21 (1.65)

1.27 (0.77)

4.73 (1.96)

1.73 (1.26)

5.63 (1.68)

1.37 (0.87)

F. 911: Bin Laden

M (N=57)

4.40 (1.78)

2.30 (1.48)

2.39 (1.54)

3.58 (2.00)

1.46 (0.87)

1.44 (0.87)

3.30 (2.03)

2.14 (1.56)

3.63 (2.07)

1.32 (0.97)

F (N=163)

3.43 (1.99)

2.37 (1.61)

2.40 (1.64)

3.01 (1.94)

1.55 (1.16)

1.22 (0.60)

2.86 (1.77)

1.54 (1.10)

3.21 (2.02)

1.17 (0.58)

F. 911: Recruitment

M (N=59)

4.15 (1.71)

2.07 (1.40)

3.07 (1.87)

2.73 (1.82)

2.14 (1.50)

1.51 (0.86)

3.17 (1.87)

2.49 (1.64)

3.68 (2.02)

1.78 (1.16)

F (N=171)

3.87 (1.73)

2.40 (1.49)

3.65 (1.73)

2.73 (1.69)

2.27 (1.63)

1.39 (0.94)

3.51 (1.83)

1.57 (1.11)

3.63 (1.90)

1.55 (1.11)

Police Brutality

M (N=54)

4.44 (1.84)

2.83 (1.86)

2.87 (1.72)

3.80 (2.21)

1.83 (1.37)

1.44 (1.00)

5.28 (2.13)

2.33 (1.79)

5.65 (1.73)

1.50 (0.93)

F (N=188)

4.07 (1.78)

3.28 (1.90)

3.65 (1.99)

4.09 (1.93)

1.77 (1.42)

1.23 (0.91)

5.04 (1.91)

1.72 (1.34)

5.55 (1.75)

1.47 (1.09)

The Road to Guantanamo

M (N=52)

3.71 (1.70)

3.12 (1.63)

3.58 (1.78)

3.19 (1.66)

2.54 (1.59)

1.40 (1.05)

3.56 (2.00)

1.65 (1.27)

4.50 (1.82)

1.60 (1.05)

F (N=187)

3.61 (1.80)

3.85 (1.88)

4.93 (1.84)

3.55 (2.01)

3.16 (1.90)

1.17 (0.71)

4.16 (1.96)

1.22 (0.76)

5.16 (1.91)a

1.53 (1.22)

Trainspotting

M (N=39)

3.49 (2.00)

1.90 (1.29)

1.79 (1.17)

4.87 (1.64)

1.41 (0.88)

2.10 (1.50)

2.15 (1.66)

3.46 (1.90)

5.79 (1.66)

1.38 (1.09)

F(N=122)

2.42 (1.67)

1.82 (1.20)

1.48 (1.01)

3.81 (2.05)

1.16 (0.54)

1.46 (1.01)

1.77 (1.45)

2.07 (1.68)

6.29 (1.24)

1.07 (0.36)

The Champ

M (N=53)

3.04 (1.65)

1.75 (1.00)

4.81 (1.81)

2.15 (1.42)

2.08 (1.56)

1.51 (0.89)

2.13 (1.44)

1.75 (1.25)

2.00 (1.39)

2.58 (1.78)

F (N=179)

3.27 (1.67)

1.84 (1.33)

5.41 (1.57)a

2.03 (1.51)

1.99 (1.58)

1.15 (0.42)

2.09 (1.39)

1.28 (0.64)

1.87 (1.37)

2.74 (1.83)

Significant t-tests of target emotion intensities between the sexes for each video are indicated by superscripts: ap<0.05; bp<0.01

Next, we tested the efficacy of the clips in eliciting the target emotions by conducting within-subject comparisons of emotion rating by film (c.f. Jenkins & Andrewes, 2012). Specifically, we conducted a ten-level within-subject ANOVA comparing the ten rated emotions within each clip, using a Bonferroni correction for multiple comparisons and a Greenhouse-Geisser correction (ε) due to sphericity assumptions being violated (Gruber, Dutra, Eidelman, Johnson, & Harvey, 2011; Uhrig et al., 2016; von Leupoldt et al., 2007). The dependent variables were average intensity ratings, and pairwise comparisons were conducted between the target emotion and each non-target emotion using contrasts. The advantage of this approach is that it best accounts for individual differences in emotion ratings when testing for the effectiveness of a given clip. Finally, because we were specifically interested in identifying clips that elicit emotions, such as anger, that often co-activate other emotions, we employed a strategy to test the degree to which mixed emotions were present in participant ratings. Following Hemenover and Schimmack (2007), we calculated a mixed-feelings score by participant for each pair of target emotions (e.g., anger and disgust in The Road to Guantanamo), consisting of the minimum shared intensity of the two target emotions.

Each method of analysis offers certain statistical benefits, although there is some overlap. For example, the success index takes into account both how discretely the film clip elicited the target emotion, in addition to how intensely the target emotion was elicited, and allows comparisons across clips to identify the most effective one for the target emotion. The ten-level within-subject ANOVA compares the mean intensities of all measured emotions within a given clip, sharing an emphasis on intensity and discreteness with the success index analysis, but giving more specific information about that single clip regarding its emotion elicitation pattern and accounting for individual differences in reporting/rating. In contrast, the mixed-feelings score attempts to capture the shared intensity of targeted emotions, so that lower scores can suggest films that are highly discrete elicitors and higher scores suggest films that more strongly elicit mixed emotions. When interpreting the results, we extrapolated across analyses with the primary goal of identifying and best characterizing clips that consistently and reliably indexed the target emotion(s) above others.

Positive emotion films: success indices

We first evaluated the success index of the film clips by combining both the mean intensity for each film clip’s target emotion as well as the corresponding discreteness for that particular emotion for that film (the percentage of participants rating each target emotion at least one point above all others). Films were evaluated twice when there were two target emotions (happiness vs. amusement; anger vs. disgust) and compared against all other clips eliciting that target emotion (see Table 2 for a summary). Surprisingly, Funny Cats was found to have the highest success index for both amusement and happiness, with the latter being primarily attributable to the high mean intensity of happiness elicited. Between Two Ferns had the second highest success index for amusement, followed closely by The Office and Whose Line is it Anyway. Regarding happiness, Alive and D2: The Mighty Ducks had, respectively, the second and third highest success rates after Funny Cats.
Table 2

Online Validation Study: Mean intensity, discreteness, and success rates

Video

Emotion

Mean intensity

Discreteness

Success

Alive

Amusement

2.32

4.46

-3.01

Between Two Ferns

Amusement

4.47

33.51

1.25

Funny Cats

Amusement

5.34

26.47

1.46

D2: The Mighty Ducks

Amusement

3.62

5.19

-1.79

The Office

Amusement

4.65

28.57

1.01

Whose Line is it Anyway?

Amusement

5.11

24.31

1.08

Alive

Happiness

3.76

42.08

1.08

Between Two Ferns

Happiness

3.72

4.86

-1.35

Funny Cats

Happiness

4.99

9.41

1.21

D2: The Mighty Ducks

Happiness

4.16

23.38

0.61

The Office

Happiness

3.62

2.75

-1.66

Whose Line is it Anyway?

Happiness

4.64

2.21

0.13

College Conspiracy

Anger

4.48

7.14

1.69

Crash

Anger

4.79

2.16

0.40

Fahrenheit 911 - Bin Laden Family

Anger

2.97

3.18

-0.88

Fahrenheit 911 - Recruitment

Anger

3.42

3.48

-0.39

Police Brutality

Anger

5.09

8.68

2.71

Road to Guantanamo

Anger

4.03

0.84

-0.68

Trainspotting

Anger

1.86

0.00

-2.85

College Conspiracy

Disgust

4.67

12.50

-0.79

Crash

Disgust

5.56

32.47

1.11

Fahrenheit 911 - Bin Laden Family

Disgust

3.32

9.55

-2.23

Fahrenheit 911 - Recruitment

Disgust

3.64

12.61

-1.77

Police Brutality

Disgust

5.57

20.25

0.48

Road to Guantanamo

Disgust

5.01

18.83

-0.13

Trainspotting

Disgust

6.17

63.35

3.33

*Because the Champ was the only clip eliciting sadness we were not able to calculate a success rate

Success rates in bold reflect the highest for a given target emotion category

Positive emotion films: within-subject ANOVAs

Within-subject ANOVAs revealed that several of the film clips did not have significant differences between interest and the target emotion, indicating high engagement of the participants with the film clip (see Table 3). Alive significantly elicited happiness more so than all other emotions F(4.91, 941.70) = 121.81, p < 0.001, ε= 0.545, and was not suited for evoking amusement. Between Two Ferns, D2: The Mighty Ducks, and Whose Line is it Anyway had happiness levels significantly different from all other emotions except interest (Between Two Ferns F(2.83, 521.15) = 178.74, p < 0.001, ε= 0.315; D2: The Mighty Ducks F(3.63, 555.12) = 217.64, p < 0.001, ε= 0.403; Whose Line is it Anyway F(3.55, 639.37) = 311.69, p < 0.001, ε= 0.395). Mean intensities of amusement and interest were also not significantly different for D2: The Mighty Ducks and The Office F(3.58, 648.06) = 175.76, p < 0.001, ε = 0.398, though all other emotions of interest were significantly lower than amusement (p < 0.001). Between Two Ferns had amusement levels significantly higher than all other emotions, including interest. Both happiness and amusement were significantly different from all other emotions, including each other, for the Funny Cats film clip F(3.48, 588.60) = 378.03, p < 0.001, ε= 0.387.
Table 3

Within-subject ANOVAs, including pairwise comparisons of target to all non-target emotions, by film, for the Online Validation Study

Film clip

N

ANOVA statistics

 

Interest

Fear

Sadness

Surprise

Guilt

Happiness

Anger

Amusement

Disgust

Affection

Alive

199

ε=0.545; F(4.91, 941.70)=121.81

Happiness

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

p<0.001

p<0.001

p<0.001

198

Amusement

p<0.001

p<0.001

ns

ns

p<0.001

p<0.001

  

p<0.001

ns

Between Two Ferns

185

ε=0.315; F(2.83, 521.15)=178.74

Happiness

ns

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

p<0.001

p<0.001

p<0.001

Amusement

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

p<0.001

Funny Cats

170

ε=0.387; F(3.48, 588.60)=378.03

Happiness

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

p<0.001

p<0.001

p<0.001

Amusement

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

p<0.001

D2: The Mighty Ducks

154

ε=0.403; F(3.63, 555.12)=217.64

Happiness

ns

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

p<0.001

p<0.001

p<0.001

Amusement

ns

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

p<0.001

The Office

182

ε=0.398; F(3.58, 648.06)=175.76

Happiness

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

p<0.001

p<0.001

p<0.001

Amusement

ns

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

p<0.001

Whose Line is it Anyway?

181

ε=0.395; F(3.55, 639.37)=311.69

Happiness

ns

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

p<0.001

p<0.001

p<0.001

Amusement

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

p<0.001

College Conspiracy

224

ε=0.686; F(6.17, 1376.13)=238.43

Anger

ns

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

ns

p<0.001

Disgust

ns

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

ns

p<0.001

 

p<0.001

Crash

231

ε=0.652; F(5.87, 1350.16)=243.09

Anger

p<0.01

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

p<0.001

p<0.001

Disgust

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

Fahrenheit 911 - Bin Laden Family

220

ε=0.586; F(5.27, 1154.56)=112.94

Anger

p<0.001

p<0.001

p<0.001

ns

p<0.001

p<0.001

 

p<0.001

p<0.001

p<0.001

Disgust

ns

p<0.001

p<0.001

ns

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

Fahrenheit 911 - Recruitment

230

ε=0.684; F(6.16, 1410.22)=117.06

Anger

p<0.01

p<0.001

ns

p<0.001

p<0.001

p<0.001

 

p<0.001

ns

p<0.001

Disgust

ns

p<0.001

ns

p<0.001

p<0.001

p<0.001

ns

p<0.001

 

p<0.001

Police Brutality

242

ε=0.676; F(6.08, 1466.09)=267.40

Anger

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

p<0.001

p<0.001

Disgust

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

The Road to Guantanamo

239

ε=0.633; F(5.70, 1356.49)=223.02

Anger

ns

ns

p<0.001

p<0.01

p<0.001

p<0.001

 

p<0.001

p<0.001

p<0.001

Disgust

p<0.001

p<0.001

p<0.01

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

Trainspotting

161

ε=0.501; F(4.51, 721.03)=225.56

Anger

p<0.01

ns

ns

p<0.001

p<0.001

ns

 

ns

p<0.001

p<0.001

Disgust

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

 

p<0.001

The Champ

232

ε=0.747; F(6.72, 1552.04)=219.09

Sadness

p<0.001

p<0.001

 

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

p<0.001

Positive emotion films: mixed-feelings scores

Mixed-feelings scores (Table 4) were generated to indicate the minimum intensity rating shared by the two target emotions (amusement, happiness) that could be co-elicited. For film clips intended for elicitation of happiness or amusement, Alive had the lowest mixed-feelings score, followed by D2: The Mighty Ducks. The low mixed-feelings score of Alive may be attributable to the overall lower intensity of reported emotions elicited by this film. Funny Cats, which had the highest success index score for both amusement and happiness, also had the highest mixed-feelings score. Whose Line is it Anyway also had a high mixed-feelings score, while The Office and Between Two Ferns had moderate mixed-feelings scores.
Table 4

Online Validation Study: Mixed-feelings scores

Positive Emotion Films

(amusement; happiness)

Mixed-Feelings Score M (SD)

 Alive

2.19 (1.37)

 Between Two Ferns

3.57 (1.99)

 D2: The Mighty Ducks

2.93 (1.75)

 Funny Cats

4.79 (1.69)

 The Office

3.44 (2.05)

 Whose Line is it Anyway

4.48 (1.95)

Negative emotion films

(anger; disgust)

Mixed Feelings Score M (SD)

 College Conspiracy

4.12 (1.85)

 Crash

4.64 (1.91)

 Fahrenheit 911- Bin Laden Family

2.78 (1.82)

 Fahrenheit 911- Recruitment

3.08 (1.78)

 Police Brutality

4.86 (2.01)

 The Road to Guantanamo

3.89 (1.97)

 Trainspotting

1.85 (1.49)

Negative emotion films: success indices

Police Brutality had the highest success index (Table 2) for anger. Trainspotting, the film clip intended to evoke disgust without confounding anger elicitation, was found to have the highest success rate for disgust. College Conspiracy and Crash had the next highest success indices for anger and disgust, respectively. The latter finding suggests Crash may have elicited moral disgust, in comparison to the physical disgust evoked by Trainspotting. Surprisingly, The Road to Guantanamo had a lower success rate for anger but a moderately higher success index when evoking disgust. Both Fahrenheit 911 film clips had low success rates for anger and disgust. As with the pilot study, the majority of film clips were without sex effects on the mean intensities of the target emotions when evaluated by t-test; exceptions are noted in Table 1.

Consistent with our pilot study as well as other reports (c.f. Gross & Levenson, 1995), The Champ elicited sadness effectively, with high intensity (M = 5.28, SD = 1.64) and discreteness (67.24%). A success index was not calculated for The Champ, as it was the only film clip evaluated for evoking sadness.

Negative emotion films: within-subject ANOVAs

For the negatively valenced film clips, Trainspotting was effective for evoking disgust higher than all other emotions F(4.51, 721.03) = 225.56, p < 0.001, ε = 0.501, whereas anger was not significantly different from fear, sadness, happiness, or amusement (Table 3). This suggests that, as expected, this film did not carry an anger confound when eliciting physical disgust. Both Crash F(5.87, 1350.16) = 243.09, p < 0.001, ε= 0.652 and Police Brutality F(6.08, 1466.09) = 267.40, p<0.001, ε= 0.676 had significantly higher levels of both anger and disgust than all other emotions, and these emotions were also significantly different from each other. College Conspiracy had mean intensities of anger and disgust that were not significantly different from interest nor from each other, but were higher than all other emotions F(6.17, 1376.13) = 238.43, p < 0.001, ε= 0.686. Anger was not significantly different from interest or fear for The Road to Guantanamo, though disgust was significantly higher than all other emotions F(5.70, 1356.49) = 223.02, p < 0.001, ε=0.633. For the Fahrenheit 911: Bin Laden Family film clip, anger and disgust were not significantly different from surprise, and disgust was also not different from interest F(5.27, 1154.56) = 112.94, p < 0.001, ε= 0.586. Furthermore, anger and disgust were not significantly different from sadness nor from each other for the Fahrenheit 911: Recruitment clip, and disgust was not significantly different from interest F(6.16, 1410.22) = 117.06, p < 0.001, ε= 0.684.

The Champ was effective at significantly (p < 0.001) eliciting greater sadness than all other emotions F(6.72, 1552.04) = 219.09, p < 0.001, ε= 0.747, in agreement with previous reports.

Negative emotion films: mixed-feelings scores

Evaluation of mixed-feelings scores (Table 4) of anger and disgust for the negative emotion films revealed that Trainspotting had the lowest score, in agreement with its ability to elicit physical disgust without evoking anger. Police Brutality and Crash had the highest mixed-feelings scores, followed by College Conspiracy and The Road to Guantanamo. The two Fahrenheit 911 clips had low-to-moderate mixed-feelings scores, but this is likely attributable to the overall low intensity of emotional responses elicited in response to viewing these clips.

Neutral film clip

Big Cat Diary, the film clip included as a potential neutral reference, was actually found to elicit significantly (p < 0.01) greater interest than all other emotions except happiness F(2.55, 542.29) = 221.48, p < 0.001, ε= 0.283. In turn, happiness was significantly greater (p < 0.001) than all other emotions except interest, amusement, and affection. Because this was the only neutral film clip evaluated, a success index was not calculated.

Discussion

In this Online Validation Study, we had participants view five randomly selected videos (out of 15 possible clips) in a randomized order to address one of the limitations we encountered in our pilot study (see Supplemental Material). These were evaluated using three complementary statistical approaches, allowing for a robust test of the utility of the candidate film clips and following established conventions (e.g., Jenkins & Andrewes, 2012). First, the success index took into consideration the relative ability of film clips, compared to other film clips being evaluated for eliciting the same emotion, to discretely and intensely elicit the target emotion. Next, the ten-level within-subject ANOVA assessed, within each film clip, the distinctive profile of emotions elicited. This permitted evaluation of the intensities of different emotions elicited by a single film clip by individual, providing an alternative way to examine the ability of a single clip to evoke a target emotion in comparison to the other emotions reported and most effectively accounting for individual differences in emotion reporting. Finally, the mixed-feelings score specifically indexed the shared intensity of co-elicited emotions (e.g., anger and disgust). Together, these three analytical methods contribute distinct yet complementary assessments of the film clips’ utility in discretely eliciting one of a pair of frequently co-elicited emotions, and all three are in agreement regarding the most effective clips out of those evaluated here.

Based on the success index, within-subjects ANOVA, and mixed-feelings score analyses for the Online Validation Study, we were able to validate the use of The Office to elicit amusement, as this film clip predominantly evoked strong amusement intensities with considerable discreteness in the absence of the similarly strong happiness elicitation achieved by Funny Cats. Similarly, our data validate use of D2: The Mighty Ducks for the evocation of happiness without a corresponding elicitation of amusement, as occurs with Funny Cats. Between Two Ferns and Alive were moderately effective in discretely eliciting amusement and happiness, respectively. However, our data do demonstrate that the Funny Cats clip is particularly powerful (and Whose Line is it Anyway is moderately useful) for circumstances where strong evocation of both happiness and amusement are desired, such as dimensional or mood induction research.

Police Brutality was the most powerful in eliciting anger when compared across the six film clips expected to evoke anger. Consistent with prior research (Schaefer et al., 2010), we were able to verify Trainspotting for elicitation of physical disgust in the absence of an anger confound. Crash was also effective in its elicitation of disgust, despite some residual evocation of anger, and is recommended for elicitation of disgust in instances when moral, rather than physical, disgust is desired. Indeed, the co-elicitation of anger and moral disgust remain significant challenges to emotion elicitation research (Ottaviani et al., 2013; Salerno & Peter-Hagene, 2013; Whitton et al., 2014). College Conspiracy was moderately effective in evoking anger. The Road to Guantanamo moderately evoked both anger and disgust in participants and, along with College Conspiracy, could be implemented to reliably elicit negative mood states. We also verified the ability of The Champ to elicit sadness.

Presentation of a film clip from Big Cat Diary was intended as a neutral film clip, but based on a previously used criterion for neutral clips (i.e., a mean intensity <2.5 on a scale of 7 for all emotions of interest; Gross & Levenson, 1995; Hewig et al., 2005) it did not qualify as neutral because mean intensities for interest, happiness, amusement, and affection were >3. Still, the mean intensity of happiness for Big Cat Diary was less than all other positively valenced films, and the mean amusement intensity was similarly lower than the comparatively positive films except for Alive. Thus, Big Cat Diary could be useful as an engaging yet mildly positive film clip, particularly in light of the drawbacks of film clips that are neutral to the point of eliciting boredom or other ambiguous emotional responses (Gross & Levenson, 1995; Hewig et al., 2005; Samson et al., 2016).

Future investigators seeking to preferentially elicit one of these pairs of frequently co-elicited emotions are therefore recommended to use the following clips: The Office for amusement, D2: The Mighty Ducks for happiness, Police Brutality for anger, Trainspotting for physical disgust, and Crash for moral disgust. Dimensional researchers may wish to validate Funny Cats for eliciting positive or pleasant affective states, and The Road to Guantanamo or College Conspiracy (depending upon target population) for negative or unpleasant emotional responses. More broadly, the present findings permit evaluation of film clips that were first piloted in a laboratory setting to elicit emotions via an online medium. To date, only two studies have validated film clips for elicitation of emotion using the internet (Aldao & Nolen-Hoeksema, 2013 ; Samson et al., 2016), and only one has evaluated the same film clips both online and in the laboratory setting as we do here (Samson et al., 2016).

Assembling a database of emotionally evocative film clips

In 1995, Gross and Levenson published what is often considered the first major set of emotion-inducing film clips, although film stimuli had certainly been used before in research (e.g., Lazarus, Speisman, Mordkoff, & Davison, 1962) and previous emotion film sets existed (e.g., Philippot, 1993). Since that time several other sets have been put forth, most emphasizing the elicitation of discrete emotional states (e.g., Gabert-Quillen et al., 2015; Hewig et al., 2005; Rottenberg et al., 2007; Schaefer et al., 2010), others emphasizing broader dimensions of affective experience (e.g., Carvalho, Leite, Galdo-Álvarez, & Gonçalves, 2012; Samson et al., 2016) and the influence of key demographic factors in the effectiveness of the stimuli (e.g., age; Hazer et al., 2015; Jenkins & Andrewes, 2012). The advantages of each film set are clear. However, to date, there has been no attempt to integrate all of these stimuli sets into one large database of emotion-eliciting film clips in order to facilitate more effective and accessible use of this growing body of work. Indeed, the valuable advantage of such a database would be to provide an initial source permitting rapid identification of highly valid clips suited precisely for each researcher’s individual needs. This is particularly useful given the variety of journals that film clip validation studies have been published in, some of which are not as prominent as others. Therefore, the complementary goal of this paper is to address an identified deficiency in the literature that we have recognized during our own research investigations. We sought to assemble the first integrated database of emotionally evocative film clips validated in prior research, and have included clips validated in our research presented here. In doing so, we provide critical data to users on the number and type of studies validating a given clip as well as improve identification of clips useful in eliciting a wide range of emotional responses. Based on the film clip validation information we provide here, researchers can rapidly identify the film clip(s) they wish to pilot for their own investigations, identify the corresponding article, and contact the respective author(s) for more details on their methodology and access to otherwise publically unavailable film clips.

Emotion theory and the organization of emotion film stimuli

Over the last several decades there has been a lively theoretical debate as to the underlying nature of emotional responses in humans (e.g., LeDoux, 2012; Panksepp, 2007). On one end of the discussion are discrete or basic emotion theorists (e.g., Ekman, 1992; Tooby & Cosmides, 2008) who posit that emotions are relatively brief and discrete episodes of loosely coordinated responses on multiple dimensions (e.g., autonomic, behavioral, experiential, etc.) that evolved to facilitate adaptation to specific environmental demands. For example, anger is associated with loosely coordinated responses that evolved to facilitate instances of goal blockage (Carver & Harmon-Jones, 2009), versus sadness which has a distinct set of loosely coordinated responses to manage instances of loss (Bonanno, Goorin, & Coifman, 2008). On the other end are dimensional or psychological constructionist theorists (e.g., Barrett, 2006) who argue instead that specific emotional responses can be understood as overlying two neurobiologically based dimensions of response to environmental demands. For example, some argue that the hedonic valence (e.g., the pleasantness vs. unpleasantness) and arousal (e.g., low vs. high) of the provoking stimuli determines an individual’s response on multiple dimensions (e.g., behavioral, autonomic etc.). As such, specific emotion labels (e.g., anger vs. fear vs. sadness) are socially-constructed, resulting from individual differences in the appraisal or conceptualization of the provocation (e.g., Barrett & Bliss-Moreau, 2009).

Although a detailed review of the literature is beyond the scope of this paper, we present these seemingly opposing theories of discrete vs. dimensional emotions to justify the inclusion and organization of a wide array of film clip stimuli that would facilitate the work of researchers who are operating from these different theoretical positions and/or have different research agendas. For example, researchers interested in experimentally manipulating mood in order to examine influences on executive attention might most benefit from employing clips validated using a dimensional framework. Conversely, researchers interested in examining the influence of fear on visual search and memory would be better served employing clips validated using a discrete emotions framework. As such, we compiled two classes of film clip stimuli researched over the last four decades into our catalog. The first includes stimuli demonstrated to elicit specific or discrete emotional responses, and the second includes stimuli demonstrated to support dimensional or psychological constructionist views. In addition, we include data from our own research that largely employs a discrete emotions framework while taking into consideration the complexities of co-elicitation of similar emotions.

Literature review and development of film stimuli catalogues

In order to identify prior research indicating evidence of effective film clip stimuli, keyword searches (e.g., “emotion elicitation,” “film clips”) yielding over 1,000 results were first conducted on PsycINFO and Google Scholar and reference sections of relevant studies were examined. From the resulting literature, validated film stimuli were included if they met the following four criteria: (1) validation data supporting the utility of each clip for emotion elicitation were provided (either as published or submitted by the author via personal communication); (2) the stimuli were validated on a healthy population; (3) the researchers generally followed established conventions (i.e., methodological, statistical) for the evaluation of the stimuli (e.g., Gross & Levenson, 1995); and (4) the data were reported in the last four decades. The total yield was 24 studies. In some cases, multiple investigations provided data in support of almost identical film clips (e.g., The Champ, MGM 1979), using the same or different indices of emotional responses (i.e., affective report, autonomic responses, facial behavior). In those cases, we cite all relevant studies supporting a given clip and indicate which indices were used in each investigation.

Validated film stimuli from the literature (including the clips validated in our Online Validation Study) are presented in two comprehensive tables. Table 5 lists all clips researched for discrete emotion elicitation. Clips are listed by emotion and basic information is included on the type of supporting data (i.e., which emotion response indices) provided by the respective investigations (these are also listed in the reference section and denoted by an *). We do not include the specific details of the construction/location of the clip (with the exception of those for which we provide data) because this can be found in the referenced article(s) which should be consulted for greater details on the methodology employed. Table 6 lists all clips researched for the elicitation of dimensions of emotional valence, arousal, and/or dominance. Clips are listed by dimension and included are information on the type of supporting data and references. In some cases, clips were evaluated for the elicitation of both discrete and dimensional emotion responses. In this case, they are only listed in Table 5 but given special designation (**). Finally, we specify in both Tables 5 and 6 the language of the clip (other than English) or if there was no language spoken, the type of adult sample the clip was validated in (community versus student), and the length of the clip, in order to facilitate easier identification of candidate film clips for future investigations.
Table 5

Comprehensive list of film clips to elicit discrete emotional responses

 

Affecta

Physiob

Facialc

Reference

AMUSEMENT

A Fish Called Wanda (2:53)

X

X

 

Boiten, 1998 abU; Schaefer et al., 2010 aF

An Officer and a Gentleman (1:51)

X

  

Maffei et al., 2014 I

Bean Camera (3:30)

X

  

Jenkins & Andrewes, 2012 CN

Benny and Joon 1 ** (2:01)

X

  

Schaefer et al., 2010 dF

Bill Cosby (2:01)

X

  

Rottenberg et al., 2007

La cité de la peur** (2:13)

X

  

Schaefer et al., 2010 dF

Le Pari (1:47)

X

  

Schaefer et al., 2010 F

Les trois frères** (2:24)

X

  

Schaefer et al., 2010 dF

On Golden Pond (0:30)

X

  

Maffei et al., 2014 I

Robin Williams Live (7:59)

X

  

Gross & Levenson, 1995

Rowan Devil (2:49)

X

  

Jenkins & Andrewes, 2012 C

The Office (4:05)

X

  

Current Study

The Dinner Game** (1:40)

X

  

Schaefer et al., 2010 dF

The Visitors (2:09)

X

  

Schaefer et al., 2010 F

There’s something about Mary [Dog](2:55)

X

  

Schaefer et al., 2010 F

There’s something about Mary [Hair](2:26)

X

  

Schaefer et al., 2010 F

When Harry Met Sally 2 ** (2:29)

X

  

Gross & Levenson, 1995 a; Hewig et al., 2005 aN; Maffei et al., 2014 aI; Rottenberg et al., 2007 a; Schaefer et al., 2010 adF; Westerink et al., 2008 dCD

Whose line is it anyway? [Hands] (U)

X

  

Rottenberg et al., 2007

ANGER

    

A Perfect World** (4:27)

X

  

Schaefer et al., 2010 dF

American History X** (1:17)

X

  

Schaefer et al., 2010 dF

Cry Freedom** (2:30)

X

  

Gross & Levenson, 1995 a; Hagemann et al., 1999 aN; Hewig et al., 2005 a; Maffei et al., 2014 aI; Rottenberg et al., 2007 a; Westerink et al., 2008 dCD

Gandhi (0:45-2:07)

X

X

 

Hagemann et al., 1999 aN; Hewig et al., 2005 aN; Maffei et al., 2014 aI ; Tomarken et al., 1990 abCN

My Bodyguard (3:56)

X

  

Maffei et al., 2014 I

Name of the Father (3:30)

X

  

Schaefer et al., 2010 F

Leaving Las Vegas (2:30)

X

  

Schaefer et al., 2010 F

Man Bites Dog (1:17)

X

  

Schaefer et al., 2010 F

My Bodyguard (3:55)

X

  

Gross & Levenson, 1995 a ;Hewig et al., 2005 aN; Rottenberg et al., 2007 a

Once WW1 (2:07)

X

  

Jenkins & Andrewes, 2012 C

Police Brutality (5:00)

X

  

Current Study

Romper (2:01)

X

  

Jenkins & Andrewes, 2012 CN

Schindler’s List [Camp]** (1:18)

X

  

Schaefer et al., 2010 dF

Schindler’s List [Balcony]** (1:55)

X

  

Schaefer et al., 2010 dF

Schindler’s List [Ghetto] (1:30)

X

  

Schaefer et al., 2010 F

Seven [Wife]** (5:52)

X

  

Schaefer et al., 2010 dF

Sleepers** (2:20)

X

  

Schaefer et al., 2010 dF

Sophie’s Choice (3:00-6:00)

X

  

Philippot, 1993 F

The Old Gun (3:00-6:00)

X

  

Philippot, 1993 F

Witness (0:45-1:30)

X

X

 

Hagemann et al., 1999 aN; Hewig et al., 2005 aN; Maffei et al., 2014 aI ; Tomarken et al., 1990 abCN

CONTENTMENT

Beach Scene (0:23)

X

  

Gross & Levenson, 1995 N

Waves (0:58)

X

  

Gross & Levenson, 1995 N

DISGUST

Accro (0:45)

X

X

X

Reynaud et al., 2012 CU

American History X** (1:17)

X

  

Schaefer et al., 2010 F

Amputation (1:02)

X

  

Aldao & Nolen-Hoeksema, 2013 aCN; Gross & Levenson, 1995 aN; Jenkins & Andrewes, 2012 aCN; Rottenberg et al., 2007 aN

Crash (5:29)

X

  

Current Study

Faces of Death [Slaughterhouse] (3:00-6:00)

X

  

Philippot, 1993 F

Faces of Death [Restaurant] (3:00-6:00)

X

  

Philippot, 1993 F

Foot Surgery (U)

X

  

Rottenberg et al., 2007 U

Hellraiser (1:30)

X

  

Schaefer et al., 2010 F

Man Bites Dog [Rape] (1:54)

X

  

Schaefer et al., 2010 F

Maria’s Lovers (0:58)

X

X

 

Hagemann et al., 1999 aN; Hewig et al., 2005 aN; Maffei et al., 2014 aI ; Tomarken et al., 1990 abCN

Misery** (3:31)

X

  

Schaefer et al., 2010 dF

Oz Beecher (1:13)

X

  

Jenkins & Andrewes, 2012 C

Pink Flamingos** (0:30)

X

  

Gross & Levenson, 1995 a; Hewig et al., 2005 aN; Maffei et al., 2014 aI ; Rottenberg et al., 2007 a; Westerink et al., 2008 dCD

Seven [Tortured] (1:43)

X

  

Schaefer et al., 2010 F

Seven [Table] (3:19)

X

  

Schaefer et al., 2010 F

Sleepers** (2:20)

X

  

Schaefer et al., 2010 dF

The Dentist (0:56)

X

  

Schaefer et al., 2010 dF

The Godfather (0:59)

X

X

 

Hagemann et al., 1999 aN; Hewig et al., 2005 aN; Maffei et al., 2014 aI ; Tomarken et al., 1990 abCN

The Meaning of Life

X

X

 

Boiten, 1998 U

The Silence of the Lambs (3:29)

X

  

Schaefer et al., 2010 F

Trainspotting [Toilet] (1:44-3:53)

X

  

Current Study; Schaefer et al., 2010 F

FEAR

A Tale of Two Sisters (0:45)

X

X

X

Reynaud et al., 2012 U

Child’s Play 2: Chucky’s Back (1:05)

X

  

Schaefer et al., 2010 F

Copycat (2:23)

X

  

Schaefer et al., 2010 F

Halloween (3:28)

X

  

Hewig et al., 2005aN; Maffei et al., 2014aI ; Philippot, 1993 aF

I Know What You Did Last Summer (10:39)

X

X

X

Kreibig et al., 2007

I Still Know What You Did Last Summer (11:13)

X

X

X

Kreibig et al., 2007

Marathon Man (2:40)

X

X

 

Boiten, 1998 abU; Hewig et al., 2005 aN ; Maffei et al., 2014 aI

Misery** (3:31)

X

  

Schaefer et al., 2010 dF

Psycho (3:00-6:00)

X

  

Philippot, 1993 F

Red Eye (2:51)

X

  

Jenkins & Andrewes, 2012 CN

Roadkill (5:00)

X

  

Jenkins & Andrewes, 2012 C

Scream** (6:33)

X

  

Schaefer et al., 2010 dF

Scream 2 (3:35)

X

  

Schaefer et al., 2010 F

Seven [Tortured] (1:43)

X

  

Schaefer et al., 2010 F

Silence of the Lambs 3 ** (3:17)

X

  

Aldao, Mennin, & McLaughlin, 2013 aC; Gross & Levenson, 1995 a; Hewig et al., 2005 aN; Maffei et al., 2014 aI ; Rottenberg et al., 2007 a; Westerink et al., 2008 dCD

The Blair Witch Project** (3:57)

X

  

Schaefer et al., 2010 dF

The Dentist (0:56)

X

  

Schaefer et al., 2010 F

The Doll (5:12)

X

X

 

Li & Chen, 2006 U

The Exorcist (1:41)

X

  

Schaefer et al., 2010 F

The Shining [Hallway] 3 **(1:19)

X

  

Gross & Levenson, 1995 a; Rottenberg et al., 2007 a; Westerink et al., 2008 dCD

The Shining [Axe]** (4:15)

X

  

Schaefer et al., 2010 dF

HAPPINESS

An Officer and a Gentleman [Factory] 4 (1:51)

X

X

 

Hagemann et al., 1999 aN; Tomarken et al., 1990 abCN

D2: The Mighty Ducks (5:38)

X

  

Current Study

Deep Dolphin (2:00)

X

  

Jenkins & Andrewes, 2012 CN

Frequency (U)

X

  

Aldao, Mennin, & McLaughlin, 2013 aC;

Le Dîner de Cons (0:45)

X

X

X

Reynaud et al., 2012 CU

Le Magnifique [Escape] (3:00-6:00)

X

  

Philippot, 1993 F

Le Magnifique [Beach] (3:00-6:00)

X

  

Philippot, 1993 F

On Golden Pond 5 (0:32)

X

X

 

Hagemann et al., 1999 aN; Tomarken et al., 1990 abCN

JOY

Tom and Jerry (6:45)

X

X

 

Li & Chen, 2006 U

NEUTRAL

Abstract Shapes** (2:00-3:25)

X

  

Gross & LevensonaN, 1995; Westerink et al., 2008 dCN

Alaska’s Wild Denali (5:02-10:01)

X

X

X

Kreibig et al., 2007 abc; Rottenberg et al., 2007 a

All the President’s Men (1:05)

X

  

Hewig et al., 2005 aN; Maffei et al., 2014 aI ; Jenkins & Andrewes, 2012 aC

Belgian Documentary [Policy] (3:00-6:00)

X

  

Philippot, 1993 F

Belgian Documentary [Interview] (3:00-6:00)

X

  

Philippot, 1993 F

Blue [aluminium] (0:16)

X

  

Schaefer et al., 2010 F

Blue [desk] (0:40)

X

  

Schaefer et al., 2010 F

Blue [escalator] (0:25)

X

  

Schaefer et al., 2010 F

Color Bars** (1:30-2:00)

X

  

Aldao, Mennin, & McLaughlin, 2013 aCU; Gross & Levenson, 1995 aN; Westerink et al., 2008 dCN

Crimes and Misdemeanors (0:57)

X

  

Hewig et al., 2005 aN; Maffei et al., 2014 aI

Führerstands mitfahrten (0:30)

X

  

Hagemann et al., 1999 N

Hannah and her Sisters (1:31)

X

  

Hewig et al., 2005 aN; Maffei et al., 2014 aI

Lost in Translation (1:36)

X

  

Jenkins & Andrewes, 2012 CN

The Lover [car] (0:43)

X

  

Schaefer et al., 2010 F

Noncommercial Screen Saver (3:26)

X

X

 

Li & Chen, 2006 U

Omagh neutral (1:00)

X

  

Jenkins & Andrewes, 2012 C

Open (1:00)

X

  

Jenkins & Andrewes, 2012 C

Open Water NV (1:05)

X

  

Jenkins & Andrewes, 2012 CN

Sticks (3:26)

X

  

Rottenberg et al., 2007 N

The Last Emperor (1:15)

X

  

Hewig et al., 2005 aN; Maffei et al., 2014 aI

PEACEFULNESS

March of the Penguins (0:45)

X

X

X

Reynaud et al., 2012 CU

SADNESS

    

A Perfect World** (4:27)

X

  

Schaefer et al., 2010 dF

An Officer and a Gentleman [Motel] (1:41)

X

X

 

Hagemann et al., 1999 aN; Hewig et al., 2005 aN; Maffei et al., 2014 aI ; Tomarken et al., 1990 abCN

Bambi (2:19)

X

  

Gross & Levenson, 1995 a; Hagemann et al., 1999 aN

City of Angels (4:15)

X

  

Schaefer et al., 2010 F

Cry Freedom (2:30)

X

  

Maffei et al., 2014 I

Dangerous Mind (2:08)

X

  

Schaefer et al., 2010 F

Dead Man Walking** (6:40)

X

  

Schaefer et al., 2010 dF

John Q (12:41)

X

  

Kreibig et al., 2007

Kramer vs. Kramer [Park] (3:00-6:00)

X

  

Philippot, 1993 F

Kramer vs. Kramer [Bedtime] (3:00-6:00)

X

  

Philippot, 1993 F

Life is Beautiful [Kill] (2:07)

X

  

Schaefer et al., 2010 F

Lion King 3 ** (1:57)

X

  

Rottenberg et al., 2007 a; Westerink et al., 2008 dCD

My Girl (2:15)

X

  

Jenkins & Andrewes, 2012 C

Philadelphia (5:28)

X

  

Schaefer et al., 2010 F

Report on famine in Biafra (0:45)

X

X

X

Reynaud et al., 2012 U

Return to Me (3:36)

X

  

Aldao, Mennin, & McLaughlin, 2013 aC; Rottenberg et al., 2007 a

Schindler’s List [Camp]** (1:18)

X

  

Schaefer et al., 2010 dF

Steel Magnolias (11:52)

X

X

X

Kreibig et al., 2007

Sylvia (3:54)

X

  

Jenkins & Andrewes, 2012 CN

Terms of Endearment (U)

X

X

 

Boiten, 1998 U

The Champ 3 ** (2:51-5:18)

X

  

Current Study a; Gross & Levenson, 1995 a; Hagemann et al., 1999 aN; Hewig et al., 2005 aN; Maffei et al., 2014 aI; Rottenberg et al., 2007a; Westerink et al., 2008 dCD

The Dead Poets Society [Schoolboy] (4:14)

X

  

Schaefer et al., 2010 F

The Killing Fields (1:23)

X

X

 

Hewig et al., 2005 aN ; Maffei et al., 2014 aI ; Tomarken et al., 1990 abCN

The Professional [Separated] (2:44)

X

  

Schaefer et al., 2010 F

SURPRISE

Capricorn One (0:49)

X

  

Gross & Levenson, 1995 a; Rottenberg et al., 2007 a

Sea of Love 3 ** (0:09)

X

  

Gross & Levenson, 1995 a; Rottenberg et al., 2007 a; Westerink et al., 2008 dCD

TENDERNESS

Elsa the Lion (U)

X

X

 

Boiten, 1998 U

E.T. (4:35)

X

  

Schaefer et al., 2010 F

Forrest Gump** (2:01)

X

  

Schaefer et al., 2010 dF

Ghost (3:35)

X

  

Schaefer et al., 2010 F

Life is Beautiful [Loud Speaker]** (1:45)

X

  

Schaefer et al., 2010 dF

Life is Beautiful [Reunited]** (4:09)

X

  

Schaefer et al., 2010 dF

Life is Beautiful [Translation]** (3:48)

X

  

Schaefer et al., 2010 dF

The Dead Poets Society [Desks]** (2:40)

X

  

Schaefer et al., 2010 dF

The Eighth Day (2:10)

X

  

Schaefer et al., 2010 F

The Professional [Separated] (2:44)

X

  

Schaefer et al., 2010 F

When a Man Loves a Woman (1:39)

X

  

Schaefer et al., 2010 F

1 Schaefer at al., 2010 validated the use of a number of clips for the elicitation of both discrete and dimensional emotional responses. These data were published in the same article and when this is the case the Schaefer et al., 2010 reference is followed by a d

2 This When Harry Met Sally clip was also validated to elicit Happiness by Hagemann et al., 1999a

3 Westerink et al., 2008 also investigated this clip using a dimensional framework; however, results indicated neutral responses from participants

4 This An Officer and a Gentleman clip was also validated to elicit Amusement by Hewig et al., 2005a

5This On Golden Pond clip was also validated to elicit Amusement by Hewig et al., 2005a

Note. All films not shown in English are denoted with a superscript after the corresponding reference as follows: D-Dutch subtitles; F-French, French-dubbed, or French subtitles; G-German, German-dubbed, or German subtitles; I-Italian, Italian-dubbed, or Italian subtitles; N-no dialogue, music only, or silent/muted; U-unclear, information not provided in paper, attempts to contact corresponding author unsuccessful. Studies were performed in university participants unless designated as C-community sample only or SC-both student and community samples

** Indicates films that have also been shown to elicit dimensional emotion responses

a Denotes the use of affective report

b Denotes the use of physiological indices of emotion response

c Denotes the use of behavioral indices of emotion response

d Denotes specific references for articles that evaluated the same clip based on a dimensional rather than discrete emotions framework

Table 6

Comprehensive list of film clips to elicit dimensional emotion responses

Validated as

Affect

Physio

Facial

Citation

PLEASANT/POSITIVE

Baby Bites Bro's Finger (0:27)

X

  

Samson et al., 2016 SC

Baby Controls Cheers (0:20)

X

  

Samson et al., 2016 LSC

Baby Dances to 50Cent (0:26)

X

  

Samson et al., 2016 LSC

Baby Dances to Beyonce (0:30)

X

  

Samson et al., 2016 LSC

Baby Dancing to R&B Music (0:29)

X

  

Samson et al., 2016 LSC

Baby Dancing to Techno Music (0:29)

X

  

Samson et al., 2016 LSC

Baby Doesnt Love His Daddy (0:26)

X

  

Samson et al., 2016 SC

Baby Eats a Lemon (0:30)

X

  

Samson et al., 2016 LSC

Baby's Evil Eye (0:21)

X

  

Samson et al., 2016 LSC

Baby Fails Hulahoop (0:30)

X

  

Samson et al., 2016 LSC

Baby Scared by his own Flatulence (0:21)

X

  

Samson et al., 2016 LSC

Baby Hiccups and Laughs (0:29)

X

  

Samson et al., 2016 NSC

Baby Singing 'Heal the World' (0:31)

X

  

Samson et al., 2016 LSC

Baby with a Pregnancy Test (0:33)

X

  

Samson et al., 2016 SC

Baby Dancing to Beatboxing (0:29)

X

  

Samson et al., 2016 LSC

Bride Falls in Water (0:28)

X

  

Samson et al., 2016 LSC

Bride Laughing During Vows (0:30)

X

  

Samson et al., 2016 SC

Bride Throws Flowers at Bridesmaid (0:29)

X

  

Samson et al., 2016 LSC

Cat Suckles Air (0:30)

X

  

Samson et al., 2016 LSC

Cookie Baby (0:30)

X

  

Samson et al., 2016 SC

Dancing Accident Kick (0:29)

X

  

Samson et al., 2016 LSC

Escalator Spinning (0:27)

X

  

Samson et al., 2016 NSC

Fail Treadmill Dance (0:30)

X

  

Samson et al., 2016 LSC

Girl Hits Her Head Bowling (0:25)

X

  

Samson et al., 2016 LSC

Girl Thrown into a Basketball Hoop (0:25)

X

  

Samson et al., 2016 LSC

Gorilla Film (1:00)

X

X

X

Ekman et al., 1980 CN

Guy Fails to Launch A Canoe (0:20)

X

  

Samson et al., 2016 LSC

Guy Wakes Up Friend by Screaming (0:30)

X

  

Samson et al., 2016 NSC

Ice Dancing (2:00)

X

X

 

Kaviani et al., 1999 CN

L’amant (2:00)

X

X

 

Codispoti et al., 2008 IU

Lamb Can’t Find Owner (0:31)

X

  

Samson et al., 2016 LSC

M*A*S*H (U)

X

 

X

McHugo et al.,1982SC

Man Crashes into Glass Door (0:23)

X

  

Samson et al., 2016 NSC

Man gets Hurt Tubing (0:30)

X

  

Samson et al., 2016 LSC

Mr. Bean (2:00)

X

X

 

Kaviani et al., 1999 CN

Musicians Play to Man on the Phone (0:28)

X

  

Samson et al., 2016 LSC

Nature Documentary (2:00)

X

X

 

Westerink et al., 2008 CD

Panda Sneezes (0:30)

X

  

Samson et al., 2016 LSC

Parachute Hits Man (0:20)

X

  

Samson et al., 2016 LSC

Puppy Film (1:00)

X

X

X

Ekman et al., 1980 CN

Red Balloon (2:10)

X

 

X

McHugo et al.,1982 SCF

Reporter gets Hit (0:29)

X

  

Samson et al., 2016 SC

Sheep Attacks Man (0:21)

X

  

Samson et al., 2016 LSC

Singing Dog (0:29)

X

  

Samson et al., 2016 LSC

Skier Crashes into Snow (0:29)

X

  

Samson et al., 2016 LSC

Sweet Home Alabama (2:00)

X

X

 

Westerink et al., 2008 CD

Tarzan (2:13)

X

X

 

Westerink et al., 2008 CD

The bear (2:00)

X

X

 

Westerink et al., 2008 CD

Thirsty Baby (0:27)

X

  

Samson et al., 2016 NSC

Wedding Photographer Fails (0:29)

X

  

Samson et al., 2016 LSC

Woman Loses Her Dress by Car Driving Away (0:23)

X

  

Samson et al., 2016 LSC

Woman Ruins Wedding (0:21)

X

  

Samson et al., 2016 LSC

PLEASANT/HIGH AROUSAL

Sports Clip (10:02)

X

X

 

Gomez et al., 2005 N

PLEASANT/HIGH AROUSAL/LOW DOMINANCE

9 Songs [Sofa; Oral] (0:40)

X

X

 

Carvalho et al., 2012 NSC

Diary of a Nymphomaniac

X

X

 

Carvalho et al., 2012 NSC

[Missionary; Chair] (0:40)

Kama Sutra: The Sensual Art of Love Making [Arch; Variant] (0:40)

X

X

 

Carvalho et al., 2012 NSC

Killing Me Softly (0:40)

X

X

 

Carvalho et al., 2012 NSC

Monamour (0:40)

X

X

 

Carvalho et al., 2012 NSC

Playboy’s Clip (0:40)

X

X

 

Carvalho et al., 2012 NSC

Underworld: Evolution (0:40)

X

X

 

Carvalho et al., 2012 NSC

PLEASANT/LOW AROUSAL

Nature Clip (6:19)

X

X

 

Gomez et al., 2005 G

PLEASANT/LOW AROUSAL/LOW DOMINANCE

Diary of a Nymphomaniac [New Home; Remembering] (0:40)

X

X

 

Carvalho et al., 2012 NSC

Disney’s Earth [Polar; Mountains; Dusk; Falls; Flowers; Storm; Trees; View; Jungle; Clouds] (0:40)

X

X

 

Carvalho et al., 2012 NSC

Good Luck Chuck (0:40)

X

X

 

Carvalho et al., 2012 NSC

Last Chance Harvey (0:40)

X

X

 

Carvalho et al., 2012 NSC

Lie With Me (0:40)

X

X

 

Carvalho et al., 2012 NSC

My Best Friend’s Girl (0:40)

X

X

 

Carvalho et al., 2012 NSC

Ruins [Beach ; Pool] (0:40)

X

X

 

Carvalho et al., 2012 NSC

The Rest Stop [Car] (0:40)

X

X

 

Carvalho et al., 2012 NSC

UNPLEASANT/NEGATIVE

Accident Film (1:00)

X

X

X

Ekman et al., 1980 CU

Broken Ankle Skating (0:30)

X

  

Samson et al., 2016 LSC

Bull Throw and Trample (0:21)

X

  

Samson et al., 2016 LSC

Croc Bites Man (0:28)

X

  

Samson et al., 2016 LSC

Gangsters in the Wood (2:00)

X

X

 

Kaviani et al., 1999 CN

Jackass The Movie (0:51)

X

X

 

Westerink et al., 2008 CD

Medical Documentary (2:00)

X

X

 

Codispoti et al., 2008 I; Palomba et al., 2000 U

Skater Snaps his Arm (0:28)

X

  

Samson et al., 2016 LSC

The Hitcher (2:12)

X

X

 

Palomba et al., 2000 U

The Hustler (4:58)

X

 

X

McHugo et al.,1982 SC

The Informer (U)

X

 

X

McHugo et al.,1982 SC

Toe Surgery (2:00)

X

X

 

Kaviani et al., 1999 CN

UNPLEASANT/HIGH AROUSAL

  

Saving Private Ryan (5:22)

X

  

Schaefer et al., 2010 F

The Deer Hunter (10:10)

X

X

 

Gomez et al., 2005 G

The Piano (0:43)

X

  

Schaefer et al., 2010 F

UNPLEASANT/HIGH AROUSAL/HIGH DOMINANCE

Cannibal Holocaust (0:40)

X

X

 

Carvalho et al., 2012 NSC

Hostel 2 (0:40)

X

X

 

Carvalho et al., 2012 NSC

Midnight Meat Train [Teeth; Ceiling; Attack] (0:40)

X

X

 

Carvalho et al., 2012 NSC

Texas Chainsaw Massacre: The Beginning [Face; Box] (0:40)

X

X

 

Carvalho et al., 2012 NSC

The Ruins (0:40)

X

X

 

Carvalho et al., 2012 NSC

UNPLEASANT/HIGH AROUSAL/LOW DOMINANCE

Hostel (0:40)

X

X

 

Carvalho et al., 2012 NSC

 The Rest Stop [Police] (0:40)

X

X

 

Carvalho et al., 2012 NSC

UNPLEASANT/LOW AROUSAL

Les Enfants du Borinage-Lettre à Henri Storck (10:52)

X

X

 

Gomez et al., 2005 G

UNPLEASANT/LOW AROUSAL/LOW DOMINANCE

American Beauty [Thora’s room; Wes and Chris] (0:40)

X

X

 

Carvalho et al., 2012 NSC

Boogeyman (0:40)

X

X

 

Carvalho et al., 2012 NSC

Boogeyman 2 (0:40)

X

X

 

Carvalho et al., 2012 NSC

Bridge to Terabithia (0:40)

X

X

 

Carvalho et al., 2012 NSC

Diary of a Nymphomaniac [Suicide] (0:40)

X

X

 

Carvalho et al., 2012 NSC

Mystic River [Argument; Porch] (0:40)

X

X

 

Carvalho et al., 2012 NSC

Own Footage [Table 1 ; Table 2 ] (0:40)

X

X

 

Carvalho et al., 2012 NSC

The Descent (0:40)

X

X

 

Carvalho et al., 2012 NSC

The Pianist (0:40)

X

X

 

Carvalho et al., 2012 NSC

NEUTRAL

Airport 2 (0:30)

X

  

Samson et al., 2016 NSC

Assembly (0:26)

X

  

Samson et al., 2016 NSC

Bart (0:31)

X

  

Samson et al., 2016 LSC

Boy Drinking Tea (0:31)

X

  

Samson et al., 2016 NSC

Broadway (0:24)

X

  

Samson et al., 2016 LSC

Bicycle Rush Hour 2 (0:20)

X

  

Samson et al., 2016 NSC

Cable Car (0:23)

X

  

Samson et al., 2016 LSC

Cafe (0:22)

X

  

Samson et al., 2016 LSC

Central Station NY (0:22)

X

  

Samson et al., 2016 NSC

City in the Night (0:20)

X

  

Samson et al., 2016 LSC

Denver Train (0:25)

X

  

Samson et al., 2016 LSC

Diving (0:23)

X

  

Samson et al., 2016 LSC

Diving II (0:25)

X

  

Samson et al., 2016 LSC

Eating Pizza (0:23)

X

  

Samson et al., 2016 SC

Eating with Chopsticks (0:29)

X

  

Samson et al., 2016 SC

Educational Program (10:18)

X

X

 

Gomez et al., 2005 G

Girls Brushing Teeth (0:21)

X

  

Samson et al., 2016 NSC

Grout (0:27)

X

  

Samson et al., 2016 SC

Hair Washing (0:26)

X

  

Samson et al., 2016 LSC

Hiking in Woods (0:26)

X

  

Samson et al., 2016 SC

Household Equipment (2:00)

X

X

 

Kaviani et al., 1999 CN

In the Kitchen (0:26)

X

  

Samson et al., 2016 NSC

In the Kitchen II (0:30)

X

  

Samson et al., 2016 SC

Knit (0:27)

X

  

Samson et al., 2016 SC

Knitting (0:32)

X

  

Samson et al., 2016 SC

Mall (0:20)

X

  

Samson et al., 2016 LSC

Man Riding Horse (0:21)

X

  

Samson et al., 2016 SC

Morning Glory (2:52)

X

 

X

McHugo et al.,1982 SC

Museum (0:24)

X

  

Samson et al., 2016 LSC

NY Street (0:29)

X

  

Samson et al., 2016 NSC

Origami 4 (0:32)

X

  

Samson et al., 2016 SC

Pancakes II (0:28)

X

  

Samson et al., 2016 SC

Pillow (0:30)

X

  

Samson et al., 2016 SC

Planting a Veggie Garden (0:29)

X

  

Samson et al., 2016 SC

Riding Horse (0:31)

X

  

Samson et al., 2016 SC

Riding in the Tube I (0:21)

X

  

Samson et al., 2016 NSC

Riding in the Tube II (0:24)

X

  

Samson et al., 2016 LSC

San Fran (0:28)

X

  

Samson et al., 2016 NSC

Scrapbook (0:23)

X

  

Samson et al., 2016 SC

Sitting on the Sofa (0:30)

X

  

Samson et al., 2016 SC

Snow (0:23)

X

  

Samson et al., 2016 LSC

Street Scenes (2:00)

X

X

 

Kaviani et al., 1999 CN

Swim Laps (0:27)

X

  

Samson et al., 2016 NSC

Switzerland- The Alpine Wonderland (2:00)

X

X

 

Codispoti et al., 2008 I; Palomba et al., 2000U

Tea (0:31)

X

  

Samson et al., 2016 SC

Treadmill (0:30)

X

  

Samson et al., 2016 SC

Van Gogh Museum (0:20)

X

  

Samson et al., 2016 NSC

Walking Through Desert (0:33)

X

  

Samson et al., 2016 LSC

Walk Through Dusseldorf (0:26)

X

  

Samson et al., 2016 LSC

Winter Wood (0:25)

X

  

Samson et al., 2016 NSC

MIXED

Boy is Hit by Playground Spring Horse (0:24)

X

  

Samson et al., 2016 LSC

Break Dancer Kicks Cat (0:25)

X

  

Samson et al., 2016 LSC

Bride Texting During Marriage (0:29)

X

  

Samson et al., 2016 LSC

Crane Drops (0:30)

X

  

Samson et al., 2016 LSC

Daughter gets Hit with a Ball (0:27)

X

  

Samson et al., 2016 LSC

Man Drops Dog (0:26)

X

  

Samson et al., 2016 SC

Note: All films not shown in English are denoted with a superscript after the corresponding reference as follows: D-Dutch subtitles; F-French, French-dubbed, or French subtitles; G-German, German-dubbed, or German subtitles; I-Italian, Italian-dubbed, or Italian subtitles; L-language independent (see Samson et al., 2016 for details); N-no dialogue, music only, or silent/muted; U-unclear, information not provided in paper, attempts to contact corresponding author unsuccessful. Studies were performed in university participants unless designated as C-community sample only or SC-both student and community samples

General discussion

Our goals for this investigation were two-fold. First, we validated several clips for discrete elicitation of particularly challenging, frequently co-elicited emotions using online presentation of film clips in randomized order. Our investigations here provide clear evidence supporting the use of the following film clips for elicitation of these discrete emotional responses: happiness – D2: The Mighty Ducks; amusement – The Office; anger – Police Brutality; disgust – Trainspotting and Crash; sadness – The Champ. The inclusion of two clips for disgust reflects inherent differences in elicitation of physical disgust versus moral disgust (respectively). Evaluation of these film clips in an online population after first being piloted in a laboratory setting strengthens the findings of the former by suggesting that the ability of these particular film clips to elicit emotional responses are generally consistent despite being presented in very different environments. Other clips demonstrated moderately consistent elicitation of anger and/or disgust or happiness and/or amusement (The Road to Guantanamo and College Conspiracy for anger and disgust; Alive for happiness; Between Two Ferns for amusement; Funny Cats and Whose Line is it Anyway for happiness and amusement). These clips can be considered as effective stimuli for use in research where they may serve the specific purposes of researchers better because of their content or because the research is not dependent upon eliciting one specific emotion. The second goal of this investigation was compilation of an extensive catalog of film clip stimuli, integrating evidence demonstrating the validation of films for elicitation of both discrete and dimensional emotion responses from research published over the past four decades.

After incorporating the six clips from the Online Validation Study here with the existing literature, we have assimilated 295 film clips from 24 articles to generate a current, unified point of reference for emotionally evocative videos. The variety within this catalog is considerable, including: clips in many languages (e.g., English, French, German, and Italian); clips with and without audio tracks; clips differing in length; clips of both professional quality and those taken of real-life occurrences; and of course, clips designed to elicit a wide array of emotional responses. This diversity is indicative of widespread interest in research on emotion and reflects the complexity of this particular methodology. Our goal was to make this resource a relatively simple yet useful reference tool in order to help support the rapidly growing field of emotion research.

Most film clips employed in the study presented here did not result in significant differences in reported intensities of emotions between the sexes. Though evaluating sex differences in responding to emotionally evocative film clips was not a primary goal of this paper, we nonetheless recognized the unique strength in our data set that would allow us to detect consistent or inconsistent differences in reported target emotion intensity between the sexes. Only one film consistently exhibited a sex difference across both the pilot study and the Online Validation Study – The Champ resulted in significantly higher reported sadness intensities in females versus males. Other clips suggesting sex differences in the present study are indicated in Tables 1, S2, and S3.

Over the course of our research we recognized some key limitations to investigations validating film clips in both the articles we sought as well as our own studies. Most important is that the vast majority of the data supporting the use of these clips was garnered using affective self-report only. Although the utility of affective or experiential report of emotion is debated (e.g., Robinson & Clore, 2002), most would agree that there are limitations associated with relying on only one index of emotional responses. Indeed, some of our own research has yielded instances where affective report is discrepant from other response modes (including autonomic activity: Coifman, Bonanno, Ray, & Gross, 2007). However, there are a number of benefits associated with validation using self-report, most critical of which is efficiency. Indeed, studies in which no a priori expectations or demands are suggested to participants and the samples are large are likely to be the strongest in terms of validity. Moreover, researchers are encouraged to employ these stimuli with the understanding that most predictable will be self-reported emotion and that reactions observed through other measures (e.g., behavioral) may or may not be consistent with self-reported responses.

An additional issue resulting from the use of self-reported affect ratings is that this tool carries inherent challenges. There are many potential emotion words that can be used for participant ratings, and here we selected ten of the most consistently used words in this type of research that are also consistent with dominant models of discrete emotion (Ekman, 1992). Had we used a larger number of words we may have increased our ability to detect emotions that were elicited, but this comes at the cost of time and attentional demands on the participant. Moreover, there is growing evidence that the capacity to differentiate emotion words may vary considerably between individuals, particularly for negative emotions (Kashdan, Barrett, & McKnight, 2015), and as such, more words may not necessarily have been more comprehensive. Further, participants can experience difficulties in disentangling similar emotional experiences, be inaccurately aware of, or improperly attend to, their emotional states, or have problems with the fluid and malleable definitions attributed to emotion words (Ellsworth & Smith, 1988; Gross & Levenson, 1995; Herring et al., 2011; Jerritta, Murugappan, Wan, & Yaacob, 2014; Maffei et al., 2014; Salerno & Peter-Hagene, 2013; Whitton et al., 2014). These challenges are particularly prevalent for emotions with positive valence (Ellsworth & Smith, 1988; Herring et al., 2011). Also challenging is the frequent co-elicitation of anger with moral disgust (Gross & Levenson, 1995; Jerritta et al., 2014; Salerno & Peter-Hagene, 2013; Whitton et al., 2014), but not with physical disgust.

It is for these reasons that we adopted a more encompassing and rigorous evaluation of the film clips in our Online Validation Study. Indeed, because we anticipated certain film clips would be more advantageous in eliciting one particular emotion over another, we sought to more objectively evaluate the data by taking into consideration these obstacles that participants face when self-reporting emotions. Consequently, we assessed emotions that are known to be evaluated and reported similarly (i.e., amusement and happiness; anger and disgust) so that we could best categorize the evocative capacity of film clips in the context of self-reported emotions. This approach permitted identification of film clips that were effective in eliciting emotional responses that may not have otherwise been evaluated (e.g., recognition of the ability of Crash to elicit disgust, though it was originally expected to elicit anger). This approach thereby facilitates the use of these film clips by future investigators that have a myriad of different experimental needs, yet ultimately are seeking to evoke a particular self-reported emotional experience in participants.

A second limitation is that the vast majority of literature validating these clips utilized undergraduate samples. This is a limitation that is evident in our own research as well. There are relatively few investigations examining the impact of age on responses to emotional film stimuli, but some data (e.g., Hazer et al., 2015; Jenkins & Andrewes, 2012) have accumulated to suggest that important differences can emerge. Indeed, gender, culture, age, and a variety of other factors should be considered when selecting clips for use with varying participant populations (see Alghowinem et al., 2014; Fernández, Pascual, Soler, & Fernández-Abascal, 2011; Gabert-Quillen et al., 2015; Liang, Hsieh, Weng, & Sun, 2013). Moreover, utilization of community, rather than college, samples through crowd-sourced venues such as Amazon Mechanical Turk might reveal different elicitation responses or altered patterns of co-elicitation, given the relative heterogeneity (Peterson, 2001) and distinctive motivations for study participation (Paolacci & Chandler, 2014) that exist in community populations. Aside from demographics, potential confounds in the form of survey satisficing, or financial or academic motivation (e.g., extra credit for study participation), should also be given substantial consideration when selecting film stimuli and sample populations (Hamby & Taylor, 2016). Despite the variation in college attributes (e.g., size, geographical location, international diversity, student culture), college students as a population could exhibit characteristically different emotional responses to film stimuli than community members. These myriad of considerations further emphasize the importance of both selecting and piloting film stimuli, an endeavor we sought to facilitate here both through our own validations and the assembly of the larger validation literature.

For the current investigation, we specifically included recent film clips that we hypothesized would be particularly salient to a college-aged population (e.g., Between Two Ferns, College Conspiracy; Fahrenheit 911: Recruitment). Our research, like much across the behavioral sciences, is highly dependent on college samples. Indeed, given the prevalence of university students in emotion elicitation research using film clips (71% of studies in Tables 5 and 6 used student samples), we chose to execute our investigation in this very population to maximize the utility of this work in future studies. However, we recognize and acknowledge the value of validating film clips in other populations, and have clear plans to do so in our future work. Moreover, we encourage researchers to apply these clips to more diverse populations with the goal of better understanding the impact of age, culture, genders, and ethnicities in emotion processing and emotion research methodology. Finally, there is increasing use of film clips validated on healthy populations for use with clinical samples. Although intuitively this makes sense and we have chosen to include only clips examined in healthy samples, there is evidence (e.g., Aldao, Mennin, & McLaughlin, 2013; Ellard, Farchione, & Barlow, 2012) suggesting that psychopathology and other illnesses evident in a given sample should also be considered as additional factors when selecting clips to elicit a given response.

A particular strength of this study is the evaluation of film clips – which were first piloted in small, in-person participant groups – in eliciting emotion when presented online to individuals. Use of online emotion elicitation methods is burgeoning in psychology (Ferrer et al., 2015), and offers advantages in sample sizes, population accessibility, and relative speed of data generation. To date, only two other studies have validated film clips for the elicitation of emotional responses in an online format (Aldao & Nolen-Hoeksema, 2013 ; Samson et al., 2016). Of course, disadvantages include less control over environmental conditions and computer capabilities, as well as reliance on the accuracy of participants’ reports of demographic values. Presence of authoritative experimenters, such as in the pilot study (see Supplemental Material), helps to curb drifting attention and prevent technical difficulties, minimizing or even preventing the need to exclude participants. This is, of course, labor-intensive and precludes the ability to individually randomize film presentation order to participants as was done in the present study. The strengths and disadvantages of web-based experimentation in psychology are discussed in recent articles (Crump et al., 2013; Ferrer et al., 2015; Gureckis et al., 2015; Mason & Suri, 2012; Woods et al., 2015), and for our Online Validation Study we employed many of the recommended checks to ensure data accuracy.

Conclusion

The purpose of this research was to fill some notable gaps in the literature regarding film clips effective in discretely eliciting frequently co-elicited, and therefore more challenging, emotions (e.g., anger) using an online medium. In our pursuit, we also developed an integrative catalog of film clip stimuli for use in future emotion elicitation research. Over the last several decades the presence of validated film sets as well as research employing them has increased exponentially. Indeed, the utility of a resource cataloging the wide array of film stimuli is evident in the rapidly increasing amount of emotion research broadly across disciplines including medicine, economics, sociology, and psychology. Though there are limitations to relying on the piloting/validation of other researchers examining responses in other samples, we believe that this catalog will serve as a useful resource for future investigations and help support the ongoing growth of emotion research.

Footnotes

  1. 1.

    There is substantial evidence for two conceptually distinct forms of disgust: physical and moral, with evidence of distinct autonomic signatures (e.g., moral disgust is characterized by sympathetic nervous system dominance whereas physical disgust is more para-sympathetically dominant: Ottaviani, Mancini, Petrocchi, Medea & Couyoudjian, 2013) as well as differing relations with other emotions (e.g., moral disgust often co-occurs with anger whereas physical disgust does not).

  2. 2.

    Publically available clips (including details on the construction of clips from commercially available films) as well as all data from the Online validation study are available at: http://personal.kent.edu/~kcoifman/resources.htm

  3. 3.

    In general, mouse clicks were not significantly correlated with the intensity of amusement, happiness, anger, disgust, or sadness for the respective film clip being investigated (see Table S10 in the Supplemental Material). Further, there was no evidence that high versus low mouse clicking influenced the association among ratings of emotions within a given clip. For example, we included high versus low (50% split) mouse clicks as a between-subjects factor in the ANOVAs and there were no significant differences. Overall, our analysis suggested that mouse clicks did not significantly influence the impact and engagement with films, nor the reliability or discrimination of emotion ratings. Finally, we also evaluated correlations between reported intensity of target emotions and prior film clip exposure to determine whether familiarity with the video content attenuated reports of intensity. As with mouse clicks, for the vast majority of the film clips there were no significant correlations (Table S10). For the very few exceptions (see Table S10), the correlations between prior exposure and intensity rating were such that familiarity with a video actually increased the intensity of the targeted emotion.

Supplementary material

13428_2016_842_MOESM1_ESM.docx (79 kb)
ESM 1 (DOCX 79 kb)

References

*indicates references that are cited in Tables 5 and/or 6.

  1. *Aldao, A., Mennin, D.S., & McLaughlin, K.A. (2013) Differentiating worry and rumination: Evidence from heart rate variability during spontaneous regulation. Cognitive Therapy and Research, 37(3):613-619.Google Scholar
  2. *Aldao, A., & Nolen-Hoeksema, S. (2013) One versus many: Capturing the use of multiple emotion regulation strategies in response to an emotion-eliciting stimulus. Cognition and Emotion, 27(4):753-760.Google Scholar
  3. Alghowinem, S., Alghuwinem, S., Alshehri, M., Al-Wabil, A., Goecke, R., & Wagner, M. (2014). Design of an Emotion Elicitation Framework for Arabic Speakers. In M. Kurosu (Ed.), Human-Computer Interaction. Advanced Interaction Modalities and Techniques: 16th International Conference, HCI International 2014, Heraklion, Crete, Greece, June 22-27, 2014, Proceedings, Part II (pp. 717–728). Cham: Springer International Publishing. http://doi.org/ 10.1007/978-3-319-07230-2_68
  4. Barrett, L. F. (2006). Are emotions natural kinds? Perspectives on Psychological Science, 1(1), 28–58.CrossRefPubMedGoogle Scholar
  5. Barrett, L. F., & Bliss-Moreau, E. (2009). Affect as a psychological primitive. In M. P. Zanna (Ed.), Advances in experimental social psychology, vol. 41 (pp. 167–218). Burlington: Academic Press.CrossRefGoogle Scholar
  6. *Boiten, F.A. (1998). The effects of emotional behaviour on components of the respiratory cycle.Biological Psychology, 49, 29-51.Google Scholar
  7. Bonanno, G. A., Goorin, L., & Coifman, K. G. (2008). Sadness and grief. In M. Lewis, J. Haviland-Jones, & L. Feldman Barrett (Eds.), The handbook of emotion (3rd ed.). New York: Guilford.Google Scholar
  8. Carver, C. S., & Harmon-Jones, E. (2009). Anger is an approach-related affect: Evidence and implications. Psychological Bulletin, 135(2), 183–204.CrossRefPubMedGoogle Scholar
  9. *Carvalho, S., Leite, J., Galdo-Álvarez, S., & Gonçalves, Ó. F. (2012). The emotional movie database (EMDB): A self-report and psychophysiological study. Applied Psychophysiology and Biofeedback, 37(4), 279-294.Google Scholar
  10. *Codispoti, M., Surcinelli, P., & Baldaro, B. (2008). Watching emotional movies: Affective reactions and gender differences. International Journal of Psychophysiology, 69(2), 90-95.Google Scholar
  11. Coifman, K. G., Bonanno, G. A., Ray, R., & Gross, J. (2007). Does repression lead to recovery and resilience? Affective-autonomic response discrepancy in bereavement. Journal of Personality and Social Psychology, 92(4), 745–758.CrossRefPubMedGoogle Scholar
  12. Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PLoS ONE, 8(3), e57410. doi: 10.1371/journal.pone.0057410 CrossRefPubMedPubMedCentralGoogle Scholar
  13. Ellsworth, P., & Smith, C. (1988). From appraisal to emotion: Differences among unpleasant feelings. Motivation and Emotion, 12(3), 271–302.CrossRefGoogle Scholar
  14. Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6(3-4), 169–200.CrossRefGoogle Scholar
  15. *Ekman, P., Freisen, W. V., & Ancoli, S. (1980). Facial signs of emotional experience. Journal of Personality and Social Psychology, 39(6), 1125-1134.Google Scholar
  16. Ellard, K. K., Farchione, T. J., & Barlow, D. H. (2012). Relative effectiveness of emotion induction procedures and the role of personal relevance in a clinical sample: A comparison of film, images, and music. Journal of Psychopathology and Behavioral Assessment, 34, 232–243.CrossRefGoogle Scholar
  17. Fernández, C., Pascual, J. C., Soler, J., & Fernández-Abascal, E. (2011). Spanish validation of an emotion-eliciting set of films. Psycothema, 23, 778–785.Google Scholar
  18. Ferrer, R. A., Grenen, E. G., & Taber, J. M. (2015). Effectiveness of internet-based affect induction procedures: A systematic review and meta-analysis. Emotion, 15(6), 752–762. doi: 10.1037/emo0000035 CrossRefPubMedGoogle Scholar
  19. Gabert-Quillen, C. A., Bartolini, E. E., Abravanel, B. T., & Sanislow, C. A. (2015). Ratings for emotion film clips. Behavior Research Methods, 47(3), 773–787.CrossRefPubMedGoogle Scholar
  20. *Gomez, P., Zimmermann, P., Guttormsen-Schär, S., & Danuser, B. (2005). Respiratory responses associated with affective processing of film stimuli. Biological Psychology, 68(3), 223-235.Google Scholar
  21. *Gross, J. J., & Levenson, R. W. (1995). Emotion elicitation using films. Cognition & Emotion, 9(1), 87-108.Google Scholar
  22. Gruber, J., Dutra, S., Eidelman, P., Johnson, S. L., & Harvey, A. G. (2011). Emotional and physiological responses to normative and idiographic positive stimuli in biopolar disorder. Journal of Affective Disorders, 133(3), 437–442.CrossRefPubMedPubMedCentralGoogle Scholar
  23. Gureckis, T.M., Martin, J., McDonnell, J., Rich, A.S., Markant, D., Coenen, A., Halpern, D., Hamrick, J.B., & Chan, P. (2015, in press) psiTurk: An open-source framework for conducting replicable behavioral experiments online. Behavioral Research Methods. doi:  10.3758/s13428-015-0642-8
  24. *Hagemann, D., Naumann, E., Maier, S., Becker, G., Lürken, A., & Bartussek, D. (1999). The assessment of affective reactivity using films: Validity, reliability and sex differences. Personality and Individual Differences, 26(4), 627-639.Google Scholar
  25. Hamby, T., & Taylor, W. (2016). Survey satisficing inflates reliability and validity measures: An experimental comparison of college and Amazon Mechanical Turk samples. Educational and Psychological Measurement, 76(6), 912–932.CrossRefGoogle Scholar
  26. Hazer, D., Ma, X., Rukavina, S., Gruss, S., Walter, S., & Traue, H. C. (2015). Emotion Elicitation Using Film Clips: Effect of Age Groups on Movie Choice and Emotion Rating. In C. Stephanidis (Ed.), HCI International 2015 - Posters’ Extended Abstracts: International Conference, HCI International 2015, Los Angeles, CA, USA, August 2-7, 2015. Proceedings, Part I (pp. 110–116). Cham: Springer International Publishing. http://doi.org/ 10.1007/978-3-319-21380-4_20
  27. Hemenover, S. H., & Schimmack, U. (2007). That's disgusting!…, but very amusing: Mixed feelings of amusement and disgust. Cognition and Emotion, 21(5), 1102–1113.CrossRefGoogle Scholar
  28. Herring, D., Burleson, M., Roberts, N., & Devine, M. (2011). Coherent with laughter: Subjective experience, behavior, and physiological responses during amusement and job. International Journal of Psychophysiology, 79(2), 211–218.CrossRefPubMedGoogle Scholar
  29. *Hewig, J., Hagemann, D., Seifert, J., Gollwitzer, M., Naumann, E., & Bartussek, D. (2005). Brief Report: A revised film set for the induction of basic emotions. Cognition & Emotion, 19(7), 1095-1109.Google Scholar
  30. *Jenkins, L. M., & Andrewes, D. G. (2012). A new set of standardised verbal and non-verbal contemporary film stimuli for the elicitation of emotions. Brain Impairment, 13(02), 212-227.Google Scholar
  31. Jerritta, S., Murugappan, M., Wan, K., & Yaacob, S. (2014). Electrocardiogram-based emotion recognition system using empirical mode decomposition and discrete Fourier transform. Expert Systems, 31(2), 110–120.CrossRefGoogle Scholar
  32. Kashdan, T. B., Barrett, L. F., & McKnight, P. E. (2015). Unpacking emotion differentiation: transforming unpleasant experience by perceiving distinctions in negativity. Current Directions in Psychological Science, 24(1), 10–16. doi: 10.1177/0963721414550708 CrossRefGoogle Scholar
  33. *Kaviani, H., Gray, J. A., Checkley, S. A., Kumari, V., & Wilson, G. D. (1999). Modulation of the acoustic startle reflex by emotionally-toned film-clips. International Journal of Psychophysiology, 32(1), 47-54.Google Scholar
  34. Koelsch, S. (2010). Towards a neural basis of music-evoked emotions. Trends in Cognitive Science, 14(3), 131–137.CrossRefGoogle Scholar
  35. *Kreibig, S. D., Wilhelm, F. H., Roth, W. T., & Gross, J. J. (2007). Cardiovascular, electrodermal, and respiratory response patterns to fear‐and sadness‐inducing films. Psychophysiology, 44(5), 787-806.Google Scholar
  36. Kring, A. M., & Gordan, A. H. (1998). Sex differences in emotion: Expression, experience, and physiology. Journal of Personality and Social Psychology, 74(3), 686–703.CrossRefPubMedGoogle Scholar
  37. Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (2008). International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical Report A-8. Gainesville: University of Florida.Google Scholar
  38. Lazarus, R. S., Speisman, J. C., Mordkoff, A. M., & Davison, L. A. (1962). A laboratory study of psychological stress produced by a motion picture film. Psychological Monographs, 76(34, Whole No. 553). Google Scholar
  39. LeDoux, J. (2012). A neuroscientist’s perspective on debates about the nature of emotion. Emotion Review, 4(4), 375–379.CrossRefGoogle Scholar
  40. Lench, H. C., & Levine, L. J. (2005). Effects of fear on risk and control judgments and memory: Implications for health promotion messages. Cognition & Emotion, 19, 1049–1069.CrossRefGoogle Scholar
  41. *Li, L., & Chen, J. H. (2006). Emotion recognition using physiological signals. In Advances in Artificial Reality and Tele-Existence (pp. 437-446). Springer Berlin Heidelberg.Google Scholar
  42. Liang, Y. C., Hsieh, S., Weng, C. Y., & Sun, C. R. (2013). Taiwan corpora of Chinese emotions and relevant psychophysiological data–Standard Chinese emotional film clips database and subjective evaluation normative data. Chinese Journal of Psychology, 55(4), 597–617.Google Scholar
  43. Logeswaran, N., & Bhattacharya, J. (2009). Crossmodal transfer of emotion by music. Neuroscience Letters, 455(2), 129–133.CrossRefPubMedGoogle Scholar
  44. *Maffei, C., Roder, E., Cortesan, C., Passera, F., Rossi, M., Segrini, M., … Fossati, A. (2014). Kinematic elicitation of basic emotions: A validation study in an Italian sample. Psychology, 5, 1065-1078. http://dx.doi.org/ 10.4236/psych.2014.59119
  45. Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44(1), 1–23.CrossRefPubMedGoogle Scholar
  46. *McHugo, G. J., Smith, C. A., & Lanzetta, J. T. (1982). The structure of self-reports of emotional responses to film segments. Motivation and Emotion, 6(4), 365-385.Google Scholar
  47. Mills, C., & D'Mello, S. (2014). On the validity of the autobiographical emotional memory task for emotion induction. PLoS ONE, 9(4), e95837. doi: 10.1371/journal.pone.0095837 CrossRefPubMedPubMedCentralGoogle Scholar
  48. Ottaviani, C., Mancini, F., Petrocchi, N., Medea, B., & Couyoumdjian, A. (2013). Autonomic correlates of physical and moral disgust. International Journal of Psychophysiology, 89(1), 57–62. doi: 10.1016/j.ijpsycho.2013.05.003 CrossRefPubMedGoogle Scholar
  49. *Palomba, D., Sarlo, M., Angrilli, A., Mini, A., & Stegagno, L. (2000). Cardiac responses associated with affective processing of unpleasant film stimuli. International Journal of Psychophysiology, 36(1), 45-57.Google Scholar
  50. Panksepp, J. (2007). Neurologizing the psychology of affects: How appraisal-based constructivism and basic emotion theory can coexist. Perspectives on Psychological Science, 2(3), 281–296.CrossRefPubMedGoogle Scholar
  51. Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23(3), 184–188.CrossRefGoogle Scholar
  52. Papa, A., & Bonanno, G. A. (2008). Smiling in the face of adversity: Interpersonal and intrapersonal functions of smiling. Emotion, 8, 1–12.CrossRefPubMedGoogle Scholar
  53. Peterson, R. A. (2001). On the use of college students in social science research: Insights from a second-order meta-analysis. Journal of Consumer Research, 28(3), 450–461.CrossRefGoogle Scholar
  54. *Philippot, P. (1993). Inducing and assessing differentiated emotion-feeling states in the laboratory. Cognition & Emotion, 7(2), 171-193.Google Scholar
  55. *Reynaud, E., El-Khoury-Malhame, M., Blin, O., & Khalfa, S. (2012). Voluntary emotion suppression modifies psychophysiological responses to films. Journal of Psychophysiology, 26(3), 116-123.Google Scholar
  56. Robinson, M. D., & Clore, G. L. (2002). Belief and feeling: Evidence for an accessibility model of emotional self-report. Psychological Bulletin, 128(6), 934–960.CrossRefPubMedGoogle Scholar
  57. *Rottenberg, J., Ray, R. D., & Gross, J. J. (2007). Emotion elicitation using films. In J.A. Coan & J.J.B. Allen (Eds). Handbook of Emotion Elicitation and Assessment. New York: Oxford University Press, pp. 9-28.Google Scholar
  58. Salas, C. E., Radovic, D., & Turnbull, O. H. (2012). Inside-out: Comparing internally generated and externally generated basic emotions. Emotion, 12(3), 568–578.CrossRefPubMedGoogle Scholar
  59. Salerno, J., & Peter-Hagene, L. C. (2013). The interactive effect of anger and disgust on moral outrage and judgments. Psychological Science, 24(10), 2069–2078.CrossRefPubMedGoogle Scholar
  60. *Samson, A.C., Kreibig, S.D., Soderstrom, B., Wade, A.A. & Gross, J.J. (2016). Eliciting positive, negative and mixed emotional states: A film library for affective scientists. Cognition and Emotion, 30(5), 827-856, doi:  10.1080/02699931.2015.1031089
  61. *Schaefer, A., Nils, F., Sanchez, X., & Philippot, P. (2010). Assessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchers. Cognition and Emotion, 24(7), 1153-1172.Google Scholar
  62. *Tomarken, A. J., Davidson, R. J., & Henriques, J. B. (1990). Resting frontal brain asymmetry predicts affective responses to films. Journal of Personality and Social Psychology, 59(4), 791.Google Scholar
  63. Tooby, J., & Cosmides, L. (2008). The evolutionary psychology of emotions and their relationship to internal regulatory variables. In M. Lewis, J. M. Haviland-Jones, & L. F. Barrett (Eds.), Handbook of emotions (3rd ed., pp. 114–137). NewYork: Guilford Press.Google Scholar
  64. Uhrig, M. K., Trautmann, N., Baumgärtner, U., Treede, R.-D., Henrich, F., Hiller, W., & Marschall, S. (2016). Emotion elicitation: A comparison of pictures and films. Frontiers in Psychology, 7, 180. doi: 10.3389/fpsyg.2016.00180 CrossRefPubMedPubMedCentralGoogle Scholar
  65. von Leupoldt, A., Rhode, J., Beregova, A., Thordsen-Sorensen, I., zur Nieden, J., & Dahme, B. (2007). Films for eliciting emotional states in children. Behavior Research Methods, 39(3), 606–609.CrossRefGoogle Scholar
  66. *Westerink, J. H., van den Broek, E. L., Schut, M. H., van Herk, J., & Tuinenbreijer, K. (2008). Computing emotion awareness through galvanic skin response and facial electromyography. In Probing Experience (pp. 149-162). Springer Netherlands.Google Scholar
  67. Whitton, A., Henry, J., Rendell, P., & Grisham, J. (2014). Disgust, but not anger provocation, enhances levator labii superioris activity during exposure to moral transgressions. Biological Psychology, 96, 48–56.CrossRefPubMedGoogle Scholar
  68. Woods, A., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015). Conducting perception research over the internet: A tutorial review. PeerJ PrePrints, 3, e1058. doi: 10.7287/peerj.preprints.921v1 CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2017

Authors and Affiliations

  • T. Lee Gilman
    • 1
  • Razan Shaheen
    • 1
  • K. Maria Nylocks
    • 1
  • Danielle Halachoff
    • 1
  • Jessica Chapman
    • 1
  • Jessica J. Flynn
    • 1
  • Lindsey M. Matt
    • 1
  • Karin G. Coifman
    • 1
  1. 1.Department of Psychological SciencesKent State UniversityKentUSA

Personalised recommendations