Comparing Reinforcement Values of Facial Expressions: An Eye-Tracking Study

  • Soichiro MatsudaEmail author
  • Takahide Omori
  • Joseph P. McCleery
  • Junichi Yamamoto
Original Article


Reinforcement values of facial expressions for visual fixation remain underdeveloped in extant literature, though numerous studies have investigated the function of facial expressions as antecedents of gaze behavior. This study used a concurrent gaze-contingent reinforcement procedure to compare reinforcement values of positive and negative facial expressions for gaze fixation. In total, 7 of the 20 typical adults fixated more frequently on the side that was followed by positive facial expressions. The results suggest that a novel application of eye tracking technology could allow for the evaluation and comparison of reinforcement values of social visual stimuli.


Gaze fixation Social reinforcers Facial expressions Eye tracking Smile 



This study was funded by Grant-in-Aid for Scientific Research on Innovative Areas No. 18H04182 “Construction of the Face-Body Studies in Transcultural Conditions” to S. M., and the CREST research project on Social Imaging (JPMJCR14E2), Japan Science and Technology Agency to J. Y.

Compliance with Ethical Standards

Conflict of Interest

All authors declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.


  1. Anderson, B. A., & Yantis, S. (2013). Persistence of value-driven attentional capture. Journal of Experimental Psychology: Human Perception and Performance, 39, 6–9.Google Scholar
  2. Anderson, B. A., Laurent, P. A., & Yantis, S. (2013). Reward prediction bias attentional selection. Frontiers in Human Neuroscience, 7, 262.Google Scholar
  3. Averbeck, B. B., & Duchaine, B. (2009). Integration of social and utilitarian factors in decision making. Emotion, 9, 599–608.CrossRefGoogle Scholar
  4. Behrens, T. E. J., Hunt, L. T., Woolrich, M. W., & Rushworth, M. F. (2008). Associative learning of social value. Nature, 456, 245–249.CrossRefGoogle Scholar
  5. Blair, R. J. R. (2003). Facial expressions, their communicatory functions and neuro-cognitive substrates. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 358, 561–572.CrossRefGoogle Scholar
  6. Call, N. A., Shillingsburg, M. A., Bowen, C. N., Reavis, A. R., & Findley, A. J. (2013). Direct assessment of preferences for social interactions in children with autism. Journal of Applied Behavior Analysis, 46, 821–826.CrossRefGoogle Scholar
  7. Cannella-Malone, H. I., Sabielny, L., & Tullis, C. A. (2015). Using eye gaze to identify reinforcers for individuals with severe multiple disabilities. Journal of Applied Behavior Analysis, 48, 680–684.CrossRefGoogle Scholar
  8. Constantino, J. N., Kennon-McGill, S., Weichselbaum, C., Marrus, N., Haider, A., Glowinski, A. L., et al. (2017). Infant viewing of social scenes is under genetic control and is atypical in autism. Nature, 547, 340–344.CrossRefGoogle Scholar
  9. Dube, W. V., Dickson, C. A., Balsamo, L. M., O’Donnell, K. L., Tomanari, G. Y., Farren, K. M., . . . McIlvane, W. J. (2010). Observing behavior and atypically restricted stimulus control. Journal of the Experimental Analysis of Behavior, 94, 297–313.CrossRefGoogle Scholar
  10. Eisenbarth, H., & Alpers, G. W. (2011). Happy mouth and sad eyes: Scanning emotional facial expressions. Emotion, 11, 860–865.CrossRefGoogle Scholar
  11. Falck-Ytter, T., Bölte, S., & Gredebäck, G. (2013). Eye tracking in early autism research. Journal of neurodevelopmental disorders, 5(1), 28. Scholar
  12. Fantz, R. L. (1965). Visual perception from birth as shown by pattern selectivity. Annals of the New York Academy of Sciences, 118, 793–814.CrossRefGoogle Scholar
  13. Feinman, S., & Lewis, M. (1983). Social referencing at ten months: A second-order effect on infants' responses to strangers. Child Development, 54, 878–887.Google Scholar
  14. Ferster, C. B., & Skinner, B. F. (1957). Schedules of reinforcement. New York, NY: Appleton-Century-Crofts.CrossRefGoogle Scholar
  15. Fisher, W., Piazza, C. C., Bowman, L. G., Hagopian, L. P., Owens, J. C., & Slevin, I. (1992). A comparison of two approaches for identifying reinforcers for persons with severe and profound disabilities. Journal of Applied Behavior Analysis, 25, 491–498.CrossRefGoogle Scholar
  16. Fleming, C. V., Wheeler, G. M., Cannella-Malone, H. I., Basbagill, A. R., Chung, Y. C., & Day, K. G. (2010). An evaluation of the use of eye gaze to measure preference of individuals with severe physical and developmental disabilities. Developmental Neurorehabilitation, 13, 266–275.CrossRefGoogle Scholar
  17. Frank, M. C., Vul, E., & Johnson, S. P. (2009). Development of infants’ attention to faces during first year. Cognition, 110, 160–170.CrossRefGoogle Scholar
  18. Furl, N., Gallagher, S., & Averbeck, B. B. (2012). A selective emotional decision-making bias elicited by facial expressions. PLoS ONE, 7, e33461.CrossRefGoogle Scholar
  19. Gliga, T., Elsabbagh, M., Andravizou, A., & Johnson, M. (2009). Faces attract infants’ attention in complex displays. Infancy, 14, 550–562.CrossRefGoogle Scholar
  20. Heerey, E. A. (2014). Learning from social rewards predicts individual differences in self-reported social ability. Journal of Experimental Psychology: General, 143, 332–339.CrossRefGoogle Scholar
  21. Hernstein, R. J. (1961). Relative and absolute strength of response as a function of frequency of reinforcement. Journal of the Experimental Analysis of Behavior, 4, 267–272.CrossRefGoogle Scholar
  22. Jones, W., & Klin, A. (2013). Attention to eyes is present but in decline in 2–6-month-old infants later diagnosed with autism. Nature, 504, 427–431.CrossRefGoogle Scholar
  23. Kelly, M. A., Roscoe, E. M., Hanley, G. P., & Schlichenmeyer, K. (2014). Evaluation of assessment methods for identifying social reinforcers. Journal of Applied Behavior Analysis, 47, 113–135.CrossRefGoogle Scholar
  24. Klin, A., Jones, W., Schultz, R., Volkmar, F., & Cohen, D. (2002). Visual fixation patterns during viewing of naturalistic social situations as predictors of social competence in individuals with autism. Archives of General Psychiatry, 59, 809–816.CrossRefGoogle Scholar
  25. Kringelbach, M., & Rolls, E. T. (2003). Neural correlates of rapid reversal learning in a simple model of human social interaction. NeuroImage, 20, 1371–1383.CrossRefGoogle Scholar
  26. LaBarbera, J. D., Izard, C. E., Vietze, P., & Parisi, S. A. (1976). Four- and six-month-old infants’ visual responses to joy, anger, and neutral expressions. Child Development, 47, 535–538.CrossRefGoogle Scholar
  27. Laidlaw, K. E. W., Foulsham, T., Kuhn, G., & Kingstone, A. (2011). Potential social interactions are important to social attention. Proceedings of the National Academy of Sciences of the United States of America, 108, 5548–5553.CrossRefGoogle Scholar
  28. Martin, T. L., Yu, C. T., Martin, G. L., & Fazzio, D. (2006). On choice, preference, and preference for choice. Behavior Analyst Today, 7, 234–241.CrossRefGoogle Scholar
  29. Matsuda, S., Minagawa, Y., & Yamamoto, J. (2015). Gaze behavior of children with ASD toward pictures of facial expressions. Autism Research & Treatment, 2015.
  30. Mazur, J. E. (1991). Choice. In H. Iverson & K. A. Lattal (Eds.), Experimental analysis of behavior (pp. 219–250). New York, NY: Elsevier.Google Scholar
  31. Paeye, C., & Madelain, L. (2011). Reinforcing saccadic amplitude variability. Journal of the Experimental Analysis of Behavior, 95, 149–162.CrossRefGoogle Scholar
  32. Paeye, C., Schütz, A. C., & Gegenfurther, K. R. (2016). Visual reinforcement shapes eye movements in visual search. Journal of Vision, 16, 15.CrossRefGoogle Scholar
  33. Pelaez, M., Virues-Ortega, J., & Gewirtz, J. L. (2012). Acquisition of social referencing via discrimination training in infants. Journal of Applied Behavior Analysis, 45, 23–36.CrossRefGoogle Scholar
  34. Perez, W. F., Endemann, P., Pessôa, C. V. B. B., & Tomanari, G. Y. (2015). Assessing stimulus control in a discrimination task with compound stimuli: Evaluating testing procedures and tracking eye fixations. The Psychological Record, 65, 83–88.CrossRefGoogle Scholar
  35. Pierce, K., Marinero, S., Hazin, R., McKenna, B., Barnes, C. C., & Malige, A. (2016). Eye tracking reveals abnormal visual preference for geometric images as an early biomarker of an autism spectrum disorder subtype associated with increased symptom severity. Biological Psychiatry, 79, 657–666.CrossRefGoogle Scholar
  36. Pourtois, G., Grandjean, D., Sander, D., & Vuilleumier, P. (2004). Electrophysiological correlates of rapid spatial orienting towards fearful faces. Cerebral Cortex, 14, 619–633.CrossRefGoogle Scholar
  37. Rosenberger, P. B. (1973). Concurrent schedule control of human visual target fixations. Journal of the Experimental Analysis of Behavior, 20, 411–416.CrossRefGoogle Scholar
  38. Schrandt, J. A., Townsend, D. B., & Poulson, C. L. (2009). Teaching empathy skills to children with autism. Journal of Applied Behavior Analysis, 42, 17–32.CrossRefGoogle Scholar
  39. Schroeder, S. R., & Holland, J. G. (1968a). Operant control of eye movements during human vigilance. Science, 161, 292–293.CrossRefGoogle Scholar
  40. Schroeder, S. R., & Holland, J. G. (1968b). Operant control of eye movements. Journal of Applied Behavior Analysis, 1, 161–166.CrossRefGoogle Scholar
  41. Schroeder, S. R., & Holland, J. G. (1969). Reinforcement of eye movement with concurrent schedules. Journal of the Experimental Analysis of Behavior, 12, 897–903.CrossRefGoogle Scholar
  42. Steingrimsdottir, H. S., & Arntzen, E. (2016). Eye movements during conditional discrimination training. The Psychological Record, 66, 201–212.CrossRefGoogle Scholar
  43. Teller, D. Y. (1979). The forced-choice preferential looking procedure: A psychophysical technique for use with human infants. Infant Behavior & Development, 2, 135–153.CrossRefGoogle Scholar
  44. Vernetti, A., Senju, A., Charman, T., Johnson, M. H., Gliga, T., & Team, B. A. S. I. S. (2018). Stimulating interaction: Using gaze-contingent eye-tracking to measure the reward value of social signals in toddlers with and without autism. Developmental Cognitive Neuroscience, 29, 21–29.CrossRefGoogle Scholar
  45. Vernetti, A., Smith, T. J., & Senju, A. (2017). Gaze-contingent reinforcement learning reveals incentive value of social signals in young children adults. Proceedings of the Royal Society of London B: Biological Sciences, 284, 20162747.CrossRefGoogle Scholar
  46. Wang, Q., Bolhuis, J., Rothkopf, C. A., Kolling, T., Knoph, M., & Triesch, J. (2012). Infants in control: Rapid anticipation of action outcomes in a gaze-contingent paradigm. PLoS ONE, 7, e30884.CrossRefGoogle Scholar
  47. Windsor, J., Piché, L. M., & Locke, P. A. (1994). Preference testing: A comparison of two presentation methods. Research in Developmental Disabilities, 15, 439–455.CrossRefGoogle Scholar
  48. de Wit, T. C. J., Falck-Ytter, T., & von Hofsten, C. (2008). Young children with autism spectrum disorder look differently at positive versus negative emotional faces. Research in Autism Spectrum Disorders, 2, 651–659.CrossRefGoogle Scholar
  49. Zahn-Waxler, C., Robinson, J. L., & Emde, R. N. (1992). The development of empathy in twins. Developmental Psychology, 28, 1038–1047.CrossRefGoogle Scholar

Copyright information

© Association for Behavior Analysis International 2019

Authors and Affiliations

  1. 1.Faculty of Human SciencesUniversity of TsukubaTsukubaJapan
  2. 2.Department of PsychologyKeio UniversityTokyoJapan
  3. 3.Department of Psychology and Kinney Cefignter for Autism Education and SupportSaint Joseph’s UniversityPhiladelphiaUSA
  4. 4.Center for Autism ResearchChildren’s Hospital of PhiladelphiaPhiladelphiaUSA

Personalised recommendations