Effects of feedback reliability on feedback-related brain activity: A feedback valuation account

Article
  • 43 Downloads

Abstract

Adaptive decision making relies on learning from feedback. Because feedback sometimes can be misleading, optimal learning requires that knowledge about the feedback’s reliability be utilized to adjust feedback processing. Although previous research has shown that feedback reliability indeed influences feedback processing, the underlying mechanisms through which this is accomplished remain unclear. Here we propose that feedback processing is adjusted by the adaptive, top-down valuation of feedback. We assume that unreliable feedback is devalued relative to reliable feedback, thus reducing the reward prediction errors that underlie feedback-related brain activity and learning. A crucial prediction of this account is that the effects of feedback reliability are susceptible to contrast effects. That is, the effects of feedback reliability should be enhanced when both reliable and unreliable feedback are experienced within the same context, as compared to when only one level of feedback reliability is experienced. To evaluate this prediction, we measured the event-related potentials elicited by feedback in two experiments in which feedback reliability was varied either within or between blocks. We found that the fronto-central valence effect, a correlate of reward prediction errors during reinforcement learning, was reduced for unreliable feedback. But this result was obtained only when feedback reliability was varied within blocks, thus indicating a contrast effect. This suggests that the adaptive valuation of feedback is one mechanism underlying the effects of feedback reliability on feedback processing.

Keywords

Feedback validity Feedback reliability Decision making Feedback-related negativity P3 

Notes

Author note

This research was supported by a grant from the Deutsche Forschungsgemeinschaft (DFG: STE 1708/3-1) to M.S. We are grateful to Johannes Fiedler, Christina Görner, and Sabine Utschick for assistance in conducting the experiments.

References

  1. Bellebaum, C., Polezzi, D., & Daum, I. (2010). It is less than you expected: The feedback-related negativity reflects violations of reward magnitude expectations. Neuropsychologia, 48, 3343–3350.CrossRefPubMedGoogle Scholar
  2. Crespi, L. P. (1942). Quantitative variation of incentive and performance in the white rat. American Journal of Psychology, 55, 467–517.CrossRefGoogle Scholar
  3. De Araujo, I. E., Rolls, E. T., Velazco, M. I., Margot, C., & Cayeux, I. (2005). Cognitive modulation of olfactory processing. Neuron, 46, 671–679.CrossRefPubMedGoogle Scholar
  4. Delorme, A., & Makeig, S. (2004). EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134, 9–21.  https://doi.org/10.1016/j.jneumeth.2003.10.009 CrossRefPubMedGoogle Scholar
  5. Doll, B. B., Hutchison, K. E., & Frank, M. J. (2011). Dopaminergic genes predict individual differences in susceptibility to confirmation bias. Journal of Neuroscience, 31, 6188–6198.CrossRefPubMedPubMedCentralGoogle Scholar
  6. Doll, B. B., Jacobs, W. J., Sanfey, A. G., & Frank, M. J. (2009). Instructional control of reinforcement learning: A behavioral and neurocomputational investigation. Brain Research, 1299, 74–94.CrossRefPubMedPubMedCentralGoogle Scholar
  7. Donchin, E., & Coles, M. G. H. (1988). Is the P300 component a manifestation of context updating? Behavioral and Brain Sciences, 11, 357–374, disc. 374–427.  https://doi.org/10.1017/S0140525X00058027 CrossRefGoogle Scholar
  8. Ernst, B., & Steinhauser, M. (2012). Feedback-related brain activity predicts learning from feedback in multiple-choice testing. Cognitive, Affective, & Behavioral Neuroscience, 12, 323–336.CrossRefGoogle Scholar
  9. Ernst, B., & Steinhauser, M. (2015). Effects of invalid feedback on learning and feedback-related brain activity in decision-making. Brain and Cognition, 99, 78–86.CrossRefPubMedGoogle Scholar
  10. Ernst, B., & Steinhauser, M. (2017). Top-down control over feedback processing: The probability of valid feedback affects feedback-related brain activity. Brain and Cognition, 115, 33–40.CrossRefPubMedGoogle Scholar
  11. Ferdinand, N. K., Mecklinger, A., Kray, J., & Gehring, W. J. (2012). The processing of unexpected positive response outcomes in the mediofrontal cortex. Journal of Neuroscience, 32, 12087–12092.CrossRefPubMedGoogle Scholar
  12. Flaherty, C. F. (1982). Incentive contrast: A review of behavioral changes following shifts in reward. Animal Learning & Behavior, 10, 409–440.CrossRefGoogle Scholar
  13. Flaherty, C. F. (1999). Incentive relativity (Vol. 15). Cambridge: Cambridge University Press.Google Scholar
  14. Foti, D., Weinberg, A., Bernat, E. M., & Proudfit, G. H. (2015). Anterior cingulate activity to monetary loss and basal ganglia activity to monetary gain uniquely contribute to the feedback negativity. Clinical Neurophysiology, 126, 1338–1347.CrossRefPubMedGoogle Scholar
  15. Grabenhorst, F., & Rolls, E. T. (2010). Attentional modulation of affective versus sensory processing: Functional connectivity and a top-down biased activation theory of selective attention. Journal of Neurophysiology, 104, 1649–1660.CrossRefPubMedGoogle Scholar
  16. Grabenhorst, F., & Rolls, E. T. (2011). Value, pleasure and choice in the ventral prefrontal cortex. Trends in Cognitive Sciences, 15, 56–67.CrossRefPubMedGoogle Scholar
  17. Grabenhorst, F., Rolls, E. T., & Bilderbeck, A. (2007). How cognition modulates affective responses to taste and flavor: Top-down influences on the orbitofrontal and pregenual cingulate cortices. Cerebral Cortex, 18, 1549–1559.CrossRefPubMedGoogle Scholar
  18. Hare, T. A., Camerer, C. F., & Rangel, A. (2009). Self-control in decision-making involves modulation of the vmPFC valuation system. Science, 324, 646–648.CrossRefPubMedGoogle Scholar
  19. Hare, T. A., O’Doherty, J., Camerer, C. F., Schultz, W., & Rangel, A. (2008). Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. Journal of Neuroscience, 28, 5623–5630.CrossRefPubMedGoogle Scholar
  20. Holroyd, C. B., & Coles, M. G. H. (2002). The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109, 679–708.  https://doi.org/10.1037/0033-295X.109.4.679 CrossRefPubMedGoogle Scholar
  21. Holroyd, C. B., Larsen, J. T., & Cohen, J. D. (2004). Context dependence of the event related brain potential associated with reward and punishment. Psychophysiology, 41, 245–253.  https://doi.org/10.1111/j.1469-8986.2004.00152.x CrossRefPubMedGoogle Scholar
  22. Holroyd, C. B., Pakzad-Vaezi, K. L., & Krigolson, O. E. (2008). The feedback correct-related positivity: Sensitivity of the event-related brain potential to unexpected positive feedback. Psychophysiology, 45, 688–697.  https://doi.org/10.1111/j.1469-8986.2008.00668.x CrossRefPubMedGoogle Scholar
  23. Hornak, J., O’Doherty, J., Bramham, J., Rolls, E. T., Morris, R. G., Bullock, P. R., & Polkey, C. E. (2004). Reward-related reversal learning after surgical excisions in orbito-frontal or dorsolateral prefrontal cortex in humans. Journal of Cognitive Neuroscience, 16, 463–478.CrossRefPubMedGoogle Scholar
  24. Johnson, J., Jr (1986). A triarchic model of P300 amplitude. Psychophysiology, 23, 367–384.CrossRefPubMedGoogle Scholar
  25. Johnson, J., Jr, & Donchin, E. (1978). On how P300 amplitude varies with the utility of the eliciting stimuli. Electroencephalography and Clinical Neurophysiology, 44, 424–437.CrossRefPubMedGoogle Scholar
  26. Kobayashi, S., de Carvalho, O. P., & Schultz, W. (2010). Adaptation of reward sensitivity in orbitofrontal neurons. Journal of Neuroscience, 30, 534–544.CrossRefPubMedPubMedCentralGoogle Scholar
  27. Kringelbach, M. L., & Rolls, E. T. (2003). Neural correlates of rapid reversal learning in a simple model of human social interaction. NeuroImage, 20, 1371–1383.CrossRefPubMedGoogle Scholar
  28. Li, J., Delgado, M. R., & Phelps, E. A. (2011). How instructed knowledge modulates the neural systems of reward learning. Proceedings of the National Academy of Sciences, 108, 55–60.CrossRefGoogle Scholar
  29. Luck, S. J., & Hillyard, S. A. (1994). Electrophysiological correlates of feature analysis during visual search. Psychophysiology, 31, 291–308.CrossRefPubMedGoogle Scholar
  30. Maxwell, F. R., Calef, R. S., Murray, D. W., Shepard, J. C., & Norville, R. A. (1976). Positive and negative successive contrast effects following multiple shifts in reward magnitude under high drive and immediate reinforcement. Animal Learning & Behavior, 4, 480–484.CrossRefGoogle Scholar
  31. Miltner, W. H. R., Braun, C. H., & Coles, M. G. H. (1997). Event-related brain potentials following incorrect feedback in a time-estimation task: Evidence for a “generic” neural system for error detection. Journal of Cognitive Neuroscience, 9, 788–798.  https://doi.org/10.1162/jocn.1997.9.6.788 CrossRefPubMedGoogle Scholar
  32. Nieuwenhuis, S. (2011). Learning, the P3, and the locus coeruleus–norepinephrine system. In R. B. Mars, J. Sallet, M. Rushworth, & N. Yeung (Eds.), Neural basis of motivational and cognitive control (pp. 209–222). Oxford: Oxford University Press.Google Scholar
  33. Nieuwenhuis, S., Aston-Jones, G., & Cohen, J. D. (2005). Decision making, the P3, and the locus coeruleus-norepinephrine system. Psychological Bulletin, 131, 510–532.  https://doi.org/10.1037/0033-2909.131.4.510 CrossRefPubMedGoogle Scholar
  34. O’Connell, R. G., Dockree, P. M., & Kelly, S. P. (2012). A supramodal accumulation-to-bound signal that determines perceptual decisions in humans. Nature Neuroscience, 15, 1729–1735.CrossRefPubMedGoogle Scholar
  35. Padoa-Schioppa, C., & Assad, J. A. (2008). The representation of economic value in the orbitofrontal cortex is invariant for changes of menu. Nature Neuroscience, 11, 95–102.CrossRefPubMedGoogle Scholar
  36. Plassmann, H., O’Doherty, J., Shiv, B., & Rangel, A. (2008). Marketing actions can modulate neural representations of experienced pleasantness. Proceedings of the National Academy of Sciences, 105, 1050–1054.CrossRefGoogle Scholar
  37. Polich, J. (2007). Updating P300: An integrative theory of P3a and P3b. Clinical Neurophysiology, 118, 2128–2148.  https://doi.org/10.1016/j.clinph.2007.04.019 CrossRefPubMedPubMedCentralGoogle Scholar
  38. Pontifex, M. B., Hillman, C. H., & Polich, J. (2009). Age, physical fitness, and attention: P3a and P3b. Psychophysiology, 46, 379–387.CrossRefPubMedPubMedCentralGoogle Scholar
  39. Rushworth, M. F., & Behrens, T. E. (2008). Choice, uncertainty and value in prefrontal and cingulate cortex. Nature Neuroscience, 11, 389–397.CrossRefPubMedGoogle Scholar
  40. Sailer, U., Fischmeister, F. P. S., & Bauer, H. (2010). Effects of learning on feedback-related brain potentials in a decision-making task. Brain Research, 1342, 85–93.CrossRefPubMedGoogle Scholar
  41. San Martín, R. (2012). Event-related potential studies of outcome processing and feedback-guided learning. Frontiers in Human Neuroscience, 6:304.  https://doi.org/10.3389/fnhum.2012.00304 CrossRefPubMedPubMedCentralGoogle Scholar
  42. Schiffer, A.-M., Siletti, K., Waszak, F., & Yeung, N. (2017). Adaptive behaviour and feedback processing integrate experience and instruction in reinforcement learning. NeuroImage, 146, 626–641.CrossRefPubMedPubMedCentralGoogle Scholar
  43. Schultz, W., Dayan, P., & Montague, P. R. (1997) A neural substrate of prediction and reward. Science, 275, 1593–1599.CrossRefPubMedGoogle Scholar
  44. Walentowska, W., Moors, A., Paul, K., & Pourtois, G. (2016). Goal relevance influences performance monitoring at the level of the FRN and P3 components. Psychophysiology, 53, 1020–1033.CrossRefPubMedGoogle Scholar
  45. Walsh, M. M., & Anderson, J. R. (2012). Learning from experience: Event-related potential correlates of reward processing, neural adaptation, and behavioral choice. Neuroscience & Biobehavioral Reviews, 36, 1870–1884.  https://doi.org/10.1016/j.neubiorev.2012.05.008 CrossRefGoogle Scholar
  46. Yeung, N., & Sanfey, A. G. (2004). Independent coding of reward magnitude and valence in the human brain. Journal of Neuroscience, 24, 6258–6264.  https://doi.org/10.1523/JNEUROSCI.4537-03.2004 CrossRefPubMedGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  1. 1.Allgemeine PsychologieCatholic University Eichstätt-IngolstadtEichstättGermany

Personalised recommendations