The relative merit of empirical priors in non-identifiable and sloppy models: Applications to models of learning and decision-making

Empirical priors
Theoretical Review
  • 80 Downloads

Abstract

Formal modeling approaches to cognition provide a principled characterization of observed responses in terms of a set of postulated processes, specifically in terms of parameters that modulate the latter. These model-based characterizations are useful to the extent that there is a clear, one-to-one mapping between parameters and model expectations (identifiability) and that parameters can be recovered from reasonably sized data using a typical experimental design (recoverability). These properties are sometimes not met for certain combinations of model classes and data. One suggestion to improve parameter identifiability and recoverability involves the use of “empirical priors”, which constrain parameters according to a previously observed distribution of values. We assessed the efficacy of this proposal using a combination of real and artificial data. Our results showed that a point-estimate variant of the empirical-prior method could not improve parameter recovery systematically. We identified the source of poor parameter recovery in the low information content of the data. As a follow-up step, we developed a fully Bayesian variant of the empirical-prior method and assessed its performance. We find that even such a method that takes the covariance structure of the parameter distributions into account cannot reliably improve parameter recovery. We conclude that researchers should invest additional efforts in improving the informativeness of their experimental designs, as many of the problems associated to impoverished designs cannot be alleviated by modern statistical methods alone.

Keywords

Identifiability Empirical priors Reinforcement learning Prospect theory 

Notes

References

  1. Ahn, W.-Y., Krawitz, A., Kim, W., Busemeyer, J. R., & Brown, J. W. (2011). A model-based fMRI analysis with hierarchical Bayesian parameter estimation. Journal of Neuroscience, Psychology, and Economics, 4, 95–110.  https://doi.org/10.1037/a0020684 CrossRefPubMedPubMedCentralGoogle Scholar
  2. Ahn, W.-Y., Vasilev, G., Lee, S.-H., Busemeyer, J. R., Kruschke, J. K., Bechara, A., & Vassileva, J. (2014). Decision-making in stimulant and opiate addicts in protracted abstinence: Evidence from computational modeling with pure users. Frontiers in Psychology, 5, 1–15.  https://doi.org/10.3389/fpsyg.2014.00849 CrossRefGoogle Scholar
  3. Ashby, N. J. S., & Rakow, T. (2016). Eyes on the prize? Evidence of diminishing attention to experienced and foregone outcomes in repeated experiential choice. Journal of Behavioral Decision Making, 29, 183–193.  https://doi.org/10.1002/bdm.1872 CrossRefGoogle Scholar
  4. Bamber, D., & van Santen, J. P. (1985). How many parameters can a model have and still be testable? Journal of Mathematical Psychology, 29, 443–473.  https://doi.org/10.1016/0022-2496(85)90005-7 CrossRefGoogle Scholar
  5. Barron, G., & Erev, I. (2003). Small feedback-based decisions and their limited correspondence to description-based decisions. Journal of Behavioral Decision Making, 16, 215–233.  https://doi.org/10.1002/bdm.443 CrossRefGoogle Scholar
  6. Batchelder, W. H., & Riefer, D. M. (1990). Multinomial processing models of source monitoring. Psychological Review, 97, 548–564.  https://doi.org/10.1037/0033-295X.97.4.548 CrossRefGoogle Scholar
  7. Bechara, A., Damasio, A. R., Damasio, H., & Anderson, S. W. (1994). Insensitivity to future consequences following damage to human prefrontal cortex. Cognition, 50, 7–15.  https://doi.org/10.1016/0010-0277(94)90018-3 CrossRefPubMedGoogle Scholar
  8. Booij, A. S., van Praag, B. M. S., & van de Kuilen, G. (2009). A parametric analysis of prospect theory’s functionals for the general population. Theory and Decision, 68, 115–148.  https://doi.org/10.1007/s11238-009-9144-4 CrossRefGoogle Scholar
  9. Broomell, S. B., & Bhatia, S. (2014). Parameter recovery for decision modeling using choice data. Decision, 1, 252–274.  https://doi.org/10.1037/dec0000020 CrossRefGoogle Scholar
  10. Brown, K. S., & Sethna, J. P. (2003). Statistical mechanical approaches to models with many poorly known parameters. Physical Review E, 68, 021904.  https://doi.org/10.1103/PhysRevE.68.021904 CrossRefGoogle Scholar
  11. Buchner, A., & Erdfelder, E. (2005). Word frequency of irrelevant speech distractors affects serial recall. Memory & Cognition, 33, 86–97.  https://doi.org/10.3758/BF03195299 CrossRefGoogle Scholar
  12. Canessa, N., Crespi, C., Motterlini, M., Baud-Bovy, G., Chierchia, G., Pantaleo, G., & Cappa, S. F. (2013). The functional and structural neural basis of individual differences in loss aversion. Journal of Neuroscience, 33, 14307–14317.  https://doi.org/10.1523/JNEUROSCI.0497-13.2013 CrossRefPubMedGoogle Scholar
  13. Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., & Riddell, A. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76, 1–32.  https://doi.org/10.18637/jss.v076.i01 CrossRefGoogle Scholar
  14. Chase, H. W., Kumar, P., Eickhoff, S. B., & Dombrovski, A. Y. (2015). Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis. Cognitive, Affective, & Behavioral Neuroscience.  https://doi.org/10.3758/s13415-015-0338-7.
  15. Cousineau, D., & Hélie, S. (2013). Improving maximum likelihood estimation using prior probabilities: A tutorial on maximum a posteriori estimation and an examination of the Weibull distribution. Tutorials in Quantitative Methods for Psychology, 9, 61–71.  https://doi.org/10.20982/tqmp.09.2.p061 CrossRefGoogle Scholar
  16. Dayan, P., & Balleine, B. W. (2002). Reward, motivation, and reinforcement learning. Neuron, 36, 285–298.  https://doi.org/10.1016/S0896-6273(02)00963-7 CrossRefPubMedGoogle Scholar
  17. Dayan, P., & Daw, N. D. (2008). Decision theory, reinforcement learning, and the brain. Cognitive, Affective, & Behavioral Neuroscience, 8, 429–453.  https://doi.org/10.3758/CABN.8.4.429 CrossRefGoogle Scholar
  18. De Martino, B., Camerer, C. F., & Adolphs, R. (2010). Amygdala damage eliminates monetary loss aversion. Proceedings of the National Academy of Sciences, 107, 3788–3792.  https://doi.org/10.1073/pnas.0910230107 CrossRefGoogle Scholar
  19. Erev, I., & Barron, G. (2005). On adaptation, maximization, and reinforcement learning among cognitive strategies. Psychological Review, 112, 912–931.  https://doi.org/10.1037/0033-295X.112.4.912 CrossRefPubMedGoogle Scholar
  20. Frank, M. J., Gagne, C., Nyhus, E., Masters, S., Wiecki, T. V., Cavanagh, J. F., & Badre, D. (2015). fMRI and EEG predictors of dynamic decision parameters during human reinforcement learning. Journal of Neuroscience, 35, 485–494.  https://doi.org/10.1523/JNEUROSCI.2036-14.2015.CrossRefPubMedPubMedCentralGoogle Scholar
  21. Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013) Bayesian data analysis, (3rd edn.) Boca Raton: CRC Press.Google Scholar
  22. Gershman, S. J. (2015). Do learning rates adapt to the distribution of rewards? Psychonomic Bulletin & Review, 22, 1320–1327.  https://doi.org/10.3758/s13423-014-0790-3 CrossRefGoogle Scholar
  23. Gershman, S. J. (2016). Empirical priors for reinforcement learning models. Journal of Mathematical Psychology, 71, 1–6.  https://doi.org/10.1016/j.jmp.2016.01.006 CrossRefGoogle Scholar
  24. Hartig, F., Minunno, F., & Paul, S. (2017). BayesianTools: General-purpose MCMC and SMC samplers and tools for Bayesian statistics. R package version 0.1.3. Retrieved from https://github.com/florianhartig/bayesiantools.
  25. Hoffman, M. D., & Gelman, A. (2014). The No-U-Turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15, 1593–1623. arXiv:1111.4246.Google Scholar
  26. Hulme, C., Roodenrys, S., Schweickert, R., Brown, G. D. A., et al., (1997). Word-frequency effects on short-term memory tasks: Evidence for a redintegration process in immediate serial recall. Journal of Experimental Psychology: Learning, Memory, and Cognition, 23, 1217–1232.  https://doi.org/10.1037//0278-7393.23.5.1217 PubMedGoogle Scholar
  27. Humphries, M. A., Bruno, R., Karpievitch, Y., & Wotherspoon, S. (2015). The expectancy valence model of the Iowa gambling task: Can it produce reliable estimates for individuals? Journal of Mathematical Psychology, 64–65, 17–34.  https://doi.org/10.1016/j.jmp.2014.10.002 CrossRefGoogle Scholar
  28. Jocham, G., Klein, T. A., & Ullsperger, M. (2011). Dopamine-mediated reinforcement learning signals in the striatum and ventromedial prefrontal cortex underlie value-based choices. Journal of Neuroscience, 31, 1606–1613.  https://doi.org/10.1523/JNEUROSCI.3904-10.2011 CrossRefPubMedGoogle Scholar
  29. Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263–292.  https://doi.org/10.2307/1914185 CrossRefGoogle Scholar
  30. Katahira, K. (2016). How hierarchical models improve point estimates of model parameters at the individual level. Journal of Mathematical Psychology, 73, 37–58.  https://doi.org/10.1016/j.jmp.2016.03.007 CrossRefGoogle Scholar
  31. Kellen, D., Mata, R., & Davis-Stober, C. P. (2017). Individual classification of strong risk attitudes: An application across lottery types and age groups. Psychonomic Bulletin & Review, 24, 1341–1349.  https://doi.org/10.3758/s13423-016-1212-5 CrossRefGoogle Scholar
  32. Kellen, D., Pachur, T., & Hertwig, R. (2016). How (in)variant are subjective representations of described and experienced risk and rewards? Cognition, 157, 126–138.  https://doi.org/10.1016/j.cognition.2016.08.020 CrossRefPubMedGoogle Scholar
  33. Lee, D., Seo, H., & Jung, M. W. (2012). Neural basis of reinforcement learning and decision making. Annual Review of Neuroscience, 35, 287–308.  https://doi.org/10.1146/annurev-neuro-062111-150512 CrossRefPubMedPubMedCentralGoogle Scholar
  34. Lee, M. D. (forthcoming). Bayesian methods in cognitive modeling. In J.T. Wixted (Ed.) The Stevens’ handbook of experimental psychology and cognitive neuroscience (4th edition, volume 5: Methodology). New York: Wiley.Google Scholar
  35. Lee, M. D., & Vanpaemel, W. (2017). Determining informative priors for cognitive models. Psychonomic Bulletin & Review.  https://doi.org/10.3758/s13423-017-1238-3.
  36. Lee, M. D., & Webb, M. R. (2005). Modeling individual differences in cognition. Psychonomic Bulletin & Review, 12, 605–621.  https://doi.org/10.3758/BF03196751 CrossRefGoogle Scholar
  37. Leong, Y. C., Radulescu, A., Daniel, R., DeWoskin, V., & Niv, Y. (2017). Dynamic interaction between reinforcement learning and attention in multidimensional environments. Neuron, 93, 451–463.  https://doi.org/10.1016/j.neuron.2016.12.040 CrossRefPubMedPubMedCentralGoogle Scholar
  38. Levy, H., & Levy, M. (2002). Experimental test of the prospect theory value function: A stochastic dominance approach. Organizational Behavior and Human Decision Processes, 89, 1058–1081.  https://doi.org/10.1016/S0749-5978(02)00011-0 CrossRefGoogle Scholar
  39. Lewandowsky, S., & Farrell, S. (2010) Computational modeling in cognition: Principles and practice. Thousand Oaks: Sage Publications Inc.Google Scholar
  40. Li, S.-C., Lewandowsky, S., & DeBrunner, V. E. (1996). Using parameter sensitivity and interdependence to predict model scope and falsifiability. Journal of Experimental Psychology: General, 125, 360–369.  https://doi.org/10.1037/0096-3445.125.4.360 CrossRefGoogle Scholar
  41. Moran, R. (2016). Thou shalt identify! The identifiability of two high-threshold models in confidence-rating recognition (and super-recognition) paradigms. Journal of Mathematical Psychology, 73, 1–11.  https://doi.org/10.1016/j.jmp.2016.03.002 CrossRefGoogle Scholar
  42. Mullen, K., Ardia, D., Gil, D., Windover, D., & Cline, J. (2011). DEoptim : An R package for global optimization by differential evolution. Journal of Statistical Software, 40, 1–17.  https://doi.org/10.18637/jss.v040.i06 CrossRefGoogle Scholar
  43. Nilsson, H., Rieskamp, J., & Wagenmakers, E.-J. (2011). Hierarchical Bayesian parameter estimation for cumulative prospect theory. Journal of Mathematical Psychology, 55, 84–93.  https://doi.org/10.1016/j.jmp.2010.08.006 CrossRefGoogle Scholar
  44. Niv, Y., Daniel, R., Geana, A., Gershman, S. J., Leong, Y. C., Radulescu, A., & Wilson, R. C. (2015). Reinforcement learning in multidimensional environments relies on attention mechanisms. Journal of Neuroscience, 35, 8145–8157.  https://doi.org/10.1523/JNEUROSCI.2978-14.2015 CrossRefPubMedPubMedCentralGoogle Scholar
  45. Niv, Y., Edlund, J. A., Dayan, P., & O’Doherty, J. P. (2012). Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain. Journal of Neuroscience, 32, 551–562.  https://doi.org/10.1523/JNEUROSCI.5498-10.2012 CrossRefPubMedGoogle Scholar
  46. Pighin, S., Bonini, N., Savadori, L., Hadjichristidis, C., & Schena, F. (2014). Loss aversion and hypoxia: Less loss aversion in oxygen-depleted environment. Stress, 17, 204–210.  https://doi.org/10.3109/10253890.2014.891103 CrossRefPubMedGoogle Scholar
  47. Plonsky, O., & Erev, I. (2017). Learning in settings with partial feedback and the wavy recency effect of rare events. Cognitive Psychology, 93, 18–43.  https://doi.org/10.1016/j.cogpsych.2017.01.002 CrossRefPubMedGoogle Scholar
  48. Plonsky, O., Teodorescu, K., & Erev, I. (2015). Reliance on small samples, the wavy recency effect, and similarity-based learning. Psychological Review, 122, 621–647.  https://doi.org/10.1037/a0039413 CrossRefPubMedGoogle Scholar
  49. Quiggin, J. (1982). A theory of anticipated utility. Journal of Economic Behavior & Organization, 3, 323–343. arXiv:1011.1669v3.  https://doi.org/10.1016/0167-2681(82)90008-7 CrossRefGoogle Scholar
  50. R Development Core Team (2008). R: A language and environment for statistical computing. R Foundation for Statistical computing, Vienna, Austria. Retrieved from http://www.r-project.org.
  51. Scheibehenne, B., & Pachur, T. (2015). Using Bayesian hierarchical parameter estimation to assess the generalizability of cognitive models of choice. Psychonomic Bulletin & Review, 22, 391–407.  https://doi.org/10.3758/s13423-014-0684-4 CrossRefGoogle Scholar
  52. Schmittmann, V. D., Dolan, C. V., Raijmakers, M. E., & Batchelder, W. H. (2010). Parameter identification in multinomial processing tree models. Behavior Research Methods, 42, 836–846.  https://doi.org/10.3758/BRM.42.3.836 CrossRefPubMedGoogle Scholar
  53. Schulze, C., van Ravenzwaaij, D., & Newell, B. R. (2015). Of matchers and maximizers: How competition shapes choice under risk and uncertainty. Cognitive Psychology, 78, 78–98.  https://doi.org/10.1016/j.cogpsych.2015.03.002 CrossRefPubMedGoogle Scholar
  54. Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461–464.  https://doi.org/10.1214/aos/1176344136 CrossRefGoogle Scholar
  55. Schweickert, R. (1993). A multinomial processing tree model for degradation and redintegration in immediate recall. Memory & Cognition, 21, 168–175.  https://doi.org/10.3758/BF03202729 CrossRefGoogle Scholar
  56. Stan Development Team (2016a). PyStan: The Python interface to Stan. Retrieved from http://mc-stan.org.
  57. Stan Development Team (2016b). RStan: The R interface to Stan. Retrieved from http://mc-stan.org.
  58. Steingroever, H., Wetzels, R., & Wagenmakers, E.-J. (2013). Validating the PVL-Delta model for the Iowa gambling task. Frontiers in Psychology, 4, 1–17.  https://doi.org/10.3389/fpsyg.2013.00898 CrossRefGoogle Scholar
  59. Steingroever, H., Wetzels, R., & Wagenmakers, E.-J. (2014). Absolute performance of reinforcement-learning models for the Iowa gambling task. Decision, 1, 161–183.  https://doi.org/10.1037/dec0000005 CrossRefGoogle Scholar
  60. Sutton, R. S., & Barto, A. G. (1998) Reinforcement learning: An introduction. Cambridge: MIT Press.Google Scholar
  61. Ter Braak, C. J. F., & Vrugt, J. A. (2008). Differential evolution Markov chain with snooker updater and fewer chains. Statistics and Computing, 18, 435–446.  https://doi.org/10.1007/s11222-008-9104-9 CrossRefGoogle Scholar
  62. Tom, S. M., Fox, C. R., Trepel, C., & Poldrack, R. A. (2007). The neural basis of loss aversion in decision-making under risk. Science, 315, 515–518.  https://doi.org/10.1126/science.1134239 CrossRefPubMedGoogle Scholar
  63. Tversky, A., & Kahneman, D. (1992). Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5, 297–323.  https://doi.org/10.1007/BF00122574 CrossRefGoogle Scholar
  64. Vehtari, A., & Lampinen, J. (2002). Bayesian model assessment and comparison using cross-validation predictive densities. Neural Computation, 14, 2439–2468.  https://doi.org/10.1162/08997660260293292 CrossRefPubMedGoogle Scholar
  65. Wakker, P. P. (2010) Prospect theory: For risk and ambiguity. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  66. Walasek, L., & Stewart, N. (2015). How to make loss aversion disappear and reverse: Tests of the decision by sampling origin of loss aversion. Journal of Experimental Psychology: General, 144, 7–11.  https://doi.org/10.1037/xge0000039 CrossRefGoogle Scholar
  67. Wetzels, R., Vandekerckhove, J., Tuerlinckx, F., & Wagenmakers, E.-J. (2010). Bayesian parameter estimation in the expectancy valence model of the Iowa gambling task. Journal of Mathematical Psychology, 54, 14–27.  https://doi.org/10.1016/j.jmp.2008.12.001 CrossRefGoogle Scholar
  68. White, C. N., Servant, M., & Logan, G. D. (2017). Testing the validity of conflict drift-diffusion models for use in estimating cognitive processes: a parameter-recovery study. Psychonomic Bulletin & Review.  https://doi.org/10.3758/s13423-017-1271-2.
  69. Worthy, D. A., Pang, B., & Byrne, K. A. (2013). Decomposing the roles of perseveration and expected value representation in models of the Iowa gambling task. Frontiers in Psychology, 4, 1–9.  https://doi.org/10.3389/fpsyg.2013.00640 CrossRefGoogle Scholar
  70. Yechiam, E., & Busemeyer, J. R. (2005). Comparison of basic assumptions embedded in learning models for experience-based decision making. Psychonomic Bulletin & Review, 12, 387–402.  https://doi.org/10.3758/BF03193783 CrossRefGoogle Scholar
  71. Yechiam, E., & Ert, E. (2007). Evaluating the reliance on past choices in adaptive learning models. Journal of Mathematical Psychology, 51, 75–84.  https://doi.org/10.1016/j.jmp.2006.11.002 CrossRefGoogle Scholar
  72. Yechiam, E., Stout, J. C., Busemeyer, J. R., Rock, S. L., & Finn, P. R. (2005). Individual differences in the response to forgone payoffs: An examination of high functioning drug abusers. Journal of Behavioral Decision Making, 18, 97–110.  https://doi.org/10.1002/bdm.487 CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  1. 1.Faculty of PsychologyUniversity of BaselBaselSwitzerland
  2. 2.Department of PsychologyUniversity of FreiburgFreiburgGermany
  3. 3.College of Arts and SciencesSyracuse UniversitySyracuseUSA

Personalised recommendations