Skip to main content
SpringerLink
Log in

An Introduction to Good Practices in Cognitive Modeling

  • Chapter
  • First Online:
Book cover An Introduction to Model-Based Cognitive Neuroscience

Abstract

Cognitive modeling can provide important insights into the underlying causes of behavior, but the validity of those insights rests on careful model development and checking. We provide guidelines on five important aspects of the practice of cognitive modeling: parameter recovery, testing selective influence of experimental manipulations on model parameters, quantifying uncertainty in parameter estimates, testing and displaying model fit, and selecting among different model parameterizations and types of models. Each aspect is illustrated with examples.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Extensive details are reported here: http://www.donvanravenzwaaij.com/Papers_files/BART_Appendix.pdf.

  2. 2.

    With only 90 trials—the standard number—parameter recovery was very poor.

  3. 3.

    The memory for the traffic light was later assessed by reminding participants that there was a traffic light at the intersection, and asking them to indicate its color.

  4. 4.

    Wagenaar and Boer put forward a similar conclusion, albeit not formalized within a Bayesian framework.

  5. 5.

    The absolute value of the deviance depends on the measurement units for time and so only relative values of deviance are meaningful. Deviance is on an exponential scale, and as a rule of thumb a difference less than 3 is negligible and and difference greater than 10 indicates a strong difference.

  6. 6.

    Note that deviance can be summed over participants, as can AIC, but BIC cannot, due to the nonlinear \(\log(n)\) term in its complexity penalty. Instead the aggregate BIC is calculated from the deviance, number of parameters and sample size summed over participants

References

  1. Brown SD, Heathcote AJ (2008) The simplest complete model of choice reaction time: linear ballistic accumulation. Cognit Psychol 57:153–178

    Google Scholar 

  2. Ratcliff R, Thapar A, McKoon G (2006) Aging, practice, and perceptual tasks: a diffusion model analysis. Psychol Aging 21:353–371

    Google Scholar 

  3. Riefer DM, Knapp BR, Batchelder WH, Bamber D, Manifold V. (2002) Cognitive psychometrics: assessing storage and retrieval deficits in special populations with multinomial processing tree models. Psychol Assess 14:184–201

    Google Scholar 

  4. Mulder M., Wagenmakers EJ, Ratcliff R, Boekel W, Forstmann BU (2012) Bias in the brain: a diffusion model analysis of prior probability and potential payoff. J Neurosci 32:2335–2343

    Google Scholar 

  5. Ratcliff R (1978) A theory of memory retrieval. Psychol Rev 85:59–108

    Google Scholar 

  6. Ratcliff R, Gomez P, McKoon G (2004) Diffusion model account of lexical decision. Psychol Rev 111:159–182

    Google Scholar 

  7. Ratcliff R, Starns JJ (2009) Modeling confidence and response time in recognition memory. Psychol Rev 116:59–83

    Google Scholar 

  8. Ratcliff R, van Dongen HPA (2009) Sleep deprivation affects multiple distinct cognitive processes. Psychon Bull Rev 16:742–751

    Google Scholar 

  9. Smith PL, Ratcliff R (2004) The psychology and neurobiology of simple decisions. Trends Neurosci 27:161–168

    Google Scholar 

  10. Ratcliff R, Tuerlinckx F (2002) Estimating parameters of the diffusion model: approaches to dealing with contaminant reaction times and parameter variability. Psychon Bull Rev 9:438–481

    Google Scholar 

  11. Wagenmakers EJ, van der Maas HJL, Grasman RPPP (2007) An EZ–diffusion model for response time and accuracy. Psychon Bull Rev 14:3–22

    Google Scholar 

  12. Donkin C, Brown S, Heathcote A, Wagenmakers EJ (2011) Different models but the same conclusions about psychological processes? Psychon Bull Rev 18:61–69

    Google Scholar 

  13. Lejuez CW, Read JP, Kahler CW, Richards JB, Ramsey SE, Stuart GL, Strong DR, Brown RA (2002) Evaluation of a behavioral measure of risk taking: the balloon analogue risk task (BART). J Exp Psychol Appl 8:75–84

    Google Scholar 

  14. Wallsten TS, Pleskac TJ, Lejuez CW (2005) Modeling behavior in a clinically diagnostic sequential risk–taking task. Psychol Rev 112:862–880

    Google Scholar 

  15. van Ravenzwaaij D, Dutilh G, Wagenmakers EJ (2011) Cognitive model decomposition of the BART: assessment and application. J Math Psychol 55:94–105

    Google Scholar 

  16. Hintze JL, Nelson RD (1998) Violin plots: a box plot–density trace synergism. Am Stat 52:181–184

    Google Scholar 

  17. Rolison JJ, Hanoch Y, Wood S (2012) Risky decision making in younger and older adults: the role of learning.Psychol Aging 27:129–140

    Google Scholar 

  18. Parks TE (1966) Signal-detectability theory of recognition-memory performance. Psychol Rev 73(1):44–58

    Google Scholar 

  19. Macmillan NA, Creelman CD (2005) Detection theory: a user’s guide, 2nd edn. Erlbaum, Mahwah

    Google Scholar 

  20. Wixted JT, Stretch V (2000) The case against a criterion-shift account of false memory. Psychol Rev 107:368–376

    Google Scholar 

  21. Ratcliff R,Rouder JN (1998) Modeling response times for two-choice decisions. Psychol Sci 9:347–356

    Google Scholar 

  22. Voss A, Rothermund K, Voss J (2004) Interpreting the parameters of the diffusion model: an empirical validation. Mem Cognit 32:1206–1220

    Google Scholar 

  23. Ho TC, Brown S, Serences JT (2009) Domain general mechanisms of perceptual decision making in human cortex. J Neurosci 29:8675–8687

    Google Scholar 

  24. Schwarz G (1978) Estimating the dimension of a model. Annals Stat 6:461–464

    Google Scholar 

  25. Lee M, Vandekerckhove J, Navarro DJ, Tuerlinckx F (2007) Presentation at the 40th Annual Meeting of the Society for Mathematical Psychology, Irvine, USA, July 2007

    Google Scholar 

  26. Starns JJ, Ratcliff R, McKoon G (2012) Evaluating the unequal-variance and dualprocess explanations of zROC slopes with response time data and the diffusion model. Cogniti Psychol 64:1–34

    Google Scholar 

  27. Rae B, Heathcote C, Donkin A, Averell L, Brown SD (in press) The hare and the tortoise: emphasizing speed can change the evidence used to make decisions. J Exp Psychol Learn Mem Cogn

    Google Scholar 

  28. Efron B, Tibshirani RJ (1993) An introduction to the bootstrap. Chapman & Hall, New York

    Google Scholar 

  29. Wagenaar WA, Boer JPA (1987) Misleading postevent information: testing parameterized models of integration in memory. Acta Psychol 66:291–306

    Google Scholar 

  30. Vandekerckhove J, Matzke D, Wagenmakers EJ 2013) In Busemeyer J, Townsend J, Wang ZJ, Eidels A (eds) Oxford handbook of computational and mathematical psychology. Oxford University Press

    Google Scholar 

  31. Lee MD, Wagenmakers EJ (in press) Bayesian modeling for cognitive science: a practical course. Cambridge University Press

    Google Scholar 

  32. Jeffreys H (1961) Theory of probability, 3rd edn. Oxford University Press, Oxford

    Google Scholar 

  33. Anscombe FJ (1973) Graphs in statistical analysis. Am Stat 27:17–21

    Google Scholar 

  34. McCullagh PN, Nelder JA (1983) Generalized linear models. Chapman & Hall, London

    Google Scholar 

  35. Raftery AE (1995) Bayesian model selection in social research. Sociol Methodol 25:111–164

    Google Scholar 

  36. Averell L, Heathcote A (2011) The form of the forgetting curve and the fate of memories. J Math Psychol 55:25–35

    Google Scholar 

  37. Brown S, Heathcote A (2003) Averaging learning curves across and within participants. Behav Res Methods Instrum Comput 35:11–21

    Google Scholar 

  38. Heathcote A, Brown S, Mewhort DJK (2000) The power law repealed: the case for an exponential law of practice. Psychon Bull Rev 7:185–207

    Google Scholar 

  39. Pachella RG (1974) The interpretation of reaction time in information–processing research. In: Kantowitz BH (ed) Human information processing: tutorials in performance and cognition. Lawrence Erlbaum Associates, Hillsdale, pp 41–82

    Google Scholar 

  40. Audley RJ, Pike AR (1965) Some alternative models of stochastic choice. Br J Math Stat Psychol 207–225

    Google Scholar 

  41. Vickers D, Caudrey D, Willson R (1971) Discriminating between the frequency of occurrence of two alternative events. ACTPSY 35:151–172

    Google Scholar 

  42. Ratcliff R, Thapar A, McKoon G (2001) The effects of aging on reaction time in a signal detection task. Psychol Aging 16:323

    Google Scholar 

  43. Wagenmakers EJ, Ratcliff R, Gómez P, McKoon G (2008) A diffusion model account of criterion shifts in the lexical decision task. J Memory Lang 58:140–159

    Google Scholar 

  44. Rae B, Heathcote A, Donkin C, Averell L, Brown SD (2013) The Hare and the tortoise: emphasizing speed can change the evidence used to make decisions. J Exp Psychol Learn Mem Cogn 1–45

    Google Scholar 

  45. Wagenmakers EJ, Ratcliff R, Gómez P, Iverson GJ (2004) Assessing model mimicry using the parametric bootstrap. J Math Psychol 48:28–50

    Google Scholar 

  46. Ratcliff R, Rouder JN (1998) Modeling response times for two–choice decisions. Psychol Sci 9:347–356

    Google Scholar 

  47. Morey RD (2008) Confidence intervals from normalized data: a correction to Cousineau. Tutor Quant Methods Psychol 4:61–64

    Google Scholar 

  48. Heathcote A, Love J (2012) Linear deterministic accumulator models of simple choice. Front Psychol 3

    Google Scholar 

  49. Starns JJ, Ratcliff R, McKoon G (2012) Evaluating the unequal-variability and dual-process explanations of zroc slopes with response time data and the diffusion model. Cognit Psychol 64:1–34

    Google Scholar 

  50. Cleveland WS (1993) Visualizing data. Hobart Press, New Jersey

    Google Scholar 

  51. Heathcote A, Brown S, Mewhort DJK (2002) Quantile maximum likelihood estimation of response time distributions. Psychon Bull Rev 9(2):394–401

    Google Scholar 

  52. Heathcote A, Brown S (2004) A theoretical basis for QML. Psychon Bull Rev 11:577–578

    Google Scholar 

  53. Speckman PL, Rouder JN (2004) A comment on Heathcote, Brown, and Mewhort’s QMLE method for response time distributions. Psychon Bull Rev 11:574–576

    Google Scholar 

  54. Kass RE, Raftery AE (1995) Bayes factors. J Am Stat Assoc 90:773–795

    Google Scholar 

  55. Lodewyckx T, Kim W, Lee MD, Tuerlinckx F, Kuppens P, Wagenmakers EJ (2011) A tutorial on Bayes factor estimation with the product space method. J Math Psychol 55:331–347

    Google Scholar 

  56. Shiffrin R, Lee M, Kim W, Wagenmakers EJ (2008) A survey of model evaluation approaches with a tutorial on Hierarchical Bayesian Methods. Cognit Sci A Multidiscip J 32:1248–1284

    Google Scholar 

  57. Spiegelhalter DJ, Best NG, Carlin BP, Van Der Linde A (2002) Bayesian measures of model complexity and fit. J Royal Stat Soc Series B (Stat Methodol) 64:583–639

    Google Scholar 

  58. Ando T (2007) Bayesian predictive information criterion for the evaluation of hierarchical Bayesian and empirical Bayes models. Biometrika 94:443–458

    Google Scholar 

  59. Myung IJ, Pitt MA (1997) Applying Occam’s razor in modeling cognition: a Bayesian approach. Psych Bull Rev 4:79–95

    Google Scholar 

  60. Burnham KP, Anderson DR (2004) Multimodel inference: understanding AIC and BIC in model selection. Sociol Methods Res 33:261–304

    Google Scholar 

  61. Donkin C, Brown S, Heathcote A (2011) Drawing conclusions from choice response time models: a tutorial using the linear ballistic accumulator. J Math Psychol 55:140–151

    Google Scholar 

  62. Turner BM, Sederberg PB, Brown SD, Steyvers M (2013) A method for efficiently sampling from distributions with correlated dimensions. Psychol Methods 18:368–384

    Google Scholar 

  63. Farrell S, Wagenmakers EJ, Ratcliff R (2006) 1/f noise in human cognition: is it ubiquitous, and what does it mean? Psych Bull Rev 13:737–741

    Google Scholar 

  64. Gilden DL (1997) Fluctuations in the time required for elementary decisions. Psychol Sci 8:296–301

    Google Scholar 

  65. Box GEP (1979) Robustness in the strategy of scientific model building. In: Launer RL, Wilkinson GN (ed) Robustness in statistics: proceedings of a workshop. Academic, New York, pp 201–236

    Google Scholar 

  66. Lewandowsky S, Farrell S (2010) Computational modeling in cognition: principles and practice. Sage, Thousand Oaks

    Google Scholar 

  67. Busemeyer JR, Diederich A (2010) Cognitive modeling. Sage, Thousand Oaks

    Google Scholar 

  68. Hunt E (2006) The mathematics of behavior. Cambridge University Press, Cambridge

    Google Scholar 

  69. Polk TA, Seifert CM (eds) (2002) Cognitive modeling. MIT Press, Cambridge

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Psychology, University of Newcastle, University Avenue, NSW 2308, Callaghan, Australia

    Andrew Heathcote & Scott D. Brown

  2. Department of Psychological Methods, University of Amsterdam, 1018 XA, Weesperplein 4, Amsterdam, The Netherlands

    Eric-Jan Wagenmakers

Authors
  1. Andrew Heathcote
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Scott D. Brown
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eric-Jan Wagenmakers
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrew Heathcote .

Editor information

Editors and Affiliations

  1. Cognitive Science Center, University of Amsterdam, Amsterdam, The Netherlands

    Birte U. Forstmann

  2. Department of Psychological Methods, University of Amsterdam, Amsterdam, The Netherlands

    Eric-Jan Wagenmakers

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer Science+Business Media, LLC

About this chapter

Cite this chapter

Heathcote, A., Brown, S., Wagenmakers, EJ. (2015). An Introduction to Good Practices in Cognitive Modeling. In: Forstmann, B., Wagenmakers, EJ. (eds) An Introduction to Model-Based Cognitive Neuroscience. Springer, New York, NY. https://doi.org/10.1007/978-1-4939-2236-9_2

Download citation

Publish with us

Policies and ethics

Search

Navigation