Too much model, too little data: How a maximum-likelihood fit of a psychometric function may fail, and how to detect and avoid this
- 42 Downloads
Maximum-likelihood estimation of the parameters of a psychometric function typically occurs through an iterative search for the maximum value in the likelihood function defined across the parameter space. This procedure is subject to failure. First, iterative search procedures may converge on a local, not global, maximum in the likelihood function. The procedure also fails when the likelihood function does not contain a maximum. This is the case when either a step function or a constant function is associated with a higher likelihood than the model can attain with finite parameter values. In such cases iterative search procedures may erroneously report having successfully converged on a maximum in the likelihood function. This will lead not only to inaccurate models for the observed data, but may also lead to inaccurate results regarding the reliability of parameter estimates, goodness-of-fit of the model, or model selection. I describe a method by which such false convergences can be reliably detected. I also present results of simulations that systematically investigate how stimulus placement, number of trials, parameters estimated, task (2AFC, 4AFC, etc.), and whether the lapse rate is allowed to vary affect the probability that the likelihood function will not contain a maximum. Based on the results of the simulations recommendations are made regarding experimental design and modeling choices. Software that implements the method is made available for downloading.
KeywordsModel selection Statistics Statistical inference
Some of the results presented here were also presented as a poster at the 17th Annual Meeting of the Vision Sciences Society. Prins, N. (2017) Journal of Vision, 17(10): 788-788.
Compliance with ethical standards
Open practices statement
All reported simulations and psychometric fits were performed using the Palamedes Toolbox version 1.9.1 available at www.palamedestoolbox.org. No experiments were preregistered.