# Too much model, too little data: How a maximum-likelihood fit of a psychometric function may fail, and how to detect and avoid this

- 42 Downloads

## Abstract

Maximum-likelihood estimation of the parameters of a psychometric function typically occurs through an iterative search for the maximum value in the likelihood function defined across the parameter space. This procedure is subject to failure. First, iterative search procedures may converge on a local, not global, maximum in the likelihood function. The procedure also fails when the likelihood function does not contain a maximum. This is the case when either a step function or a constant function is associated with a higher likelihood than the model can attain with finite parameter values. In such cases iterative search procedures may erroneously report having successfully converged on a maximum in the likelihood function. This will lead not only to inaccurate models for the observed data, but may also lead to inaccurate results regarding the reliability of parameter estimates, goodness-of-fit of the model, or model selection. I describe a method by which such false convergences can be reliably detected. I also present results of simulations that systematically investigate how stimulus placement, number of trials, parameters estimated, task (2AFC, 4AFC, etc.), and whether the lapse rate is allowed to vary affect the probability that the likelihood function will not contain a maximum. Based on the results of the simulations recommendations are made regarding experimental design and modeling choices. Software that implements the method is made available for downloading.

## Keywords

Model selection Statistics Statistical inference## Notes

### Acknowledgements

Some of the results presented here were also presented as a poster at the 17th Annual Meeting of the Vision Sciences Society. Prins, N. (2017) *Journal of Vision, 17*(10): 788-788.

### Compliance with ethical standards

### Open practices statement

All reported simulations and psychometric fits were performed using the Palamedes Toolbox version 1.9.1 available at www.palamedestoolbox.org. No experiments were preregistered.

## References

- Albert, A., & Anderson, J.A. (1984). On the existence of maximum likelihood estimates in logistic regression models.
*Biometrika, 71*(1), 1-10.CrossRefGoogle Scholar - Efron, B. & Tibshirani, R.J. (1993). An introduction to the bootstrap. Chapman & Hall/CRC, Boca Raton, FL.CrossRefGoogle Scholar
- Fründ, I., Haenel, N.V., & Wichmann, F.A. (2011). Inference for psychometric functions in the presence of nonstationary behavior. Journal of Vision, 11(6), 16, 1-19.CrossRefPubMedGoogle Scholar
- Hall, J.L. (1981). Hybrid adaptive procedure for estimation of psychometric functions.
*Journal of the Acoustical Society of America, 69*(6), 1763-1769.CrossRefPubMedGoogle Scholar - Kingdom, F.A.A. & Prins, N. (2016). Psychophysics: A Practical Introduction, 2nd Ed. Academic Press, San Diego, CA.Google Scholar
- Kolda, T.G., Lewis, R.M., & Torczon, V. (2003). Optimization by direct search: new perspectives on some classical and modern methods.
*SIAM Rev. 45*, 385-482.CrossRefGoogle Scholar - Kontsevich, L.L. & Tyler, C.W. (1999). Bayesian adaptive estimation of psychometric slope and threshold.
*Vision Research, 39*, 2729-2737.CrossRefPubMedGoogle Scholar - Kuss, M., Jäkel, F., & Wichmann, F.A. (2005). Bayesian inference for psychometric functions.
*Journal of Vision, 5*, 478-492.CrossRefPubMedGoogle Scholar - Lagarias, J.C., Reeds, J.A., Wright, M.H., & Wright, P.E. (1998). Convergence properties of the Nelder-Mead simplex method in low dimensions.
*SIAM J Optim, 9*(1), 112-147.CrossRefGoogle Scholar - Linares, D. & López-Moliner, J. (2016). quickpsy: An R Package to Fit Psychometric Functions for Multiple Groups.
*The R Journal, 8*:1, 122-131.CrossRefGoogle Scholar - Manny, R.E. & Klein, S.A. (1985). A three alternative tracking paradigm to measure Vernier acuity of older infants.
*Vision Research, 25*(9), 1245-1252.CrossRefPubMedGoogle Scholar - May K. A., Solomon J. A. (2013). Four theorems on the psychometric function.
*PLoS One*, 8 (10), e74815.CrossRefPubMedPubMedCentralGoogle Scholar - Nelder, J.A. & Mead, R. (1965). A simplex method for function minimization.
*Comput. J. 7*, 308-313.CrossRefGoogle Scholar - Pratt, J.W. (1981). Concavity of the Log Likelihood.
*Journal of the American Statistical Association, 76*(373), 103-106.CrossRefGoogle Scholar - Prins, N. (2012). The psychometric function: The lapse rate revisited.
*Journal of Vision, 12*(6): 25, 1-16CrossRefPubMedGoogle Scholar - Prins, N. (2013). The psi-marginal adaptive method: How to give nuisance parameters the attention they deserve (no more, no less).
*Journal of Vision, 13*(7): 3, 1-17.CrossRefPubMedGoogle Scholar - Prins, N. & Kingdom, F.A.A. (2018). Applying the model-comparison approach to test specific research hypotheses in psychophysical research using the Palamedes Toolbox.
*Frontiers in Psychology, 9*:1250.CrossRefPubMedPubMedCentralGoogle Scholar - Schütt, H.H., Harmeling, S., Macke, J.H., & Wichmann, F.A. (2016). Painfree and accurate Bayesian estimation of psychometric functions for (potentially) overdispersed data.
*Vision Research, 122*, 105-123.CrossRefPubMedGoogle Scholar - Swanson, W.H. & Birch, E.E. (1992). Extracting thresholds from noisy psychophysical data.
*Perception & Psychophysics, 51*(5), 409-422.CrossRefGoogle Scholar - Swets, J.A. (1961). Is there a sensory threshold?
*Science, 134*, 168-177.CrossRefPubMedGoogle Scholar - Treutwein, B. & Strasburger, H. (1999). Fitting the psychometric function.
*Perception & Psychophysics, 61*(1), 87-106.CrossRefGoogle Scholar - van Driel, J., Knapen, T., van Es, D.M., & Cohen, M.X. (2014). Interregional alpha-band synchrony supports temporal cross-modal integration.
*Neuroimage, 101*, 404-415.CrossRefPubMedGoogle Scholar - Wichmann, F. A., & Hill, N. J. (2001). The psychometric function: I. fitting, sampling, and goodness of fit. Perception and Psychophysics, 63(8), 1293–1313.CrossRefPubMedGoogle Scholar
- Williamson, R.E. & Trotter, H.F. (1996). Multivariable Mathematics, 3rd Ed. Prentice Hall: Upper Saddle River, NJGoogle Scholar