Advertisement

Too much model, too little data: How a maximum-likelihood fit of a psychometric function may fail, and how to detect and avoid this

  • Nicolaas PrinsEmail author
Article
  • 42 Downloads

Abstract

Maximum-likelihood estimation of the parameters of a psychometric function typically occurs through an iterative search for the maximum value in the likelihood function defined across the parameter space. This procedure is subject to failure. First, iterative search procedures may converge on a local, not global, maximum in the likelihood function. The procedure also fails when the likelihood function does not contain a maximum. This is the case when either a step function or a constant function is associated with a higher likelihood than the model can attain with finite parameter values. In such cases iterative search procedures may erroneously report having successfully converged on a maximum in the likelihood function. This will lead not only to inaccurate models for the observed data, but may also lead to inaccurate results regarding the reliability of parameter estimates, goodness-of-fit of the model, or model selection. I describe a method by which such false convergences can be reliably detected. I also present results of simulations that systematically investigate how stimulus placement, number of trials, parameters estimated, task (2AFC, 4AFC, etc.), and whether the lapse rate is allowed to vary affect the probability that the likelihood function will not contain a maximum. Based on the results of the simulations recommendations are made regarding experimental design and modeling choices. Software that implements the method is made available for downloading.

Keywords

Model selection Statistics Statistical inference 

Notes

Acknowledgements

Some of the results presented here were also presented as a poster at the 17th Annual Meeting of the Vision Sciences Society. Prins, N. (2017) Journal of Vision, 17(10): 788-788.

Compliance with ethical standards

Open practices statement

All reported simulations and psychometric fits were performed using the Palamedes Toolbox version 1.9.1 available at www.palamedestoolbox.org. No experiments were preregistered.

References

  1. Albert, A., & Anderson, J.A. (1984). On the existence of maximum likelihood estimates in logistic regression models. Biometrika, 71(1), 1-10.CrossRefGoogle Scholar
  2. Efron, B. & Tibshirani, R.J. (1993). An introduction to the bootstrap. Chapman & Hall/CRC, Boca Raton, FL.CrossRefGoogle Scholar
  3. Fründ, I., Haenel, N.V., & Wichmann, F.A. (2011). Inference for psychometric functions in the presence of nonstationary behavior. Journal of Vision, 11(6), 16, 1-19.CrossRefPubMedGoogle Scholar
  4. Hall, J.L. (1981). Hybrid adaptive procedure for estimation of psychometric functions. Journal of the Acoustical Society of America, 69(6), 1763-1769.CrossRefPubMedGoogle Scholar
  5. Kingdom, F.A.A. & Prins, N. (2016). Psychophysics: A Practical Introduction, 2nd Ed. Academic Press, San Diego, CA.Google Scholar
  6. Kolda, T.G., Lewis, R.M., & Torczon, V. (2003). Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45, 385-482.CrossRefGoogle Scholar
  7. Kontsevich, L.L. & Tyler, C.W. (1999). Bayesian adaptive estimation of psychometric slope and threshold. Vision Research, 39, 2729-2737.CrossRefPubMedGoogle Scholar
  8. Kuss, M., Jäkel, F., & Wichmann, F.A. (2005). Bayesian inference for psychometric functions. Journal of Vision, 5, 478-492.CrossRefPubMedGoogle Scholar
  9. Lagarias, J.C., Reeds, J.A., Wright, M.H., & Wright, P.E. (1998). Convergence properties of the Nelder-Mead simplex method in low dimensions. SIAM J Optim, 9(1), 112-147.CrossRefGoogle Scholar
  10. Linares, D. & López-Moliner, J. (2016). quickpsy: An R Package to Fit Psychometric Functions for Multiple Groups. The R Journal, 8:1, 122-131.CrossRefGoogle Scholar
  11. Manny, R.E. & Klein, S.A. (1985). A three alternative tracking paradigm to measure Vernier acuity of older infants. Vision Research, 25(9), 1245-1252.CrossRefPubMedGoogle Scholar
  12. May K. A., Solomon J. A. (2013). Four theorems on the psychometric function. PLoS One, 8 (10), e74815.CrossRefPubMedPubMedCentralGoogle Scholar
  13. Nelder, J.A. & Mead, R. (1965). A simplex method for function minimization. Comput. J. 7, 308-313.CrossRefGoogle Scholar
  14. Pratt, J.W. (1981). Concavity of the Log Likelihood. Journal of the American Statistical Association, 76(373), 103-106.CrossRefGoogle Scholar
  15. Prins, N. (2012). The psychometric function: The lapse rate revisited. Journal of Vision, 12(6): 25, 1-16CrossRefPubMedGoogle Scholar
  16. Prins, N. (2013). The psi-marginal adaptive method: How to give nuisance parameters the attention they deserve (no more, no less). Journal of Vision, 13(7): 3, 1-17.CrossRefPubMedGoogle Scholar
  17. Prins, N. & Kingdom, F.A.A. (2018). Applying the model-comparison approach to test specific research hypotheses in psychophysical research using the Palamedes Toolbox. Frontiers in Psychology, 9:1250.CrossRefPubMedPubMedCentralGoogle Scholar
  18. Schütt, H.H., Harmeling, S., Macke, J.H., & Wichmann, F.A. (2016). Painfree and accurate Bayesian estimation of psychometric functions for (potentially) overdispersed data. Vision Research, 122, 105-123.CrossRefPubMedGoogle Scholar
  19. Swanson, W.H. & Birch, E.E. (1992). Extracting thresholds from noisy psychophysical data. Perception & Psychophysics, 51(5), 409-422.CrossRefGoogle Scholar
  20. Swets, J.A. (1961). Is there a sensory threshold? Science, 134, 168-177.CrossRefPubMedGoogle Scholar
  21. Treutwein, B. & Strasburger, H. (1999). Fitting the psychometric function. Perception & Psychophysics, 61(1), 87-106.CrossRefGoogle Scholar
  22. van Driel, J., Knapen, T., van Es, D.M., & Cohen, M.X. (2014). Interregional alpha-band synchrony supports temporal cross-modal integration. Neuroimage, 101, 404-415.CrossRefPubMedGoogle Scholar
  23. Wichmann, F. A., & Hill, N. J. (2001). The psychometric function: I. fitting, sampling, and goodness of fit. Perception and Psychophysics, 63(8), 1293–1313.CrossRefPubMedGoogle Scholar
  24. Williamson, R.E. & Trotter, H.F. (1996). Multivariable Mathematics, 3rd Ed. Prentice Hall: Upper Saddle River, NJGoogle Scholar

Copyright information

© The Psychonomic Society, Inc. 2019

Authors and Affiliations

  1. 1.Department of PsychologyUniversity of MississippiOxfordUSA

Personalised recommendations