Advertisement

Recovering the variance of d' from hit and false alarm statistics

  • Juan BotellaEmail author
  • Manuel Suero
Article

Abstract

Sometimes the reports of primary studies that are potentially analyzable within the signal detection theory framework do not report sample statistics for its main indexes, especially the sample variance of d'. We describe a procedure for estimating the variance of d' from other sample statistics (specifically, the mean and variance of the observed rates of hit and false alarm). The procedure acknowledges that individuals can be heterogeneous in their sensitivity and/or decision criteria, and it does not adopt unjustifiable or needlessly complex assumptions. In two simulation studies reported here, we show that the procedure produces certain biases, but, when used in meta-analysis, it produces very reasonable results. Specifically, the weighted estimate of the mean sensitivity is very accurate, and the coverage of the confidence interval is very close to the nominal confidence level. We applied the procedure to 20 experimental groups or conditions from seven articles (employing recognition memory or attention tasks) that reported statistics for both the hit and false alarm rates, as well as for d'. In most of these studies the assumption of homogeneity was untenable. The variances estimated by our method, based on the hit and false alarm rates, approximate reasonably to the variances in d' reported in those articles. The method is useful for estimating unreported variances of d', so that the associated studies can be retained for meta-analyses.

Keywords

SDT d-prime variance Meta-analysis 

Notes

References

  1. Aleman, A., Hijman, R., de Haan, E. H., & Kahn, R. S. (1999). Memory impairment in schizophrenia: a meta-analysis. American Journal of Psychiatry, 156, 1358–1366.PubMedGoogle Scholar
  2. Bora, E., Yucel, M., & Pantelis, C. (2009). Cognitive endophenotypes of bipolar disorder: A meta-analysis of neuropsychological deficits in euthymic patients and their first-degree relatives. Journal of Affective Disorders, 113, 1–20.CrossRefGoogle Scholar
  3. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis, Chichester, UK: Wiley.CrossRefGoogle Scholar
  4. Brown, G. S., & White, K. G. (2005). The optimal correction for estimating extreme discriminability. Behavior Research Methods, 37, 436–449.  https://doi.org/10.3758/BF03192712 CrossRefPubMedGoogle Scholar
  5. Bröder, A., & Schütz, J. (2009). Recognition ROCs are curvilinear—or are they? On premature arguments against the two-high-threshold model of recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 587–606.  https://doi.org/10.1037/a0015279 CrossRefPubMedGoogle Scholar
  6. Cooper, H. M., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.Google Scholar
  7. Dobbins, I. G., & Kroll, N. E. A. (2005). Distinctiveness and the recognition mirror effect: Evidence for an item-based criterion placement heuristic. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31, 1186–1198.  https://doi.org/10.1037/0278-7393.31.6.1186 CrossRefPubMedGoogle Scholar
  8. Ennis, D. M., & Bi, J. (1998). The beta-binomial model: Accounting for inter-trial variation in replicated difference and preference tests. Journal of Sensory Studies, 13, 389–412.CrossRefGoogle Scholar
  9. Fett, A. K. J., Viechtbauer, W., Penn, D. L., van Os, J., & Krabbendam, L. (2011). The relationship between neurocognition and social cognition with functional outcomes in schizophrenia: A meta-analysis. Neuroscience & Biobehavioral Reviews, 35, 573–588.CrossRefGoogle Scholar
  10. Gourevitch, V., & Galanter, E. (1967). A significance test for one parameter isosensitivity functions. Psychometrika, 32, 25–33.CrossRefGoogle Scholar
  11. Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York, NY: Wiley.Google Scholar
  12. Hautus, M. J., & Lee, A. (2006). Estimating sensitivity and bias in a yes/no task. British Journal of Mathematical and Statistical Psychology, 59, 257–273.CrossRefGoogle Scholar
  13. Hays, W. L. (1988). Statistics (4th ed.). Philadelphia, PA: Holt, Rinehart & Winston.Google Scholar
  14. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.Google Scholar
  15. Higham, P. A., Perfect, T. J., & Bruno, D. (2009). Investigating strength and frequency effects in recognition memory using type-2 signal detection theory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 57–80.  https://doi.org/10.1037/a0013865 CrossRefPubMedGoogle Scholar
  16. Hooks, K., Milich, R., & Lorch, E. P. (1994). Sustained and selective attention in boys with attention deficit hyperactivity disorder. Journal of Clinical Child Psychology, 23, 69–77.  https://doi.org/10.1207/s15374424jccp2301_9
  17. Johnson, N. L., Kemp, A. W., & Kotz, S. (2005). Univariate discrete distributions (3rd ed.). Hoboken, NJ: Wiley.CrossRefGoogle Scholar
  18. Johnson, N. L., Kotz, S., & Balakrishnan, N. (1995). Continuous univariate distributions (2nd ed., Vol. 2). Hoboken, NJ: Wiley.Google Scholar
  19. Kleinman, J. C. (1973). Proportions with extraneous variance: Single and independent samples. Journal of the American Statistical Association, 68, 46–54.Google Scholar
  20. Macmillan, N. A., & Creelman, C. D. (2005). Detection theory: A user’s guide (2nd ed.). Mahwah, NJ: Erlbaum.Google Scholar
  21. Matthews, G., Jones, D. M., & Chamberlain, A. G. (1989). Interactive effects of extraversion and arousal on attentional task performance: Multiple resources or encoding processes. Journal of Personality and Social Psychology, 56, 629–639.CrossRefGoogle Scholar
  22. Miller, J. (1996). The sampling distribution of d'. Perception & Psychophysics, 58, 65–72.  https://doi.org/10.3758/BF03205476 CrossRefGoogle Scholar
  23. Murdock, B. B., Jr., & Ogilvie, J. C. (1968). Binomial variability in short-term memory. Psychological Bulletin, 70, 256–260.  https://doi.org/10.1037/h0026259 CrossRefPubMedGoogle Scholar
  24. Nuechterlein, K. H., & Asarnow, R. F. (1992). Manual and computer program for the UCLA Continuous Performance Test: Version 4. Los Angeles, CA: University of California, Los Angeles.Google Scholar
  25. R Core Team. (2015). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from www.R-project.org Google Scholar
  26. Rhodes, M. G., & Anastasi, J. S. (2012). The own-age bias in face recognition: A meta-analytic and theoretical review. Psychological Bulletin, 138, 146–174.  https://doi.org/10.1037/a0025750 CrossRefPubMedGoogle Scholar
  27. Rhodes, M. G., & Jacoby, L. L. (2007). On the dynamic nature of response criterion in recognition memory: Effects of base rate, awareness, and feedback. Journal of Experimental Psychology: Learning, Memory, and Cognition, 33, 305–320.  https://doi.org/10.1037/0278-7393.33.2.305 CrossRefPubMedGoogle Scholar
  28. Roitman, S. E., Cornblatt, B. A., Bergman, A., Obuchowski, M., Mitropoulou, V., Keefe, R. S., . . . Siever, L. J. (1997). Attentional functioning in schizotypal personality disorder. American Journal of Psychiatry, 154, 655–660.CrossRefGoogle Scholar
  29. Rosvold, H. E., Mirsky, A. F., Sarason, I., Bransome, E. D., Jr., & Beck, L. H. (1956). A continuous performance test of brain damage. Journal of Consulting Psychology, 20, 343–350.CrossRefGoogle Scholar
  30. Rotello, C. M. (2017). Signal detection theories of recognition memory. In J. T. Wixted (ed.), Learning and memory: A comprehensive reference (pp. 201–225). Amsterdam: Elsevier.  https://doi.org/10.1016/B978-0-12-809324-5.21044-4
  31. Rubio-Aparicio, M., Marín-Martínez, F., Sánchez-Meca, J., & López-López, J. A. (2018). A methodological review of meta-analyses of the effectiveness of clinical psychology treatments. Behavior Research Methods, 50, 2057–2073.  https://doi.org/10.3758/s13428-017-0973-8 CrossRefPubMedGoogle Scholar
  32. Skellam, J. G. (1948). A probability distribution derived from the binomial distribution by regarding the probability of success as variable between the sets of trials. Journal of the Royal Statistical Society: Series B, 10, 257–261.Google Scholar
  33. Snodgrass, J. G., & Corwin, J. (1988). Pragmatics of measuring recognition memory: Applications to dementia and amnesia. Journal of Experimental Psychology: General, 117, 34–50.  https://doi.org/10.1037/0096-3445.117.1.34
  34. Suero, M., Botella, J., & Privado, J. (2018). Estimating the expected value and variance of SDT indexes with heterogeneous individuals. Journal of Mathematical Psychology, 83, 12–23.CrossRefGoogle Scholar
  35. Suero, M., Privado, J., & Botella, J. (2017). Methods to estimate the variance of some indices of the signal detection theory: A simulation study. Psicológica, 18, 149–175.Google Scholar
  36. Tanner, T. A., Jr., Haller, R. W., & Atkinson, R. C. (1967). Signal recognition as influenced by presentation schedules. Perception & Psychophysics, 2, 349–358.  https://doi.org/10.3758/BF03210070 CrossRefGoogle Scholar
  37. Treisman, M. (1987). Effects of the setting and adjustment of decision criteria on psychophysical performance. In E. E. Roskam & R. Suck (Eds.), Progress in mathematical psychology (pp. 253–297). New York, NY: Elsevier Science.Google Scholar
  38. Treisman, M., & Faulkner, A. (1984). The setting and maintenance of criteria representing levels of confidence. Journal of Experimental Psychology: Human Perception and Performance, 10, 119–139.  https://doi.org/10.1037/0096-1523.10.1.119 CrossRefGoogle Scholar
  39. Treisman, M., & Faulkner, A. (1985). Can decision criteria interchange locations? Some positive evidence. Journal of Experimental Psychology: Human Perception and Performance, 11, 187–208.  https://doi.org/10.1037/0096-1523.11.2.187 CrossRefGoogle Scholar
  40. Treisman, M., & Williams, T. C. (1984). A theory of criterion setting with an application to sequential dependencies. Psychological Review, 91, 68–111.  https://doi.org/10.1037/0033-295X.91.1.68 CrossRefGoogle Scholar
  41. Wickens, T. D. (2001). Elementary signal detection theory. New York, NY: Oxford University Press.CrossRefGoogle Scholar

Copyright information

© The Psychonomic Society, Inc. 2019

Authors and Affiliations

  1. 1.Universidad Autónoma de MadridMadridSpain

Personalised recommendations