Advertisement

Quality & Quantity

, Volume 47, Issue 1, pp 257–270 | Cite as

Is there life after P<0.05? Statistical significance and quantitative sociology

  • Athena Engman
Article

Abstract

The overwhelming majority of quantitative work in sociology reports levels of statistical significance. Often, significance is reported with little or no discussion of what it actually entails philosophically, and this can be problematic when analyses are interpreted. Often, significance is understood to represent the probability of the null hypothesis (usually understood as a lack of relationship between two or more variables). This understanding is simply erroneous. The first section of this paper deals with this common misunderstanding. The second section gives a history of significance testing in the social sciences, with reference to the historical foundations of many common misinterpretations of significance testing. The third section is devoted to a discussion of the consequences of misinterpreting statistical significance for sociology. It is argued that reporting statistical significance provides sociology with very little value, and that the consequences of misinterpreting significance values outweighs the benefits of their use.

Keywords

Statistical significance History of statistics Probability Epistemology 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bakan D.: The test of significance in psychological research. Psychol. Bull. 66(6), 423–437 (1996)CrossRefGoogle Scholar
  2. Camilleri S.F.: Theory, probability, and induction in social research. In: Morrison, D.E., Henkel, R.E. (eds) The Significance Test Controversy, Aldine, Chicago (1970)Google Scholar
  3. Campbell S.K.: Flaws and fallacies in statistical thinking. Prentice-Hall, Englewood Cliffs (1974)Google Scholar
  4. Carter B., New C.: Realist social theory and empirical research. In: Carter, B., New, C. (eds) Making Realism Work, Routledge, New York (2004)Google Scholar
  5. Carver R.P.: The case against statistical significance testing. Harvard Educ. Rev. 48, 378–399 (1978)Google Scholar
  6. Cohen J.: Things I have learned (so far). Am. Psychol. 45(12), 1304–1312 (1990)CrossRefGoogle Scholar
  7. Cohen, J.: The earth is round (p < 0.05). In: Harlow, L.L., Mulail, S.A., Steiger, J.H. (eds.) What if there were no Significance Tests? Lawrence Erlbaum, Mahwah (1997)Google Scholar
  8. Copas J.B., Li H.G.: Inference for non-random samples. J. Roy. Stat. Soc. 59(1), 55–95 (1997)CrossRefGoogle Scholar
  9. Daniel L.G.: Statistical significance testing: a historical overview of misuse and misinterpretation with implications for the editorial policies of educational journals. Res. Schools 5(2), 23–32 (1998)Google Scholar
  10. Dawes R.M.: Rational choice in an uncertain world. Harcourt Brace Jovanovich, Toronto (1988)Google Scholar
  11. Duggan, T.J., Dean, C.W.: Common misinterpretations of significance levels in sociological journals. In: Morrison, D.E., Henkel, D.E. (eds.) The Significance Test Controversy. Aldine, Chicago (1970)Google Scholar
  12. Falk R., Greenbaum C.W.: Significance tests die hard: the amazing persistence of a probabilistic misconception. Theory Psychol. 5(1), 75–98 (1995)CrossRefGoogle Scholar
  13. Fisher R.A.: The design of experiments. Oliver and Boyd, Edinburgh (1935)Google Scholar
  14. Fisher R.A.: Statistical methods and scientific inference. Oliver & Boyd, Edinburough (1956)Google Scholar
  15. Gigerenzer, G., Krauss, S., Vitouch, O.: The null ritual: What you always wanted to know about significance testing but were afraid to ask. In: Kaplan, D. (ed.) The Sage Handbook of Quantitative Methodology for the Social Sciences. Sage, Thousand Oaks (2004)Google Scholar
  16. Gigerenzer G.: Mindless statistics. J. Socio-Econ. 33, 587–606 (2004)CrossRefGoogle Scholar
  17. Gigerenzer, G.: The superego, the ego, and the ID in statistical reasoning. In: Keren, G., Lewis, C. (eds.) A Handbook for Data Analysis in the Behavioural Sciences: Methodological Issues. Lawrence Erlbaum, Hillsdale (1993)Google Scholar
  18. Gigerenzer G., Murray D.J.: Cognition as intuitive statistics. Lawrence Erldaum Associates, Hillsdale (1987)Google Scholar
  19. Gigerenzer, G., Swijtink, Z., Porter, T., Daston, L., Beatty, J., Krüger, L. (ed.): The empire of chance: How probability changed science and everyday life. Cambridge University Press, New York (1989)Google Scholar
  20. Gold, D.: Statistical tests and substantive significance. In: Morrison, D.E., Henkel, R.E. (eds.) The Significance Test Controversy. Aldine, Chicago (1970)Google Scholar
  21. Hacking I.: Logic of statistical inference. Cambridge University Press, New York (1965)Google Scholar
  22. Hacking, I.: Theory of probable inference: Neyman, Pierce and Braithwaite. In: Miller, D.H. (ed.) Science, Belief, and Behaviour. Cambridge University Press, New York (1980)Google Scholar
  23. Harding S.: Whose science? Whose knowledge? Thinking from women’s lives. Open University Press, Milton (1991)Google Scholar
  24. Howson C., Urbach P.: Scientific reasoning: the Bayesian approach. Open Court, La Salle (1989)Google Scholar
  25. Kass E., Wasserman L.: The selection of prior distributions by formal rules. J. Am. Stat. Assoc. 91(435), 1343–1370 (1996)CrossRefGoogle Scholar
  26. Kish L.: Some statistical problems in research design. Am. Sociol. Rev. 24, 328–338 (1959)CrossRefGoogle Scholar
  27. Leahey E.: Alphas and asterisks: the development of statistical significance testing standards in sociology. Social Forces 84(1), 1–24 (2005)CrossRefGoogle Scholar
  28. McGinnis, R.: Randomization and inference in sociological research. In: Morrison, D.E., Henkel, R.E. (eds) The Significance Test Controversy. Aldine, Chicago (1970)Google Scholar
  29. Meier A.M.: Adolescent fist sex and subsequent mental health. Am. J. Sociol. 112(6), 1811–1847 (2007)CrossRefGoogle Scholar
  30. Miller J.E., van der Meulen Rogers Y.: Economic importance and statistical significance: guidelines for communicating empirical research. Feminist Econ. 14(2), 117–149 (2008)CrossRefGoogle Scholar
  31. Mulaik, S.A., Raju, N.S., Harshman, R.A.: There is a time and a place for significance testing. In: Harlow, L.L., Mulail, S.A., Steiger, J.H. (eds.) What if there were no Significance Tests? Lawrence Erlbaum, Mahwah (1997)Google Scholar
  32. Olsen, W. Medthological triangulation and realist research: an Indian exemplar. In: Carter, B., New, C. (eds.) Making Realism Work. Routledge, New York (2004)Google Scholar
  33. Olsen W., Morgan J.: A critical epistemology of analytical statistics: Addressing the sceptical realist. J. Theory Social Behav. 35(3), 255–284 (2005)CrossRefGoogle Scholar
  34. Porter T.M.: Signifying little. Science 320(6), 1292 (2008)CrossRefGoogle Scholar
  35. Prukez, R.M.: An introduction to Bayesian inference and its applications. In: Harlow, L.L., Mulail, S.A., Steiger, J.H. (eds.) What if there were no Significance Tests? Lawrence Erlbaum, Mahwah (1997)Google Scholar
  36. Reese A.: Does significance matter?. Significance 1, 39–40 (2004)CrossRefGoogle Scholar
  37. Reichardi, C.S., Gollob, H.F.: When confidence intervals should be used instead of significance tests, and visa versa. In: Harlow, L.L., Mulail, S.A., Steiger, J.H. (eds.) What if there were no Significance Tests? Lawrence Erlbaum, Mahwah (1997)Google Scholar
  38. Richards S.: Philosophy & sociology of science: an introduction. Basil Blackwell, Oxford (1983)Google Scholar
  39. Rindskopf, D.M.: Testing ‘small’, not null, hypotheses: classical and Bayesian approaches. In: Harlow, L.L., Mulail, S.A., Steiger, J.H. (eds.) What if there were no Significance Tests? Lawrence Erlbaum, Mahwah (1997)Google Scholar
  40. Roskam E.E.: Operationalization, a superfluous concept. Qual. Quant. 28, 237–275 (1989)Google Scholar
  41. Rozenbloom W.W.: The fallacy of the null-hypothesis significance test. Psychol. Bull. 57(5), 416–428 (1960)CrossRefGoogle Scholar
  42. Saris W.E., Gallhofer I.: Operationalization of social science concepts by intuition. Qual. Quant. 38, 235–258 (2004)CrossRefGoogle Scholar
  43. Schmidt, F.L., Hunter, J.E.: Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In: Harlow, L.L., Mulail, S.A., Steiger, J.H. (eds.) What if there were no Significance Tests? Lawrence Erlbaum, Mahwah (1997)Google Scholar
  44. Selvin H.C.: A critique of tests of significance in survey research. In: Morrison, D.E., Henkel, R.E. (eds) The significance test controversy, Aldine, Chicago (1970)Google Scholar
  45. Sterne J.A.C., Smith G.D.: Sifting the evidence—what’s wrong with significance tests?. Br. Med. J. 322, 226–231 (2001)CrossRefGoogle Scholar
  46. Tyler R.W.: What is statistical significance?. Educat. Res. Bull. 10, 115–118 (1931)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2011

Authors and Affiliations

  1. 1.Department of SociologyUniversity of TorontoTorontoCanada

Personalised recommendations