Abstract
The overwhelming majority of quantitative work in sociology reports levels of statistical significance. Often, significance is reported with little or no discussion of what it actually entails philosophically, and this can be problematic when analyses are interpreted. Often, significance is understood to represent the probability of the null hypothesis (usually understood as a lack of relationship between two or more variables). This understanding is simply erroneous. The first section of this paper deals with this common misunderstanding. The second section gives a history of significance testing in the social sciences, with reference to the historical foundations of many common misinterpretations of significance testing. The third section is devoted to a discussion of the consequences of misinterpreting statistical significance for sociology. It is argued that reporting statistical significance provides sociology with very little value, and that the consequences of misinterpreting significance values outweighs the benefits of their use.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bakan D.: The test of significance in psychological research. Psychol. Bull. 66(6), 423–437 (1996)
Camilleri S.F.: Theory, probability, and induction in social research. In: Morrison, D.E., Henkel, R.E. (eds) The Significance Test Controversy, Aldine, Chicago (1970)
Campbell S.K.: Flaws and fallacies in statistical thinking. Prentice-Hall, Englewood Cliffs (1974)
Carter B., New C.: Realist social theory and empirical research. In: Carter, B., New, C. (eds) Making Realism Work, Routledge, New York (2004)
Carver R.P.: The case against statistical significance testing. Harvard Educ. Rev. 48, 378–399 (1978)
Cohen J.: Things I have learned (so far). Am. Psychol. 45(12), 1304–1312 (1990)
Cohen, J.: The earth is round (p < 0.05). In: Harlow, L.L., Mulail, S.A., Steiger, J.H. (eds.) What if there were no Significance Tests? Lawrence Erlbaum, Mahwah (1997)
Copas J.B., Li H.G.: Inference for non-random samples. J. Roy. Stat. Soc. 59(1), 55–95 (1997)
Daniel L.G.: Statistical significance testing: a historical overview of misuse and misinterpretation with implications for the editorial policies of educational journals. Res. Schools 5(2), 23–32 (1998)
Dawes R.M.: Rational choice in an uncertain world. Harcourt Brace Jovanovich, Toronto (1988)
Duggan, T.J., Dean, C.W.: Common misinterpretations of significance levels in sociological journals. In: Morrison, D.E., Henkel, D.E. (eds.) The Significance Test Controversy. Aldine, Chicago (1970)
Falk R., Greenbaum C.W.: Significance tests die hard: the amazing persistence of a probabilistic misconception. Theory Psychol. 5(1), 75–98 (1995)
Fisher R.A.: The design of experiments. Oliver and Boyd, Edinburgh (1935)
Fisher R.A.: Statistical methods and scientific inference. Oliver & Boyd, Edinburough (1956)
Gigerenzer, G., Krauss, S., Vitouch, O.: The null ritual: What you always wanted to know about significance testing but were afraid to ask. In: Kaplan, D. (ed.) The Sage Handbook of Quantitative Methodology for the Social Sciences. Sage, Thousand Oaks (2004)
Gigerenzer G.: Mindless statistics. J. Socio-Econ. 33, 587–606 (2004)
Gigerenzer, G.: The superego, the ego, and the ID in statistical reasoning. In: Keren, G., Lewis, C. (eds.) A Handbook for Data Analysis in the Behavioural Sciences: Methodological Issues. Lawrence Erlbaum, Hillsdale (1993)
Gigerenzer G., Murray D.J.: Cognition as intuitive statistics. Lawrence Erldaum Associates, Hillsdale (1987)
Gigerenzer, G., Swijtink, Z., Porter, T., Daston, L., Beatty, J., Krüger, L. (ed.): The empire of chance: How probability changed science and everyday life. Cambridge University Press, New York (1989)
Gold, D.: Statistical tests and substantive significance. In: Morrison, D.E., Henkel, R.E. (eds.) The Significance Test Controversy. Aldine, Chicago (1970)
Hacking I.: Logic of statistical inference. Cambridge University Press, New York (1965)
Hacking, I.: Theory of probable inference: Neyman, Pierce and Braithwaite. In: Miller, D.H. (ed.) Science, Belief, and Behaviour. Cambridge University Press, New York (1980)
Harding S.: Whose science? Whose knowledge? Thinking from women’s lives. Open University Press, Milton (1991)
Howson C., Urbach P.: Scientific reasoning: the Bayesian approach. Open Court, La Salle (1989)
Kass E., Wasserman L.: The selection of prior distributions by formal rules. J. Am. Stat. Assoc. 91(435), 1343–1370 (1996)
Kish L.: Some statistical problems in research design. Am. Sociol. Rev. 24, 328–338 (1959)
Leahey E.: Alphas and asterisks: the development of statistical significance testing standards in sociology. Social Forces 84(1), 1–24 (2005)
McGinnis, R.: Randomization and inference in sociological research. In: Morrison, D.E., Henkel, R.E. (eds) The Significance Test Controversy. Aldine, Chicago (1970)
Meier A.M.: Adolescent fist sex and subsequent mental health. Am. J. Sociol. 112(6), 1811–1847 (2007)
Miller J.E., van der Meulen Rogers Y.: Economic importance and statistical significance: guidelines for communicating empirical research. Feminist Econ. 14(2), 117–149 (2008)
Mulaik, S.A., Raju, N.S., Harshman, R.A.: There is a time and a place for significance testing. In: Harlow, L.L., Mulail, S.A., Steiger, J.H. (eds.) What if there were no Significance Tests? Lawrence Erlbaum, Mahwah (1997)
Olsen, W. Medthological triangulation and realist research: an Indian exemplar. In: Carter, B., New, C. (eds.) Making Realism Work. Routledge, New York (2004)
Olsen W., Morgan J.: A critical epistemology of analytical statistics: Addressing the sceptical realist. J. Theory Social Behav. 35(3), 255–284 (2005)
Porter T.M.: Signifying little. Science 320(6), 1292 (2008)
Prukez, R.M.: An introduction to Bayesian inference and its applications. In: Harlow, L.L., Mulail, S.A., Steiger, J.H. (eds.) What if there were no Significance Tests? Lawrence Erlbaum, Mahwah (1997)
Reese A.: Does significance matter?. Significance 1, 39–40 (2004)
Reichardi, C.S., Gollob, H.F.: When confidence intervals should be used instead of significance tests, and visa versa. In: Harlow, L.L., Mulail, S.A., Steiger, J.H. (eds.) What if there were no Significance Tests? Lawrence Erlbaum, Mahwah (1997)
Richards S.: Philosophy & sociology of science: an introduction. Basil Blackwell, Oxford (1983)
Rindskopf, D.M.: Testing ‘small’, not null, hypotheses: classical and Bayesian approaches. In: Harlow, L.L., Mulail, S.A., Steiger, J.H. (eds.) What if there were no Significance Tests? Lawrence Erlbaum, Mahwah (1997)
Roskam E.E.: Operationalization, a superfluous concept. Qual. Quant. 28, 237–275 (1989)
Rozenbloom W.W.: The fallacy of the null-hypothesis significance test. Psychol. Bull. 57(5), 416–428 (1960)
Saris W.E., Gallhofer I.: Operationalization of social science concepts by intuition. Qual. Quant. 38, 235–258 (2004)
Schmidt, F.L., Hunter, J.E.: Eight common but false objections to the discontinuation of significance testing in the analysis of research data. In: Harlow, L.L., Mulail, S.A., Steiger, J.H. (eds.) What if there were no Significance Tests? Lawrence Erlbaum, Mahwah (1997)
Selvin H.C.: A critique of tests of significance in survey research. In: Morrison, D.E., Henkel, R.E. (eds) The significance test controversy, Aldine, Chicago (1970)
Sterne J.A.C., Smith G.D.: Sifting the evidence—what’s wrong with significance tests?. Br. Med. J. 322, 226–231 (2001)
Tyler R.W.: What is statistical significance?. Educat. Res. Bull. 10, 115–118 (1931)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Engman, A. Is there life after P<0.05? Statistical significance and quantitative sociology. Qual Quant 47, 257–270 (2013). https://doi.org/10.1007/s11135-011-9516-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11135-011-9516-z