Advertisement

A Holistic Approach to Empirical Analysis: The Insignificance of P, Hypothesis Testing and Statistical Significance*

  • Morris AltmanEmail author
Conference paper
  • 49 Downloads
Part of the Springer Proceedings in Mathematics & Statistics book series (PROMS, volume 313)

Abstract

It is well documented that academics and practitioners focus on statistical significance (typically represented by P tests) and statistical hypothesis testing to determine if their non-statistical analytical hypothesis is correct or likely to be correct. Moreover, statistical significance is relied upon to determine which variables should be used in their models or analyses. In spite of ongoing criticism, this practice continues to the detriment of robust scientific analysis. I discuss the significant limitations of statistical significance in scientific analysis, irrespective of discipline, with some focus on economics. I place statistical significance in a broader analytical context, discussing other analytical procedures that need to be followed and emphasized for one’s analysis to be scientifically robust. This relates the development of models, assumptions underlying the models, the data collected and constructed, the relationship between statistical significance and causality and the importance of non-statistical theory to the identification of pertinent modelling variables. Analytical significance (size effects and variability) is core to any robust scientific analysis, but only in the context of all of the other prior steps in the applied research project being in place. In this broader analytical framework, the statistical significance becomes relatively insignificant. I also address why statistical significance and statistical hypothesis testing dominates the applied analytical landscape even though this dominance is not best practice. Of critical importance are the mental models of best practice and the worldview and preferences of decision makers determining what gets published and who is successful in securing grants and employment.

Keywords

Statistical significance Analytical significance Size effects P-values Theory Causality Herding Mental models Power relationships 

Notes

Acknowledgements

Thanks to Hannah Altman and Louise Lamontagne for their helpful comments and suggestions.

References

  1. 1.
    Akerlof, G., Kranton, R.: Economics and identity. Q. J. Econ. 115, 715–753 (2000)Google Scholar
  2. 2.
    Akerlof, G.A., Shiller, R.J.: Animal Spirits: How Human Psychology Drives the Economy, and Why It Matters for Global Capitalism. Princeton University Press, Princeton (2009)Google Scholar
  3. 3.
    Altman, M.: The methodology of economics and the survivor principle revisited and revised: some welfare and public policy implications of modeling the economic agent. Rev. Soc. Econ. 57, 427–449 (1999)Google Scholar
  4. 4.
    Altman, M.: Introduction to special issue on statistical significance. J. Socio-Econ. 33, 523–525 (2004)Google Scholar
  5. 5.
    Altman, M.: Statistical significance, path dependency, and the culture of journal publication. J. Socio-Econ. 33, 651–663 (2004)Google Scholar
  6. 6.
    Altman, M.: Mental models, bargaining power and institutional change. In: Paper present at the, World Interdisciplinary Network for Institutional Research Conference, Old Royal Naval College, Greenwich, London, UK, 11–14 September 2014Google Scholar
  7. 7.
    Altman, M.: A bounded rationality assessment of the new behavioral economics. In: Frantz, R., Chen, S.-H., Dopfer, K., Heukelom, F., Mousavi, S. (eds.) Routledge Handbook of Behavioral Economics, pp. 179–193. Routledge, London (2017)Google Scholar
  8. 8.
    Altman, M.: A more scientific approach to applied economics: reconstructing statistical, analytical significance, and correlation analysis. Available at SSRN (2018). https://ssrn.com/abstract=3126147 or  https://doi.org/10.2139/ssrn.3126147
  9. 9.
    American Statistical Association. American statistical association releases statement on statistical significance and p-values: provides principles to improve the conduct and interpretation of quantitative science (2016). https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf
  10. 10.
    Arrow, K.J.: Decision Theory and the Choice of a Level of Significance for the t-Test. In: Olkin, I., et al. (eds.) Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling, pp. 70–78. Stanford University Press, Stanford (1960)Google Scholar
  11. 11.
    Baddeley, M.: Herding, social influence and expert opinion. J. Econ. Methodol. 20, 35–44 (2013)Google Scholar
  12. 12.
    Bailey, D.H., Borwein, J.M., Brent, R.P., Reisi, M.: Reproducibility in computational science: a case study: randomness of the digits of Pi. Exp. Math. (2016).  https://doi.org/10.1080/10586458.2016.1163755
  13. 13.
    Becker, G.: Accounting for Tastes. Harvard University Press, Cambridge (1996); Coase, R.: Essays on Economics and Economists. University of Chicago Press, Chicago (1994)Google Scholar
  14. 14.
    Coe, R.: Its the effect size, stupid: what effect size is and why it is important. In: Paper presented at the 2002 Annual Conference of the British Educational Research Association, University of Exeter, Exeter, Devon, England, September 12–14 (2002). http://www.leeds.ac.uk/educol/documents/00002182.htm
  15. 15.
    Cohen, J.: Statistical Power Analysis for the Behavioral Sciences. Routledge, New York (1977)Google Scholar
  16. 16.
    Deaton, A.: Instruments, randomization, and learning about development. J. Econ. Lit. 48, 424–455 (2010)Google Scholar
  17. 17.
    Fanelli, D.: How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLOS One 4 (2009). https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0005738
  18. 18.
    Fidler, F., Thomason, N., Cumming, G., Finch, S., Lee, J.: Editors can lead researchers to confidence intervals, but can’t make them think: statistical reform lessons from medicine. Psychol. Sci. 15, 119–126 (2004)Google Scholar
  19. 19.
    Friedman, M.: The methodology of positive economics. In: Friedman, M. (ed.) Essays in Positive Economics, pp. 3–43. University of Chicago Press, Chicago (1953)Google Scholar
  20. 20.
    Gallo, A.: A refresher on statistical significance. Harvard Business Review (2016). https://hbr.org/2016/02/a-refresher-on-statistical-significance#comment-section
  21. 21.
    Harford, T.: Big data: are we making a big mistake? Financial Times (2014). https://www.ft.com/content/21a6e7d8-b479-11e3-a09a-00144feabdc0
  22. 22.
    Keynes, J.M.: The General Theory of Employment, Interest and Money. Macmillan, London (1936)Google Scholar
  23. 23.
    Leibenstein, H.: Allocative efficiency vs. X-efficiency. Am. Econ. Rev. 56, 392–415 (1966)Google Scholar
  24. 24.
    Munafõ, M.R., Smith, G.D.: Replication is not enough. Nature: Int. J. Sci. 553, 400–401 (2018). https://www.nature.com/magazine-assets/d41586-018-01023-3/d41586-018-01023-3.pdf
  25. 25.
    McCloskey, D.: The loss function has been mislaid: the rhetoric of significance tests. Am. Econ. Rev. 75, 201–205 (1985)Google Scholar
  26. 26.
    McCloskey, D.: The insignificance of statistical significance. Sci. Am. 72, 32–33 (1995)CrossRefGoogle Scholar
  27. 27.
    McCloskey, D.N., Ziliak, S.: The standard error of regressions. J. Econ. Lit. 34, 97–114 (1996)Google Scholar
  28. 28.
    McCrum-Gardner, E.: Sample size and power calculations made simple. Int. J. Ther. Rehabil. 17, 10–14 (2010)Google Scholar
  29. 29.
    Morrison, D.E., Henkel, R.E.: The Significance Test Controversy: A Reader. Aldine, Chicago (1970)Google Scholar
  30. 30.
    Qualtrics. Calculating sample size (2018). https://www.qualtrics.com/blog/calculating-sample-size/
  31. 31.
    Simon, H.A.: Behavioral Economics. In: Eatwell, J., Millgate, M., Newman, P. (eds.) The New Palgrave: A Dictionary of Economics. Macmillan, London (1987)Google Scholar
  32. 32.
    Taleb, N.N.: The Black Swan: The Impact of the Highly Improbable. Random House, New York (2007)Google Scholar
  33. 33.
    Thompson, B.: Why encouraging effect size reporting is not working: the etiology of researcher resistance to changing practices. J. Psychol. 133, 133–141 (1999)Google Scholar
  34. 34.
    Wasserstein, R.L., Lazar, N.A.: The ASA’s statement on p-values: context, process, and purpose. Am. Stat. 70, 129–133 (2016).  https://doi.org/10.1080/00031305.2016.1154108
  35. 35.
    Ziliak, S., McCloskey, D.N.: Size matters: the standard error of regressions in the American economic. Rev. J. Socio-Econ. 33, 527–546Google Scholar
  36. 36.
    Ziliak, S., McCloskey, D.N.: The Cult of Statistical Significance: How the Standard Error Costs Jobs, Justice, and Lives. University of Michigan Press, Ann Arbor (2008)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Dean, University of Dundee School of Business (UDSB)DundeeScotland, UK
  2. 2.Chair Professor of Behavioural and Institutional Economics, and Co-operativesDundeeScotland, UK

Personalised recommendations