Abstract
It is well documented that academics and practitioners focus on statistical significance (typically represented by P tests) and statistical hypothesis testing to determine if their non-statistical analytical hypothesis is correct or likely to be correct. Moreover, statistical significance is relied upon to determine which variables should be used in their models or analyses. In spite of ongoing criticism, this practice continues to the detriment of robust scientific analysis. I discuss the significant limitations of statistical significance in scientific analysis, irrespective of discipline, with some focus on economics. I place statistical significance in a broader analytical context, discussing other analytical procedures that need to be followed and emphasized for one’s analysis to be scientifically robust. This relates the development of models, assumptions underlying the models, the data collected and constructed, the relationship between statistical significance and causality and the importance of non-statistical theory to the identification of pertinent modelling variables. Analytical significance (size effects and variability) is core to any robust scientific analysis, but only in the context of all of the other prior steps in the applied research project being in place. In this broader analytical framework, the statistical significance becomes relatively insignificant. I also address why statistical significance and statistical hypothesis testing dominates the applied analytical landscape even though this dominance is not best practice. Of critical importance are the mental models of best practice and the worldview and preferences of decision makers determining what gets published and who is successful in securing grants and employment.
Keywords
Jon was one of the most creative, critical, energetic and open-minded academics I ever met and we hit it off almost immediately. Four years ago we started our collaboration on workshops and a book, on underlying fundamental problems to applied statistics, and math from a multidisciplinary perspective. What a severe loss his passing was to the community of critical thinkers and thought leaders. And, I lost a friend.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Akerlof, G., Kranton, R.: Economics and identity. Q. J. Econ. 115, 715–753 (2000)
Akerlof, G.A., Shiller, R.J.: Animal Spirits: How Human Psychology Drives the Economy, and Why It Matters for Global Capitalism. Princeton University Press, Princeton (2009)
Altman, M.: The methodology of economics and the survivor principle revisited and revised: some welfare and public policy implications of modeling the economic agent. Rev. Soc. Econ. 57, 427–449 (1999)
Altman, M.: Introduction to special issue on statistical significance. J. Socio-Econ. 33, 523–525 (2004)
Altman, M.: Statistical significance, path dependency, and the culture of journal publication. J. Socio-Econ. 33, 651–663 (2004)
Altman, M.: Mental models, bargaining power and institutional change. In: Paper present at the, World Interdisciplinary Network for Institutional Research Conference, Old Royal Naval College, Greenwich, London, UK, 11–14 September 2014
Altman, M.: A bounded rationality assessment of the new behavioral economics. In: Frantz, R., Chen, S.-H., Dopfer, K., Heukelom, F., Mousavi, S. (eds.) Routledge Handbook of Behavioral Economics, pp. 179–193. Routledge, London (2017)
Altman, M.: A more scientific approach to applied economics: reconstructing statistical, analytical significance, and correlation analysis. Available at SSRN (2018). https://ssrn.com/abstract=3126147 or https://doi.org/10.2139/ssrn.3126147
American Statistical Association. American statistical association releases statement on statistical significance and p-values: provides principles to improve the conduct and interpretation of quantitative science (2016). https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf
Arrow, K.J.: Decision Theory and the Choice of a Level of Significance for the t-Test. In: Olkin, I., et al. (eds.) Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling, pp. 70–78. Stanford University Press, Stanford (1960)
Baddeley, M.: Herding, social influence and expert opinion. J. Econ. Methodol. 20, 35–44 (2013)
Bailey, D.H., Borwein, J.M., Brent, R.P., Reisi, M.: Reproducibility in computational science: a case study: randomness of the digits of Pi. Exp. Math. (2016). https://doi.org/10.1080/10586458.2016.1163755
Becker, G.: Accounting for Tastes. Harvard University Press, Cambridge (1996); Coase, R.: Essays on Economics and Economists. University of Chicago Press, Chicago (1994)
Coe, R.: Its the effect size, stupid: what effect size is and why it is important. In: Paper presented at the 2002 Annual Conference of the British Educational Research Association, University of Exeter, Exeter, Devon, England, September 12–14 (2002). http://www.leeds.ac.uk/educol/documents/00002182.htm
Cohen, J.: Statistical Power Analysis for the Behavioral Sciences. Routledge, New York (1977)
Deaton, A.: Instruments, randomization, and learning about development. J. Econ. Lit. 48, 424–455 (2010)
Fanelli, D.: How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLOS One 4 (2009). https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0005738
Fidler, F., Thomason, N., Cumming, G., Finch, S., Lee, J.: Editors can lead researchers to confidence intervals, but can’t make them think: statistical reform lessons from medicine. Psychol. Sci. 15, 119–126 (2004)
Friedman, M.: The methodology of positive economics. In: Friedman, M. (ed.) Essays in Positive Economics, pp. 3–43. University of Chicago Press, Chicago (1953)
Gallo, A.: A refresher on statistical significance. Harvard Business Review (2016). https://hbr.org/2016/02/a-refresher-on-statistical-significance#comment-section
Harford, T.: Big data: are we making a big mistake? Financial Times (2014). https://www.ft.com/content/21a6e7d8-b479-11e3-a09a-00144feabdc0
Keynes, J.M.: The General Theory of Employment, Interest and Money. Macmillan, London (1936)
Leibenstein, H.: Allocative efficiency vs. X-efficiency. Am. Econ. Rev. 56, 392–415 (1966)
Munafõ, M.R., Smith, G.D.: Replication is not enough. Nature: Int. J. Sci. 553, 400–401 (2018). https://www.nature.com/magazine-assets/d41586-018-01023-3/d41586-018-01023-3.pdf
McCloskey, D.: The loss function has been mislaid: the rhetoric of significance tests. Am. Econ. Rev. 75, 201–205 (1985)
McCloskey, D.: The insignificance of statistical significance. Sci. Am. 72, 32–33 (1995)
McCloskey, D.N., Ziliak, S.: The standard error of regressions. J. Econ. Lit. 34, 97–114 (1996)
McCrum-Gardner, E.: Sample size and power calculations made simple. Int. J. Ther. Rehabil. 17, 10–14 (2010)
Morrison, D.E., Henkel, R.E.: The Significance Test Controversy: A Reader. Aldine, Chicago (1970)
Qualtrics. Calculating sample size (2018). https://www.qualtrics.com/blog/calculating-sample-size/
Simon, H.A.: Behavioral Economics. In: Eatwell, J., Millgate, M., Newman, P. (eds.) The New Palgrave: A Dictionary of Economics. Macmillan, London (1987)
Taleb, N.N.: The Black Swan: The Impact of the Highly Improbable. Random House, New York (2007)
Thompson, B.: Why encouraging effect size reporting is not working: the etiology of researcher resistance to changing practices. J. Psychol. 133, 133–141 (1999)
Wasserstein, R.L., Lazar, N.A.: The ASA’s statement on p-values: context, process, and purpose. Am. Stat. 70, 129–133 (2016). https://doi.org/10.1080/00031305.2016.1154108
Ziliak, S., McCloskey, D.N.: Size matters: the standard error of regressions in the American economic. Rev. J. Socio-Econ. 33, 527–546
Ziliak, S., McCloskey, D.N.: The Cult of Statistical Significance: How the Standard Error Costs Jobs, Justice, and Lives. University of Michigan Press, Ann Arbor (2008)
Acknowledgements
Thanks to Hannah Altman and Louise Lamontagne for their helpful comments and suggestions.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Altman, M. (2020). A Holistic Approach to Empirical Analysis: The Insignificance of P, Hypothesis Testing and Statistical Significance*. In: Bailey, D., et al. From Analysis to Visualization. JBCC 2017. Springer Proceedings in Mathematics & Statistics, vol 313. Springer, Cham. https://doi.org/10.1007/978-3-030-36568-4_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-36568-4_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36567-7
Online ISBN: 978-3-030-36568-4
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)