Skip to main content
SpringerLink
Log in

Science’s Reproducibility and Replicability Crisis: International Business Is Not Immune

  • Chapter
  • First Online:
Research Methods in International Business

Part of the book series: JIBS Special Collections ((JIBSSC))

Abstract

International business is not immune to science’s reproducibility and replicability crisis. We argue that this crisis is not entirely surprising given the methodological practices that enhance systematic capitalization on chance. This occurs when researchers search for a maximally predictive statistical model based on a particular dataset and engage in several trial-and-error steps that are rarely disclosed in published articles. We describe systematic capitalization on chance, distinguish it from unsystematic capitalization on chance, address five common practices that capitalize on chance, and offer actionable strategies to minimize the capitalization on chance and improve the reproducibility and replicability of future IB research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    As noted by an anonymous reviewer, multilevel modeling is as susceptible to capitalization on chance as other methods, including OLS regression. Although the existence of a dependent data structure allows multilevel modeling to produce more accurate standard errors compared to OLS regression (Aguinis and Culpepper 2015), this is an improvement regarding unsystematic but not systematic capitalization on chance.

  2. 2.

    These are different ways to “manage outliers.” Winsorization involves transforming extreme values to a specified percentile of the data (e.g., a 90th percentile Winsorization would transform all the data below the 5th percentile to the 5th percentile, and all the data above the 95th percentile would be set at the 95th percentile). Studentised residuals are computed by dividing a residual by an estimate of its standard deviation, and Cook’s D measures the effect of deleting a given observation.

References

  • Aguinis, H., and S.A. Culpepper. 2015. An expanded decision making procedure for examining cross-level interaction effects with multilevel modeling. Organizational Research Methods 18 (2): 155–176.

    Article  Google Scholar 

  • Aguinis, H., and R.J. Vandenberg. 2014. An ounce of prevention is worth a pound of cure: Improving research quality before data collection. Annual Review of Organizational Psychology and Organizational Behavior 1 (1): 569–595.

    Article  Google Scholar 

  • Aguinis, H., C.A. Pierce, F.A. Bosco, and I.S. Muslin. 2009a. First decade of Organizational Research Methods: Trends in design, measurement, and data-analysis topics. Organizational Research Methods 12 (1): 69–112.

    Article  Google Scholar 

  • Aguinis, H., C.A. Pierce, and S.A. Culpepper. 2009b. Scale coarseness as a methodological artifact: Correcting correlation coefficients attenuated from using coarse scales. Organizational Research Methods 12 (4): 623–652.

    Article  Google Scholar 

  • Aguinis, H., S. Werner, J.L. Abbott, C. Angert, J.H. Park, and D. Kohlhausen. 2010. Customer-centric science: Reporting significant research results with rigor, relevance, and practical impact in mind. Organizational Research Methods 13 (3): 515–539.

    Article  Google Scholar 

  • Aguinis, H., R.K. Gottfredson, and H. Joo. 2013. Best-practice recommendations for defining, identifying, and handling outliers. Organizational Research Methods 16 (2): 270–301.

    Article  Google Scholar 

  • Aguinis, H., D.L. Shapiro, E. Antonacopoulou, and T.G. Cummings. 2014. Scholarly impact: A pluralist conceptualization. Academy of Management Learning & Education 13 (4): 623–639.

    Article  Google Scholar 

  • Andersson, U., A. Cuervo-Cazurra, and B.B. Nielsen. 2014. From the editors: Explaining interaction effects within and across levels of analysis. Journal of International Business Studies 45 (9): 1063–1071.

    Article  Google Scholar 

  • Bakker, M., and J.M. Wicherts. 2011. The (mis)reporting of statistical results in psychology journals. Behavior Research Methods 43 (3): 666–678.

    Article  Google Scholar 

  • Bakker, M., A. van Dijk, and J.M. Wicherts. 2012. The rules of the game called psychological science. Perspectives on Psychological Science 7 (6): 543–554.

    Article  Google Scholar 

  • Bamberger, P., and S. Ang. 2016. The quantitative discovery: What is it and how to get it published. Academy of Management Discoveries 2 (1): 1–6.

    Article  Google Scholar 

  • Banks, G.C., E.H. O’Boyle Jr., et al. 2016a. Questions about questionable research practices in the field of management: A guest commentary. Journal of Management 42 (1): 5–20.

    Article  Google Scholar 

  • Banks, G.C., S.G. Rogelberg, H.M. Woznyj, R.S. Landis, and D.E. Rupp. 2016b. Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business and Psychology 31 (3): 323–338.

    Article  Google Scholar 

  • Becker, T.E. 2005. Potential problems in the statistical control of variables in organizational research: A qualitative analysis with recommendations. Organizational Research Methods 8 (3): 274–289.

    Article  Google Scholar 

  • Bedeian, A.G., S.G. Taylor, and A.N. Miller. 2010. Management science on the credibility bubble: Cardinal sins and various misdemeanors. Academy of Management Learning & Education 9 (4): 715–725.

    Google Scholar 

  • Bentler, P.M. 2007. On tests and indices for evaluating structural models. Personality and Individual Differences 42 (5): 825–829.

    Article  Google Scholar 

  • Bergh, D.D., B. Sharp, and M. Li. 2017. Tests for identifying “red flags” in empirical findings: Demonstration and recommendations for authors, reviewers and editors. Academy of Management Learning & Education 16 (1): 110–124.

    Article  Google Scholar 

  • Bernerth, J., and H. Aguinis. 2016. A critical review and best-practice recommendations for control variable usage. Personnel Psychology 69 (1): 229–283.

    Article  Google Scholar 

  • Bettis, R.A., S. Ethiraj, A. Gambardella, C. Helfat, and W. Mitchell. 2016. Creating repeatable cumulative knowledge in strategic management. Strategic Management Journal 37 (2): 257–261.

    Article  Google Scholar 

  • Bobko, P. 2001. Correlation and regression. 2nd ed. Thousand Oaks: Sage.

    Book  Google Scholar 

  • Boellis, A., S. Mariotti, A. Minichilli, and L. Piscitello. 2016. Family involvement and firms’ establishment mode choice in foreign markets. Journal of International Business Studies 47 (8): 929–950.

    Article  Google Scholar 

  • Bosco, F.A., H. Aguinis, J.G. Field, C.A. Pierce, and D.R. Dalton. 2016. HARKing’s threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology 69 (3): 709–750.

    Article  Google Scholar 

  • Breaugh, J.A. 2008. Important considerations in using statistical procedures to control for nuisance variables in non-experimental studies. Human Resource Management Review 18 (4): 282–293.

    Article  Google Scholar 

  • Butler, N., H. Delaney, and S. Spoelstra. 2017. The grey zone: Questionable research practices in the business school. Academy of Management Learning & Education 16 (1): 94–109.

    Article  Google Scholar 

  • Carlson, K.D., and J. Wu. 2012. The illusion of statistical control: Control variable practice in management research. Organizational Research Methods 15 (3): 413–435.

    Article  Google Scholar 

  • Cascio, W.F. 2012. Methodological issues in international HR management research. International Journal of Human Resource Management 23 (12): 2532–2545.

    Article  Google Scholar 

  • Cascio, W.F., and H. Aguinis. 2005. Test development and use: New twists on old questions. Human Resource Management 44 (3): 219–235.

    Article  Google Scholar 

  • Cascio, W.F., and S. Zedeck. 1983. Open a new window in rational research planning: Adjust alpha to maximize statistical power. Personnel Psychology 36 (3): 517–526.

    Article  Google Scholar 

  • Chang, S.J., A. van Wittleloostuijn, and L. Eden. 2010. From the editors: Common method variance in international business research. Journal of International Business Studies 41 (2): 178–184.

    Article  Google Scholar 

  • Chen, E.E., and S.P. Wojcik. 2016. A practical guide to big data research in psychology. Psychological Methods 21 (4): 458–474.

    Article  Google Scholar 

  • Cortina, J.M. 2002. Big things have small beginnings: An assortment of “minor” methodological misunderstandings. Journal of Management 28 (3): 339–362.

    Article  Google Scholar 

  • Cortina, J.M., J.P. Green, K.R. Keeler, and R.J. Vandenberg. 2017a. Degrees of freedom in SEM: Are we testing the models that we claim to test? Organizatonal Research Methods20 (3): 350–378.

    Article  Google Scholar 

  • Cortina, J.M., H. Aguinis, and R.P. DeShon. 2017b. Twilight of dawn or of evening? A century of research methods in the Journal of Applied Psychology. Journal of Applied Psychology 102 (3): 274–290.

    Article  Google Scholar 

  • Credé, M., and P.D. Harms. 2015. 25 years of higher-order confirmatory factor analysis in the organizational sciences: A critical review and development of reporting recommendations. Journal of Organizational Behavior 36 (6): 845–872.

    Article  Google Scholar 

  • Cuervo-Cazurra, A., U. Andersson, M.Y. Brannen, B. Nielsen, and A.R. Reuber. 2016. From the editors: Can I trust your findings? Ruling out alternative explanations in international business research. Journal of International Business Studies 47 (8): 881–997.

    Article  Google Scholar 

  • Dalton, D.R., H. Aguinis, C.A. Dalton, F.A. Bosco, and C.A. Pierce. 2012. Revisiting the file drawer problem in metaanalysis: An empirical assessment of published and non-published correlation matrices. Personnel Psychology 65 (2): 221–249.

    Article  Google Scholar 

  • Davis, G.F. 2015. What is organizational research for? Administrative Science Quarterly 60 (2): 179–188.

    Article  Google Scholar 

  • Edwards, J.R., and J.W. Berry. 2010. The presence of something or the absence of nothing: Increasing theoretical precision in management research. Organizational Research Methods 13 (4): 668–689.

    Article  Google Scholar 

  • Eysenck, H.J. 1978. An exercise in mega-silliness. American Psychologist 33 (5): 517.

    Article  Google Scholar 

  • Fanelli, D. 2009. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS One 4: e5738. https://doi.org/10.1371/journal.pone.0005738.

    Article  Google Scholar 

  • Fischer, R., and A. Mansell. 2009. Commitment across cultures: A meta-analytical approach. Journal of International Business Studies 40 (8): 1339–1358.

    Article  Google Scholar 

  • Fisher, G., and H. Aguinis. 2017. Using theory elaboration to make theoretical advancements. Organizational Research Methods 20 (3): 438–464.

    Article  Google Scholar 

  • Fitzsimmons, S., Y. Liao, and D. Thomas. 2017. From crossing cultures to straddling them: An empirical examination of outcomes for multicultural employees. Journal of International Business Studies 48 (1): 63–89.

    Article  Google Scholar 

  • Freese, J. 2007. Replication standards for quantitative social science: Why not sociology. Sociological Methods & Research 36 (2): 153–172.

    Article  Google Scholar 

  • Friedman, D., and S. Sunder. 1994. Experimental methods: A primer for economists. New York: Cambridge University Press.

    Book  Google Scholar 

  • Fung, S.K., G. Zhou, and X.J. Zhu. 2016. Monitor objectivity with important clients: Evidence from auditor opinions around the world. Journal of International Business Studies 47 (3): 263–294.

    Article  Google Scholar 

  • Funk, C.A., J.D. Arthurs, L.J. Treviño, and J. Joireman. 2010. Consumer animosity in the global value chain: The effect of international production shifts on willingness to purchase hybrid products. Journal of International Business Studies 41 (4): 639–651.

    Article  Google Scholar 

  • George, G. 2014. Rethinking management scholarship. Academy of Management Journal 57 (1): 1–6.

    Article  Google Scholar 

  • Goldfarb, B., and A.A. King. 2016. Scientific apophenia in strategic management research: Significance tests & mistaken inference. Strategic Management Journal 37 (1): 167–176.

    Article  Google Scholar 

  • Harlow, L.L., and F.L. Oswald. 2016. Big data in psychology: Introduction to the special issue. Psychological Methods 21 (4): 447–457.

    Article  Google Scholar 

  • Hollenbeck, J.H., and P.M. Wright. 2016. Harking, sharking, and tharking: Making the case for post hoc analysis of scientific data. Journal of Management 43 (1): 5–18.

    Article  Google Scholar 

  • Hunter, J.E., and F.L. Schmidt. 2015. Methods of meta-analysis: Correcting error and bias in research findings. 3rd ed. Thousand Oaks: Sage.

    Google Scholar 

  • Hurley, A.E., et al. 1997. Exploratory and confirmatory factor analysis: Guidelines, issues, and alternatives. Journal of Organizational Behavior 18 (6): 667–683.

    Article  Google Scholar 

  • Ioannidis, J.P.A. 2005. Why most published research findings are false. PLoS Medicine 2 (8): e124. https://doi.org/10.1371/journal.pmed.0020124.

    Article  Google Scholar 

  • John, L.K., G. Loewenstein, and D. Prelec. 2012. Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science 23 (5): 524–532.

    Article  Google Scholar 

  • Kepes, S., and M.A. McDaniel. 2013. How trustworthy is the scientific literature in industrial and organizational psychology? Industrial and Organizational Psychology 6 (3): 252–268.

    Article  Google Scholar 

  • Kerr, N.L. 1998. HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review 2 (3): 196–217.

    Article  Google Scholar 

  • Lisak, A., M. Erez, Y. Sui, and C. Lee. 2016. The positive role of global leaders in enhancing multicultural team innovation. Journal of International Business Studies 47 (6): 655–673.

    Article  Google Scholar 

  • Macaskill, P., S. Walter, and L. Irwig. 2001. A comparison of methods to detect publication bias in meta-analysis. Statistics in Medicine 20 (4): 641–654.

    Article  Google Scholar 

  • MacCallum, R.C., M. Roznowski, and L.B. Necowitz. 1992. Model modification in covariance structure analysis: The problem of capitalization on chance. Psychological Bulletin 111 (3): 490–504.

    Article  Google Scholar 

  • Meyer, K.E., A. van Witteloostuijn, and S. Beugelsdijk. 2017. What’s in a p? Reassessing best practices for conducting and reporting hypothesis-testing research. Journal of International Business Studies 48 (5): 535–551.

    Article  Google Scholar 

  • Nosek, B.A., J.R. Spies, and M. Motyl. 2012. Scientific utopia II: Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science 7 (6): 615–631.

    Article  Google Scholar 

  • Nuijten, M.B., C.H. Hartgerink, M.A. Assen, S. Epskamp, and J.M. Wicherts. 2015. The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods 48 (4): 1–22.

    Google Scholar 

  • O’Boyle, E.H., G.C. Banks, and E. Gonzalez-Mule. 2017. The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management 43 (2): 376–399.

    Article  Google Scholar 

  • Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349 (6251): aac4716. https://doi.org/10.1126/science.aac4716.

    Article  Google Scholar 

  • Orr, J.M., P.R. Sackett, and C.L.Z. DuBois. 1991. Outlier detection and treatment in I/O psychology: A survey of researcher beliefs and an empirical illustration. Personnel Psychology 44 (3): 473–486.

    Article  Google Scholar 

  • Reeb, D., M. Sakakibara, and I.P. Mahmood. 2012. Endogeneity in international business research. Journal of International Business Studies 43 (3): 211–218.

    Article  Google Scholar 

  • Richard, P.J., T.M. Devinney, G.S. Yip, and G. Johnson. 2009. Measuring organizational performance: Towards methodological best practice. Journal of Management 35 (3): 718–804.

    Article  Google Scholar 

  • Rousseeuw, P.J., and A.M. Leroy. 2003. Robust regression and outlier detection. Hoboken: Wiley.

    Google Scholar 

  • Schwab, A., and W.H. Starbuck. 2017. A call for openness in research reporting: How to turn covert practices into helpful tools. Academy of Management Learning & Education 16 (1): 125–141.

    Article  Google Scholar 

  • Sijtsma, K. 2016. Playing with data – Or how to discourage questionable research practices and stimulate researchers to do things right. Psychometrika 81 (1): 1–15.

    Article  Google Scholar 

  • Simmons, J.P., L.D. Nelson, and U. Simonsohn. 2011. Falsepositive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22 (11): 1359–1366.

    Article  Google Scholar 

  • Spector, P.E., and M.T. Brannick. 2011. Methodological urban legends: The misuse of statistical control variables. Organizational Research Methods 14 (2): 287–305.

    Article  Google Scholar 

  • Stahl, G.K., M.L. Maznevski, A. Voigt, and K. Jonsen. 2010. Unraveling the effects of cultural diversity in teams: A meta-analysis of research on multicultural work groups. Journal of International Business Studies 41 (4): 690–709.

    Article  Google Scholar 

  • Starbuck, W.H. 2016. 60th anniversary essay: How journals could improve research practices in social science. Administrative Science Quarterly 61 (2): 165–183.

    Article  Google Scholar 

  • van Essen, M., P.P. Heugens, J. Otten, and J. van Oosterhout. 2012. An institution-based view of executive compensation: A multilevel meta-analytic test. Journal of International Business Studies 43 (4): 396–423.

    Article  Google Scholar 

  • Verbeke, A., M.Y. Von Glinow, and Y. Luo. 2017. Becoming a great reviewer: Four actionable guidelines. Journal of International Business Studies 48 (1): 1–9.

    Article  Google Scholar 

  • Waldman, I.D., and S.O. Lilienfeld. 2016. Thinking about data, research methods, and statistical analyses: Commentary on Sijtsma’s (2014) “playing with data”. Psychometrika 81 (1): 16–26.

    Article  Google Scholar 

  • Wright, P.M. 2016. Ensuring research integrity: An editor’s perspective. Journal of Management 42 (5): 1037–1043.

    Article  Google Scholar 

Download references

Acknowledgements

We thank Alain Verbeke and two Journal of International Business Studies anonymous reviewers for their highly constructive feedback that allowed us to improve our manuscript.

Author information

Authors and Affiliations

  1. George Washington University, Washington, DC, USA

    Herman Aguinis

  2. University of Colorado Denver, Denver, CO, USA

    Wayne F. Cascio

  3. Purdue University Northwest, Hammond, IN, USA

    Ravi S. Ramani

Authors
  1. Herman Aguinis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wayne F. Cascio
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ravi S. Ramani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Herman Aguinis .

Editor information

Editors and Affiliations

  1. Department of Management, Texas A&M University, College Station, TX, USA

    Lorraine Eden

  2. The University of Sydney Business School, The University of Sydney, Sydney, NSW, Australia

    Bo Bernhard Nielsen

  3. Copenhagen Business School, Frederiksberg, Denmark

    Bo Bernhard Nielsen

  4. University of Calgary, Calgary, AB, Canada

    Alain Verbeke

  5. University of Reading, Reading, UK

    Alain Verbeke

  6. Vrije Universiteit Brussel, Brussels, Belgium

    Alain Verbeke

Rights and permissions

Reprints and permissions

Copyright information

© 2020 The Author(s)

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Aguinis, H., Cascio, W.F., Ramani, R.S. (2020). Science’s Reproducibility and Replicability Crisis: International Business Is Not Immune. In: Eden, L., Nielsen, B.B., Verbeke, A. (eds) Research Methods in International Business. JIBS Special Collections. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-030-22113-3_2

Download citation

Publish with us

Policies and ethics

Search

Navigation