Journal of Business and Psychology

, Volume 26, Issue 1, pp 105–121 | Cite as

Meta-analytic Decisions and Reliability: A Serendipitous Case of Three Independent Telecommuting Meta-analyses

  • Levi R. G. Nieminen
  • Jessica M. Nicklin
  • Tara K. McClure
  • Madhura Chakrabarti



Despite the potential for researcher decisions to negatively impact the reliability of meta-analysis, very few methodological studies have examined this possibility. The present study compared three independent and concurrent telecommuting meta-analyses in order to determine how researcher decisions affected the process and findings of these studies.


A case study methodology was used, in which three recent telecommuting meta-analyses were re-examined and compared using the process model developed by Wanous et al. (J Appl Psychol 74:259–264, 1989).


Results demonstrated important ways in which researcher decisions converged and diverged at stages of the meta-analytic process. The influence of researcher divergence on meta-analytic findings was neither evident in all cases, nor straightforward. Most notably, the overall effects of telecommuting across a range of employee outcomes were generally consistent across the meta-analyses, despite substantial differences in meta-analytic samples.


Results suggest that the effect of researcher decisions on meta-analytic findings may be largely indirect, such as when early decisions guide the specific moderation tests that can be undertaken at later stages. However, directly comparable “main effect” findings appeared to be more robust to divergence in researcher decisions. These results provide tentative positive evidence regarding the reliability of meta-analytic methods and suggest targeted areas for future methodological studies.


This study presents unique insight into a methodological issue that has not received adequate research attention, yet has potential implications for the reliability and validity of meta-analysis as a method.


Meta-analysis Methodological Replication Reliability Validity Telecommuting 



We would like to thank Boris Baltes and Christopher Berry for their constructive comments on a previous draft of this manuscript.


  1. Aguinis, H., Dalton, D. A., Bosco, F. A., Pierce, C. A., & Dalton, C. M (2009, August). Meta-analytic choices and judgment calls: Implications for theory and scholarly impact. Paper presented at the meeting of the Academy of Management, Chicago, IL.Google Scholar
  2. Allen, M., & Preiss, R. (1993). Replication and meta-analysis: A necessary connection. Journal of Social Behavior and Personality, 8, 9–20.Google Scholar
  3. Barrick, M. R., & Mount, M. K. (1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1–26.CrossRefGoogle Scholar
  4. Barrick, M. R., Mount, M. K., & Judge, T. A. (2001). Personality and performance at the beginning of the new millennium: What do we know and where do we go next? International Journal of Selection and Assessment, 9, 9–30.CrossRefGoogle Scholar
  5. Beal, D. J., Corey, D. M., & Dunlap, W. P. (2002). On the bias of Huffcutt and Arthur’s (1995) procedure for identifying outliers in the meta-analysis of correlations. Journal of Applied Psychology, 87, 583–589.CrossRefPubMedGoogle Scholar
  6. Beaman, A. L. (1991). An empirical comparison of meta-analytic and traditional reviews. Personality and Social Psychology Bulletin. Special Issue: Meta-Analysis in Personality and Social Psychology, 17, 252–257.Google Scholar
  7. Bobko, P., & Roth, P. L. (2008). Psychometric accuracy and (the continuing need for) quality thinking in meta-analysis. Organizational Research Methods, 11, 114–126.CrossRefGoogle Scholar
  8. Briggs, D. C. (2005). Meta-analysis: A case study. Evaluation Review, 29, 87–127.CrossRefPubMedGoogle Scholar
  9. Bullock, R. J., & Svyantek, D. J. (1985). Analyzing meta-analysis: Potential problems, an unsuccessful replication, and evaluation criteria. Journal of Applied Psychology, 70, 108–115.CrossRefGoogle Scholar
  10. Burke, M. J., & Landis, R. S. (2003). Methodological and conceptual challenges in conducting and interpreting meta-analyses. In K. R. Murphy (Ed.), Validity generalization: A critical review (pp. 287–310). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  11. Campion, M. A. (1993). Article review checklist: A criterion checklist for reviewing research articles in applied psychology. Personnel Psychology, 46, 705–718.CrossRefGoogle Scholar
  12. Cortina, J. M. (2002). Big things have small beginnings: An assortment of “minor” methodological misunderstandings. Journal of Management, 28, 339–362.Google Scholar
  13. Cortina, J. M. (2003). Apples and oranges (and pears, oh my!): The search for moderators in meta-analysis. Organizational Research Methods, 6, 415–439.CrossRefGoogle Scholar
  14. Cortina, J. M., & Dunlap, W. P. (1997). On the logic and purpose of significance testing. Journal of Applied Psychology, 2, 161–172.Google Scholar
  15. Cree, L. H. (1999). Work/family balance of telecommuters. Dissertation Abstracts International Section B: The Sciences and Engineering, 59(11-B), 6100.Google Scholar
  16. Dieckmann, N. F., Malle, B. F., & Bodner, T. E. (2009). An empirical assessment of meta-analytic practice. Review of General Psychology, 13, 101–115.CrossRefGoogle Scholar
  17. Eden, D. (2002). Replication, meta-analysis, scientific progress, and AMJ’s publication policy. Academy of Management Journal, 45, 841–846.Google Scholar
  18. Egger, M., & Smith, G. D. (1998). Meta-analysis bias in location and selection of studies. British Medical Journal, 316, 61–66.PubMedGoogle Scholar
  19. Field, A. P. (2001). Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed- and random-effects methods. Psychological Methods, 6, 161–180.CrossRefPubMedGoogle Scholar
  20. Gajendran, R. S., & Harrison, D. A. (2007). The good, the bad, and the unknown about telecommuting: Meta-analysis of psychological mediators and individual consequences. Journal of Applied Psychology, 92, 1524–1541.CrossRefPubMedGoogle Scholar
  21. Geyskens, I., Krishnan, R., Steenkamp, J. B. E. M., & Cunha, P. V. (2009). A review and evaluation of meta-analysis practices in management research. Journal of Management, 35, 393–419.CrossRefGoogle Scholar
  22. Greenhouse, J. B., & Iyengar, S. (1994). Sensitivity analysis and diagnostics. In H. M. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 503–520). New York: Russell Sage Foundation.Google Scholar
  23. Greenhalgh, T., & Peacock, R. (2005). Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. British Medical Journal, 331, 1064–1065.CrossRefPubMedGoogle Scholar
  24. Hackett, R. D., & Guion, R. M. (1985). A re-evaluation of the absenteeism-job satisfaction relationship. Organizational Behavior and Human Decision Processes, 35, 340–381.CrossRefPubMedGoogle Scholar
  25. Hackman, J. R., & Oldham, G. R. (1975). Development of the job diagnostic survey. Journal of Applied Psychology, 60, 159–170.CrossRefGoogle Scholar
  26. Hale, J., & Dillard, J. (1991). The uses of meta-analysis: Making knowledge claims and setting research agendas. Communication Monographs, 58, 463–471.CrossRefGoogle Scholar
  27. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.Google Scholar
  28. Hunter, J. E., & Schmidt, F. L. (2000). Fixed effects vs. random effects meta-analysis models: Implications for cumulative research knowledge. International Journal of Selection and Assessment, 8, 275–292.CrossRefGoogle Scholar
  29. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.Google Scholar
  30. Kisamore, J., & Brannick, M. (2008). An illustration of the consequences of meta-analysis model choice. Organizational Research Methods, 11, 35–53.CrossRefGoogle Scholar
  31. Konstantopoulos, S., & Hedges, L. V. (2009). Analyzing effect sizes: Fixed-effects models. In H. M. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 279–293). New York: Russell Sage Foundation.Google Scholar
  32. Kromrey, J. D., & Rendina-Gobioff, G. (2006). On knowing what we do not know: An empirical comparison of methods to detect publication bias in meta-analysis. Educational and Psychological Measurement, 66, 357–373.CrossRefGoogle Scholar
  33. Miner, J. B., & Raju, N. S. (2004). Risk propensity differences between managers and entrepreneurs and between low- and high-growth entrepreneurs: A reply in a more conservative vein. Journal of Applied Psychology, 89, 3–13.CrossRefPubMedGoogle Scholar
  34. Nicklin, J. M., Mayfield, C. O., Caputo, P. M., Arboleda, M. A., Cosentino, R. E., Lee, M., et al. (2009). Does telecommuting increase organizational attitudes and outcomes: A meta-analysis. Pravara Management Review, 8, 2–16.Google Scholar
  35. Nieminen, L. R. G., Chakrabarti, M., McClure, T. K., & Baltes, B. B. (2008). A meta-analysis of the effects of telecommuting on employee outcomes. Paper presented at the 23rd Annual Conference of the Society for Industrial and Organizational Psychology, San Francisco, CA.Google Scholar
  36. Ones, D. S., Mount, M. K., Barrick, M. R., & Hunter, J. E. (1994). Personality and job performance: A critique of the Tett, Jackson, and Rothstein (1991) meta-analysis. Personnel Psychology, 47, 147–156.CrossRefGoogle Scholar
  37. Overton, R. C. (1998). A comparison of fixed-effects and mixed (random-effects) models for meta-analysis tests of moderator variable effects. Psychological Methods, 3, 354–379.CrossRefGoogle Scholar
  38. Oyer, E. J. (1997). Validity and impact of meta-analyses in early intervention research. Dissertation Abstracts International Section A: Humanities and Social Sciences. 57(7-A), 2859.Google Scholar
  39. Raghuram, S., & Weisenfeld, B. (2004). Work-nonwork conflict and job stress among virtual workers. Human Resource Management, 43, 259–277.CrossRefGoogle Scholar
  40. Rothstein, H. R., & McDaniel, M. A. (1989). Guidelines for conducting and reporting meta-analyses. Psychological Reports, 65, 759–770.Google Scholar
  41. Rothstein, H., Sutton, A. J., & Bornstein, M. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment, and adjustments. Chichester, UK: Wiley.Google Scholar
  42. Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55, 68–78.CrossRefPubMedGoogle Scholar
  43. Schmidt, F. L., & Hunter, J. E. (1999). Comparison of three meta-analysis methods revisited: An analysis of Johnson, Mullen, and Salas (1995). Journal of Applied Psychology, 84, 144–148.CrossRefGoogle Scholar
  44. Schulze, R. (2004). Meta-analysis: A comparison of approaches. Cambridge: Hogrefe & Huber.Google Scholar
  45. Schulze, R. (2007). Current methods for meta-analysis: Approaches, issues, and developments. Journal of Psychology. Special Issue: The State and the Art of Meta-Analysis, 215(2), 90–103.Google Scholar
  46. Scott, K. D., & Taylor, D. S. (1985). An examination of conflicting findings on the relationship between job satisfaction and absenteeism: A meta-analysis. Academy of Management Journal, 28, 599–612.CrossRefGoogle Scholar
  47. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton-Mifflin.Google Scholar
  48. Staples, D. S. (2001). A study of remote workers and their differences from non-remote workers. Journal of End User Computing, 13, 3–14.Google Scholar
  49. Stewart, W. H., Jr., & Roth, P. L. (2001). Risk propensity differences between entrepreneurs and managers: A meta-analytic review. Journal of Applied Psychology, 86, 145–153.CrossRefPubMedGoogle Scholar
  50. Stewart, W. H., Jr., & Roth, P. L. (2004). Data quality affects meta-analytic conclusions: A response to Miner and Raju (2004) concerning entrepreneurial risk propensity. Journal of Applied Psychology, 89, 14–21.CrossRefPubMedGoogle Scholar
  51. Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job performance: A meta-analytic review. Personnel Psychology, 44, 703–742.CrossRefGoogle Scholar
  52. Wanous, J. P., Sullivan, S. E., & Malinak, J. (1989). The role of judgment calls in meta-analysis. Journal of Applied Psychology, 74, 259–264.CrossRefGoogle Scholar
  53. Wells, K., & Littell, J. H. (2009). Study quality assessment in systematic reviews of research on intervention effects. Research on Social Work Practice, 19, 52–62.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Levi R. G. Nieminen
    • 1
  • Jessica M. Nicklin
    • 2
  • Tara K. McClure
    • 1
  • Madhura Chakrabarti
    • 1
  1. 1.Department of PsychologyWayne State UniversityDetroitUSA
  2. 2.Department of PsychologyUniversity of HartfordWest HartfordUSA

Personalised recommendations