Occupational Health Science

, Volume 3, Issue 2, pp 125–143 | Cite as

How Often Do I Agree: an Experimental Test of Item Format Method Variance in Stress Measures

  • Paul E. SpectorEmail author
  • Ashley E. Nixon


Three studies experimentally investigated response formats of agreement (A) and frequency (F) to determine their effect on correlations between often used measures of job stressors and strains. Study 1 used a within-subject design where respondents were given two sets of the same 6 stressor measures that varied in A versus F formats, as well as measures of 6 strains. Study 2 replicated Study 1 using a between-subject design where each respondent was randomly assigned to either the A or F stressor scale format. Study 3 utilized a 2 (A vs. F for stressors) by 2 (A vs. F for strains) design to test the assumption that matched response formats would result in higher correlations than unmatched due to common method variance. The first two studies provide little evidence for a systematic difference between A and F formats in predicting strains. Study 3 found that matching formats more often resulted in lower, not higher, correlations between stressor and strain pairs. Taken together, these experimental studies suggest that response format has little effect on correlations, means, standard deviations, and internal consistencies, and is not a method variance concern, at least with these commonly used measures of job stressors and strains.


Research method Method variance Measurement Construct validity Response style Response bias Acquiescence 


Compliance with Ethical Standards

Conflict of Interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.


  1. Amstad, F. T., Meier, L. L., Fasel, U., Elfering, A., & Semmer, N. K. (2011). A meta-analysis of work–family conflict and various outcomes with a special emphasis on cross-domain versus matching-domain relations. Journal of Occupational Health Psychology, 16(2), 151–169. Scholar
  2. Baumgartner, H., & Steenkamp, J.-B. E. (2001). Response styles in marketing research: a cross-national investigation. Journal of Marketing Research, 38(2), 143–156. Scholar
  3. Blau, G., & Andersson, L. (2005). Testing a measure of instigated workplace incivility. Journal of Occupational and Organizational Psychology, 78(4), 595–614. Scholar
  4. Bowling, N. A., Alarcon, G. M., Bragg, C. B., & Hartman, M. J. (2015). A meta-analytic examination of the potential correlates and consequences of workload. Work and Stress, 29(2), 95–113. Scholar
  5. Cammann, C., Fichman, M., Jenkins, D., & Klesh, J. (1979). The Michigan organizational assessment questionnaire. Unpublished manuscript, University of Michigan, Ann Arbor. University of Michigan, Ann Arbor.Google Scholar
  6. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. Scholar
  7. Caplan, R. D., Cobb, S., French, J. R. P., Van Harrison, R., & Penneau, S. R. (1980). Job demands and worker health. An Arbor: University of Michigan, Institute for Social Research.Google Scholar
  8. Dalal, R. S. (2005). A meta-analysis of the relationship between organizational citizenship behavior and counterproductive work behavior. Journal of Applied Psychology, 90(6), 1241–1255. Scholar
  9. De Beuckelaer, A., Weijters, B., & Rutten, A. (2010). Using ad hoc measures for response styles: a cautionary note. Quality & Quantity: International Journal of Methodology, 44(4), 761–775. Scholar
  10. Fiske, D. W. (1987). On understanding our methods and their effects. Diagnostica, 33(3), 188–194.Google Scholar
  11. Guidroz, A. M., Burnfield-Geimer, J. L., Clark, O., Schwetschenau, H. M., & Jex, S. M. (2010). The nursing incivility scale: development and validation of an occupation-specific measure. Journal of Nursing Measurement, 18(3), 176–200.CrossRefPubMedGoogle Scholar
  12. House, R. J., Schuler, R. S., & Levanoni, E. (1983). Role conflict and ambiguity scales: reality or artifacts? Journal of Applied Psychology, 68(2), 334–337. Scholar
  13. Judge, T. A., Locke, E. A., & Durham, C. C. (1997). The dispositional causes of job satisfaction: a core evaluations approach. Research in Organizational Behavior, 19, 151–188.Google Scholar
  14. Malhotra, N. K., Schaller, T. K., & Patil, A. (2017). Common method variance in advertising research: when to be concerned and how to control for it. Journal of Advertising, 46(1), 193–212. Scholar
  15. Moorman, R. H., & Podsakoff, P. M. (1992). A meta-analytic review and empirical test of the potential confounding effects of social desirability response sets in organizational behaviour research. Journal of Occupational and Organizational Psychology, 65(2), 131–149.CrossRefGoogle Scholar
  16. Murphy, K. R., Jako, R. A., & Anhalt, R. L. (1993). Nature and consequences of halo error: a critical analysis. Journal of Applied Psychology, 78(2), 218–225. Scholar
  17. Netemeyer, R. G., Boles, J. S., & McMurrian, R. (1996). Development and validation of work–family conflict and family–work conflict scales. Journal of Applied Psychology, 81(4), 400–410. Scholar
  18. Nixon, A. E., Mazzola, J. J., Bauer, J., Krueger, J. R., & Spector, P. E. (2011). Can work make you sick? A meta-analysis of the relationships between job stressors and physical symptoms. Work and Stress, 25(1), 1–22. Scholar
  19. Pace, V. L. (2010). Method variance from the perspectives of reviewers: poorly understood problem or overemphasized complaint? Organizational Research Methods, 13(3), 421–434. Scholar
  20. Pindek, S., & Spector, P. E. (2016). Organizational constraints: a meta-analysis of a major stressor. Work and Stress, 30(1), 7–25. Scholar
  21. Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903.CrossRefPubMedGoogle Scholar
  22. Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539–569. Scholar
  23. Rorer, L. G. (1965). The great response-style myth. Psychological Bulletin, 63(3), 129–156. Scholar
  24. Spector, P. E., & Jex, S. M. (1998). Development of four self-report measures of job stressors and strain: interpersonal conflict at work scale, organizational constraints scale, quantitative workload inventory, and physical symptoms inventory. Journal of Occupational Health Psychology, 3(4), 356–367. Scholar
  25. Spector, P. E., Dwyer, D. J., & Jex, S. M. (1988). Relation of job stressors to affective, health, and performance outcomes: a comparison of multiple data sources. Journal of Applied Psychology, 73(1), 11–19. Scholar
  26. Spector, P. E., Bauer, J. A., & Fox, S. (2010). Measurement artifacts in the assessment of counterproductive work behavior and organizational citizenship behavior: do we know what we think we know? Journal of Applied Psychology, 95(4), 781–790. Scholar
  27. Spector, P. E., Rosen, C. C., Richardson, H. A., Williams, L. J., & Johnson, R. E. (2019). A new perspective on method variance: a measure-centric approach. Journal of Management, 45(3), 855–880. Scholar
  28. Weijters, B., Cabooter, E., & Schillewaert, N. (2010a). The effect of rating scale format on response styles: the number of response categories and response category labels. International Journal of Research in Marketing, 27(3), 236–247. Scholar
  29. Weijters, B., Geuens, M., & Schillewaert, N. (2010b). The individual consistency of acquiescence and extreme response style in self-report questionnaires. Applied Psychological Measurement, 34(2), 105–121. Scholar
  30. Weijters, B., Geuens, M., & Schillewaert, N. (2010c). The stability of individual response styles. Psychological Methods, 15(1), 96–110. Scholar
  31. Weitz, J. (1952). A neglected concept in the study of job satisfaction. Personnel Psychology, 5, 201–205. Scholar
  32. Williams, E. J. (1959). The comparison of regression variables. Journal of the Royal Statistical Society (Series B), 21, 396–399.Google Scholar
  33. Williams, L. J., Gavin, M. B., & Williams, M. L. (1996). Measurement and nonmeasurement processes with negative affectivity and employee attitudes. Journal of Applied Psychology, 81(1), 88–101. Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Psychology, PCD4118University of South FloridaTampaUSA
  2. 2.Atkinson Graduate School of ManagementWillamette UniversitySalemUSA

Personalised recommendations