Skip to main content
SpringerLink
Log in

Establishing Content-Related Validity Evidence for Assessments in Counseling: Application of a Sequential Mixed-Method Approach

  • ORIGINAL ARTICLE
  • Published:
International Journal for the Advancement of Counselling Aims and scope Submit manuscript

Abstract

Establishing content-related validity evidence is considered essential to draw conclusions about a scale’s quality. The purpose of this conceptual article is to provide best practice recommendations in the content validation procedure of scale development in the field of counseling, incorporating both qualitative and quantitative content review phases as a sequential mixed-method approach. In line with this goal, a description of the four basic sequential stages of the content validation procedure is provided, including a brief overview of content validity and Lawshe’s (1975) technique as a quantitative approach to content validity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

    Google Scholar 

  • Ayre, C., & Scally, J. A. (2014). Critical values for Lawshe’s content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47, 79–86. https://doi.org/10.1177/0748175613513808.

    Article  Google Scholar 

  • Butt, Z. (2016). In pursuit of empirically supported assessment for use in medical settings. Clinical Psychology Science and Practice, 23, 382–402. https://doi.org/10.1111/cpsp.12176.

    Article  Google Scholar 

  • Carretero-Dios, H., & Perez, C. (2007). Standards for the development and review of instrumental studies: Considerations about test selection in psychological research. International Journal of Clinical and Health Psychology, 7, 863–882.

    Google Scholar 

  • Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7(3), 309–319.

    Article  Google Scholar 

  • Cohen, R. J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.

    Article  Google Scholar 

  • Davis, L. (1992). Instrument review. Getting the most from a panel of experts. Applied Nursing Research, 5(4), 194–197.

    Article  Google Scholar 

  • Delgado-Rico, E., Carretero-Dios, H., & Ruch, W. (2012). Content validity evidences in test development: An applied perspective. International Journal of Clinical and Health Psychology, 12(3), 449–460.

    Google Scholar 

  • DeVellis, R. F. (2017). Scale development: Theory and applications. Los Angeles: Sage.

  • Doris, M. R., Berg-Weger, M., Tebb, S. S., Lee, E. S., & Rauch, S. (2003). Objectifying content validity: Conducting a content validity study in social work research. Social Work Research, 27(2), 94–104.

    Article  Google Scholar 

  • Drost, E. A. (2011). Validity and reliability in social science research. Education Research and Perspectives, 38, 105–123.

    Google Scholar 

  • Flood, E. (2014). Mixed methods for establishing content validity: A value added approach. Asia Pacific Biotech News, 18(9), 58–62.

    Article  Google Scholar 

  • Gable, R. K., & Wolf, J. W. (1993). Instrument development in the affective domain: Measuring attitudes and values in corporate and school settings. Boston: Kluwer Academic.

    Book  Google Scholar 

  • Gajewski, B. J., Price, L. R., Coffland, V., Boyle, D. K., & Bott, M. J. (2013). Integrated analysis of content and construct validity of psychometric instruments. Quality & Quantity, 47, 57–78. https://doi.org/10.1007/s11135-011-9503-4.

    Article  Google Scholar 

  • Greeley, A. T., Johnson, E., Seem, S., Braver, M., Dias, L., Evans, K., Kincade, E., & Pricken, P. (1989). Research self-efficacy scale. Unpublished scale presented at the conference of the Association for Women in Psychology, Bethesda, MD.

  • Gilbert, G. E., & Prion, S. (2016). Making sense of methods and measurement: Lawshe’s content validity index. Clinical Stimulation in Nursing, 12, 530–531. https://doi.org/10.1016/j.ecns.2016.08.002.

    Article  Google Scholar 

  • Haynes, S. N., Richard, D. C. S., & Kubany, E. S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7(3), 238–247.

    Article  Google Scholar 

  • Heppner, P. P., Wampold, B. E., Owen, J., Wang, K. T., & Thompson, M. N. (2016). Research design in counseling (4th ed.). Boston: Cengage Learning.

    Google Scholar 

  • Hood, S. B. (2009). Validity in psychological testing and scientific realism. Theory and Psychology, 19(4), 451–473. https://doi.org/10.1177/0959354309336320.

    Article  Google Scholar 

  • Hoyt, W. T., Warbasse, R. E., & Chu, E. (2006). Construct validation in counseling psychology research. The Counseling Psychologist, 34(6), 769–805. https://doi.org/10.1177/0011000006287389.

    Article  Google Scholar 

  • James, L. R., Demaree, R. G., & Wolf, G. (1993). Rwg: An assessment of within-group interrater agreement. Journal of Applied Psychology, 78, 306–309.

    Article  Google Scholar 

  • Kline, T. J. B. (2005). Psychological testing: A practical approach to design and evaluation. Thousand Oaks: Sage. https://doi.org/10.4135/9781483385693.

    Book  Google Scholar 

  • Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 563–575.

    Article  Google Scholar 

  • Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23, 127–135.

    Article  Google Scholar 

  • Lynn, M. (1986). Determination and quantification of content validity. Nursing Research, 35, 382-385.

    Article  Google Scholar 

  • McKenzie, J. F., Wood, M. L., Kotecki, J. E., Clark, J. K., & Brey, R. A. (1999). Establishing content validity: Using qualitative and quantitative steps. American Journal of Health Behavior, 23, 311–318.

    Article  Google Scholar 

  • Mullen, P. R., Lambie, G. W., & Conley, A. H. (2013). Development of the ethical and legal issues in counseling self-efficacy scale. Measurement and Evaluation in Counseling and Development, 47(1), 62–78. https://doi.org/10.1177/0748175613513807.

    Article  Google Scholar 

  • Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Research in Nursing & Health, 29, 489–497. https://doi.org/10.1002/nur.10147.

    Article  Google Scholar 

  • Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30, 459–467.

    Article  Google Scholar 

  • Smith, B. W., Dalen, J., Wiggins, K., Tooley, E., Christopher, P., & Bernard, J. (2008). The brief resilience scale: Assessing the ability to bounce back. International Journal of Behavioral Medicine, 15(3):194–200.  

    Article  Google Scholar 

  • Streiner, D. L., Norman, G. R., & Cairney, J. (2015). Health measurement scales: A practical guide to their development and use (5th ed.). Oxford: Oxford University Press.

  • Sullivan, G. M. (2011). A primer on the validity of assessment instruments. Journal of Graduate Medical Education, 3(2), 119–120. https://doi.org/10.4300/JGME-D-11-00075.1.

    Article  PubMed  PubMed Central  Google Scholar 

  • Swank, J. M., & Lambie, G. W. (2016). Development of the research competencies scale. Measurement and Evaluation in Counseling and Development, 49(2), 91–108. https://doi.org/10.1177/0748176615625749.

    Article  Google Scholar 

  • Tilden, V., Nelson, C., & May, B. (1990). Use of qualitative methods to enhance content validity. Nursing Research, 39(3), 172-175.

    Article  Google Scholar 

  • Tinsley, H. E. A., & Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology, 22, 358–376.

    Article  Google Scholar 

  • Veneziano, L., & Hooper, J. (1997). A method for quantifying content validity of health-related questionnaires. American Journal of Health Behavior, 21(1), 67–70.

    Google Scholar 

  • Wallace, L. S., Blake, G. H., Parham, J. S., & Baldridge, R. E. (2003). Development and content validation of family practice residency recruitment questionnaires. Family Medicine, 35(7), 495–499.

    Google Scholar 

  • Weems, G. H., & Onwuegbuzie, A. J. (2001). The impact of midpoint responses and reverse coding on survey data. Measurement and Evaluation in Counseling and Development, 34(3), 166–176.

    Google Scholar 

  • Wilson, F. R., Pan, W., & Schumsky, D. A. (2012). Recalculation of the critical values for Lawshe’s content validity ratio. Measurement and Evaluation in Counseling and Development, 45, 197–210. https://doi.org/10.1177/0748175612440286.

    Article  Google Scholar 

  • Worthington, R. L., & Whittaker, T. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806–838. https://doi.org/10.1177/0011000006288127.

    Article  Google Scholar 

  • Wynd, C. A., Schmidt, B., & Schaefer, M. A. (2003). Two quantitative approaches for estimating content validity. Western Journal of Nursing Research, 25, 508–518.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Counseling and Educational Psychology, Texas A&M University- Corpus Christi, 6300 Ocean Drive, Corpus Christi, TX, 78412-5834, USA

    Hulya Ermis-Demirtas

Authors
  1. Hulya Ermis-Demirtas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hulya Ermis-Demirtas.

Ethics declarations

Conflict of Interest

The authors declared that they have no conflict of interest.

Appendices

Appendix 1

Table 1 Qualitative review criteria of instrument components dispatched to the experts
Full size table

Appendix 2

Table 2 Quantitative review example provided to experts
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ermis-Demirtas, H. Establishing Content-Related Validity Evidence for Assessments in Counseling: Application of a Sequential Mixed-Method Approach. Int J Adv Counselling 40, 387–397 (2018). https://doi.org/10.1007/s10447-018-9332-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10447-018-9332-4

Keywords

Search

Navigation