Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Expanding the evidence of content validity for the Teacher Behavior Checklist using the IRAP

TBC’s content validity using the IRAP

Abstract

The Teacher Behavior Checklist (TBC) consists of 28 items mainly used to measure college teachers’ performance or to investigate the primary qualities of good teachers. Psychometric studies using the TBC have presented evidence of its validity and reliability in different cultures. However, TBC content validity is exclusively based on self-reported measurements, which presents limitations that are widely discussed in the literature. Our study sought to investigate an alternative measurement for TBC content validity using a latency-based task, namely the Implicit Relational Assessment Procedure (IRAP). The main objective was to assess the strength of the relationship between the concept of “Good Teacher” and “Bad Teacher” and six positive and negative features derived from TBC items using an IRAP preparation. The second objective was to investigate the correlation between IRAP and TBC scores. The participants were 64 undergraduate students (M = 21; F = 43), aged 16 to 36. The IRAP trials included six target stimuli selected from the TBC that could be either “Good Teacher” or “Bad Teacher,” and six labels that could be either positive or negative features. The IRAP revealed that the “Good Teacher” is “Positive,” not “Negative.” On the other hand, the “Bad Teacher” is “Negative,” but participants could not deny that this teacher has “Positive” features. Only one statistically significant correlation was found between the Good Teacher-Negative IRAP trial type and the TBC score. Future studies should expand the use of implicit measurements in psychometric studies using the TBC.

This is a preview of subscription content, log in to check access.

Fig. 1.
Fig. 2.

Notes

  1. 1.

    The effective teacher is one who favors learning without turning educational stimuli into an aversive value for the student. Moreover, what is taught must increase the chances of the learner attenuating or solving problem situations (for her/himself and also for the society of which she/he is a part) outside the classroom. Although we theoretically defend the use of the term “effective teacher” (see Henklain, 2017), studies on the TBC have employed expressions such as “Excellent Teacher,” “Master Teacher,” and sometimes “Good Teacher,” which is easier for participants to understand. The expression “Good Teacher,” for instance, is the smallest and easiest one. For this reason, hereafter we will use the term “Good Teacher” as the construct being evaluated by the TBC.

References

  1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

  2. Barnes-Holmes, D. (2012). IRAP (Versão 2012) [Software]. Maynooth: Maynooth University, Ireland.

  3. Barnes-Holmes, D., Barnes-Holmes, Y., Power, P., Hayden, E., Milne, R., & Stewart, I. (2006). Do you really know what you believe? Developing the Implicit Relational Assessment Procedure (IRAP) as a direct measure of implicit beliefs. Irish Psychologist, 32(7), 169–177 Retrieved May 15, 2018, from http://bit.ly/2Gx9e5r.

  4. Barnes-Holmes, D., Hayden, E., Barnes-Holmes, Y., & Stewart, I. (2008). The Implicit Relational Assessment Procedure (IRAP) as a response- time and event-related-potentials methodology for testing natural verbal relations: A preliminary study. The Psychological Record, 58, 497–516. https://doi.org/10.1007/BF03395634.

  5. Buskist, W., & Keeley, J. W. (2018). Searching for universal principles of excellence in college and university teaching. New Directions for Teaching and Learning, 156, 95–105. https://doi.org/10.1002/tl.20321.

  6. Buskist, W., Sikorski, J., Buckley, T., & Saville, B. K. (2002). Elements of master teaching. In S. F. Davis & W. Buskist (Eds.), The teaching of psychology: Essays in honor of Wilbert J. McKeachie and Charles L. Brewer (pp. 30–39). New York, NY: Psychology Press.

  7. Busler, J., Kirk, C., Keeley, J., & Buskist, W. (2017). What constitutes poor teaching? A preliminary inquiry into the misbehaviors of not-so-good instructors. Teaching of Psychology, 44, 1–5. https://doi.org/10.1177/0098628317727907.

  8. Cassettari, N. (2014). Avaliação de professors: Uma questão de escolhas [Teacher evaluation: A matter of choices]. Estudos em Avaliação Educacional, 25(57), 166–197. https://doi.org/10.18222/eae255720142829.

  9. Chan, G., Barnes-Holmes, D., Barnes-Holmes, Y., & Stewart, I. (2009). Implicit attitudes to work and leisure among North American and Irish individuals: A preliminary study. Internacional Journal of Psychology & Psychological Therapy, 9(3), 317–334 Retrieved May 15, 2018, from https://tinyurl.com/ycuog9t9.

  10. Cunha, C. M., de Almeida-Neto, O. P., & Stackfleth, R. (2015). Principais métodos de avaliação psicométrica da validade de instrumentos de medida [Main psychometric evaluation methods of the validity of measuring instruments]. Revista de Atenção de Saúde, 14(47), 75–83. https://doi.org/10.13037/rbcs.vol14n47.3391.

  11. Finn, M., Barnes-Holmes, D., Hussey, I., & Graddy, J. (2016). Exploring the behavioral dynamics of the Implicit Relational Assessment Procedure: The impact of three types of introductory rules. The Psychological Record, 66(2), 309–321. https://doi.org/10.1007/2Fs40732-016-0173-4.

  12. Finn, M., Barnes-Holmes, D., & McEnteggart, C. (2018). Exploring the single-trial-type-dominance-effect on the IRAP: Developing a differential arbitrarily applicable relational responding effects (DAARRE) model. The Psychological Record, 68, 11–25. https://doi.org/10.1007/s40732-017-0262-z.

  13. Ford, C. R. (2016). Identifying effective teaching behaviors of pharmacy faculty master teachers (Doctoral Thesis, Auburn University). Retrieved from https://goo.gl/Y2PbEa

  14. Furnham, A. (1986). Response bias, social desirability and dissimulation. Personality and Individual Differences, 7, 385–400. https://doi.org/10.1016/0191-8869(86)90014-0.

  15. Henklain, M. H. O. (2017). Aplicações do Teacher Behavior Checklist à formação de avaliação de professores de nível superior: Contribuições analítico-comportamentais e psicométricas [Teacher Behavior Checklist applications to the training and assessment of higher education teachers: Behavior-analytic and psychometrical contributions] (Doctoral dissertation). Retrieved from https://repositorio.ufscar.br/handle/ufscar/9886

  16. Hussey, I., Thompson, M., McEnteggart, C., Barnes-Holmes, D., & Barnes-Holmes, Y. (2015). Interpreting and inverting with less cursing: A guide to interpreting IRAP data. Journal of Contextual Behavioral Science, 4(3), 1–6. https://doi.org/10.1016/j.jcbs.2015.05.001.

  17. Ismail, E. A., & Groccia, J. E. (2017). Foreign and U.S.-educated faculty members’ views on what constitutes excellent teaching: Effects of gender and discipline. To Improve the Academy, 36(1), 20–38. https://doi.org/10.1002/tia2.20056.

  18. Kavanagh, D., Barnes-Holmes, Y., Barnes-Holmes, Y., McEnteggart, C., & Finn, M. (2018). Exploring differential trial-type effects and the impact of a read-aloud procedure on deictic relational responding on the IRAP. The Psychological Record, 68, 163–176. https://doi.org/10.1007/s40732-018-0276-1.

  19. Keeley, J., Christopher, A. N., & Buskist, W. (2012). Emerging evidence for excellent teaching across borders. In J. E. Groccia, M. Al-Sudairy, & W. Buskist (Eds.), Handbook of college and university teaching: Global perspectives (pp. 374–390). Thousand Oaks, CA: Sage.

  20. Keeley, J., Smith, D., & Buskist, W. (2006). The teacher behaviors checklist: Factor analysis of its utility for evaluating teaching. Teaching of Psychology, 33(2), 84–91. https://doi.org/10.1207/s15328023top3302_1.

  21. Lammers, W. J., Savina, E., Skotko, D., & Churlyaeva, M. (2010). Faculty and student perceptions of outstanding university teachers in the USA and Russia. Educational Psychology: An International Journal of Experimental Educational Psychology, 30(7), 803–815. https://doi.org/10.1080/01443410.2010.512382.

  22. Laros, J. A. (2005). O uso da análise fatorial: Algumas diretrizes para pesquisadores [The use of factorial analysis: Some guidelines to researchers]. In L. Pasquali (Org.), Análise fatorial para pesquisadores (pp. 163–184). Brasília, Brazil: LabPAM.

  23. Liu, S., Keeley, J., & Buskist, W. (2015). Chinese college students’ perceptions of characteristics of excellent teachers. Teaching of Psychology, 42(1), 83–86. https://doi.org/10.1177/0098628315620888.

  24. McGovern, T. V., & Miller, S. L. (2008). Integrating teacher behaviors with character strengths and virtues for faculty development. Teaching of Psychology, 35, 278–285. https://doi.org/10.1080/00986280802374609.

  25. Mizael, T. M., de Almeida, J. H., Silveira, C. C., & De Rose, J. C. (2016). Changing racial bias by transfer of functions in equivalence classes. The Psychological Record, 66, 451–462. https://doi.org/10.1007/s40732-016-0185-0.

  26. Pacico, J. C. (2015). Como é feito um teste? Produção de itens [How do we built a test? Item development]. In C. S. Hutz, D. R. Bandeira, & C. M. Trentini (Eds.), Psicometria [Psychometry] (pp. 55–70). Porto Alegre, Brazil: Artmed.

  27. Perez, W. F., de Almeida, J. H., De Rose, J., Dorigon, A. H., de Vasconcelos, E. L., da Silva, M. A., . . . Barnes-Holmes, D. (2018). Implicit and explicit measures of transformation of function from facial expressions of fear and of happiness via equivalence relations. The Psychological Record, 69(13), 1–12. https://doi.org/10.1007/s40732-018-0304-1

  28. Perez, W. F., Kovac, R., Linares, I. M. P., Barbosa, S. F. U., Gomes, C. T., dos Santos, G. A. R., & de Almeida, J. H. (2017). Utilizando o AAQ-II e o IRAP para prever a ocorrência de esquiva experiencial [Using the AAQ-II and the IRAP to predict the occurrence of experiential avoidance]. Revista Brasileira de Terapia Comportamental e Cognitiva, 19(3), 34–48. https://doi.org/10.31505/rbtcc.v19i3.1052.

  29. Perez, W. F., Kovac, R., Nico, Y. C., Fidalgo, A. P., Caro, D., Linares, I. M. P., ... de Almeida, J. H. (2017). Utilizando o Implicit Relational Assessment Procedure (IRAP) para avaliar relações verbais implicadas na esquiva experiencial: Um estudo piloto [Using the Implicit Relational Assessment Procedure (IRAP) to assess the verbal relations entailed on experiential avoidance: A pilot study]. Revista Brasileira de Terapia Comportamental e Cognitiva, 19(3), 77–91. https://doi.org/10.31505/rbtcc.v19i3.1056

  30. R Development Core Team. (2017). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/

  31. Rabelo, L. Z., Bortoloti, R., & Souza, D. H. (2014). Dolls are for girls and not for boys: Evaluating the appropriateness of the Implicit Relational Assessment Procedure for school-age children. The Psychological Record, 64, 71–77. https://doi.org/10.1007/s40732-014-0006-2.

  32. Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22–42. https://doi.org/10.1016/j.stueduc.2016.08.007.

  33. Vahey, N. A., Nicholson, E., & Barnes-Holmes, D. (2015). A meta-analysis of criterion effects for the Implicit Relational Assessment Procedure (IRAP) in the clinical domain. Journal of Behavior Therapy and Experimental Psychiatry, 48, 59–65. https://doi.org/10.1016/j.jbtep.2015.01.004.

Download references

Funding

This research was supported by the São Paulo Research Foundation (FAPESP 2016/05935-6, 2017/10037-0, 2008/57705-8) and by the Brazilian National Research Council (CNPq, Grants 573972/2008-7, and 465686/2014-1); this study is also part of the research program of the National Institute of Science and Technology on Behavior, Cognition, and Teaching (INCT-ECCE).

Author information

Correspondence to Marcelo H. O. Henklain.

Ethics declarations

Conflict of Interest

The authors declare they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This research was approved by the Brazilian platform for ethics committees (Plataforma Brasil, Certificate of Submission for Ethical Appraisal No. 54448416.6.0000.5302).

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Henklain, M.H.O., Haydu, V.B., Carmo, J.S. et al. Expanding the evidence of content validity for the Teacher Behavior Checklist using the IRAP. Psychol Rec 69, 205–214 (2019). https://doi.org/10.1007/s40732-019-00334-9

Download citation

Keywords

  • Psychometry
  • Content validity
  • Teacher Behavior Checklist
  • Implicit Relational Assessment Procedure