Advertisement

A Modification of the IRT-Based Standard Setting Method

  • Pilar RodríguezEmail author
  • Mario Luzardo
Conference paper
Part of the Springer Proceedings in Mathematics & Statistics book series (PROMS, volume 265)

Abstract

We present a modification of the IRT-based standard setting method proposed by García, Abad, Olea & Aguado (Psicothema 25(2):238–244, 2013), which we have combined with the cloud delphi method (Yang, Zeng, & Zhang in IJUFKBS 20(1):77–97, 2012). García et al. (Psicothema 25(2):238–244, 2013) calculate the average characteristic curve of each level, to determine cutoff scores on the basis of the joint characteristic curve. In the proposed new method, the influence of each item on the average item characteristic curve is weighted according to its proximity to the next level. Performance levels are placed on a continuous scale, with each judge asked to determine an interval for each item. The cloud delphi method is used until a stable final interval is achieved. From these judgments, the weights of each item in the scale are calculated. Then, a family of weighted average characteristic curves is calculated and in the next step, joint weighted averaged ICC are calculated. The cutoff score is determined by finding the ability where the joint weighted averaged ICC reach a certain predefined probability level. This paper compares the performance of this new procedure for a math test with the classic Bookmarking method. We will show that this modification to the method improves cutoff score estimation.

Keywords

Performance standard setting Item response theory Delphi method 

References

  1. Berk, R. A. (1986). A consumer’s guide to setting performance standards on criterion. Review of Educational Research, 56(1), 137–172.CrossRefGoogle Scholar
  2. Cizek, G. J., & Bunch, M. B. (2007). Standard setting. A guide to establishing and evaluating performance standards on tests. Thousand Oak, CA: Sage Publications.Google Scholar
  3. García, P. E., Abad, F. J., Olea, J., & Aguado, D. (2013). A new IRT-based standard setting method: Application to elath-listening. Psicothema, 25(2), 238–244.Google Scholar
  4. Glass, G. (1978). Standards and criteria. Journal of Educational Measurement, 15(4), 237–261.CrossRefGoogle Scholar
  5. Hambleton, R. K. (1978). The use of cut-off scores with criterion-referenced tests in instructional settings. Journal of Educational Measurement, 15(4), 277–290.CrossRefGoogle Scholar
  6. Hambleton, R. K., Jaeger, R. M., Plake, B. S., & Mills, C. (2000). Setting performance standards on complex educational assessments. Applied Psychological Measurement, 24(4), 355–366.CrossRefGoogle Scholar
  7. Jaeger, R. M. (1989). Certification of student competence. In R. L. Linn (Ed.), Educational measurement (pp. 485–514). New York: American Council on Education and Macmillan.Google Scholar
  8. Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64(3), 425–461.CrossRefGoogle Scholar
  9. Linn, R. (2003). Performance standards: Utility for different uses of assessments. Education Policy Analysis Archives, 11(31). Retrieved from: http://epaa.asu.edu/epaa/v11n31/.CrossRefGoogle Scholar
  10. Margolis, M. J., & Clauser, B. E. (2014). The impact of examinee performance information on judges’ cut scores in modified Angoff standard setting exercises. Educational Measurement Issues and Practice, 33(1), 15–22.CrossRefGoogle Scholar
  11. Mousavi, A., Cui, Y., & Rogers, T. (2018). An examination of different methods of setting cutoff values in person fit research. International Journal of Testing, 1–22.  https://doi.org/10.1080/15305058.2018.1464010.CrossRefGoogle Scholar
  12. Rodríguez, P. (2017). Creación, desarrollo y resultados de la aplicación de pruebas de evaluación basadas en estándares para diagnosticar competencias en matemática y lectura al ingreso a la universidad. Revista Iberoamericana de Evaluación Educativa, 10(1), 89–107.  https://doi.org/10.15366/riee2017.10.1.005.
  13. Wang, G., Xu, C., & Li, D. (2014). Generic normal cloud model. Information Sciences, 280, 1–15.MathSciNetCrossRefGoogle Scholar
  14. Yang, X. J., Zeng, L., & Zhang, R. (2012). Cloud delphi method. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 20(1), 77–97.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Eastern Regional University Center, University of RepublicMaldonadoUruguay
  2. 2.School of PsychologyUniversity of RepublicMontevideoUruguay

Personalised recommendations