Abstract
We present a modification of the IRT-based standard setting method proposed by García, Abad, Olea & Aguado (Psicothema 25(2):238–244, 2013), which we have combined with the cloud delphi method (Yang, Zeng, & Zhang in IJUFKBS 20(1):77–97, 2012). García et al. (Psicothema 25(2):238–244, 2013) calculate the average characteristic curve of each level, to determine cutoff scores on the basis of the joint characteristic curve. In the proposed new method, the influence of each item on the average item characteristic curve is weighted according to its proximity to the next level. Performance levels are placed on a continuous scale, with each judge asked to determine an interval for each item. The cloud delphi method is used until a stable final interval is achieved. From these judgments, the weights of each item in the scale are calculated. Then, a family of weighted average characteristic curves is calculated and in the next step, joint weighted averaged ICC are calculated. The cutoff score is determined by finding the ability where the joint weighted averaged ICC reach a certain predefined probability level. This paper compares the performance of this new procedure for a math test with the classic Bookmarking method. We will show that this modification to the method improves cutoff score estimation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Berk, R. A. (1986). A consumer’s guide to setting performance standards on criterion. Review of Educational Research, 56(1), 137–172.
Cizek, G. J., & Bunch, M. B. (2007). Standard setting. A guide to establishing and evaluating performance standards on tests. Thousand Oak, CA: Sage Publications.
García, P. E., Abad, F. J., Olea, J., & Aguado, D. (2013). A new IRT-based standard setting method: Application to elath-listening. Psicothema, 25(2), 238–244.
Glass, G. (1978). Standards and criteria. Journal of Educational Measurement, 15(4), 237–261.
Hambleton, R. K. (1978). The use of cut-off scores with criterion-referenced tests in instructional settings. Journal of Educational Measurement, 15(4), 277–290.
Hambleton, R. K., Jaeger, R. M., Plake, B. S., & Mills, C. (2000). Setting performance standards on complex educational assessments. Applied Psychological Measurement, 24(4), 355–366.
Jaeger, R. M. (1989). Certification of student competence. In R. L. Linn (Ed.), Educational measurement (pp. 485–514). New York: American Council on Education and Macmillan.
Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64(3), 425–461.
Linn, R. (2003). Performance standards: Utility for different uses of assessments. Education Policy Analysis Archives, 11(31). Retrieved from: http://epaa.asu.edu/epaa/v11n31/.
Margolis, M. J., & Clauser, B. E. (2014). The impact of examinee performance information on judges’ cut scores in modified Angoff standard setting exercises. Educational Measurement Issues and Practice, 33(1), 15–22.
Mousavi, A., Cui, Y., & Rogers, T. (2018). An examination of different methods of setting cutoff values in person fit research. International Journal of Testing, 1–22. https://doi.org/10.1080/15305058.2018.1464010.
Rodríguez, P. (2017). Creación, desarrollo y resultados de la aplicación de pruebas de evaluación basadas en estándares para diagnosticar competencias en matemática y lectura al ingreso a la universidad. Revista Iberoamericana de Evaluación Educativa, 10(1), 89–107. https://doi.org/10.15366/riee2017.10.1.005.
Wang, G., Xu, C., & Li, D. (2014). Generic normal cloud model. Information Sciences, 280, 1–15.
Yang, X. J., Zeng, L., & Zhang, R. (2012). Cloud delphi method. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 20(1), 77–97.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Rodríguez, P., Luzardo, M. (2019). A Modification of the IRT-Based Standard Setting Method. In: Wiberg, M., Culpepper, S., Janssen, R., González, J., Molenaar, D. (eds) Quantitative Psychology. IMPS IMPS 2017 2018. Springer Proceedings in Mathematics & Statistics, vol 265. Springer, Cham. https://doi.org/10.1007/978-3-030-01310-3_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-01310-3_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-01309-7
Online ISBN: 978-3-030-01310-3
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)