Advertisement

cpm.4.CSE/IRT: Compact process model for measuring competences in computer science education based on IRT models

  • Andreas Zendler
Article

Abstract

cpm.4.CSE/IRT (compact process model for Competence Science Education based on IRT models) is a process model for competence measurement based on IRT models. It allows the efficient development of measuring instruments for computer science education. Cpm.4.CSE/IRT consists of four sub processes: B1 determine items, B2 test items, B3 analyze items according to Rasch model, and B4 interpret items by criteria. Cpm.4.CSE/IRT is modeled in IDEF0, a process modeling language that is standardized and widely used. It is implemented in R, an open-source software optimized for statistical calculations and graphics that allows users to interact using the web application framework Shiny. Through coordinated processes, cpm.4.CSE/IRT ensures the quality and comparability of test instruments in competence measurement. Cpm.4.CSE/IRT is demonstrated using an example from the competence area of Modeling.

Keywords

Computer science education Educational process model Item response theory IRT model Competence-based education Competence mode Competence assessment Shiny 

Notes

References

  1. ACER ConQuest 4 (2018). ConQuest. Retrieved January 2, 2018, from https://www.acer.edu.au/conquest.
  2. ACM (Association for Computing Machinery) (2014). ITiCSE´14 (proceedings of the ACM conference on innovation and technology in computer science education). New York: ACM.Google Scholar
  3. ACM (Association for Computing Machinery) (2015). ITiCSE´15 (proceedings of the ACM conference on innovation and technology in computer science education). New York: ACM.Google Scholar
  4. ACM (Association for Computing Machinery) (2016). ITiCSE´16 (proceedings of the ACM conference on innovation and technology in computer science education). New York: ACM.Google Scholar
  5. Andersen, E. B. (1973). A goodness of fit test for the Rasch model. Psychometrika, 38, 123–140.MathSciNetCrossRefGoogle Scholar
  6. Anderson, L. W., Krathwohl, D. R., & Airasian, P. W. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. New York: Longman.Google Scholar
  7. Bartolucci, F., Bacci, S., & Gnaldi, M. (2016). Statistical analysis of questionnaires: A unified approach based on R and Stata. New York: Chapman & Hall.Google Scholar
  8. Beaton, E., & Allen, N. (1992). Interpreting scales through scale anchoring. Journal of Educational Statististics, 17, 191–204.Google Scholar
  9. Beeley, C. (2016). Web application development with R using shiny. Birningham: Packt Publishing.Google Scholar
  10. Berges, M. & Hubwieser, P. (2015). Evaluation of source code with item response theory. In ITiCSE '15 proceedings of the 2015 ACM conference on innovation and Technology in Computer Science Education (pp. 51–56). ACM: New York.
  11. Bigsteps (2018). Bigsteps. Retrieved January 2, 2018, from http://www.winsteps.com/bigsteps.htm.
  12. Booch, G., Rumbaugh, J. & Jacobson, I. (2005). The unified modeling language user guide. New York: Addison-Wesley.Google Scholar
  13. Borg, I., & Staufenbiel, T. (2007). Lehrbuch Theorien und Methoden der Skalierung. Bern: Huber.Google Scholar
  14. Botturi, L. (2008). E2ML: A tool for sketching instructional designs. In L. Botturi & S. T. Stubbs (Eds.), Handbook of visual languages for instructional design (pp. 112–132). New York: Information Science Reference.CrossRefGoogle Scholar
  15. Bühner, M. (2011). Einführung in die test- und Fragebogenkonstruktion. München: Pearson Studium.Google Scholar
  16. Burnham, K., & Anderson, D. R. (2002). Model selection and multimodel inference: A practical information-theoretic approach. New York: Springer.zbMATHGoogle Scholar
  17. Chang, W. (2013). R graphics cookbook. Beijing: O'Reilly and Associates.Google Scholar
  18. ConstructMap (2018). ConstructMap. Retrieved January 2, 2018, from http://bearcenter.berkeley.edu/software/constructmap.
  19. CRAN (Comprehensive R Archive Network) (2018). CRAN Task Views. Retrieved January 2, 2018 from https://cran.r-project.org/AS.
  20. De Ayala, R. J. (2009). The theory and practice of item response theory. New York: Guilford press.Google Scholar
  21. Derntl, M., & Motschnig-Pitrik, R. (2008). CoUML: A visual language for modeling cooperative environments. In L. Botturi & S. T. Stubbs (Eds.), Handbook of visual languages for instructional design (pp. 155–184). New York: Information Science Reference.Google Scholar
  22. Eid, M., & Schmid, K. (2014). Testtheorie und Testkonstruktion. Göttingen: Hogrefe.Google Scholar
  23. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.Google Scholar
  24. Fischer, G. H., & Molenaar, I. (Eds.). (1995). Rasch models - foundations, recent developments, and applications. Berlin: Springer.Google Scholar
  25. GI. Gesellschaft für Informatik (2008). Grundsätze und Standards für die Informatik in der Schule. Bildungsstandards Informatik für die Sekundarstufe I. LOG IN, 28 (150/151) supplement.Google Scholar
  26. GI. Gesellschaft für Informatik (2016). Bildungsstandards Informatik für die Sekundarstufe II. LOG IN, 36(183/184) supplement.Google Scholar
  27. Glas, C. A. W., & Verhelst, N. D. (1995). Testing the Rasch model. In G. H. Fischer & J. W. Molenaar (Eds.), Rasch models: Their foundations, recent developments and applications (pp. 69–95). New York: Springer.CrossRefGoogle Scholar
  28. Goldhammer, F., & Hartig, J. (2012). Interpretation von Testresultaten und Testeichung. In H. Moosbrugger, & A, Kelava (Eds.), Testtheorie und Fragebogenkonstruktion (pp. 173–201). Berlin. Springer.Google Scholar
  29. Griffin, P. (2007). The comfort of competence and the uncertainty of assessment. Studies in Educational Evaluation, 33, 87–99.CrossRefGoogle Scholar
  30. Haladyna, T. (2004). Developing and validating multiple choice test items. London: Lawrence Erlbaum Associates Publisher.Google Scholar
  31. Horn, R. A. J. (2004). Standards. New York: Lang.Google Scholar
  32. Hsieh, S.-C., Lin, J.-S., & Lee, H.-C. (2012). Analysis on literature review of competency. International Review of Business and Economics, 2, 25–50.Google Scholar
  33. Hubwieser, P. (1999). Modellierung in der Schulinformatik. LOG IN, 24–29.Google Scholar
  34. Hubwieser, P. (2007). Didaktik der Informatik. Grundlagen, Konzepte, Beispiele. Berlin: Springer.Google Scholar
  35. Ihaka, R., & Gentleman, R. (1996). R: A language for data analysis and graphics. Journal of Computational and Graphical Statistics, 5(3), 299–314.Google Scholar
  36. Institute for Objective Measurement (2018). Tools overview. Retrieved January 2, 2018, from http://www.rasch.org/software.htm.
  37. Irtel, H. (1996). Entscheidungs- und testtheoretische Grundlagen der Psychologischen Diagnostik. Frankfurt am Amain: Lang.Google Scholar
  38. Jonkisz, E., Moosbrugger, H., & Brandt, H. (2012). Planung und Entwicklung von Tests und Fragebogen. In H. Moosbrugger & A. Kelava (Eds.), Testtheorie und Fragebogenkonstruktion (Kapitel 3). Berlin: Springer.Google Scholar
  39. Klieme, E., & Maag Merki, K. (2008). Introduction of educational standards in German-speaking countries. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 305–314). Göttingen: Hogrefe.Google Scholar
  40. Klieme, E., Hartig, J., & Rauch, D. (2008). The concept of competence in educational contexts. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of of competencies in educational contexts (pp. 3–22). Göttingen: Hogrefe.Google Scholar
  41. Knowledge Based Systems (1993). Integrated Definition for Function Modeling (IDEF0). Retrieved March 1, 2014, from http://www.itl.nist.gov/fipspubs/idef02.doc.
  42. Koller, I., & Hatzinger, R. (2013). Nonparametric tests for the Rasch model: Explanation, development, and application of quasi-exact tests for small samples. InterStat, 11, 1–16.Google Scholar
  43. Koller, I., Alexandrowicz, R., & Hatzinger, R. (2012). Das Rasch Modell in der praxis. Wien: Facultas.Google Scholar
  44. Kron, F. W. (2008). Grundwissen Didaktik. Stuttgart: UTB.Google Scholar
  45. Lersch, R., & Schreder, G. (2013). Grundlagen kompetenzorientierten Unterrichtens: Von den Bildungsstandards zum Schulcurriculum. Opladen: Budrich.Google Scholar
  46. Linacre, J. M. (1994). Sample size and item calibration stability. Rasch Measurement Transactions, 7, 328.Google Scholar
  47. Mair, P., & Hatzinger, R. (2007). CML based estimation of extended Rasch models with the eRm package in R. Psychology Science, 49, 26–43.Google Scholar
  48. Martin-Löf, P. (1973). Statistika modeller. Stockholm: Instituet för Försäkringsmathematik och Mathematisk Statistisk vid Stockholms Universitet.Google Scholar
  49. Mayer, R., Painter, M., & de Witte, P. (1992). IDEF family for concurrent engineering and business reengineering applications. New York: Knowledge Based Systems.Google Scholar
  50. Menzel, C., & Mayer, R. (2005). The IDEF family of languages. In P. Bernus, K. Martins, & G. Schmidt (Eds.), Handbook on architectures of information systems (pp. 215–250). Berlin: Springer.Google Scholar
  51. Ministep (2018). Ministep. Retrieved January 2, 2018, from http://www.winsteps.com/ministep.htm.
  52. Molenaar, J. W. (1995). TSome background for item response theory and the Rasch model. In G. H. Fischer & J. W. Molenaar (Eds.), Rasch models: Their foundations, recent developments and applications (pp. 3–14). New York: Springer.CrossRefGoogle Scholar
  53. Moosbrugger, H., & Kelava, A. (Eds.) (2012). Testtheorie und Fragebogenkonstruktion. Berlin. Springer.Google Scholar
  54. Mühling, A., Ruf, A., & Hubwieser, P. (2015). Design and first results of a psychometric test for measuring basic programming abilities. In Proceedings of the Workshop in Primary and Secondary Computing Education. WiPSCE’15 (pp. 2–10). New York, NY: ACM.Google Scholar
  55. Mullis, I. V. S., Martin, M. O., Foy, P., & Arora, A. (2011). TIMSS 2011 international results in mathematics. Chestnut Hill, MA: TIMMS & Pierls International Study Center.Google Scholar
  56. OECD iLibrary (2018). Retrieved January 2, 2018, from http://www.oecd-ilibrary.org/education/pisa-2009-ergebnisse_9789264095335-de.
  57. Paquette, G., Léonard, M., & Lundgren-Cayrol, K. (2008). The MOT+ visual language for knowledge-based instructional design. In L. Botturi & S. T. Stubbs (Eds.), Handbook of visual languages for instructional design (pp. 133–154). New York: Information Science Reference.Google Scholar
  58. Ponocny, I. (2001). Nonparametric goodness of fit tests for the Rasch model. Psychometrika, 66, 437–460.MathSciNetCrossRefGoogle Scholar
  59. Preinerstorfer, D., & Formann, A. (2012). Parameter recovery and model selection in mixed Rasch models. British Journal of Mathematical and Statistical Psychology, 65, 251–262.MathSciNetCrossRefGoogle Scholar
  60. Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Kopenhagen. Danish Institute for Educational Research.Google Scholar
  61. Ravitch, D. (1995). National standards in American education. Washington, D.C.: Brookings Institution Press.Google Scholar
  62. Rizopoulos, R. (2006). Ltm: An R package for latent variable modelling and item response theory analyses. Journal of Statistical Software, 17, 1–25.CrossRefGoogle Scholar
  63. Robinsohn, S. B. (1971). Bildungsreform als revision des curriculums und ein Strukturkonzept für curriculums-entwicklung. Berlin: Luchterhand.Google Scholar
  64. RStudio (2018). Shiny. Retrieved January 2, 2018, from http://shiny.rstudio.com/.
  65. Rumbaugh, J., Jacobson, I., & Booch, G. (2010). The unified modeling language reference manual. New York: Addison-Wesley.Google Scholar
  66. Rumm2030 (2018). Rumm2030. Retrieved January 2, 2018, from http://www.rummlab.com.au/.
  67. Rychen, S., & Salganik, L. H. (2003). Definition and selection of competencies: theoretical and conceptual foundations – summary of the final report “key competencies for a successful life and a well-functioning society. Retrieved January 2, 2018 from http://www.netuni.nl/courses/hre/uploads/File/deseco_finalreport_summary.pdf.
  68. Sarris, W. E., & Gallhofer, I.N. (2014). Design, evaluation, and analysis of questionnaires for survey research. New York: Wiley.Google Scholar
  69. Seifert, A. (2015). Kompetenzforschung in den Fachdidaktiken auf der Grundlage von IRT-Modellen. In U. Riegel, S. Schubert, G. Siebert-Ott, & K. Macha (Eds.), Kompetenzmodellierung und Kompetenzmessung in den Fachdidaktiken (pp. 131–161). Münster: Waxmann.etrobl, C. (2015). Das Rasch-Modell. München: Hampp.Google Scholar
  70. Strobl, C. (2015). Das Rasch-Modell. München: Hampp.Google Scholar
  71. Sudol, A., & Studer, C. (2010). Analyzing test items: Using item response theory to validate assessments. In Proceedings of the 41st ACM Technical Symposium on Computer Science Education, SIGCSE '10 (pp. 436–440). New York, NY: ACM.Google Scholar
  72. Teetor, R. (2011). R cookbook. Beijing: O'Reilly and Associates.Google Scholar
  73. Tew, A. E., & Guzdial, M. (2011). The fcs1: A language independent assessment of cs1 knowledge. In Proceedings of the 42Nd ACM Technical Symposium on Computer Science Education, SIGCSE '11 (pp. 111–116). New York, NY: ACM.Google Scholar
  74. van der Linden, W., & Hambleton, R. K. (1997). Handbook of modern item-response theory. Berlin: Springer.Google Scholar
  75. van der Linden, W., & Hambleton, R. K. (2016). Handbook of item-response theory (three volume set). New York: CRC Press.Google Scholar
  76. Weinert, F. E. (1998). Vermittlung von Schlüsselqualifikationen. In S. Matalik & D. Schade (Eds.), Entwicklungen in Aus- und Weiterbildung - (pp. 23–43). Baden-Baden: Nomos.Google Scholar
  77. Wendt, H., Bos, W., Selter, C., Köller, O., Schwippert, C., & Kaspar, D. (Eds.) (2016). TIMSS 2015 Mathematische und naturwissenschaftliche Kompetenzen von Grundschulkindern in Deutschland im internationalen Vergleich. Münster: Waxmann.Google Scholar
  78. Wickham, H. (2013). R packages. Beijing: O'Reilly and Associates.Google Scholar
  79. Wilson, M. (2005). Constructing measures. An item response modelling approach. Mawah: Lawrence Erlbaum Associates.Google Scholar
  80. Winsteps (2018). Winsteps. Retrieved January 2, 2018, from http://www.winsteps.com/winsteps.htm.
  81. Winters, T., & Payne, T. (2005). What do students know? An outcomes-based assessment system. In Proceedings of the First International Workshop on Computing Education Research, ICER '05 (pp. 65–72). New York, NY: ACM.Google Scholar
  82. Winters, T., & Payne, T. (2006). Closing the loop on test creation: A question assessment mechanism for instructors. SIGCSE Bulletin, 38(1), 169–172.CrossRefGoogle Scholar
  83. Zendler, A., & Hubwieser, P. (2013). The influence of teacher training programs on evaluations of central computer science concepts. Teacher and Teaching Education, 34(August), 130–142.CrossRefGoogle Scholar
  84. Zendler, A., Spannagel, C., & Klaudt, D. (2011). Marrying content and process in computer science education. IEEE Transactions on Education, 54(3), 387–397.CrossRefGoogle Scholar
  85. Zendler, A., Klaudt, D., & Seitz, C. (2014). Empirical determination of competence areas to computer science education. Journal of Educational Computing Research, 51(1), 71–89.CrossRefGoogle Scholar
  86. Zendler, A., McClung, O. W., & Klaudt, D. (2015). A cross-cultural comparison of concepts in computer science education: The US–Germany experience. The International Journal of Information and Learning Technology, 32(4), 235–256.CrossRefGoogle Scholar
  87. Zendler, A., Seitz, C., & Klaudt, D. (2016). Process-based development of competence models to computer science education. Journal of Educational Computing Research, 54(4), 563–597.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Professor of Computer ScienceUniversity of Education LudwigsburgLudwigsburgGermany

Personalised recommendations